Skip to main content

Parse log file as input and export the data to database as output.

Project description


Parse log file as input and export the data to database as output.


pip install logio

Installed Command Utils

  • logio


C:\Workspace\logio>logio --help
Usage: logio [OPTIONS] COMMAND [ARGS]...

  Parse log file as input and export the data to database as output.

  -c, --config TEXT  Config file path. The config file must in yaml format.

  --help             Show this message and exit.

  scan    Try example settings on a test file.
  server  Start log handler server.
  test    Parse the example text and print out parse result.


  • input
    • type: stdin, file, tail
    • ignore-blank-lines: true, false
    • encoding: utf-8, gb18030, ...
    • for type=file
    • filename
    • for type=tail
    • filename
    • offset-file
    • read-from-end
    • backup-patterns
    • sleep-interval
    • non-blocking
    • blocking
  • output
    • type: mysql, stdout, print-not-matched-line
    • buffer-size
    • for type=mysql
    • ignore-not-matched-lines
    • keep-failed-lines
    • inserts: list<string, string>
      • key: DEFAULT, some other key
      • sql_template
  • parser
    • type: regex, json
    • keeep-not-matched-lines: bool
    • for type=regex
    • use-default-rules
    • transforms
    • rules
    • matches: list<string, string>
      • matched_name
      • regex

Example config

  type: tail
  filename: tests\nginx.access.log
  type: regex
    simple: "{SIMPLE_NGINX_ACCESS}"
    - field: time_local
      type: strftime
      strptime: "%d/%b/%Y:%H:%M:%S %z"
    - field: request_time
      type: str2number
    - field: request_time
      type: formula
      formula: "{request_time} * 1000000"
    - field: request_time
      type: int
  type: mysql
    port: 3306
    user: root
    password: password
    database: database
      sql: insert into table_name (add_time, path, response_time) values (%s, %s, %s)
        - time_local
        - request_path
        - request_time


v0.1.18 2021/02/22

  • Set default buffer-size to 10.
  • Add log in loop.
  • Use fastutils.logutils.setup.

V0.1.13 2021/02/18

  • Add Nginx Access Log Rules.
  • Add LogFilter subsystem.
  • Add transforms support in LogParser.

v0.1.10 2020/10/10

  • Add json parser.

v0.1.9 2020/09/10

  • Change package's home url.

v0.1.8 2020/09/09

  • Translate help informations to english.
  • Add License.
  • Add LogToStdout.


  • Some bad old release, ignore them...

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for logio, version 0.1.18
Filename, size File type Python version Upload date Hashes
Filename, size logio-0.1.18.tar.gz (14.6 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page