logio

Parse log file as input and export the data to database as output.

Install

pip install logio

Installed Command Utils

• logio

Usage

C:\Workspace\logio>logio --help
Usage: logio [OPTIONS] COMMAND [ARGS]...

  Parse log file as input and export the data to database as output.

Options:
  -c, --config TEXT  Config file path. The config file must in yaml format.
                     [required]

  --help             Show this message and exit.

Commands:
  scan    Try example settings on a test file.
  server  Start log handler server.
  test    Parse the example text and print out parse result.

Settings

• input
  • type: stdin, file, tail
  • ignore-blank-lines: true, false
  • encoding: utf-8, gb18030, ...
  • for type=file
  • filename
  • for type=tail
  • filename
  • offset-file
  • read-from-end
  • backup-patterns
  • sleep-interval
  • non-blocking
  • blocking
• output
  • type: mysql, stdout, print-not-matched-line
  • buffer-size
  • for type=mysql
  • ignore-not-matched-lines
  • keep-failed-lines
  • inserts: list<string, string>
    • key: DEFAULT, some other key
    • sql_template
• parser
  • type: regex, json
  • keeep-not-matched-lines: bool
  • for type=regex
  • use-default-rules
  • transforms
  • rules
  • matches: list<string, string>
    • matched_name
    • regex

Example config

input:
  type: tail
  filename: tests\nginx.access.log
parser:
  type: regex
  matches:
    simple: "{SIMPLE_NGINX_ACCESS}"
  transforms:
    - field: time_local
      type: strftime
      strptime: "%d/%b/%Y:%H:%M:%S %z"
    - field: request_time
      type: str2number
    - field: request_time
      type: formula
      formula: "{request_time} * 1000000"
    - field: request_time
      type: int
output:
  type: mysql
  mysql:
    host: 127.0.0.1
    port: 3306
    user: root
    password: password
    database: database
  inserts:
    simple:
      sql: insert into table_name (add_time, path, response_time) values (%s, %s, %s)
      fields:
        - time_local
        - request_path
        - request_time

Releases

v0.1.21 2025/10/24

• Add RequestBody transformer.
• Add LogToHttp output.

v0.1.19 2022/01/18

• Add sympy in requirements.txt.
• Remove mysqlclient from requirements.txt.
• Fix license file missing problem.

v0.1.18 2021/02/22

• Set default buffer-size to 10.
• Add log in loop.
• Use fastutils.logutils.setup.

V0.1.13 2021/02/18

• Add Nginx Access Log Rules.
• Add LogFilter subsystem.
• Add transforms support in LogParser.

v0.1.10 2020/10/10

• Add json parser.

v0.1.9 2020/09/10

• Change package's home url.

v0.1.8 2020/09/09

• Translate help informations to english.
• Add License.
• Add LogToStdout.

v0.1.7

• Some bad old release, ignore them...