Skip to main content

Parse log file as input and export the data to database as output.

Project description

logio

Parse log file as input and export the data to database as output.

Install

pip install logio

Installed Command Utils

  • logio

Usage

C:\Workspace\logio>logio --help
Usage: logio [OPTIONS] COMMAND [ARGS]...

  Parse log file as input and export the data to database as output.

Options:
  -c, --config TEXT  Config file path. The config file must in yaml format.
                     [required]

  --help             Show this message and exit.

Commands:
  scan    Try example settings on a test file.
  server  Start log handler server.
  test    Parse the example text and print out parse result.

Settings

  • input
    • type: stdin, file, tail
    • ignore-blank-lines: true, false
    • encoding: utf-8, gb18030, ...
    • for type=file
    • filename
    • for type=tail
    • filename
    • offset-file
    • read-from-end
    • backup-patterns
    • sleep-interval
    • non-blocking
    • blocking
  • output
    • type: mysql, stdout, print-not-matched-line
    • buffer-size
    • for type=mysql
    • ignore-not-matched-lines
    • keep-failed-lines
    • inserts: list<string, string>
      • key: DEFAULT, some other key
      • sql_template
  • parser
    • type: regex, json
    • keeep-not-matched-lines: bool
    • for type=regex
    • use-default-rules
    • transforms
    • rules
    • matches: list<string, string>
      • matched_name
      • regex

Example config

input:
  type: tail
  filename: tests\nginx.access.log
parser:
  type: regex
  matches:
    simple: "{SIMPLE_NGINX_ACCESS}"
  transforms:
    - field: time_local
      type: strftime
      strptime: "%d/%b/%Y:%H:%M:%S %z"
    - field: request_time
      type: str2number
    - field: request_time
      type: formula
      formula: "{request_time} * 1000000"
    - field: request_time
      type: int
output:
  type: mysql
  mysql:
    host: 127.0.0.1
    port: 3306
    user: root
    password: password
    database: database
  inserts:
    simple:
      sql: insert into table_name (add_time, path, response_time) values (%s, %s, %s)
      fields:
        - time_local
        - request_path
        - request_time

Releases

v0.1.18 2021/02/22

  • Set default buffer-size to 10.
  • Add log in loop.
  • Use fastutils.logutils.setup.

V0.1.13 2021/02/18

  • Add Nginx Access Log Rules.
  • Add LogFilter subsystem.
  • Add transforms support in LogParser.

v0.1.10 2020/10/10

  • Add json parser.

v0.1.9 2020/09/10

  • Change package's home url.

v0.1.8 2020/09/09

  • Translate help informations to english.
  • Add License.
  • Add LogToStdout.

v0.1.7

  • Some bad old release, ignore them...

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for logio, version 0.1.18
Filename, size File type Python version Upload date Hashes
Filename, size logio-0.1.18.tar.gz (14.6 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page