logfmt1

To manually craft a regex:

import logfmt1, json
fmt = json.load(open("/.../access.log.fmt", "r"))
rx = logfmt1.regex(fmt)
rx = logfmt1.rx2re(rx)   # turn into Python regex

Or with plain old guesswork / presuming a standard log format:

rx = logfmt1.regex({"class": "apache combined"})

Though that’s of course not the intended use case, and hinges on predefined formats in /usr/share/logfmt/.

logfmt1.logopen()

logopen(fn=…) is basically a file-like iterator that yields dictionaries rather than text strings.

for row in logfmt1.logopen(".../access.log"):
print(row["request_time"])

And it provides a basic regex/formatstring debugging feature (via debug=True parameter or with logex -D):

logex

Very crudementary extractor for log files:

logex .../access.log --tab @host @date +id

Which also handles the .fmt implicitly. (Kinda the whole point of this project.)

update-logfmt

The Python package does bundle a run-parts wrapper, but just the apache collector, and a local Python copy of the format database. It should discover all (Apache) *.log files nonetheless and pair them with .fmt declarations.

And that’s sort of the main aspect of this project. Establish .log.fmt files until application vendors come around to making logs parseable. The rules database structure is subject to change, and only one possible implementation. There might also be simpler approaches (grok mapping) to generate regexps for format strings.