petit-log-crunchtools 4.7.0

Download and Information
===============================================================================
http://crunchtools.com/software/petit/

Installation
===============================================================================
    pip install petit-log-crunchtools
    # or: uv tool install petit-log-crunchtools
    # or: pipx install petit-log-crunchtools

Installs the petit command and the petit library. The PyPI distribution is
named petit-log-crunchtools, not petit, because "petit" on PyPI belongs to
an unrelated project. It follows the same naming convention as the rest of
the crunchtools fleet (gatehouse-crunchtools, mcp-gemini-crunchtools) — see
CHANGELOG.md's 3.0.0 and 3.1.1 entries.

Native packages, from the signed repositories at
https://crunchtools.github.io/packages, so updates arrive with the rest of
the system:

    # Fedora, RHEL 8/9/10 and rebuilds, Amazon Linux 2023
    sudo curl -fsSLo /etc/yum.repos.d/crunchtools.repo \
        https://crunchtools.github.io/packages/rpm/crunchtools.repo
    sudo dnf install petit

    # SUSE Linux Enterprise 15 SP7 and 16
    sudo zypper addrepo https://crunchtools.github.io/packages/rpm/crunchtools.repo
    sudo zypper install petit

    # Debian 12 and 13, Ubuntu 24.04 and 26.04
    sudo install -d /etc/apt/keyrings
    sudo curl -fsSLo /etc/apt/keyrings/crunchtools.asc \
        https://crunchtools.github.io/packages/crunchtools.asc
    sudo curl -fsSLo /etc/apt/sources.list.d/crunchtools.sources \
        https://crunchtools.github.io/packages/deb/crunchtools.sources
    sudo apt update && sudo apt install petit

The signing key's fingerprint is
588C E8BF 2F36 D77E 1B1E  545C C04F 530D 3931 F683. Each GitHub release also
carries the .rpm and .deb for a one-off install (dnf install ./petit-*.rpm,
apt install ./petit_*.deb). The packages install petit in /usr/lib/petit
and run it on the newest Python 3.11 or later the system has; on RHEL 8 and
9 that pulls in the python3.12 package. Ubuntu 22.04 and Debian 11 have no
supported Python 3.11, so use pipx there.

A container image is also published, for CI or isolated execution:

    podman run --rm -v $(pwd):/data:ro,Z quay.io/crunchtools/petit --hash /data/some.log


Using petit as a library
===============================================================================
    from petit import hash_lines, detect_format

    with open("/var/log/messages") as log:
        for group in hash_lines(log):
            print(group.count, group.pattern)

hash_lines() groups lines by fingerprint, most frequent first, reading them
one at a time: memory follows the number of groups, not the size of the
input. hash_text() and analyze_text() do the same for a string already in
memory, and analyze_lines() is analyze_text() for lines. detect_format()
reports which driver claims the text; "RawEntry" means no driver recognised it.
analyze_text() returns the same groups plus how they were produced, and takes
the options the CLI has: hash_mode ("daemon", "host", "wordcount"),
collapse_fingerprints, and framer. A JSON array, JSON Lines, a mail thread, or
a log whose messages run over several lines is grouped per object or per
message rather than per line, and the Analysis accounts for both records and
source lines (see "How petit reads a log"). Normalization is chosen by the driver for the format
unless you pass filter_name or stopwords. docs/drivers.md explains how
drivers decide what to collapse.

Text goes in, data comes out. Nothing here reads a file, writes to stdout, or
exits the process — failures raise PetitError subclasses (EmptyLogError,
ParseError, DataFileError) for the caller to handle.


Public API and versioning
===============================================================================
Petit follows Semantic Versioning. Two things are public and covered by it:

  * the petit command line, its options and its output
  * the names exported from the petit package: analyze_text, analyze_lines,
    hash_text, hash_lines, detect_format, Analysis, Group, and the
    PetitError hierarchy

Everything else is internal. The driver classes in petit.CrunchLog and
the hash classes in petit.LogHash may change in any release; they are
where new log formats get added, and pinning them would freeze that. If you
need something from them, ask for it to be exported rather than importing it
directly.


Why
===============================================================================
Log analysis is something that all systems administrators know they need to do. 
Many of us come to this point, either because there is a problem, there is a 
security requirement from the organization, or it keeps you up all night 
wanting to know what is going on in all of that data.

Looking for best practices for log analysis on this Internet is difficult at 
best. Many years ago, I discovered a script that hashed log files by removing 
all of their numbers and replacing them with "#" characters. The results of 
this simple algorithm were phenomenal, logs could be reduced by a factor of 
ten. This was much more readable, yet left much of the quality data that I 
needed to determine if there was a problem.

In the years since I discovered that simple algorithm, I have come to discover 
many techniques on text analysis which are commonly used in linguistics and 
anthropology to analyze natural languages. This has led me to develop very 
simple best practices for analyzing logs.

The Basics
===============================================================================
 1. Logs are made up of output which are programmed by human beings. There 
are no real restraints on what is output, other than, some cultural rules on 
being professional. This makes the output from programs very much a natural i
language. This also makes the output of someones program an approximation of 
the reality of what is happening inside a program. This is important to 
remember, logs are not perfect.

 2. When a systems administrator analyzes logs by changing them, he is 
creating an approximation of an approximation of reality in side a working 
program. This is not necessarily a bad thing, especially, when the programmer 
never gives you better than his approximation of reality anyway.

 3. In practice logs are made up of certainty and uncertainty. For example, I 
know what OpenSSH puts in the log during a login, because it is common. On the 
other hand, I do not now what a Compaq DL380 G3 will put in the log when it has 
a disk controller error. This is important to remember.

 4. The basic log analysis algorithm in Petit works to remove certainty, while 
leaving uncertainty. Stated another way, Petit quantitatively removes certainty, 
thereby leaving uncertainty, which by necessity requires qualitative analysis 
from a systems administrator

 5. After the algorithm has been applied, the output must be read by a systems 
administrator to determine if it is a normal or abnormal. Then abnormal entries 
can be acted on, hopefully before there is noticeable impact to your system. 


How petit reads a log
===============================================================================
Every input goes through the same stages:

	text -> lines -> framer -> records -> entry driver -> hash driver
	                                                   -> graphs

 1. The text is split into lines.

 2. A framer cuts the lines into records. A record is one log message:
    usually one line, but not always. petit tries each framer in turn and
    the first that recognises the whole input wins:

	json       a JSON array of objects, or JSON Lines: one record per object
	message    a mail thread or mbox: one record per message
	multiline  log messages that run over several lines: one record per
	           message, continuation lines included (see below)
	line       anything else: one record per line

    --framer json|message|multiline|line forces one.

 3. An entry driver reads each record and picks out the time, host, daemon
    and message. petit votes on which driver fits (syslog, rsyslog, Apache
    access and error, secure, Snort, ...). If one record in the input can't
    be read by the chosen driver, the whole input falls back to RawEntry,
    which reads anything but finds no times, hosts or daemons in it.

 4. A hash driver turns each entry into a pattern, and entries with the same
    pattern are one group in --hash. The graphs count entries per slice of
    time instead.

Counts are kept for both: an Analysis reports lines_in (source lines) and
records_in (records), which are the same unless a framer joined lines.

Big inputs. petit streams: it never holds the whole input, only each
group's count and a few sample lines, so a 96 MB log runs in about 30 MB of
memory, and so does one four times its size. A file is read more than once —
once to choose the framer, once to sample records for the drivers' vote,
once to parse — so every choice is made on the whole file. A pipe can only
be read once. petit holds its first 4 MB; a pipe that ends there is read
exactly like a file, and one that runs longer is framed and parsed by
whatever its first 4 MB chose, with any later record that driver can't
read falling back to RawEntry on its own.

Multi-line messages. Stack traces, Python tracebacks and journalctl's
indented continuation lines put one message over several lines, and only the
first carries a timestamp:

	Sep 19 15:41:27 host01 ModemManager[1151]: <msg> couldn't check support...
	Sep 19 15:41:27 host01 gnome-shell[2856]: Object .GProxyVolume ... disposed
	                                          == Stack trace for context 0x5566 ==
	                                          #0   556694310aa8 i   resource:///...
	                                          #1   5566943109f8 i   resource:///...
	Sep 19 15:41:28 host01 gnome-shell[2856]: Object .GProxyVolume ... disposed
	                                          == Stack trace for context 0x5566 ==

Read line by line, that is seven records, four of which have no time, so the
syslog driver can't read them and the whole log falls back to RawEntry. The
multiline framer makes it three: a record starts at every line that begins
with a timestamp, and every other line belongs to the record above it. Each
crash is then one entry at 15:41:27, grouped with the other crashes like it,
and counted once in a graph.

It recognises the timestamps of syslog and journalctl, RFC 3339/5424, Python
logging, log4j/logback, Go, nginx, Apache, Snort, Kubernetes, the kernel,
Tomcat, java.util.logging, Redis and Unix time (docs/drivers.md has the
list). It only switches on when some continuation line is indented, which is
what a real multi-line log looks like; a log without one is framed line by
line, exactly as before. journalctl's "-- Boot ... --" lines are skipped.

docs/drivers.md covers each stage in detail and how to add to them.


Routine Operations
===============================================================================

Hash a syslog, removing reboots and all standard filters. By default petit will 
show a sample for all entries which are found three or less times.

	petit --hash --fingerprint /var/log/messages

Hash an Apache log

	petit --hash /var/log/httpd/access_log

Get a daemons report

	petit --daemon /var/log/messages

Get a host report

	petit --host /var/log/messages

Find qualitatively important words in your log. This is especially useful to 
help determine what should be monitored in swatch.

	petit --wordcount /var/log/messages

Graph the whole log, sized to the terminal (see Graphs, below)

	petit --graph /var/log/messages

Track a special word you are interested in by minute

	cat /var/log/messages | grep error | petit --mgraph

Show samples for each entry

	petit --hash --allsample /var/log/messages

Graphs
===============================================================================
A graph counts log entries per slice of time and draws one column per slice,
tallest where the log was busiest:

	petit --graph /var/log/httpd/error_log
	    #                       #                #
	    #                       ##               #    #
	#   #             #         ##  # #        # #    #  #
	#  ###            #         ##  # #####    # ##   #  ####  #
	# ####  # ## #### # #   #  ###### ######  #####   # ##### ##
	############################################################
	04                            16                           02

	Start Time:	 2011-04-10 04:00:00 		Minimum Value: 0
	End Time:	 2011-04-15 02:00:00 		Maximum Value: 7
	Duration:	 120 hours (2-hour columns) 			Scale: 1.1666666666666667

That is five days of an Apache error log on an 80-column terminal: 60
two-hour columns, labelled 04:00 on the 10th, 16:00 on the 12th and 02:00
on the 15th.

Time units. Every graph is built from one unit. The axis under the graph
labels the first, middle and last column with that column's starting value in
its unit, so an hour graph reads 00-23, not a date; Start Time and End Time
give the full dates.

	Unit    Fixed graph            --span   --graph column sizes   Axis label
	second  --sgraph   60 columns  30s      1, 5, 15, 30 s         second of the minute
	minute  --mgraph   60 columns  45m      1, 5, 15, 30 m         minute of the hour
	hour    --hgraph   24 columns  36h      1, 2, 3, 6, 12 h       hour of the day, 00-23
	day     --dgraph   31 columns  45d      1, 2, 7 d              day of the month
	month   --mograph  12 columns  18mo     1, 3, 6 mo             month, 01-12
	year    --ygraph   10 columns  12y      1, 5, 10 y             year, last two digits

Three ways to choose the window:

  --sgraph ... --ygraph  A fixed number of one-unit columns starting at the
                         first line of the log.
  --span N<unit>         N one-unit columns starting at the first line, e.g.
                         --span 90m. Units: s, m, h, d, mo, y. N is at least 6,
                         and the graph must fit the terminal or petit exits 2.
  --graph                The whole log, earliest entry to latest. petit picks
                         the finest column size from the table that fits the
                         terminal and draws only the columns the log covers.
                         Three and a half days is 84 one-hour columns on a
                         120-column terminal, or 42 two-hour columns on 80.
                         Logs out of time order are fine: the window runs
                         from the earliest entry, wherever it is.

Where a column starts. The window starts at the entry's time floored to the
unit: 10:07:12 becomes 10:07 for minutes, 10:00 for hours, the 1st of the
month for months. When a column spans several units it also starts on a round
value: 15-minute columns at :00/:15/:30/:45, 2-hour columns on even hours,
3-month columns in Jan/Apr/Jul/Oct, 5-year columns on years ending in 0 or 5.
Days are the exception: multi-day columns start on the entry's own day,
because months don't divide into 2 or 7 days. Months and years are counted on
the calendar, so every month column is exactly one calendar month.

The summary lines. Start Time and End Time are the starts of the first and
last columns. Duration is the whole window, with the column size when a
column spans several units. Minimum and Maximum Value are the fewest and
most entries in any one column, and Scale is how many entries one row of the
graph stands for.

Width. The terminal width comes from $COLUMNS or the terminal itself, and is
80 when petit's output is piped. Two characters are kept for the axis labels,
which run past the last column. --wide draws each column two characters wide,
so it fits half as many. --tick changes the character used to draw.

Lines petit can't read a time from are stamped with the year 1900, so they
fall outside any window that starts at a real time. --graph ignores them
unless no line in the log has a time.


Special Operations
===============================================================================

Create an on the fly driver for a nonstandard file format, then pipe it to Petit. 
Petit can hash files of non-standard types ok, but graphing requires the time 
values to be in the correct columns.

	cat /var/log/httpd/error_log | awk '{$1="";$5="";print}' | petit --sgraph