petit-log-crunchtools 4.3.0

Download and Information
===============================================================================
http://crunchtools.com/software/petit/

Installation
===============================================================================
    pip install petit-log-crunchtools
    # or: uv tool install petit-log-crunchtools
    # or: pipx install petit-log-crunchtools

Installs the petit command and the petit library. The PyPI distribution is
named petit-log-crunchtools, not petit, because "petit" on PyPI belongs to
an unrelated project. It follows the same naming convention as the rest of
the crunchtools fleet (gatehouse-crunchtools, mcp-gemini-crunchtools) — see
CHANGELOG.md's 3.0.0 and 3.1.1 entries.

A container image is also published, for CI or isolated execution:

    podman run --rm -v $(pwd):/data:ro,Z quay.io/crunchtools/petit --hash /data/some.log


Using petit as a library
===============================================================================
    from petit import hash_text, detect_format

    for group in hash_text(open("/var/log/messages").read()):
        print(group.count, group.pattern)

hash_text() groups lines by fingerprint, most frequent first. detect_format()
reports which driver claims the text; "RawEntry" means no driver recognised it.
analyze_text() returns the same groups plus how they were produced, and takes
the options the CLI has: hash_mode ("daemon", "host", "wordcount"),
collapse_fingerprints, and framer. A JSON array, JSON Lines, or a mail thread
is grouped per object or per message rather than per line, and the Analysis
accounts for both records and source lines. Normalization is chosen by the driver for the format
unless you pass filter_name or stopwords. docs/drivers.md explains how
drivers decide what to collapse.

Text goes in, data comes out. Nothing here reads a file, writes to stdout, or
exits the process — failures raise PetitError subclasses (EmptyLogError,
ParseError, DataFileError) for the caller to handle.


Public API and versioning
===============================================================================
Petit follows Semantic Versioning. Two things are public and covered by it:

  * the petit command line, its options and its output
  * the names exported from the petit package: analyze_text, hash_text,
    detect_format, Analysis, Group, and the PetitError hierarchy

Everything else is internal. The driver classes in petit.CrunchLog and
the hash classes in petit.LogHash may change in any release; they are
where new log formats get added, and pinning them would freeze that. If you
need something from them, ask for it to be exported rather than importing it
directly.


Why
===============================================================================
Log analysis is something that all systems administrators know they need to do. 
Many of us come to this point, either because there is a problem, there is a 
security requirement from the organization, or it keeps you up all night 
wanting to know what is going on in all of that data.

Looking for best practices for log analysis on this Internet is difficult at 
best. Many years ago, I discovered a script that hashed log files by removing 
all of their numbers and replacing them with "#" characters. The results of 
this simple algorithm were phenomenal, logs could be reduced by a factor of 
ten. This was much more readable, yet left much of the quality data that I 
needed to determine if there was a problem.

In the years since I discovered that simple algorithm, I have come to discover 
many techniques on text analysis which are commonly used in linguistics and 
anthropology to analyze natural languages. This has led me to develop very 
simple best practices for analyzing logs.

The Basics
===============================================================================
 1. Logs are made up of output which are programmed by human beings. There 
are no real restraints on what is output, other than, some cultural rules on 
being professional. This makes the output from programs very much a natural i
language. This also makes the output of someones program an approximation of 
the reality of what is happening inside a program. This is important to 
remember, logs are not perfect.

 2. When a systems administrator analyzes logs by changing them, he is 
creating an approximation of an approximation of reality in side a working 
program. This is not necessarily a bad thing, especially, when the programmer 
never gives you better than his approximation of reality anyway.

 3. In practice logs are made up of certainty and uncertainty. For example, I 
know what OpenSSH puts in the log during a login, because it is common. On the 
other hand, I do not now what a Compaq DL380 G3 will put in the log when it has 
a disk controller error. This is important to remember.

 4. The basic log analysis algorithm in Petit works to remove certainty, while 
leaving uncertainty. Stated another way, Petit quantitatively removes certainty, 
thereby leaving uncertainty, which by necessity requires qualitative analysis 
from a systems administrator

 5. After the algorithm has been applied, the output must be read by a systems 
administrator to determine if it is a normal or abnormal. Then abnormal entries 
can be acted on, hopefully before there is noticeable impact to your system. 


Routine Operations
===============================================================================

Hash a syslog, removing reboots and all standard filters. By default petit will 
show a sample for all entries which are found three or less times.

	petit --hash --fingerprint /var/log/messages

Hash an Apache log

	petit --hash /var/log/httpd/access_log

Get a daemons report

	petit --daemon /var/log/messages

Get a host report

	petit --host /var/log/messages

Find qualitatively important words in your log. This is especially useful to 
help determine what should be monitored in swatch.

	petit --wordcount /var/log/messages

Graph the whole log, sized to the terminal (see Graphs, below)

	petit --graph /var/log/messages

Track a special word you are interested in by minute

	cat /var/log/messages | grep error | petit --mgraph

Show samples for each entry

	petit --hash --allsample /var/log/messages

Graphs
===============================================================================
A graph counts log entries per slice of time and draws one column per slice,
tallest where the log was busiest:

	petit --graph /var/log/httpd/error_log
	    #                       #                #
	    #                       ##               #    #
	#   #             #         ##  # #        # #    #  #
	#  ###            #         ##  # #####    # ##   #  ####  #
	# ####  # ## #### # #   #  ###### ######  #####   # ##### ##
	############################################################
	04                            16                           02

	Start Time:	 2011-04-10 04:00:00 		Minimum Value: 0
	End Time:	 2011-04-15 02:00:00 		Maximum Value: 7
	Duration:	 120 hours (2-hour columns) 			Scale: 1.1666666666666667

That is five days of an Apache error log on an 80-column terminal: 60
two-hour columns, labelled 04:00 on the 10th, 16:00 on the 12th and 02:00
on the 15th.

Time units. Every graph is built from one unit. The axis under the graph
labels the first, middle and last column with that column's starting value in
its unit, so an hour graph reads 00-23, not a date; Start Time and End Time
give the full dates.

	Unit    Fixed graph            --span   --graph column sizes   Axis label
	second  --sgraph   60 columns  30s      1, 5, 15, 30 s         second of the minute
	minute  --mgraph   60 columns  45m      1, 5, 15, 30 m         minute of the hour
	hour    --hgraph   24 columns  36h      1, 2, 3, 6, 12 h       hour of the day, 00-23
	day     --dgraph   31 columns  45d      1, 2, 7 d              day of the month
	month   --mograph  12 columns  18mo     1, 3, 6 mo             month, 01-12
	year    --ygraph   10 columns  12y      1, 5, 10 y             year, last two digits

Three ways to choose the window:

  --sgraph ... --ygraph  A fixed number of one-unit columns starting at the
                         first line of the log.
  --span N<unit>         N one-unit columns starting at the first line, e.g.
                         --span 90m. Units: s, m, h, d, mo, y. N is at least 6,
                         and the graph must fit the terminal or petit exits 2.
  --graph                The whole log, earliest entry to latest. petit picks
                         the finest column size from the table that fits the
                         terminal and draws only the columns the log covers.
                         Three and a half days is 84 one-hour columns on a
                         120-column terminal, or 42 two-hour columns on 80.
                         Logs out of time order are fine: the window runs
                         from the earliest entry, wherever it is.

Where a column starts. The window starts at the entry's time floored to the
unit: 10:07:12 becomes 10:07 for minutes, 10:00 for hours, the 1st of the
month for months. When a column spans several units it also starts on a round
value: 15-minute columns at :00/:15/:30/:45, 2-hour columns on even hours,
3-month columns in Jan/Apr/Jul/Oct, 5-year columns on years ending in 0 or 5.
Days are the exception: multi-day columns start on the entry's own day,
because months don't divide into 2 or 7 days. Months and years are counted on
the calendar, so every month column is exactly one calendar month.

The summary lines. Start Time and End Time are the starts of the first and
last columns. Duration is the whole window, with the column size when a
column spans several units. Minimum and Maximum Value are the fewest and
most entries in any one column, and Scale is how many entries one row of the
graph stands for.

Width. The terminal width comes from $COLUMNS or the terminal itself, and is
80 when petit's output is piped. Two characters are kept for the axis labels,
which run past the last column. --wide draws each column two characters wide,
so it fits half as many. --tick changes the character used to draw.

Lines petit can't read a time from are stamped with the year 1900, so they
fall outside any window that starts at a real time. --graph ignores them
unless no line in the log has a time.


Special Operations
===============================================================================

Create an on the fly driver for a nonstandard file format, then pipe it to Petit. 
Petit can hash files of non-standard types ok, but graphing requires the time 
values to be in the correct columns.

	cat /var/log/httpd/error_log | awk '{$1="";$5="";print}' | petit --sgraph