petit-log-crunchtools 4.1.0

Download and Information
===============================================================================
http://crunchtools.com/software/petit/

Installation
===============================================================================
    pip install petit-log-crunchtools
    # or: uv tool install petit-log-crunchtools
    # or: pipx install petit-log-crunchtools

Installs the petit command and the petit library. The PyPI distribution is
named petit-log-crunchtools, not petit, because "petit" on PyPI belongs to
an unrelated project. It follows the same naming convention as the rest of
the crunchtools fleet (gatehouse-crunchtools, mcp-gemini-crunchtools) — see
CHANGELOG.md's 3.0.0 and 3.1.1 entries.

A container image is also published, for CI or isolated execution:

    podman run --rm -v $(pwd):/data:ro,Z quay.io/crunchtools/petit --hash /data/some.log


Using petit as a library
===============================================================================
    from petit import hash_text, detect_format

    for group in hash_text(open("/var/log/messages").read()):
        print(group.count, group.pattern)

hash_text() groups lines by fingerprint, most frequent first. detect_format()
reports which driver claims the text; "RawEntry" means no driver recognised it.
analyze_text() returns the same groups plus how they were produced, and takes
the options the CLI has: hash_mode ("daemon", "host", "wordcount"),
collapse_fingerprints, and framer. A JSON array, JSON Lines, or a mail thread
is grouped per object or per message rather than per line, and the Analysis
accounts for both records and source lines. Normalization is chosen by the driver for the format
unless you pass filter_name or stopwords. docs/drivers.md explains how
drivers decide what to collapse.

Text goes in, data comes out. Nothing here reads a file, writes to stdout, or
exits the process — failures raise PetitError subclasses (EmptyLogError,
ParseError, DataFileError) for the caller to handle.


Public API and versioning
===============================================================================
Petit follows Semantic Versioning. Two things are public and covered by it:

  * the petit command line, its options and its output
  * the names exported from the petit package: analyze_text, hash_text,
    detect_format, Analysis, Group, and the PetitError hierarchy

Everything else is internal. The driver classes in petit.CrunchLog and
the hash classes in petit.LogHash may change in any release; they are
where new log formats get added, and pinning them would freeze that. If you
need something from them, ask for it to be exported rather than importing it
directly.


Why
===============================================================================
Log analysis is something that all systems administrators know they need to do. 
Many of us come to this point, either because there is a problem, there is a 
security requirement from the organization, or it keeps you up all night 
wanting to know what is going on in all of that data.

Looking for best practices for log analysis on this Internet is difficult at 
best. Many years ago, I discovered a script that hashed log files by removing 
all of their numbers and replacing them with "#" characters. The results of 
this simple algorithm were phenomenal, logs could be reduced by a factor of 
ten. This was much more readable, yet left much of the quality data that I 
needed to determine if there was a problem.

In the years since I discovered that simple algorithm, I have come to discover 
many techniques on text analysis which are commonly used in linguistics and 
anthropology to analyze natural languages. This has led me to develop very 
simple best practices for analyzing logs.

The Basics
===============================================================================
 1. Logs are made up of output which are programmed by human beings. There 
are no real restraints on what is output, other than, some cultural rules on 
being professional. This makes the output from programs very much a natural i
language. This also makes the output of someones program an approximation of 
the reality of what is happening inside a program. This is important to 
remember, logs are not perfect.

 2. When a systems administrator analyzes logs by changing them, he is 
creating an approximation of an approximation of reality in side a working 
program. This is not necessarily a bad thing, especially, when the programmer 
never gives you better than his approximation of reality anyway.

 3. In practice logs are made up of certainty and uncertainty. For example, I 
know what OpenSSH puts in the log during a login, because it is common. On the 
other hand, I do not now what a Compaq DL380 G3 will put in the log when it has 
a disk controller error. This is important to remember.

 4. The basic log analysis algorithm in Petit works to remove certainty, while 
leaving uncertainty. Stated another way, Petit quantitatively removes certainty, 
thereby leaving uncertainty, which by necessity requires qualitative analysis 
from a systems administrator

 5. After the algorithm has been applied, the output must be read by a systems 
administrator to determine if it is a normal or abnormal. Then abnormal entries 
can be acted on, hopefully before there is noticeable impact to your system. 


Routine Operations
===============================================================================

Hash a syslog, removing reboots and all standard filters. By default petit will 
show a sample for all entries which are found three or less times.

	petit --hash --fingerprint /var/log/messages

Hash an Apache log

	petit --hash /var/log/httpd/access_log

Get a daemons report

	petit --daemon /var/log/messages

Get a host report

	petit --host /var/log/messages

Find qualitatively important words in your log. This is especially useful to 
help determine what should be monitored in swatch.

	petit --wordcount /var/log/messages

Graph first 60 seconds in a syslog

	petit --sgraph /var/log/messages

Track a special work you are interested in by minute

	cat /var/log/messages | grep error | petit --mgraph

Show samples for each entry

	petit --hash --allsample /var/log/messages

Special Operations
===============================================================================

Create an on the fly driver for a nonstandard file format, then pipe it to Petit. 
Petit can hash files of non-standard types ok, but graphing requires the time 
values to be in the correct columns.

	cat /var/log/httpd/error_log | awk '{$1="";$5="";print}' | petit --sgraph