Tee_Logger

A Python logger that writes to a dated log file and optionally mirrors output to stdout with ANSI colors. Supports in-place compression, automatic log archival, and smart caller line attribution.

Features

• Tee logging — file-only (debug, info, warning, error, critical, exception, …) or stdout+file (teeprint, teewarning, teeerror, teeexception, …)
• Dated layout — {programName}_log/YYYY-MM-DD/{programName}_YYYY-MM-DD_HH-MM-SS.log, rolling over to a new day at midnight
• Caller attribution — log lines include abbreviated file:line of the direct caller
• Compression — write .zst/.xz/.gz/.bz2 logs directly (crash-safe, see below), or tar.xz archive old day-folders
• Maintenance — auto-compress and delete logs by age, in a background thread

Installation

pip install tee-logger

No runtime dependencies; src/Tee_Logger.py also works when copied on its own.

Requires Python 3.6+. In-place zstd compression requires Python 3.14+ with zstd support in the build; otherwise it falls back to xz.

Quick start

from Tee_Logger import teeLogger

tl = teeLogger(systemLogFileDir='.', programName='MyApp')
tl.info('file only')
tl.teeprint('stdout and file')
tl.teeerror('error to both')

Compressed logs:

tl = teeLogger(
    programName='MyApp',
    in_place_compression=True,  # first available of zstd, xz, gzip, bz2; or name one
    compression_level=3,        # optional, backend-specific
)

Compressed logs are written in batches so they compress well, and flushed to disk:

• immediately for WARNING and above (warning, error, critical, exception, …),
• otherwise at most flush_interval seconds (default 30) after a record is written, even if the program goes quiet.

So if the process is killed (OOM, SIGKILL, segfault), at most the last flush_interval seconds of DEBUG/INFO records are lost; warnings and errors are already on disk. gzip and zstd flush mid-stream; xz and bz2 cannot, so each flush finishes the current stream and starts a new one in the same file (xz -dc, bzip2 -dc, lzma.open, bz2.open read them all). A file that is still being written may end in an unfinished stream: zcat/zstd -dc print everything and then warn about it. flush_interval=0 flushes every record, which costs a lot of compression (gzip/zstd files about twice as large).

Disable file logging:

tl = teeLogger(noLog=True)

Configuration

Per-call override: tl.info('msg', callerStackDepth=3).

Shared logger name

programName is passed to logging.getLogger(programName). A second teeLogger with the same name reuses that logger and replaces the previous file handler so only one active log file is attached. That is intentional (one live log per name). Use distinct programName values if two components need independent log files at the same time.

Log layout

MyApp_log/
├── MyApp_latest.log          # symlink to active log (Unix); .zst/.xz/.gz/.bz2 suffix when compressed
├── 2024-11-30.tar.xz         # archived day-folder
└── 2025-02-10/
    └── MyApp_2025-02-10_01-33-26.log

A process that runs past midnight closes its file and continues in the new day's folder; the first line there names the file it continued from.

Log files are opened without following symlinks, so a link planted at a predictable log path cannot redirect writes into another file. Still, avoid pointing systemLogFileDir (or running with the default .) at a directory other users can write to.

The logger's own messages (the Log file: … banner, fallbacks, cleanup failures) go to stderr, so they never mix into a tool's piped stdout. The banner is hidden when suppressPrintout is on.

Log line format:

2025-02-10 01:33:26,623 [INFO    ] [MyApp:42      ] message text

Caller stack depth

With the default callerStackDepth=-1, teeLogger skips its own frames and records the direct caller — your code or your wrapper, not internal Tee_Logger methods.

def my_wrapper(msg):
    tl.info(msg)          # log shows this line

def business():
    my_wrapper('hello')   # log shows my_wrapper, not business

Use a non-negative callerStackDepth or a per-call override for advanced tuning.

Log maintenance

On initialization and at each midnight rollover, {programName}_log/ is scanned for YYYY-MM-DD folders (age comes from the folder name):

1. Delete folders and archives older than deleteLogAfterYears
2. Compress folders older than compressLogAfterMonths to .tar.xz

Folders with a file modified in the last 24 hours are left alone, in case another process is still writing there. The work runs in a background thread; the program exits only after an archive in progress is finished. cleanup_old_logs() runs the same maintenance synchronously.

Compression uses system tar+xz when available, otherwise Python's tarfile. It is safe against crashes and concurrent cleanups: the folder is first renamed to YYYY-MM-DD.compressing-<pid>/, the archive is written to a temporary file and renamed into place, and only then is the folder deleted. If archiving fails, the folder is put back; a claim left by a crashed process is picked up again after 24 hours.

API reference

Full docstrings and examples live in the module:

python -c "import Tee_Logger; help(Tee_Logger.teeLogger)"

Logging methods (all accept callerStackDepth=):

Public helpers: abbreviate_filename, pretty_format_table, printWithColor, getCallerInfo, compress_folder, teeLogger.

Testing

Primary entry point (pytest, including doctests in src/Tee_Logger.py):

uv sync --extra dev
uv run pytest

Secondary entry points:

python src/Tee_Logger.py --test
python src/Tee_Logger.py -V

CI runs on Python 3.9, 3.11, 3.12, and 3.14, plus 3.6 in a python:3.6 container.

Upgrading to 6.43 from 6.42

• When a compressed log file will use it (in_place_compression set, noLog off, and a log directory other than /dev/null, /dev/stdout or /dev/stderr), flush_interval must be a number: NaN raises ValueError and a non-number (None, '30') raises TypeError when the logger is made, and the value is kept as a float. 6.42 accepted them: None and '30' then printed a logging error for every record below WARNING, and NaN kept its flusher thread busy on a CPU core while records reached disk only on WARNING and above and at close. Otherwise the value is still not checked.
• flush_interval=float('inf') no longer prints a traceback from the flusher thread; compressed logs then reach disk on WARNING and above and at close, as in 6.42.
• Console and diagnostic lines are written with one write() each instead of through print(), so lines from several threads no longer interleave, and a program that replaces builtins.print no longer changes tee-logger's output.
• Fixed: a failed background flush is reported once instead of stopping the flusher for good (gzip and zstd then retry it; with xz and bz2 the records of the stream that failed are still lost); a failed close of the old log at midnight, or a stderr that cannot be written, no longer makes a log call raise or leaves the logger without a log file; records other threads log during the midnight switch are no longer lost; a signal handler that logs while its own thread is rolling the log over at midnight no longer deadlocks the process.

Upgrading to 6.42 from 6.40 or earlier

• The default programName drops the file extension: myscript.py now logs to myscript_log/ instead of myscript.py_log/. Pass programName='myscript.py' to keep the old folder, or move it.
• in_place_compression=True picks zstd on Python 3.14+ (previously always xz), so the file suffix may change.
• compression_level now takes effect; it was silently ignored before.
• The Log file: banner and other internal messages moved from stdout to stderr.
• python-dateutil is no longer a dependency.
• When the log directory is not writable, logs fall back to $TMPDIR/teeLogger-<uid>/ instead of /tmp directly.

License

GPL-3.0-or-later