ColTrace

Static Pandas column lineage and potential impact analysis for Python projects. ColTrace reads source files, identifies supported transformations, and connects column versions across direct function calls. It never imports or executes the project or reads the project's input data.

Development preview: 0.1.0.dev4. The supported subset is deliberately small. Unsupported operations produce diagnostics and may interrupt lineage. A graph with no edge is not proof that a dependency does not exist.

Install locally

Requires Python 3.11+. From this checkout:

python -m venv .venv
# Activate .venv using your shell, then:
python -m pip install -e ".[dev,viz]"
coltrace --help

On Windows without activation, use .\.venv\Scripts\python.exe -m coltrace. The viz extra supplies PyVis for HTML export; basic analysis and JSON need only NetworkX and Typer. Pandas is not a runtime dependency. Publication to PyPI is a separate release step; these commands install the local project.

Try the complete example

coltrace scan examples/pricing_pipeline
coltrace trace preco_referencia
coltrace impact preco_referencia
coltrace graph preco_referencia --open
coltrace export preco_referencia --format json --output .coltrace/pricing.json

The example connects extraction, pricing, and feature functions through explicit calls in pipeline.py. No prices.csv file is needed to analyze it.

valor_produto -> preco_bruto -> preco_referencia -> margem
                                      |              |
                                      +--------------+-> margem_percentual
custo ---------------------------------------------> margem

impact preco_referencia identifies margem and margem_percentual as direct potential dependents. There is also a longer path through margem. A changed input does not necessarily change every downstream value.

scan writes .coltrace/scan.json by default. --out DIRECTORY selects another snapshot directory. Queries load that snapshot explicitly; they do not silently rescan the project. Rescan after editing source, or use --project DIRECTORY to analyze current files in memory. --from FILE selects an existing snapshot.

coltrace trace margem --project examples/pricing_pipeline --upstream --depth 2
coltrace impact preco_referencia --from .coltrace/scan.json --evidence confirmed
coltrace graph --from .coltrace/scan.json --output .coltrace/all.html

HTML files contain their visualization resources and work locally without a server or CDN. A selected column opens its journey: direct reads, applications, writes, derived columns and destinations. Frame passage is labeled separately from a direct column read. Callback control and configuration appear in the expandable creation history, rather than as direct uses of the selected column. The graph uses generously spaced, numbered blocks in a vertical flow. The scrollable sidebar contains the trace history, source code, and locations; selecting a step highlights its matching graph block, and vice versa. Open Details or Diagnostics without losing the current selection. The Outcomes tab shows derived-column destinations and the DataFrames carrying the selected column. Zoom in/out, use Fit all for an overview, or Reset view to return to a readable scale. --view columns shows the column dependency projection; --operations includes operation nodes in that projection. A graph without a selected column shows the whole project's column dependencies.

coltrace scan examples/text_pipeline
coltrace trace comment_text --function process_train
coltrace graph comment_text --function process_train --open

trace defaults to direct column actions, derived-column outcomes, and containing DataFrames. --full-history includes creation ancestry and callback internals, with control/configuration distinguished from value transformations. --diagnostics also displays dependency lists and relevant diagnostics. Rescan after an upgrade: schema 0.2 snapshots have no recorded history. See the column history guide and current checkpoint. The graph workspace checkpoint describes the current layout and additional column checks. See the column journey checkpoint for the usage-first view and its acceptance cases.

Identity and ambiguity

Different DataFrames keep different identities, even when both contain a column named score. Assignments create column versions. frame["a"] = frame["a"] + 1 connects the old and new versions; frame["a"] = 0 cuts the old value's lineage. History still records that overwrite without adding a false value dependency.

Selection by name uses the latest observed version within each DataFrame. If several unrelated identities remain, the CLI lists candidates and exits with status 3. Same-name copies connected by proven pass-through relations form one selectable family, whose original observed column is selected. A rename, value derivation, or shared label alone does not establish that equivalence. Select one explicitly using --column-id ID or --dataframe ID. Use --version for an older version, and --module or --function for exact source filters. IDs are opaque, deterministic within equivalent scans, and may change when source locations or call contexts change. They are not permanent database keys.

--depth counts column-to-column hops. --relation derive|rename|pass filters relations, while --evidence confirmed|all filters their evidence. Filtering out intermediate relations can disconnect a path. Direct and transitive lists use the shortest eligible distance, so a node reached both ways appears as direct.

Supported patterns

See the supported subset and boundaries for the exact contract. Calls, branches, dynamic columns, and Pandas operations outside that contract produce located diagnostics rather than invented connections.

The text pipeline example demonstrates independent train/evaluation flows, multi-output text cleaning, row filters, and exports. See the NLP checkpoint for what was validated against a real project and which downstream third-party operations remain opaque.

Evidence and completeness

• confirmed: supported syntax and receiver identity establish the relation.
• inferred: the relation depends on an explicit assumption, such as an uncalled function's DataFrame annotation or an unknown external schema during rename.
• unresolved: insufficient evidence across an unsupported boundary.

complete, partial, and unresolved describe how much of an operation is understood; evidence and completeness are separate. External column reads are marked as external/unknown origins. They do not identify the real upstream data source's column definition or guarantee its presence.

Use --diagnostics to inspect warnings and errors. Query output also includes relevant gaps. An inconclusive trace prints Not determined for empty dependency lists and shows the recognized DataFrame origin separately from column evidence. See troubleshooting incomplete traces. The analyzer is not a Python type checker or an execution validator. Its grammar is limited by the interpreter running ColTrace. Pandas imports are assumed to refer to the real library; environment/module impersonation is not resolved.

Python API

from coltrace import analyze_project
from coltrace.graph import LineageGraph

report = analyze_project("examples/pricing_pipeline")
graph = LineageGraph(report)
column = graph.select("preco_referencia")
impact = graph.impact(column.id)
print(report.to_json())

The earlier scan_project API remains available for raw occurrences and receiver evidence. Its discovery report (0.1) is a different format from the lineage snapshot (0.3). See discovery JSON and lineage JSON. Public facts are dataclasses; their serialized form is independent of NetworkX and contains no AST or executable objects.

Configuration

Only the scanned root's pyproject.toml is read:

[tool.coltrace]
exclude = ["generated", "legacy/*.py", "*_test.py"]

Glob patterns match case-sensitive, project-relative POSIX paths. Patterns with no / also match individual path components. Built-in exclusions cover .venv, venv, .git, __pycache__, .coltrace, pytest/mypy/Ruff caches, build, and dist. Linked files/directories and Windows reparse points are skipped. ScanConfig(exclude=("generated",)) overrides TOML settings and extends built-ins.

Development

python -m pytest -q
python -m ruff check src tests examples
python -m ruff format --check src tests examples
python -m mypy
python -m build

Tests include independent expected dependencies, forbidden edges, alias/version semantics, cross-module calls, CLI, malformed snapshots, offline HTML, and a sentinel proving that analyzed source is not executed. See the accuracy protocol, current checkpoint, decision log, and backlog.

The GitHub Actions workflow configures Windows/Linux with Python 3.11–3.13. It must run in the repository before those combinations can be claimed as verified.

Licensed under the MIT License. Copyright 2026 Cristiano Diniz.