SciLineage
Lineage tracking for Python data pipelines.
SciLineage is a lightweight library for building data processing pipelines with automatic provenance tracking. It captures the full computational lineage of your results, enabling reproducibility and intelligent caching.
Features
- Automatic Lineage Tracking: Every computation captures its inputs and function, building a complete provenance graph
- Input Classification: Automatically distinguishes variable inputs from constants for accurate lineage
- Pluggable Caching: Register a backend via
configure_backend()to enable cache lookups via lineage hashes - Lightweight: Core dependency is only
scicanonicalhash - Type Safe: Full type hints throughout
Installation
pip install scilineage
Quick Start
Basic Usage
from scilineage import lineage_fcn
@lineage_fcn
def process(data, factor):
return data * factor
# Call returns a LineageFcnResult, not the raw result
result = process([1, 2, 3], 2)
# Access the computed value
print(result.data) # [2, 4, 6]
# Access lineage information
print(result.invoked.inputs) # {'arg_0': [1, 2, 3], 'arg_1': 2}
print(result.invoked.fcn.fcn.__name__) # 'process'
Multi-Output Functions
@lineage_fcn(unpack_output=True)
def split_data(data):
mid = len(data) // 2
return data[:mid], data[mid:]
first, second = split_data([1, 2, 3, 4])
print(first.data) # [1, 2]
print(second.data) # [3, 4]
Chaining Computations
@lineage_fcn
def normalize(data):
max_val = max(data)
return [x / max_val for x in data]
@lineage_fcn
def scale(data, factor):
return [x * factor for x in data]
raw = [10, 20, 30, 40]
normalized = normalize(raw)
scaled = scale(normalized, 100)
print(scaled.data) # [25.0, 50.0, 75.0, 100.0]
Extracting Lineage
from scilineage import extract_lineage, get_upstream_lineage
@lineage_fcn
def step1(x):
return x + 1
@lineage_fcn
def step2(x):
return x * 2
result = step2(step1(5))
lineage = extract_lineage(result)
print(lineage.function_name) # 'step2'
print(lineage.function_hash) # SHA-256 of function bytecode
chain = get_upstream_lineage(result)
for record in chain:
print(f"{record['function_name']}: inputs={record['inputs']}")
Manual Interventions
from scilineage import manual
# Step outside the pipeline for a manual correction
edited_data = [1, 2, 3]
# Re-enter the pipeline — the intervention is documented in lineage
corrected = manual(
edited_data,
label="outlier_removal",
reason="amplitude < 0.1 in trial 3 is sensor artifact",
)
API Reference
@lineage_fcn(unpack_output=False, unwrap=True, generates_file=False)
Decorator to convert a function into a LineageFcn.
unpack_output: Whether to unpack a tuple return into separateLineageFcnResultsunwrap: If True, automatically unwrapLineageFcnResultinputs to their raw datagenerates_file: If True, marks the function as producing files as side effects
LineageFcnResult
Wrapper around computed values that carries lineage.
.data: The actual computed value.invoked: TheLineageFcnInvocationthat produced this.hash: Unique hash based on computation lineage.output_num: Index for multi-output functions
LineageFcnInvocation
Represents a specific function invocation with captured inputs.
.fcn: The parentLineageFcn(function wrapper).inputs: Dict of captured input values.outputs: Tuple ofLineageFcnResultresults.compute_lineage_hash(): Generate lineage hash for cache key computation
LineageFcn
The decorated function wrapper.
.fcn: The original wrapped function.hash: SHA-256 hash of function bytecode.invocations: AllLineageFcnInvocations created from this
License
MIT License - see LICENSE for details.
1 maintainer
Metadata
1 maintainer
Release files for scilineage 0.1.47
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| scilineage-0.1.47-py3-none-any.whl | Python 3 | none | any | Details |
Release files / scilineage-0.1.47-py3-none-any.whl
| Download URL | scilineage-0.1.47-py3-none-any.whl |
|---|---|
| Size | 7.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
cdca12c75bda7428728be4745b68ca54ea094e453933910a1871e2e3cfc1bf56
|
|
BLAKE2b-256 checksum How to use checksums |
d8360622180703380bed1777bc92bf9cce710c2b8f8d0f75d5d27f9e70f4ea48
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.
Transparency logSecurity for scilineage 0.1.47
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 1 file in this release has been uploaded with provenance metadata.
Attestation for scilineage-0.1.47-py3-none-any.whl
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 20, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
