Skip to main content

SciLineage

Lineage tracking for Python data pipelines.

SciLineage is a lightweight library for building data processing pipelines with automatic provenance tracking. It captures the full computational lineage of your results, enabling reproducibility and intelligent caching.

Features

  • Automatic Lineage Tracking: Every computation captures its inputs and function, building a complete provenance graph
  • Input Classification: Automatically distinguishes variable inputs from constants for accurate lineage
  • Pluggable Caching: Register a backend via configure_backend() to enable cache lookups via lineage hashes
  • Lightweight: Core dependency is only scicanonicalhash
  • Type Safe: Full type hints throughout

Installation

pip install scilineage

Quick Start

Basic Usage

from scilineage import lineage_fcn

@lineage_fcn
def process(data, factor):
    return data * factor

# Call returns a LineageFcnResult, not the raw result
result = process([1, 2, 3], 2)

# Access the computed value
print(result.data)  # [2, 4, 6]

# Access lineage information
print(result.invoked.inputs)         # {'arg_0': [1, 2, 3], 'arg_1': 2}
print(result.invoked.fcn.fcn.__name__)  # 'process'

Multi-Output Functions

@lineage_fcn(unpack_output=True)
def split_data(data):
    mid = len(data) // 2
    return data[:mid], data[mid:]

first, second = split_data([1, 2, 3, 4])
print(first.data)   # [1, 2]
print(second.data)  # [3, 4]

Chaining Computations

@lineage_fcn
def normalize(data):
    max_val = max(data)
    return [x / max_val for x in data]

@lineage_fcn
def scale(data, factor):
    return [x * factor for x in data]

raw = [10, 20, 30, 40]
normalized = normalize(raw)
scaled = scale(normalized, 100)

print(scaled.data)  # [25.0, 50.0, 75.0, 100.0]

Extracting Lineage

from scilineage import extract_lineage, get_upstream_lineage

@lineage_fcn
def step1(x):
    return x + 1

@lineage_fcn
def step2(x):
    return x * 2

result = step2(step1(5))

lineage = extract_lineage(result)
print(lineage.function_name)   # 'step2'
print(lineage.function_hash)   # SHA-256 of function bytecode

chain = get_upstream_lineage(result)
for record in chain:
    print(f"{record['function_name']}: inputs={record['inputs']}")

Manual Interventions

from scilineage import manual

# Step outside the pipeline for a manual correction
edited_data = [1, 2, 3]

# Re-enter the pipeline — the intervention is documented in lineage
corrected = manual(
    edited_data,
    label="outlier_removal",
    reason="amplitude < 0.1 in trial 3 is sensor artifact",
)

API Reference

@lineage_fcn(unpack_output=False, unwrap=True, generates_file=False)

Decorator to convert a function into a LineageFcn.

  • unpack_output: Whether to unpack a tuple return into separate LineageFcnResults
  • unwrap: If True, automatically unwrap LineageFcnResult inputs to their raw data
  • generates_file: If True, marks the function as producing files as side effects

LineageFcnResult

Wrapper around computed values that carries lineage.

  • .data: The actual computed value
  • .invoked: The LineageFcnInvocation that produced this
  • .hash: Unique hash based on computation lineage
  • .output_num: Index for multi-output functions

LineageFcnInvocation

Represents a specific function invocation with captured inputs.

  • .fcn: The parent LineageFcn (function wrapper)
  • .inputs: Dict of captured input values
  • .outputs: Tuple of LineageFcnResult results
  • .compute_lineage_hash(): Generate lineage hash for cache key computation

LineageFcn

The decorated function wrapper.

  • .fcn: The original wrapped function
  • .hash: SHA-256 hash of function bytecode
  • .invocations: All LineageFcnInvocations created from this

License

MIT License - see LICENSE for details.

Metadata

Release files for scilineage 0.1.59

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scilineage 0.1.59
File Size Uploaded Provenance Actions
scilineage-0.1.59.tar.gz 7.7 kB With provenance Details for scilineage-0.1.59.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scilineage 0.1.59
File Interpreter ABI Platform Actions
scilineage-0.1.59-py3-none-any.whl Python 3 none any Details for scilineage-0.1.59-py3-none-any.whl

Total release size: 15.1 kB

Release files / scilineage-0.1.59.tar.gz

Download URL scilineage-0.1.59.tar.gz
Size 7.7 kB
Tags Source
SHA-256 checksum
How to use checksums
83ee8b28eb282bf3dab9a55620cdf8aa48c4406f6462078de337e74ad227c8b5
BLAKE2b-256 checksum
How to use checksums
fae9e1b6e8ff9f928b9f21ecd4318bb5e2335700a52a86058669ff0cd0bb8bb0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Release files / scilineage-0.1.59-py3-none-any.whl

Download URL scilineage-0.1.59-py3-none-any.whl
Size 7.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1c25a50e92894443aacedfea9ec15ec413c3a3917f80ef4a9782bf5ccdcd365d
BLAKE2b-256 checksum
How to use checksums
5d7984af5ff73ae1c87cb321251c63589976cb3276b4078d864edd404e144e5c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Security for scilineage 0.1.59

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for scilineage-0.1.59.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.59 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page