barca

Runs Python functions as a dependency graph and caches their results.

Barca runs Python functions as a dependency graph and caches their results.

You mark functions with @asset, @sensor or @task and declare their inputs. Qualified imports (@barca.asset()) and imported aliases (@a() after from barca import asset as a) work for decorators and helpers. The barca binary, written in Rust, reads the source without importing it, works out what needs to run, and runs it in Python worker processes. Results are stored as files under .barca/. An asset runs again only when its code or its inputs change; sensors and tasks run every time. Project helper references in defaults, decorators, exception handler types, set literals, match patterns, assignment targets and loop else: blocks contribute to cache hashes. Module-level control-flow bindings, unpacked/augmented constants and helper-module star imports retain their dependencies conservatively.

There is no server to run and no configuration file to write. barca serve adds schedules, an HTTP API and a web UI when you want them.

# pipeline.py
from barca import asset


@asset()
def raw_data() -> list[dict]:
    return [{"x": 1}, {"x": 2}, {"x": 3}]


@asset(inputs={"data": raw_data})
def summary(data: list[dict]) -> dict:
    return {"count": len(data), "total": sum(d["x"] for d in data)}

$ barca get pipeline.py
[barca] 2/2 steps | done in 0.0s
Run 51fabebab1c0 | all assets in 0.135s (2 steps, 1 phase)

Value:
{
  "count": 3,
  "total": 6
}

$ barca get pipeline.py
Run 51fa8fe09d90 | all assets in 0.005s (0 steps, 1 phase)
...

The second run executes 0 steps: both results come from the cache. Output shown in this file is from barca 0.18.0.

Install

uv add barca          # or: pip install barca

Python 3.12 or later. The wheel contains the barca binary, the decorators and a Python API; wheels are published for macOS on Apple Silicon and for Linux on x86-64 and arm64, glibc and musl. uv is recommended, not required: barca runs your steps with the Python of the environment it is installed in, or python3 on PATH.

Optional extras: barca[parquet] (pyarrow, for pandas DataFrames), and barca[s3], barca[gcs], barca[azure] or barca[remote] for a remote store. For barca sql, install SQL support with uv add 'barca[sql]'.

Where results live

.barca/metadata.db                          run history and the record of what is cached
.barca/artifacts/<node>/<run_hash>.<ext>    one file per result: .json, .pkl, .parquet or .csv
.barca/envs/<env>/                          the same two, for each --env other than the default

• .barca/ is created in the project root: the nearest directory at or above the current one that holds a barca.toml (an empty file is enough), else the current directory.
• Do not commit .barca/. Barca writes a .gitignore inside it, so git already ignores it and nothing needs adding to your own.
• Results are cached by run hash, which covers the function's code and its inputs, not the bytes of the output.
• Results are always written locally first. A remote store is an optional shared copy: set BARCA_REMOTE_URI=s3://my-bucket/barca/my-project (or gs://, abfs://) and other machines using the same location get cache hits for what this one computed.
• Required history writes commit terminal status and step outcomes together. A write failure reports an infrastructure error and preserves earlier history and recorded progress.
• Completed results are recorded locally during a run, after confirmed upload when using a remote store. A killed run can reuse recorded results; new shared progress is published every minute in healthy operation, then the final outcome at run end.
• --env <name> (or BARCA_ENV) keeps a separate cache and history, so dev and prod do not share results.

barca get summary pipeline.py --env prod
barca status pipeline.py --env prod

More: barca docs cache, barca docs remote.

For one process, BARCA_REMOTE=off barca get total pipeline.py uses local artifacts and history. BARCA_STATE=off barca serve pipeline.py keeps metadata local while retaining configured artifact sharing; optimistic shared metadata remains refused by serve.

Seeing what barca sees

Ask the CLI instead of reading source files or opening files under .barca/. None of these commands runs a step.

barca list                                # every asset, sensor and task, with its inputs
barca status                              # per node: cached or stale and why, last run, rows and columns
barca get summary --dry-run               # what this exact command would run or serve from cache
barca plan                                # the execution plan as JSON
barca sql "select * from raw_data"        # query a cached result with DuckDB

After changing summary in the example above:

$ barca status
NAME      KIND   STATE   WHY           LAST RUN                           SHAPE           DEPS
raw_data  asset  cached  materialized  success 2026-10-07 18:14:53 0.00s  3 rows x 1 col  -
summary   asset  stale   changed       success 2026-10-07 18:14:53 0.00s  dict (2 keys)   raw_data

1 cached, 1 stale, 0 never run, 0 partial, 0 unknown, 0 always run

barca sql makes every node with a json, CSV or parquet result a view named after its declared name=, or its function when unset, so you can inspect data without writing a step or a script. Partitioned views include current known keys; removed-key results remain in history:

$ barca sql "select count(*) as n, sum(x) as total from raw_data"
n  total
3  6

$ barca sql "describe raw_data"
column_name  column_type  null  key  default  extra
x            BIGINT       YES

At most 100 rows are returned unless you pass --limit N or --all. Pickled results cannot be queried.

Scripts and AI agents. get, run and the inspection commands print their result on stdout: a table or summary in a terminal, JSON when piped or captured (--json and --pretty override). Progress and errors go to stderr; in JSON mode an error is one JSON line with error, code, kind and remediation. Exit codes: 0 ok, 1 a step failed, 2 usage error, 3 barca or infrastructure failure, 130 cancelled. A reader that stops early (barca list | head -1) is not an error: nothing panics, a run still finishes and is recorded, and the exit code is unchanged.

$ barca sql "select * from raw_data where x > 1" --json
{"columns": ["x"], "rows": [{"x": 2}, {"x": 3}], "total": 2, "truncated": false}

(Printed indented.) An agent should load SKILL.md, also printed by barca docs skill. barca docs agents is the full reference, and barca docs contract lists every command, flag, exit code and JSON schema, each marked stable or experimental.

More: barca docs status, barca docs sql.

Recomputes also check artifact bytes: if unchanged code and inputs produce a different SHA-256 digest, barca warns and sets steps[].bytes_changed: true. barca status retains the last such event. This is a byte comparison; serializer metadata can also differ.

Data from outside comes in through a sensor

An asset that reads a file, a bucket or a table in its own body has the same code and the same inputs on every run, so it is computed once and then served from cache, even after the data changes. Put a @sensor in front of it that returns something identifying the current version of the data, and make the asset take the sensor as an input:

import hashlib
from pathlib import Path

from barca import asset, sensor


@sensor()
def orders_version() -> tuple[bool, str]:
    # anything that identifies the version: an etag, a last-modified time
    return True, hashlib.md5(Path("orders.csv").read_bytes()).hexdigest()


@asset(inputs={"version": orders_version})
def orders(version: str) -> list:
    return Path("orders.csv").read_text().splitlines()[1:]


@asset(inputs={"rows": orders})
def order_count(rows: list) -> int:
    return len(rows)

barca get order_count pipeline.py    # 3 steps run
barca get order_count pipeline.py    # 1 step runs: the sensor. orders and order_count are cached
# ... a row is appended to orders.csv ...
barca get order_count pipeline.py    # 3 steps run: the sensor returned a new value

A sensor runs every time, and the value it returns is part of the run hash of the assets that read it. Return only what identifies the data: a value that changes on every run, such as a timestamp, re-runs those assets every time.

To recompute by hand, name the assets:

barca get order_count pipeline.py --refresh orders                # orders and everything downstream of it
barca get order_count pipeline.py --refresh orders --no-cascade   # only orders; barca warns that order_count is out of date
barca get order_count pipeline.py --refresh-all                   # every asset the target depends on

--refresh takes one comma-separated list (--refresh a,b). Use --refresh-all to force all assets to recompute. Do not delete files under .barca/ to force a recompute.

Cache hashes follow project references in assignment receivers and indices, loop else: blocks, match patterns and ordinary helper decorators. Unpacked/augmented module constants, module-level control-flow bindings and helper-module star imports retain their dependencies conservatively; edits to other definitions in those modules can also invalidate consumers.

More: barca docs cache, which also lists what the run hash does not see (installed packages and environment variables not declared with env=).

Large inputs

A DataFrame, Arrow table or DuckDB relation is stored as parquet by default. Use @asset(serializer="csv") for CSV artifacts or @sink("exports/orders.csv") for a CSV export. CSV has a header and no pandas index; readers infer types. The annotation on the consuming parameter decides how it is read back:

import duckdb
from barca import asset


@asset()
def events() -> duckdb.DuckDBPyRelation:
    return duckdb.sql("select range as id, range % 10 as bucket from range(100000)")


@asset(inputs={"events": events})
def per_bucket(events: duckdb.DuckDBPyRelation) -> duckdb.DuckDBPyRelation:
    # only the `bucket` column is read
    return events.aggregate("bucket, count(*) as n").order("bucket")

A declared input is always loaded, whether or not the function uses it. Barca warns when it plans a step that never uses one:

[barca] warning: pipeline.py:report never uses its input `events`. It is still loaded in full each time the step runs, and it counts toward the step's cache key. Use it, remove it from inputs=, or rename the parameter `_events` if it is there for ordering only (a `_` input is not loaded and never flagged)

json and pickle inputs are always read whole.

More: barca docs big-inputs, barca docs types.

Commands

Run JSON reports elapsed time and attempts for executed steps; slow imports are reported separately as import_seconds. barca docs status explains retry and partition timing; barca history --json and barca stats expose recorded attempts.

The target comes first, then files or directories. Both are optional: with no files, barca reads every .py file under the project root that imports barca (barca docs discovery). A project star import with a literal __all__ preserves Barca decorator and group bindings when its exports do not collide with their local names.

barca get summary pipeline.py     # an asset and what it depends on
barca get pipeline.py             # every asset and sensor in the file; tasks are skipped
barca run publish pipeline.py     # a task and what it depends on

get is for assets and uses the cache. run is for tasks: the task always runs, and the assets it depends on come from the cache as with get. Using the wrong one is a usage error (exit 2) that names the right one. barca pipeline.py is short for barca get pipeline.py.

Failed runs end stderr with the failed step, its first error line and the count of skipped downstream nodes. JSON retains the detailed step and target outcomes.

--fields a,b keeps only those keys on each item of the JSON output. Every barca <command> --help ends with runnable examples. For a full refresh use --refresh-all (or refresh_all=True in Python); the deprecated alias is removed in 0.24.0.

Partitions

Partitions run one asset once per key. Each key is cached on its own.

from barca import asset, collect, partitions

REGIONS = ["emea", "amer", "apac"]


@asset(partitions={"region": partitions(REGIONS)})
def sales(region: str) -> dict:
    return {"region": region, "revenue": len(region) * 100}


@asset(inputs={"all_sales": collect(sales)})    # every key's result, as one list
def total(all_sales: list[dict]) -> dict:
    return {"regions": len(all_sales), "revenue": sum(s["revenue"] for s in all_sales)}

barca get total pipeline.py runs 4 steps, then 0 on a second run. After "latam" is added to REGIONS it runs 2: sales for latam, then total. The same holds when the keys are a literal list written inside the decorator (up to 0.18 that form ran every key again). No single key can be targeted or refreshed, and barca get sales returns one key's value. In barca sql a partitioned asset is one view with a partition column (region=emea).

More: barca docs partitions.

Schedules and the server

barca serve runs a cron scheduler, an HTTP API and a web UI. It binds to 127.0.0.1 by default. --host 0.0.0.0 listens on every interface for containers and VMs. There is no authentication; keep a reachable server on a private network or behind an authenticating proxy.

# job.py
from barca import task, Schedule


@task(freshness=Schedule("*/10 * * * *"))    # every 10 minutes; a 6-field cron starts with seconds
def refresh() -> None:
    print("refreshing")

barca list job.py                        # shows each schedule and its next fire time
barca serve job.py                       # port 8274; the web UI is at /ui/
barca serve job.py --timezone utc        # evaluate cron in UTC (default: local time)
barca serve job.py --host 0.0.0.0         # every interface; authenticate at the proxy

Schedules fire only while barca serve is running; barca get and barca run do not look at the clock. A scheduled task runs on every tick. A scheduled asset is brought up to date on every tick, which means it runs only if it is not cached: data from outside still has to come in through a sensor. A scheduled sensor's tick runs the sensor and does not trigger the assets that read it. The other freshness values, Always (the default) and Manual, are recorded and shown by barca list and have no effect at run time today.

Runs started over HTTP are asynchronous: POST returns a run_id, and you poll /status/<run_id>.

curl localhost:8274/health                  # {"read_only":false,"scheduler":true,"status":"ok","version":"0.23.0"}
curl localhost:8274/schedule                # each schedule: last and next fire, last status
curl -XPOST localhost:8274/run              # every asset and sensor (tasks are skipped) -> {"run_id":"..."}
curl -XPOST localhost:8274/get/summary      # one asset and what it depends on
curl -XPOST localhost:8274/run/publish      # one task; this recomputes every asset it depends on
curl localhost:8274/status/<run_id>         # {"status":"complete","result":{...}}
curl -XDELETE localhost:8274/run/<run_id>   # cancel a run in flight (409 if it already finished)

More: barca docs scheduling, the server API reference, and Deploying for running it behind nginx.

Python API

import barca

barca.get("pipeline.py")              # every asset; returns the last asset's value: {"count": 3, "total": 6}
barca.get("summary", "pipeline.py")   # one asset, cache-aware: {"count": 3, "total": 6}
barca.plan("pipeline.py")["total_steps"]   # 2

The decorators return the function unchanged, so decorated functions can be called and unit tested as ordinary Python.

Datadog job traces

Set BARCA_TELEMETRY=datadog to report job runs and steps to a Datadog Agent. Install barca[datadog] in your job environment to add named Python executions and nested library traces in the same job trace. In the <DD_SERVICE>-python service, select barca.execute to see jobs by their canonical names. See barca docs telemetry for setup, tags, and delivery limits.

barca logs RUN_ID --pretty reads recorded step stdout without progress messages. Use --step report for one function or exact node id, or --json for [{node_id, line}]. JSON get/run output suppresses step prints; other modes prefix them with the node id on stderr.

The manual

barca docs lists the topics and barca docs <topic> prints one. Start with overview. The site, barca.sh, has the same material plus a guide and patterns.

Development

git clone https://github.com/barca-orc/barca.git
cd barca
uv venv
source .venv/bin/activate
uv pip install maturin
maturin develop --uv --release --extras test   # builds the binary and installs it into .venv
cargo test

See Architecture and Contributing. benchmarks/ holds the same pipelines written for barca, Dagster and Prefect; benchmarks/RESULTS.md records measurements with their date and machine. Barca is pre-1.0: a breaking change to the CLI ships in a minor release with a "Breaking" line in the release notes.

License

MIT