dlt-source-aquabyte

An installable dlt source package that loads aquaculture data from the Aquabyte API v3 into any dlt destination: sites, biomass, lice counts, welfare scores, behaviour and environmental readings.

Records land as the API returns them — nothing renamed, nothing dropped, no invented child tables. What is added is named up front rather than discovered later: _dlt_valid_from and _dlt_valid_to on the versioned sites table. Column names are the API's own field names, in dlt's usual snake_case.

Everything else is mechanics: auth, pagination, envelope unwrapping, incremental cursors, and overridable key and write-disposition defaults. Reshaping belongs in your transform layer, where you can change it without waiting for a release. Its only dependency is dlt itself — destination, orchestrator, secrets manager and log routing stay your choices.

Install

uv add dlt-source-aquabyte "dlt[duckdb]"   # any dlt destination works; DuckDB is the one below

Quick start

Put the API base URL and your key in a .dlt/ directory beside the script you are about to run.

.dlt/config.toml:

[sources.aquabyte]
base_url = "https://api.aquabyte.ai/v3/"
initial_date = "2020-01-01"             # first-run start for the date-based cursors
initial_time = "2020-01-01T00:00:00Z"   # first-run start for the time-based cursors

.dlt/secrets.toml:

[sources.aquabyte]
api_key = "your-api-key-here"

Then load every resource:

import dlt
from dlt_source_aquabyte import aquabyte_source

pipeline = dlt.pipeline(
    pipeline_name="aquabyte",
    destination="duckdb",     # any dlt destination
    dataset_name="aquabyte_data",
)
print(pipeline.run(aquabyte_source()))

The two initial_* values are the first-run start for the resources that keep an incremental cursor; a run of only sites or environmental_latest needs neither, and a cursor resource missing one fails with an error naming it. How far back your data goes differs per endpoint and per account, and setting a start earlier than that costs empty requests, not errors.

What it loads

sites reads every site, and each site record carries its pens the way the API nests them — every pen, active or not. There is no separate pens table, because the API serves no pens endpoint; where pen history lives, and what the source does and does not promise about a pen or site that stops being reported. sites is versioned rather than replaced: a row is retired, never deleted, because a pen leaves /sites as soon as it is emptied (what that means for your queries). Nested objects land as one JSON column each, and welfare_scores is not unpivoted (why, and how to override it).

Configuring a resource

Each resource takes its endpoint's params in snake_case, except the window ones, which the incremental cursor drives:

source = aquabyte_source()
source.sites.bind(site_id="site-001")     # switches to GET /sites/{siteId}
source.biomass.bind(pen_id="pen-abc", bucket_size=250)
pipeline.run(source)

• pen_id defaults to "all" — the API's own value for "every pen", in one request. Pass one id to read a single pen.
• site_id is the one path param, not a query param: binding it moves sites to the per-site endpoint, and both write the same table.
• The window is the incremental cursor's, not a parameter of its own: a daily run resumes where it left off, and a backfill binds the window on the resource's incremental_* argument — see the reference.
• params is on every resource and merged into the query string last: the escape hatch for a query param the API grows later, no release needed. It wins over every named param, penId included.

Params can also be set in config, per resource:

[sources.aquabyte.environmental]
period = "15min"

Two params decide the grain of the data itself

period and bucket_size change what the API computes for you, not which rows you ask for. Both are worth deciding before the first load: a coarse setting is not wrong, but the detail under it never lands, and getting it later means re-loading that history the backfill way.

⚠️ Changing period later leaves both grains in the table. The key is penId + fromTime + toTime, so hourly rows do not merge over the daily ones they cover — both sit there. Pick a period per resource and keep it, or re-load the history behind the change.

bucket_size adds no rows. weightDist covers only the weights observed, so a pen of smolt returns a couple of buckets and a harvest-size pen at 250 g a few dozen — one JSON column either way (what the arrays hold).

The package emits no log records of its own. dlt logs the window each run asked for and every request it made, on its own dlt logger, and routing them is dlt's [runtime] settings rather than anything here: what to set, and dlt's own logging documentation.

Examples

One concept each, readable on GitHub. From a checkout, run one with python examples/<name>.py.

Compatibility

The two numbers are unrelated — the package version is ordinary SemVer and never mirrors the API's. Built against that version's specs/openapi.json and run against the live API (last on 2026-08-17). A later backwards-compatible version is expected to work and is not verified here; run the suite first.

Read next

• Reference — the versioned site registry, nesting, backfilling, column types, and what the source deliberately does not expose.
• API quirks worth knowing — where the live API departs from its own OpenAPI document, including which identifiers to join on. Some of them change what a correct query looks like, so read it before your first one.
• Changelog and contributing.

License

Apache-2.0. specs/openapi.json is Aquabyte's own OpenAPI document, included as the spec this package is built against; it is their material, and the licence does not extend to it.