dlt-source-aquabyte

A dlt source for the Aquabyte API v3: sites, biomass, lice counts, welfare scores, behaviour and environmental readings, into any dlt destination.

Records land as the API returns them — nothing renamed, nothing dropped, no invented child tables. Column names are the API's own, in dlt's usual snake_case. Reshaping belongs in your transform layer, where you can change it without waiting for a release.

The package handles auth, pagination, envelope unwrapping, incremental cursors and window splitting. Its only dependency is dlt.

How to start

uv add dlt-source-aquabyte "dlt[duckdb]"   # any dlt destination works; DuckDB is the one below

Then two files in a .dlt/ directory beside your script.

.dlt/config.toml:

[sources.aquabyte]
base_url = "https://api.aquabyte.ai/v3/"
initial_date = "2020-01-01"            # first-run start, date resources
initial_time = "2020-01-01T00:00:00Z"  # first-run start, time resources

.dlt/secrets.toml:

[sources.aquabyte]
api_key = "your-api-key-here"

For the two start values, choose a date that predates when you first lowered an Aquabyte camera into the sea. A start earlier than your first record costs empty requests, not errors, and step 2 below replaces the guess with measurements.

Then four steps, one example each. That is everything needed to get running, a page of code each.

1. Discover — discover_history.py. How far back your account goes, how much is there, and how fresh each resource is. Its output decides steps 2 and 3.
2. Configure again, now from measurements. period and bucket_size per resource for the granularity you want (Configuring a resource). And move initial_date/initial_time forward to the day step 3 will end, because they are where the daily load starts: a backfill leaves no cursor behind, so a daily load still pointed at the old date would re-request every year you are about to load.
3. Backfill — backfill.py. Put the earliest dates from step 1 at the top of the file and run it once. The stored cursor is left alone, so this can be re-run at any time without disturbing step 4.
4. Daily load — daily_load.py. The same script on a timer from then on, each run resuming from the cursor.

No step computes a window: the source splits a multi-year span into requests the API accepts. From a checkout, run one with python examples/<name>.py.

Two resources need more than the start values above:

• welfare_scores refuses any start before 2024-04-20, so an older initial_date fails that one resource on every run. Give it a start of its own — both discover_history.py and backfill.py already do:

  source.welfare_scores.bind(incremental_date=dlt.sources.incremental(initial_value="2024-04-20"))
  

• sites and environmental_latest need neither value, keeping no cursor. A cursor resource missing one fails with an error naming it.

What it loads

Three defaults worth knowing before your first query:

• There is no pens table. The API serves no pens endpoint, so each site record carries its pens as the API nests them, active or not — where pen history lives.
• sites is versioned, not replaced. A pen leaves /sites as soon as it is emptied, so a row is retired rather than deleted — what that means for your queries.
• Nested objects land as one JSON column each, welfare_scores included — why, and how to override it.

Configuring a resource

Each resource takes its endpoint's params in snake_case. The window is the exception: the incremental cursor drives it, and a backfill binds it on the resource's incremental_* argument (how).

source = aquabyte_source()
source.sites.bind(site_id="site-001")  # switches to GET /sites/{siteId}
source.biomass.bind(pen_id="pen-abc", bucket_size=250)
pipeline.run(source)

Params can also be set in config, per resource:

[sources.aquabyte.environmental]
period = "15min"

Two params decide the granularity of the data itself

period and bucket_size change what the API computes for you, not which rows you ask for. Decide both before the first load: a coarse setting is not wrong, but the detail under it never lands, and getting it later means re-loading that history the backfill way.

⚠️ Changing period later leaves both granularities in the table. The key is penId + fromTime + toTime, so hourly rows do not merge over the daily ones they cover. Keep one period per resource, or re-load the history behind the change. That is also why behaviour_breathing_index drops toTime from its key: it is daily-only, so its granularity cannot change.

period also sets the widest window the API accepts — 7 days at 15min, 31 at h, 366 at D — and the source splits its requests to fit, so a long catch-up works at any granularity (detail).

weightDist covers only the weights observed, so a smolt pen returns two buckets and a harvest-size pen at 250 g a few dozen (what the arrays hold).

Compatibility

The two numbers are unrelated: the package version is ordinary SemVer. Built against that API version's specs/openapi.json and run live against it, last on 2026-08-31. A later backwards-compatible API version should work, but run the suite first.

Read next

• API quirks worth knowing — where the live API departs from its OpenAPI document, and which identifiers to join on. Some change what a correct query looks like, so read it before your first one.
• Reference — site versioning, nesting, backfilling, window splitting, logging and column types.
• Changelog and contributing.

License

Apache-2.0. specs/openapi.json is Aquabyte's own OpenAPI document, included as the spec this package is built against. It is their material, and the licence does not extend to it.