jgtpricedb-util

Run-to-completion jobs for a jgtpricedb price store.

jgtpricedb is the store — schema, bar identity, the trading-session period grid, and the anchor-based refresh engine that writes only what moved. jgtpricedb-util is the steward — the handful of things an operator actually runs against that store, each one a job that starts, does a stated amount of work, prints one greppable verdict, and stops.

Nothing here is a daemon. light counts its ticks and exits. freshness and probe-relabel measure and exit. bootstrap and refresh run one pass. That is what makes the same image safe from cron, from a compose one-shot, and from a person's hands at six in the morning: we run it, it stops when it is done, we don't delete it.

The verdict line

Every job's last line of stdout is:

JGTPDB <job> OK|FAIL <summary>

and the exit code agrees with it:

1 and 2 are separate because "this data is wrong" and "I could not look" call for different responses, and a caller that retries the first is wasting its time. Grep the line, or read $?; they never disagree, because both come from the same object.

The line is printed on every exit, including the ones nobody plans: a usage error, an unreadable file, an unexpected exception. The traceback goes to stderr where a person can read it; stdout still ends with the line, because a caller that greps for it and finds nothing cannot tell a crashed job from one that has not finished yet.

The jobs

bootstrap and refresh are one engine, two authorities

They share a body because jgtpricedb shares one: refresh_series reads what the store already holds for the periods a source offers and picks its own algorithm — bootstrap when the range is empty, incremental update when it is not. Forcing that choice from outside would be guessing at state the store can simply read.

What differs is where the list of series comes from, and that is the whole difference the two names carry:

• bootstrap --all-series means every <INSTRUMENT>_<TF>.csv in the holdings directory, and registers them.
• refresh --all-series means every series already registered in the store, and registers nothing. A CSV that appears in the source directory unannounced is not silently adopted mid-week.

--now ISO (0.1.1) is on bootstrap, refresh and light, the way it has always been on freshness: it decides bar completeness as of a named instant instead of the wall clock, and moves the calendar with it. "Is this bar still forming?" is a question about an instant, and a run that answered it from the clock while being handed a different one was answering a different question — a captured holdings file then reads as complete in January and forming in August.

light — bounded, and quiet when there is nothing to move

--loop N is a count of ticks, not a duration, and there is no --forever. The job spends its count and exits, which is what makes the same image safe from a timer, where the restart is somebody's stated policy rather than a while-loop's accident.

Each tick finds the series' anchor and rewrites it if the values moved. It never appends. Two things are therefore normal rather than wrong, and the verdict line counts them apart from errors:

• deferred= — the series has no forming bar, so a full refresh owns it. Over a weekend every series is here. A run of nothing but deferrals still exits 0, because a job that failed every Saturday would be a job nobody reads on Monday.
• rollovers= — the period advanced while the loop was running. The append belongs to refresh, so the tick says so instead of writing a second forming bar.

errors= is what fails the run, and divergences= fails it only under --strict.

export — the store becomes authoritative and nothing downstream is told

The Phase-3 bridge, from the operator's side. jgtpy, jgtml and jgt-data-server keep opening $JGTPY_DATA/pds/<INSTRUMENT>_<TF>.csv; what changes is which side of that file is the truth.

# the served window, every active series
jgtpdb export --db /work/prices.db --out /work/pds --all-series --bars 330

# one series, whole history, line endings pinned
jgtpdb export --db /work/prices.db --out /work/pds \
              --instrument EUR-USD --timeframe H1 --full --terminator lf

The claim is checkable and was checked: a holdings file read into a store and written back out is byte-for-byte itself — 89 of 89 live series across 13 instruments and all seven timeframes, 95 208 bars, with the live forming bar as the last row in 42 of them. The format is measured rather than assumed, and rispecs/03-csv-export.spec.md in the store repository holds the measurement:

• quote columns keep the double the feed published — the holdings themselves carry 151.04600000000002, and rounding it would write a different number than the store holds;
• the derived columns are quantized at the instrument's display precision +1 (Open/High/Low/Close) and +2 (Median), which is the exact decimal width of a mid and of a median, so the quantum removes the binary noise the division introduces and nothing else — and because the quantum is the exact width, no rounding tie ever arises to be broken one way or the other;
• the line terminator belongs to the file being replaced. Both CRLF and LF live in the holdings directory today, written by two writers, and a file's ending changes when the other one touches it — three files flipped LF to CRLF in a single day of the last measurement. --terminator match is the default; crlf and lf pin it, and a pinned choice outranks what the file carries.

Two refusals are worth knowing before an operator meets them:

• a series with no bars is refused rather than written. A header-only CSV is what a year of bars looks like after an unpopulated store wrote over it, and it overwrites without raising anything;
• a --db that does not exist is a refusal, not an empty database created on the spot and exported from under an OK line.

Each file is written beside itself and moved into place, so a consumer reading mid-run gets the previous file rather than half of the next one. Cost is around 70 µs a bar — 22 ms for a 300-bar window, 169 ms for a seven-timeframe set — next to nothing beside the recomputation the store exists to make incremental.

One writer. The export becomes the writer of a served directory when the current writer stops, never beside it. Two processes writing one price file is the failure freshness was built to catch.

freshness — the job today's incident is made of

A writer container and a reader container each mounted a different host directory at the same container path, because a stray shell variable was set when one of them was launched. Both processes were healthy. Both logs were clean. The writer wrote current bars all day and the reader served bars from days earlier, and nothing inside either container could tell, because from inside, /data/current/pds is /data/current/pds.

Three questions catch it, and no one of them is enough alone:

1. Identity — st_dev/st_ino. Same inode: one file, no split possible. Different inode: not yet a fault, a copy is legitimate — but now the other two questions carry the weight.
2. Drift — does the served file's newest bar match the expected one's? This is the direct signature: two files, each internally consistent, describing different moments. One period of slack, because the writer may legitimately be a bar ahead of a file the server copied a moment ago — which is also this check's honest limit: a split whose two sides have not yet parted by a whole period is invisible to it, and the alternative is failing every server that serves from a copy.
3. Staleness — is the newest bar older than this timeframe's budget? This catches what the first two cannot: both paths being the same stale directory, where nothing diverges because nothing is being written.

The weekend is not an outage. Forex closes Friday 17:00 New York and opens Sunday 17:00, so a Saturday probe correctly finds every file hours old. The store's own ForexCalendar is consulted and an age is minutes of open market — the closure is not counted, because no bar could have been written in it.

Not counting it only over the weekend is not enough, and the difference is a whole class of false alarm: holding the clock still until Sunday 17:00 and then releasing it puts a 48-hour cliff at the reopen, where a file holding Friday's last bar reads one minute old all weekend and 2 881 minutes old the instant the market opens — every series on the feed failing at once, every week, with nothing wrong and no bar missed. Elapsed open time has no such edge, and it equals the wall clock whenever the market never shut in between.

Defaults are roughly three periods plus slack per timeframe, overridable:

jgtpdb freshness --served /data/current/pds --expected /writer/pds \
                 --max-age-minutes H1=120,m15=40

probe-relabel — the measurement that produced jgtpricedb 0.1.1

A store cut on naive UTC renamed 100% of the H4, D1, W1 and M1 bars it was given, because the feed cuts those periods on the 17:00 America/New_York session boundary. It renamed them silently: the bar kept its prices, period_start moved its timestamp, the key came from the moved timestamp, and no exception was raised anywhere.

The probe is one line of arithmetic run over real files, using the library's own grid:

period_start(timeframe, ts, session_for(instrument)) == ts

Run it against holdings rather than a fixture. A fixture encodes what the grid believed on the day it was written; the CSVs encode what the broker actually published, which is the only authority there is.

oanda-fetch — session alignment is not optional

OANDA aligns D, W, M and H4 candles to dailyAlignment in alignmentTimezone, and the default is 00:00 UTC — not the feed's boundary. Left at the default, every session-anchored bar would land an offset away from the holdings and the store would key it as a different bar. This job sends dailyAlignment=17, alignmentTimezone=America/New_York and weeklyAlignment=Saturday for exactly those timeframes, all three read off jgtpricedb's own NY_TRADING_SESSION rather than restated here.

weeklyAlignment is Saturday, and the reason is worth a paragraph. The trading week is labelled Sunday — that is session.week_opens_on — but the boundary that opens it is the evening before, Saturday 17:00 New York. OANDA's parameter names the boundary's weekday, not the label's. Asking for Sunday returns a weekly candle stamped one day late: measured against the live practice API, every W1 bar of both a summer and a winter window landed off period_start's grid. The store recovers, because identity floors a stamp before it keys anything — which is exactly what made it silent. The CSV on disk disagreed with the holdings for the same week, and probe-relabel over that directory reported 100% on W1, a defect belonging to the fetch rather than to the grid. tests/test_session_grid.py keeps both responses, correct and wrong, as captured evidence.

Median has no OANDA field, so it is derived — from bid and ask, the authoritative sides, by the store's own formula. Not from OANDA's mid candle: that candle is built from mid ticks rather than averaged from its own bid and ask, so its High and Low genuinely disagree with (bid + ask) / 2. Those cells are written through as what the broker published and CsvSource reports them as validate divergences, which is the design — the store keeps its own derivation and says where the source differed. A Median computed off that mid would inherit the same disagreement and report it against a column that was never the problem; over 149 captured live candles that was six false reports.

An --oanda refresh therefore prints a handful of divergences= on the mid columns as a matter of course. That is a finding about the broker's mid stream, not a failed run — and --strict, which turns findings fatal, will fail such a run on purpose.

Quickstart — pip

pip install jgtpricedb-util

# fill a store from the existing holdings
jgtpdb bootstrap --db /tmp/prices.db --holdings $JGTPY_DATA/pds \
                 --instrument EUR-USD --timeframe H1

# ask the broker for today, then take the delta
export OANDA_TOKEN=...            # practice by default
jgtpdb oanda-fetch --instrument EUR-USD --timeframe H1 \
                   --since 2026-07-29 --out /tmp/oanda
jgtpdb refresh --db /tmp/prices.db --holdings /tmp/oanda \
               --instrument EUR-USD --timeframe H1

# watch the forming bar move, ten times, then stop
jgtpdb light --db /tmp/prices.db --holdings /tmp/oanda \
             --instrument EUR-USD --timeframe H1 --loop 10 --interval 60 --oanda

# write the store back out as the CSV every consumer already reads
jgtpdb export --db /tmp/prices.db --out /tmp/pds --all-series --bars 330
jgtpdb verify --a /tmp/pds/EUR-USD_H1.csv --b $JGTPY_DATA/pds/EUR-USD_H1.csv

Quickstart — container

The image is a job container: restart: "no", one job per invocation.

cd jgt-pricedb-util

# build from PyPI (the normal path)
docker compose build

# run a job — the entrypoint is jgtpdb, so the command is just the job
docker compose run --rm jgt-pricedb-util probe-relabel --holdings /data/current/pds
docker compose run --rm jgt-pricedb-util bootstrap --db /work/prices.db \
    --holdings /data/current/pds --instrument EUR-USD --timeframe H1
docker compose run --rm jgt-pricedb-util freshness \
    --served /data/current/pds --expected /data/current/pds

To run this checkout instead of the published release — before a release exists, or to prove a change — export both variables before building, and leave them exported for the run:

export JGT_PRICEDB_UTIL_LOCAL=1 JGT_PRICEDB_UTIL_TAG=local
docker compose build
docker compose run --rm jgt-pricedb-util probe-relabel --holdings /data/current/pds

Both, not one. LOCAL decides what gets installed and TAG decides what the image is called; setting only the first builds this checkout under the released version's name, and setting only the second is a rename of the PyPI build. Exporting them once covers the build and every run after it — forget them on the run and compose quietly reaches for the released image instead.

The plain-docker equivalent, if you would rather not export anything:

docker build --build-arg LOCAL=1 -t jgtpricedb-util:local .
docker run --rm -v "$JGTPY_DATA:/data/current:ro" -v "$PWD/work:/work" \
    jgtpricedb-util:local probe-relabel --holdings /data/current/pds

Mounts, as composed:

/data/current is mounted read-only deliberately. This package reads the holdings and writes to /work; a job that could rewrite the pipeline's own output is a job that can cause the incident freshness exists to detect.

OANDA_TOKEN, OANDA_ACCOUNT_ID and OANDA_ENVIRONMENT pass through from the host environment. The environment defaults to practice when unset, in this tool as in every other jgt tool.

derive-cds — one provider boundary for historical chart data

The PriceDB export is the retained source. derive-cds applies the canonical Williams engine to the whole retained window, verifies that dense Alligator, AO and AC columns exist on every output row, then atomically replaces CDS. An invalid PDS, failed calculation, empty result or indicator gap leaves the previous live CDS untouched. The chart server only reads the resulting file.

jgtpdb derive-cds --pds /data/current/pds --out /data/current/cds \
  --instrument EUR-USD,GBP-USD --timeframe m15,H1 --bars 10000

--bars N publishes the last N complete rows and reads only N plus the 610-bar indicator warmup, so a 1.6M-row m5 archive is not loaded to refresh its tail. Without it the whole export is projected. The jgt-data-server updater builds its rolling CDS through this same materialize_cds seam (10 000 bars by default — the price service holds thousands of bars per timeframe, not the broker's 500), and oanda-fetch --count above 5000 pages backward through the v20 API to reach that depth.

Dependencies

Ordinary jobs remain jgtpricedb plus the standard library. No requests — urllib. The optional [indicators] extra adds the canonical Williams engine for derive-cds; that dependency is confined to the provider job and is never loaded by bootstrap, refresh, export, freshness or chart-serving code.

jgtpricedb itself brings sqlalchemy and pandas, so the image already contains the tabular runtime used to validate and atomically publish CDS.

Related

• jgtpricedb — the store this stewards.
• The session-grid decision, its evidence and its measurement live in jgtpricedb's rispecs/01-price-store.spec.md, The Session Grid.