jgtpricedb-util

Run-to-completion jobs for a jgtpricedb price store.

jgtpricedb is the store — schema, bar identity, the trading-session period grid, and the anchor-based refresh engine that writes only what moved. jgtpricedb-util is the steward — the handful of things an operator actually runs against that store, each one a job that starts, does a stated amount of work, prints one greppable verdict, and stops.

Nothing here is a daemon. light counts its ticks and exits. freshness and probe-relabel measure and exit. bootstrap and refresh run one pass. That is what makes the same image safe from cron, from a compose one-shot, and from a person's hands at six in the morning: we run it, it stops when it is done, we don't delete it.

The verdict line

Every job's last line of stdout is:

JGTPDB <job> OK|FAIL <summary>

and the exit code agrees with it:

code	meaning
0	OK — the job ran and measured no failure
1	FAIL — the job ran and measured a failure
2	refused — the job could not be performed (bad arguments, absent directory, unreadable file, broker unreachable, or an abort part-way through)

1 and 2 are separate because "this data is wrong" and "I could not look" call for different responses, and a caller that retries the first is wasting its time. Grep the line, or read $?; they never disagree, because both come from the same object.

The line is printed on every exit, including the ones nobody plans: a usage error, an unreadable file, an unexpected exception. The traceback goes to stderr where a person can read it; stdout still ends with the line, because a caller that greps for it and finds nothing cannot tell a crashed job from one that has not finished yet.

The jobs

job	what it does
bootstrap	Registers every named series from a holdings directory and runs one refresh pass. How a store first learns what exists.
refresh	One delta pass over series the store already knows. Registers nothing new. --oanda fetches candles into the source directory first.
light	Bounded forming-bar loop: --loop N --interval S, then exit. Rewrites the anchor when its values move; never appends.
oanda-fetch	OANDA v20 candles into the exact 15-column jgt CSV, at the filename CsvSource reads.
freshness	Does the directory a server serves hold the bars the writer wrote? Inode identity, newest-bar drift, per-timeframe staleness.
probe-relabel	Does the period grid rename the broker's bars? Any timeframe above 0% exits 1.
snapshot	Tar the paths a run is about to touch, before it touches them. Prints the archive's sha256.
verify	Byte-compare two files; names the first offset that differs.

bootstrap and refresh are one engine, two authorities

They share a body because jgtpricedb shares one: refresh_series reads what the store already holds for the periods a source offers and picks its own algorithm — bootstrap when the range is empty, incremental update when it is not. Forcing that choice from outside would be guessing at state the store can simply read.

What differs is where the list of series comes from, and that is the whole difference the two names carry:

• bootstrap --all-series means every <INSTRUMENT>_<TF>.csv in the holdings directory, and registers them.
• refresh --all-series means every series already registered in the store, and registers nothing. A CSV that appears in the source directory unannounced is not silently adopted mid-week.

light — bounded, and quiet when there is nothing to move

--loop N is a count of ticks, not a duration, and there is no --forever. The job spends its count and exits, which is what makes the same image safe from a timer, where the restart is somebody's stated policy rather than a while-loop's accident.

Each tick finds the series' anchor and rewrites it if the values moved. It never appends. Two things are therefore normal rather than wrong, and the verdict line counts them apart from errors:

• deferred= — the series has no forming bar, so a full refresh owns it. Over a weekend every series is here. A run of nothing but deferrals still exits 0, because a job that failed every Saturday would be a job nobody reads on Monday.
• rollovers= — the period advanced while the loop was running. The append belongs to refresh, so the tick says so instead of writing a second forming bar.

errors= is what fails the run, and divergences= fails it only under --strict.

freshness — the job today's incident is made of

A writer container and a reader container each mounted a different host directory at the same container path, because a stray shell variable was set when one of them was launched. Both processes were healthy. Both logs were clean. The writer wrote current bars all day and the reader served bars from days earlier, and nothing inside either container could tell, because from inside, /data/current/pds is /data/current/pds.

Three questions catch it, and no one of them is enough alone:

1. Identity — st_dev/st_ino. Same inode: one file, no split possible. Different inode: not yet a fault, a copy is legitimate — but now the other two questions carry the weight.
2. Drift — does the served file's newest bar match the expected one's? This is the direct signature: two files, each internally consistent, describing different moments. One period of slack, because the writer may legitimately be a bar ahead of a file the server copied a moment ago — which is also this check's honest limit: a split whose two sides have not yet parted by a whole period is invisible to it, and the alternative is failing every server that serves from a copy.
3. Staleness — is the newest bar older than this timeframe's budget? This catches what the first two cannot: both paths being the same stale directory, where nothing diverges because nothing is being written.

The weekend is not an outage. Forex closes Friday 17:00 New York and opens Sunday 17:00, so a Saturday probe correctly finds every file hours old. The store's own ForexCalendar is consulted and an age is minutes of open market — the closure is not counted, because no bar could have been written in it.

Not counting it only over the weekend is not enough, and the difference is a whole class of false alarm: holding the clock still until Sunday 17:00 and then releasing it puts a 48-hour cliff at the reopen, where a file holding Friday's last bar reads one minute old all weekend and 2 881 minutes old the instant the market opens — every series on the feed failing at once, every week, with nothing wrong and no bar missed. Elapsed open time has no such edge, and it equals the wall clock whenever the market never shut in between.

Defaults are roughly three periods plus slack per timeframe, overridable:

jgtpdb freshness --served /data/current/pds --expected /writer/pds \
                 --max-age-minutes H1=120,m15=40

probe-relabel — the measurement that produced jgtpricedb 0.1.1

A store cut on naive UTC renamed 100% of the H4, D1, W1 and M1 bars it was given, because the feed cuts those periods on the 17:00 America/New_York session boundary. It renamed them silently: the bar kept its prices, period_start moved its timestamp, the key came from the moved timestamp, and no exception was raised anywhere.

The probe is one line of arithmetic run over real files, using the library's own grid:

period_start(timeframe, ts, session_for(instrument)) == ts

Run it against holdings rather than a fixture. A fixture encodes what the grid believed on the day it was written; the CSVs encode what the broker actually published, which is the only authority there is.

oanda-fetch — session alignment is not optional

OANDA aligns D, W, M and H4 candles to dailyAlignment in alignmentTimezone, and the default is 00:00 UTC — not the feed's boundary. Left at the default, every session-anchored bar would land an offset away from the holdings and the store would key it as a different bar. This job sends dailyAlignment=17, alignmentTimezone=America/New_York and weeklyAlignment=Saturday for exactly those timeframes, all three read off jgtpricedb's own NY_TRADING_SESSION rather than restated here.

weeklyAlignment is Saturday, and the reason is worth a paragraph. The trading week is labelled Sunday — that is session.week_opens_on — but the boundary that opens it is the evening before, Saturday 17:00 New York. OANDA's parameter names the boundary's weekday, not the label's. Asking for Sunday returns a weekly candle stamped one day late: measured against the live practice API, every W1 bar of both a summer and a winter window landed off period_start's grid. The store recovers, because identity floors a stamp before it keys anything — which is exactly what made it silent. The CSV on disk disagreed with the holdings for the same week, and probe-relabel over that directory reported 100% on W1, a defect belonging to the fetch rather than to the grid. tests/test_session_grid.py keeps both responses, correct and wrong, as captured evidence.

Median has no OANDA field, so it is derived — from bid and ask, the authoritative sides, by the store's own formula. Not from OANDA's mid candle: that candle is built from mid ticks rather than averaged from its own bid and ask, so its High and Low genuinely disagree with (bid + ask) / 2. Those cells are written through as what the broker published and CsvSource reports them as validate divergences, which is the design — the store keeps its own derivation and says where the source differed. A Median computed off that mid would inherit the same disagreement and report it against a column that was never the problem; over 149 captured live candles that was six false reports.

An --oanda refresh therefore prints a handful of divergences= on the mid columns as a matter of course. That is a finding about the broker's mid stream, not a failed run — and --strict, which turns findings fatal, will fail such a run on purpose.

Quickstart — pip

pip install jgtpricedb-util

# fill a store from the existing holdings
jgtpdb bootstrap --db /tmp/prices.db --holdings $JGTPY_DATA/pds \
                 --instrument EUR-USD --timeframe H1

# ask the broker for today, then take the delta
export OANDA_TOKEN=...            # practice by default
jgtpdb oanda-fetch --instrument EUR-USD --timeframe H1 \
                   --since 2026-07-29 --out /tmp/oanda
jgtpdb refresh --db /tmp/prices.db --holdings /tmp/oanda \
               --instrument EUR-USD --timeframe H1

# watch the forming bar move, ten times, then stop
jgtpdb light --db /tmp/prices.db --holdings /tmp/oanda \
             --instrument EUR-USD --timeframe H1 --loop 10 --interval 60 --oanda

Quickstart — container

The image is a job container: restart: "no", one job per invocation.

cd jgt-pricedb-util

# build from PyPI (the normal path)
docker compose build

# run a job — the entrypoint is jgtpdb, so the command is just the job
docker compose run --rm jgt-pricedb-util probe-relabel --holdings /data/current/pds
docker compose run --rm jgt-pricedb-util bootstrap --db /work/prices.db \
    --holdings /data/current/pds --instrument EUR-USD --timeframe H1
docker compose run --rm jgt-pricedb-util freshness \
    --served /data/current/pds --expected /data/current/pds

To run this checkout instead of the published release — before a release exists, or to prove a change — export both variables before building, and leave them exported for the run:

export JGT_PRICEDB_UTIL_LOCAL=1 JGT_PRICEDB_UTIL_TAG=local
docker compose build
docker compose run --rm jgt-pricedb-util probe-relabel --holdings /data/current/pds

Both, not one. LOCAL decides what gets installed and TAG decides what the image is called; setting only the first builds this checkout under the released version's name, and setting only the second is a rename of the PyPI build. Exporting them once covers the build and every run after it — forget them on the run and compose quietly reaches for the released image instead.

The plain-docker equivalent, if you would rather not export anything:

docker build --build-arg LOCAL=1 -t jgtpricedb-util:local .
docker run --rm -v "$JGTPY_DATA:/data/current:ro" -v "$PWD/work:/work" \
    jgtpricedb-util:local probe-relabel --holdings /data/current/pds

Mounts, as composed:

container path	host	mode
/data/current	${JGTPY_DATA:-/b/trading/jgtml/data/current}	read-only
/work	./work	read-write — the store, fetched candles, snapshots

/data/current is mounted read-only deliberately. This package reads the holdings and writes to /work; a job that could rewrite the pipeline's own output is a job that can cause the incident freshness exists to detect.

OANDA_TOKEN, OANDA_ACCOUNT_ID and OANDA_ENVIRONMENT pass through from the host environment. The environment defaults to practice when unset, in this tool as in every other jgt tool.

Dependencies

jgtpricedb, and otherwise the standard library. No requests — urllib. No pandas in this package's own code — csv. A job container that pulls a second dependency tree is a job container that one day fails for a reason that has nothing to do with the data.

jgtpricedb itself brings sqlalchemy and pandas, so the image contains them; what this package adds on top is nothing.

Related

• jgtpricedb — the store this stewards.
• The session-grid decision, its evidence and its measurement live in jgtpricedb's rispecs/01-price-store.spec.md, The Session Grid.