Oh My Data (OMD)

English | 中文说明

ohmydata is an offline-first market-data ingestion SDK and CLI. Provider endpoint adapters accept already initialized official-compatible clients; credentials are never loaded by this library.

Supported Python and installation

Python 3.11 and 3.12 are supported (>=3.11,<3.13). From a source checkout:

uv sync
uv run python -c "import ohmydata; print(ohmydata.__version__)"

The core has no runtime dependencies. Install ohmydata[tushare] for the Pandas-backed Tushare adapter, ohmydata[yfinance] for US/global market data and fundamentals, ohmydata[sec-cli] for the SEC N-PORT batch CLI, or ohmydata[sec-financials] for company 10-K/10-Q financial statements and Parquet dataset writer. Provider tests use fake clients and never call a network.

Core Architecture (offline & immutable)

ohmydata.core provides canonical request identities, classified retry with total-attempt semantics, explicit instance-scoped rate limiters, dataframe-free provenance, and immutable APPEND/FROZEN snapshots. Request parameters reject secret-bearing keys before serialization. Snapshot callers provide exact bytes; the core never contacts providers or loads credentials.

Availability evidence is represented by the dataframe-free AvailabilityEvidence value object. Source-declared timestamps are the only evidence marked pit_proven; inferred schedules, date-only declarations, and provider-first-observed fallbacks remain conservative. Snapshot construction uses validated observation receipts and normalizes datetimes to UTC.

from datetime import UTC, datetime
from pathlib import Path

from ohmydata.core import (
    RateLimiter,
    RateLimitPolicy,
    RequestSpec,
    RetryPolicy,
    SnapshotMode,
    SnapshotStore,
    execute_with_retry,
)

try:
    RequestSpec("demo", "bars", {"api_token": "never-serialize"})
except ValueError:
    pass
limiter = RateLimiter(RateLimitPolicy(0.1))
limiter.acquire()
result = execute_with_retry(lambda: "ok", RetryPolicy(max_attempts=1))
store = SnapshotStore(Path("snapshots"))
store.write(
    RequestSpec("demo", "bars", {}), b"[]", datetime.now(UTC), "json-v1", SnapshotMode.APPEND
)
store.write(
    RequestSpec("demo", "bars", {}), b"[]", datetime.now(UTC), "json-v1", SnapshotMode.FROZEN
)

RetryPolicy(max_attempts=3) counts the first call. APPEND preserves distinct observations; FROZEN permits one response identity. SnapshotStore.observe() adds immutable, ordered fetch receipts without changing snapshot bytes; SnapshotRef.fact_version identifies the exact request, payload, and serialization. provider_first_observed_at() reports when OMD first persisted those exact bytes, not provider publication time or consumer usability. Limiter state is per instance.

Raw provider rows can be wrapped in RawFactEnvelope, preserving the response-level fact_version separately from a canonical row hash. Revision status remains conservative until an explicit same-key prior row is supplied; point-in-time and date-only availability quality flags are serialized too. The offline characterization matrix in tests/characterization/test_pit_fail_closed.py also proves that late arrivals, date-only evidence, replay mismatches, pagination truncation, and historical-vintage claims remain fail-closed. OMD does not choose consumer cutoffs, calendars, dataset commits, or usable sessions.

Dated instrument identity declarations

InstrumentIdentityIndex separates issuer IDs, securities and dated provider aliases. Queries require an exact provider/alias/venue, effective date and knowledge cutoff; missing mappings raise CoverageError. All interval overlaps reject the proposed catalog, even if recorded later. Retain prior catalogs for replay; v1 has no metadata revision or revocation model. Caller evidence is not verified listing history, market availability or financial approval.

from datetime import UTC, date, datetime
from ohmydata.core import (
    InstrumentIdentity,
    InstrumentIdentityIndex,
    InstrumentType,
    IssuerIdentity,
    ProviderInstrumentAlias,
)
from ohmydata.providers.sec import SecDataVersionId, SecDataVersionKind

recorded = datetime(2025, 1, 1, tzinfo=UTC)
issuer = IssuerIdentity("demo:issuer", "evidence:issuer", recorded)
security = InstrumentIdentity(
    "demo:class-a",
    issuer.issuer_id,
    InstrumentType.COMMON_SHARE,
    "evidence:security",
    recorded,
)
alias = ProviderInstrumentAlias(
    "demo",
    "EXAMPLE",
    "XNAS",
    "USD",
    security.instrument_id,
    date(2025, 1, 1),
    date(2026, 1, 1),
    "evidence:listing",
    recorded,
)
catalog = InstrumentIdentityIndex([issuer], [security], [alias])
resolution = catalog.resolve(
    provider="demo",
    alias="EXAMPLE",
    venue="XNAS",
    effective_date=date(2025, 1, 1),
    knowledge_cutoff=recorded,
)
dependency_input = SecDataVersionId(
    SecDataVersionKind.INSTRUMENT_IDENTITY,
    resolution.binding_identity,
)
# Explicit inputs for a SecMetricTerminalDeclaration or SecMetricExternalInput:
security_basis = resolution.instrument.instrument_id
declaration_reference = resolution.resolution_identity

binding_identity includes selected issuer/security/alias evidence and is stable across unrelated catalog changes and query cutoffs. resolution_identity additionally binds the complete catalog and query. Opaque issuer IDs do not encode a CIK contract; SEC callers supply CIK separately. Effective intervals are required half-open dates, not timestamps or trading sessions. Limits and scope are in the identity contract.

Tushare Provider (A-Share & China ETF Ingestion)

Pass an already initialized official-client-compatible object. The adapter does not create clients or read credentials; this fake-client example is safe to run offline:

import pandas as pd
from ohmydata.providers.tushare import EmptyPolicy, FundDailyRequest, TushareClient


class FakeClient:
    def fund_daily(self, **kwargs):
        return pd.DataFrame(
            {
                "ts_code": ["FAKE.ETF"],
                "trade_date": ["20240102"],
                "open": [1.0],
                "high": [1.1],
                "low": [0.9],
                "close": [1.05],
                "pre_close": [1.0],
                "change": [0.05],
                "pct_chg": [5.0],
                "vol": [100],
                "amount": [250.0],
            }
        )


request = FundDailyRequest(
    empty_policy=EmptyPolicy.ERROR, ts_code="FAKE.ETF", start_date="20240101", end_date="20240102"
)
result = TushareClient(FakeClient()).fetch_fund_daily(request)

The typed etf_basic endpoint preserves Tushare's provider-native metadata and requires an explicit empty policy. Its official filters are ts_code, index_code, list_date, list_status, exchange, and mgr; market is forwarded only as a compatibility filter for callers that already use it.

from ohmydata.providers.tushare import EtfBasicRequest

request = EtfBasicRequest(empty_policy=EmptyPolicy.ERROR, market="E", list_status="L")
result = TushareClient(FakeClient()).fetch_etf_basic(request)

Typed stock dividend events are available through StockDividendRequest and fetch_stock_dividend. Select at least one of ts_code, ann_date, record_date, ex_date, or imp_ann_date; selectors may be combined. The response preserves provider-native dates, process states, values, units, nulls, and revision or duplicate rows, and does not infer point-in-time availability.

Values and nulls retain Tushare's native semantics: fund daily OHLC and change/pct_chg are provider values, vol is in hands, and amount is in thousand yuan. Empty responses must be selected explicitly with EmptyPolicy.ALLOW or EmptyPolicy.ERROR. fund_share.fd_share remains provider-native in ten-thousand shares (万份); fund_adj and fund_nav values are likewise preserved without adjustment or imputation. FundNavRequest and FundShareRequest validate real calendar dates and reject provider rows outside the requested symbol/date scope. NAV revisions (including exact duplicates) remain intact; a 2,000-row fund_share response is rejected as an ambiguous provider cap. Announcement and trade dates are date-only evidence and do not prove an intraday availability timestamp.

Stock daily and adjustment endpoints

The Tushare adapter exposes typed, injected-client daily and adj_factor requests:

from ohmydata.providers.tushare import (
    EmptyPolicy,
    StockAdjustmentRequest,
    StockDailyRequest,
    TushareClient,
)

daily = TushareClient(client).fetch_stock_daily(
    StockDailyRequest(empty_policy=EmptyPolicy.ALLOW, ts_code="000001.SZ")
)
adjustment = TushareClient(client).fetch_stock_adjustment(
    StockAdjustmentRequest(empty_policy=EmptyPolicy.ALLOW, trade_date="20240102")
)

Both requests require exactly one symbol (optionally date-bounded) or one exact trade date, and return stable ts_code/trade_date ordering. Fields are explicit and ordered; custom lists must retain both identity fields. Values, units, and nulls remain provider-native: daily pct_chg is a percentage, vol is hands, amount is thousand yuan, and adj_factor is unmodified. Suspended rows are not synthesized, and no adjusted-price calculation or point-in-time availability claim is made.

ETF PCF constituent endpoints

EtfShConsRequest and EtfSzConsRequest expose the exchange-native etf_sh_cons and etf_sz_cons schemas. Shanghai uses sca (CNY replacement amount); Shenzhen uses sub_cc and red_cc (CNY subscription/redemption replacement amounts). Quantities are shares and cpr/rdr are percentages. Provider values, nulls, sentinels, and duplicate observations remain unchanged.

Each endpoint rejects an ambiguous exactly-3000-row response. Use fetch_etf_pcf_history with an explicit exchange, date range, and EmptyPolicy to recursively bisect calendar windows without offsets. The recipe reports successful leaf provenances, request and truncation counts, and returns a defensive provider-native Pandas frame. A trade_date is date-only provider evidence; availability timestamps, point-in-time lag, cross-exchange normalization, and published dataset policy remain consumer responsibilities.

from ohmydata.providers.tushare import (
    EmptyPolicy,
    EtfPcfHistoryRequest,
    fetch_etf_pcf_history,
)

history = fetch_etf_pcf_history(
    client,
    EtfPcfHistoryRequest(
        ts_code="510050.SH",
        exchange="SH",
        start_date="20240101",
        end_date="20240131",
        empty_policy=EmptyPolicy.ALLOW,
    ),
)
frame = history.frame

Look-through source facts and vintage plane

capture_tushare_result serializes an already validated TushareFetchResult deterministically into an append-only SnapshotStore and returns an immutable TushareObservedResult binding provenance, snapshot/observation/fact identities, availability evidence, and a content hash:

from datetime import UTC, datetime
from pathlib import Path

from ohmydata.core import SnapshotStore
from ohmydata.providers.tushare import (
    EmptyPolicy,
    EtfBasicRequest,
    TushareClient,
    capture_tushare_result,
)

request = EtfBasicRequest(empty_policy=EmptyPolicy.ERROR, ts_code="510050.SH")
result = TushareClient(client).fetch_etf_basic(request)
observed = capture_tushare_result(
    SnapshotStore(Path("snapshots")),
    request,
    result,
    observed_at=datetime.now(UTC),
)

observed_at is explicit and timezone-aware; the capture path never reads credentials or calls a provider. Typed stock_basic and index_member_all endpoints add current stock identity and dated Shenwan industry membership facts; broad all-market requests require an explicit opt-in, and responses at the documented row caps (6000 / 2000) fail closed.

The pure recipes build_etf_index_mapping_observations, audit_index_weight_vintage, and build_lookthrough_source_bundle report observed ETF→index mapping versions, per-vintage weight/count/retrieval diagnostics, and a manifest-only source bundle. They never infer historical effective dates, first_usable_session, weight renormalization, industry backfill, style/cluster labels, or portfolio exposure. Every emitted fact remains PIT_UNPROVEN unless an auditable provider contract proves otherwise; index_weight retrieval completeness is unproven in this release by contract. Bundle manifests report mapped indices without a captured weight vintage and record industry-observation coverage for every captured component.

Consumer gaps are documented field-by-field in docs/v0.1.2-lookthrough-migration.md.

The optional ohmydata[vintage-plane] extra provides the ETF benchmark and constituent vintage artifact assembler. It accepts only caller-constructed, bounded requests and synthetic captured observations; it never discovers credentials or universes. Current Tushare mappings are current-only evidence, not historical point-in-time availability, and index_weight retrieval and economic completeness remain explicitly unproven under the provider contract.

from datetime import UTC, datetime
from pathlib import Path

from ohmydata.core import SnapshotStore, SourceFactRegistry
from ohmydata.providers.tushare import (
    EtfBenchmarkConstituentScope,
    assemble_etf_benchmark_constituent_vintages,
)

# Offline synthetic assembly; no credentials or network are used.
bundle = assemble_etf_benchmark_constituent_vintages(
    [],
    store=SnapshotStore(Path("snapshots")),
    registry=SourceFactRegistry(Path("registry")),
    scope=EtfBenchmarkConstituentScope(),
    cutoff=datetime(2026, 1, 1, tzinfo=UTC),
    output_dir=Path("bundle"),
)

OMD owns source evidence and immutable lineage only. Consumers own canonical cutoffs, trading calendars, session alignment, normalized datasets, and all strategy or portfolio semantics.

Adjusted ETF bars recipe

fetch_adjusted_etf_bars composes fund_daily with provider-native fund_adj factors. Choose AdjustmentCoveragePolicy.STRICT (the default) or PRESERVE_MISSING_FACTOR; raw OHLC and adj_factor remain available beside the explicitly derived adjusted OHLC columns. The recipe is offline-testable when supplied an injected TushareClient and does not claim point-in-time availability. Tushare adjustment responses may contain extra dates for the requested symbol; the recipe ignores those factor-only dates while strict coverage still requires a finite factor for every returned daily bar. Rows for foreign symbols fail.

from ohmydata.providers.tushare import (
    AdjustedEtfBarsRequest,
    AdjustmentCoveragePolicy,
    EmptyPolicy,
)

request = AdjustedEtfBarsRequest(
    "FAKE.ETF",
    EmptyPolicy.ERROR,
    AdjustmentCoveragePolicy.STRICT,
    start_date="20240101",
    end_date="20240131",
)

Offline weighted dividend yield recipes

build_portfolio_dividend_yield and build_index_dividend_yield calculate a provider-semantic weighted yield from already-downloaded Pandas frames. Portfolio mkv is yuan; index weight and daily_basic.dv_ttm are provider percentages. The returned dividend_yield is a decimal ratio (sum((w_i / W) * dv_ttm_i) / 100), where W is the provider-native total weight. Choose DividendYieldCoveragePolicy.REQUIRE_COMPLETE to reject missing finite yield coverage, PRESERVE_INCOMPLETE to return None, or the explicitly named NORMALIZE_SUPPORTED policy to divide only by finite supported weight while still reporting the original finite_weight_coverage. Callers own any minimum coverage threshold and must not present a normalized partial estimate as full coverage. Zero supported coverage remains unknown. Inputs are not modified, and dates are identity checks only: the recipe does not infer point-in-time availability or report selection.

from ohmydata.providers.tushare import (
    DividendYieldCoveragePolicy,
    build_index_dividend_yield,
)

result = build_index_dividend_yield(
    index_weights_df,
    daily_basic_df,
    DividendYieldCoveragePolicy.REQUIRE_COMPLETE,
)
print(result.dividend_yield)

The typed IndexWeightRequest accepts either one exact observation date or a complete inclusive range within one calendar month. Responses are checked for the requested index and date scope and sorted by index, observation date, and constituent. weight remains the provider-native percentage, including null or non-finite values; no effective period, availability timestamp, or weight renormalization is inferred.

SEC N-PORT Holdings Provider & CLI

The ohmydata.providers.sec package provides offline-testable primitives for the official quarterly N-PORT data set: caller-selected series, immutable artifact retention, native Decimal/date/null values, and EDGAR acceptance metadata. It deliberately does not classify funds, resolve tickers, compute trading sessions, or perform an implicit live download; network access requires an explicit caller-created client and contact User-Agent.

Install the Parquet writer and run the batch CLI with a reviewed equity-ETF universe containing exact CIK/series identities:

uv sync --extra sec-cli

# Run full-history sync via a configuration file:
uv run omd sec nport sync --config artifacts/sec-sync.yaml

# Or inspect plan, fetch, or validate using the same configuration:
uv run omd sec nport plan --config artifacts/sec-sync.yaml
uv run omd sec nport validate --config artifacts/sec-sync.yaml

# Or with explicit flags (--quarters full expands 2019q4 through the latest completed quarter):
uv run omd sec nport sync \
  --quarters full \
  --root artifacts/sec-nport \
  --universe artifacts/sec-equity-etfs.json \
  --user-agent-file /path/to/private-sec-contact.txt \
  --availability-policy accepted-at-plus-lag \
  --lag-days 0

--config FILE accepts .json, .yaml, .yml, or .toml mappings (such as artifacts/sec-sync.yaml with quarters: full, root: artifacts/sec-nport-full, universe: artifacts/sec-equity-etf-universe.json, user_agent_file: artifacts/sec-contact.txt, availability_policy: accepted-at-plus-lag, and lag_days: 0) to eliminate repetitive flags; explicit CLI options override configuration values. --quarters full expands to 2019q4 through the latest completed calendar quarter, while single-quarter tokens (e.g. --quarters 2026q2, or --quarter latest for inspect) are supported as shorthands.

fetch retains each replay-valid quarterly ZIP and the exact EDGAR metadata closure, while build works offline from those retained artifacts. sync processes one quarter at a time and resumes without downloading a completed quarter again. validate checks the complete local artifact/catalog closure; inspect prints safe summaries and only emits holding rows when --rows is explicitly supplied. fetch, build, and sync emit safe progress milestones to stderr (quarter indices, download bytes, and build timing), which can be silenced with --quiet. Quarterly ZIP responses are streamed to immutable storage; build and sync also accept positive --max-selected-rows and --max-output-bytes limits (defaulting to 5,000,000 rows and 2 GiB) and fail before partition publication when either bound is exceeded.

Each immutable core partition contains fund_vintages.parquet, holdings.parquet, and identifiers.parquet, plus quality and manifest JSON. Provider-native percentages remain percentage points, numeric text is preserved beside exact Decimal values, and missing values remain missing. Consumers own security-master mapping, exchange calendars, first-usable-session alignment, and strategy features.

The output root must be outside the repository or already Git-ignored. Contact data is accepted only through a file or explicit stdin and is never written to artifacts or output. Universe classification is caller-reviewed; the intended equity-ETF research universe excludes GLD, bond ETFs, and money-market/currency ETFs.

SEC Company Financials (10-K & 10-Q PIT via EdgarTools)

The sec-financials extra wraps edgartools with strict credential injection, core isolation, and explicit filing and availability metadata for the three core financial statements (Balance Sheet, Income Statement, and Cash Flow Statement):

uv sync --extra sec-financials

# Sync company financials via an OMD configuration file:
uv run omd sec financials sync --config artifacts/sec-financials.yaml

# Or inspect local partitions:
uv run omd sec financials inspect --root artifacts/sec-financials --symbol AAPL --rows

# Or validate local Parquet partition checksums:
uv run omd sec financials validate --root artifacts/sec-financials

Python SDK example with injected credentials:

from ohmydata.providers.sec import (
    SecFinancialsClient,
    SecFinancialsRequest,
    SecHttpClient,
    write_financials_partition,
)

# Injected client reading identity strictly from contact info (no .env):
user_agent = "MyResearchApp/1.0 (contact@example.com)"
client = SecFinancialsClient(user_agent, http_client=SecHttpClient(user_agent))

request = SecFinancialsRequest(
    symbols=("AAPL", "MSFT"),
    forms=("10-K", "10-Q"),
    availability_policy="accepted-at-plus-lag",
    lag_days=0,
)

vintages = client.fetch_company_financials(request)

# Partitioned Parquet data lake writing:
for symbol in ("AAPL", "MSFT"):
    sym_vintages = [v for v in vintages if v.symbol == symbol]
    write_financials_partition("artifacts/sec-financials", symbol, sym_vintages)

Each vintage records EDGAR's official accepted_at timestamp and computes availability_anchor = accepted_at + lag_days. Financial statement rows preserve native line item labels and concepts (concept, label, value_native). The native presentation-tree path currently sets standard_concept to the native concept; this field does not establish a cross-company taxonomy mapping. Compatibility inputs may supply a separate standard_concept, whose semantics still require explicit validation before quantitative comparisons.

Live requests default to parser_version="sec-live-financial-parser-v2-edgartools-5.56.0". This mode requires the selected filing's bounded raw XML instance, corroborates native facts against it, and preserves complete units. Missing, ambiguous or mismatched unit evidence raises SecUnitEvidenceError; the call does not return a partial batch or silently switch versions. The raw instance download has its own byte budget; this is not a hard deadline on edgartools' other filing requests.

Evidenced vintages carry SecFinancialUnitEvidence and use v4 identities and Parquet partitions. Statement rows in v4 explicitly identify their vintage, including mixed old/new results for the same accession. V3-only writes and old vintage identities remain unchanged. Existing immutable roots must be retained; write changed data to a new root. Older SDK validators may reject v4, so consumers must adopt a compatible pinned version and validate their reruns before switching. Evidence records the original instance identity, without retaining its bytes or claiming authenticated origin, financial quality, publication time, or live replay.

For explicit legacy behavior, select sec-live-financial-parser-v1-edgartools-5.56.0 in the request or the financials CLI's --parser-version option. V1 retains its known compound-unit limitations. Standalone parse_statement_rows retains its legacy default; explicit v2 also requires raw_instance and a native statement. See the live unit evidence contract.

For offline, explicitly versioned PIT research, the SEC package also exposes serialize_sec_typed_rows_projection, SecNormalizedFinancialFactVersion, SecPitPolicy, and select_sec_financial_versions. The caller writes the canonical typed-row projection into its SnapshotStore, supplies exact SOURCE_DECLARED timestamp evidence, then explicitly selects MARKET_KNOWN or SYSTEM_REPLAY. This projection binds an accession, typed row and caller-attested source-artifact identity, but is not original SEC XBRL/SGML validation. SYSTEM_REPLAY additionally requires an exact PASS quality record and consumer commit before the requested cutoff. See the SEC financial production PIT contract.

For a runnable offline introduction, execute uv run python examples/sec_offline_research.py from a repository checkout. The synthetic research example creates four synthetic quarters, writes and reloads an immutable PIT bundle, and demonstrates publication, consumer-commit and later-quarantine cutoffs. It verifies a TTM of 100.00 USD with four input identities. All timestamps and quality/commit records are synthetic declarations; this does not qualify real SEC data for market reconstruction or prove a research strategy. It uses temporary local files and needs no credentials or network access.

compute_sec_four_quarter_ttm provides offline revenue and net-income sums from exactly four declared independent fiscal quarters. SecQuarterTtmConfig binds their normalized version IDs, native concepts, fiscal dates, accounting scope, comparability cohort and declaration references to an explicit PIT mode, cutoff and policy. The function reruns the existing selector and returns exact Decimal output, input lineage and an input_availability_bound. That bound describes the selected inputs; the consumer supplies the derived result's publication and first usable session. Issuer ownership, quarter independence and comparability remain explicit caller assertions. The recipe validates structural agreement with the selected rows. See the four-quarter TTM contract for the input, arithmetic and declaration limits.

from ohmydata.providers.sec import compute_sec_four_quarter_ttm

# config is a SecQuarterTtmConfig containing four caller declarations and references.
ttm = compute_sec_four_quarter_ttm(
    config=config,
    versions=normalized_versions,
    quality_records=quality_records,
    consumer_commits=consumer_commits,
    max_input_records=10000,
)

compute_sec_metric_graph evaluates a bounded, topologically ordered graph of fixed recipes: four-quarter or FY+current-YTD−prior-YTD revenue/net-income TTM, CFO−CapEx, YoY, margins, PE, PS and FPE. SEC terminals pass through the existing PIT selector once; market-cap, security-price and forecast-EPS inputs remain distinct caller attestations with explicit source, observation, quality and optional commit references. The graph does not authenticate those attestations.

Every terminal declares period/fiscal labels, accounting and attribution scope, currency, dimensions, comparability cohort and security basis. CapEx sign, denominator policy, division precision and rounding are explicit. PE/PS require company-total equity and a graph-derived TTM denominator; FPE preserves the forecast horizon and requires identical currency/security units and security basis. No implicit FX, share-class/ADR conversion or PE×EPS price reconstruction is performed. Domain policies can return a labeled missing final value for an invalid denominator; missing source coverage and incompatible basis fail.

Results retain complete selected SEC evidence or labeled external attestations, transitive input identities and input availability bounds. Monetary arithmetic is exact; division uses an isolated Decimal context and reports a conservative absolute error bound. Rounded ratio outputs are final outputs only. See the neutral metric graph contract.

from ohmydata.providers.sec import compute_sec_metric_graph

metrics = compute_sec_metric_graph(
    config=metric_graph_config,
    versions=normalized_versions,
    quality_records=quality_records,
    consumer_commits=consumer_commits,
    external_inputs=external_attestations,
)
final_metric = metrics.final

discover_sec_filing_events replays a retained SEC submissions root and every historical file declared by that root. The caller supplies a CIK, selected forms, UTC acceptance window, overlap duration and incremental/reconciliation mode. It preserves acceptance metadata, emits filing events and exact 8-K item 2.02 events, and fails on missing pages or conflicting facts. Acceptance is not proof of the website's first publication time.

SecEventDiscoveryLedger atomically stores discovered events and their cursor in immutable generations. On POSIX filesystems, appenders cooperate through a writer lock; load() replays the retained source observations and reconstructs the entire committed chain. An exact retry returns its original receipt, even after later appends. The ledger records discovery; execution, retry scheduling and consumer publication remain separate. Resource caps and snapshot request bindings are specified in the event discovery contract.

from ohmydata.providers.sec import SecEventDiscoveryLedger, discover_sec_filing_events

ledger = SecEventDiscoveryLedger(ledger_path, store=snapshot_store)
head, discovered = ledger.load()
batch = discover_sec_filing_events(
    snapshot_store,
    retained_root_source,
    retained_history_sources,
    policy=discovery_policy,
    prior_cursor=None if head is None else head.cursor,
)
receipt = ledger.append(batch, expected_receipt_id=None if head is None else head.receipt_id)

SecEventWorkLedger records explicit caller reports through DISCOVERED, QUEUED, FETCHING and VALIDATING to READY, QUARANTINED or FAILED, with bounded transient retry waits. It binds one immutable work specification and discovery batch to a separate caller-selected directory. Entering FETCHING counts a total attempt; the caller supplies timestamps and retry deadlines. Replaying the journal reconstructs every transition from retained discovery evidence. READY preserves the previously reported outputs and requires quality-reference IDs; the report does not issue quality approval or publish data.

SecEventExecutor drives that journal synchronously with injected acquisition and validation callbacks. Both callbacks support explicit COMPLETE/NOT_STARTED/ UNKNOWN recovery and return retained operation-bound receipts created through retain_sec_execution_outputs or retain_sec_execution_validation. Uncertain side effects fail without automatic retry. Declared transient acquisition failures wait for an explicit retry time; there is no background loop or internal sleep. Warm READY/QUARANTINED reuse replays output and validation bytes without invoking callbacks. Output RAW_FACT IDs bind snapshot bytes and do not certify financial quality. See the execution and recovery contract for callback interfaces, fixed receipts, lock scope and cooperative resource limits.

from ohmydata.providers.sec import SecEventExecutor

executor = SecEventExecutor(work_ledger, store=execution_snapshot_store, clock=utc_clock)
result = executor.run(
    discovery_batch,
    work_spec,
    acquisition_handler,
    validation_handler,
    deadline=operation_deadline,
    max_steps=16,
)
print(result.status, result.state)

SecDependencyIndex holds immutable, caller-declared typed version edges. plan_sec_event_invalidation(index, changed_inputs, known_at=...) returns exact reachable output IDs and traversed edge IDs using only edges recorded by that cutoff. Recipe/configuration versions can be explicit inputs. Unknown inputs have no known dependent output; that result does not prove complete lineage. The plan identifies existing outputs for reconsideration without changing their historical availability. See the work and invalidation contract.

from ohmydata.providers.sec import SecEventWorkLedger, plan_sec_event_invalidation

work = SecEventWorkLedger(work_path, store=snapshot_store)
head, state, registered_edges = work.load()
report_receipt = work.append(
    batch,
    work_spec,
    work_command,
    expected_receipt_id=None if head is None else head.receipt_id,
)
invalidation = plan_sec_event_invalidation(
    dependency_index, changed_version_ids, known_at=knowledge_cutoff
)

write_sec_pit_bundle can freeze a caller-selected receipt closure through SnapshotStore; load_sec_pit_bundle rebuilds it only from an injected observation-ID resolver and source store, without persisting source bytes or paths. Both functions default to at most 10,000 input receipts and 8 MiB per bundle or replayed source payload; max_records and max_bytes accept stricter positive integer limits. See the bundle replay contract.

SecQualityFinding and select_sec_quality_findings additionally provide a separate, bounded history of caller-authored normalized-row quality assertions. Their evidence references are only caller-supplied observation/fact identities: they do not prove source correctness. Passing a non-empty quality_findings to write_sec_pit_bundle writes a v2 bundle and requires the same injected source store and observation resolver used at replay; every referenced observation is replay-verified without persisting payload bytes or paths. evaluate_sec_structural_quality can create bounded OPEN findings for absent row identity fields, incomplete or unknown periods, and exact duplicate-row value disagreements in supplied replay-bound normalized versions. Its report records the exact versions and fixed rules checked; it does not prove a filing or value is correct, establish availability, or change a PIT/quality-policy decision. Automatic accounting checks and financial-value corrections are not implemented. The quality-finding contract, structural-rule contract, and v2 bundle contract define the history, assertions, and closure requirements.

For retained traditional-XBRL full submissions, produce_sec_financials_from_sgml builds the same typed projection from one offline raw observation. The source must use this exact request shape and serialization; the call accepts no URL, path, credential, or caller-supplied publication time:

The SGML producer uses filing acceptance as an availability proxy. SEC acceptance does not prove first website publication. MARKET_KNOWN raises ValueError if any supplied version uses sec-sgml-financial-adapter-v1 or sec-sgml-financial-adapter-v2, including versions that would otherwise be excluded by policy, cutoff, or quality. This applies to previously saved versions as well. SYSTEM_REPLAY retains its existing time, quality, and consumer-commit requirements; it does not establish first publication. Do not relabel these versions to bypass the check. Affected market backtests need review and reruns with qualifying evidence before their results can be relied on. Snapshots and identities remain unchanged. The availability decision describes the migration and the separate known-by production capability described below.

from ohmydata.core import RequestSpec
from ohmydata.providers.sec import SecSgmlFinancialsRequest, produce_sec_financials_from_sgml

raw_observation = source_store.observe(
    RequestSpec(
        "sec",
        "company-filing-sgml",
        {
            "cik": "0000320193",
            "accession_number": "0000320193-24-000006",
            "form": "10-Q",
        },
    ),
    retained_full_sgml_bytes,
    observed_at,
    "sec-filing-sgml-v1",
)
production = produce_sec_financials_from_sgml(
    source_store=source_store,
    source_observation=raw_observation,
    projection_store=projection_store,
    request=SecSgmlFinancialsRequest(
        "AAPL",
        "0000320193",
        "0000320193-24-000006",
        "10-Q",
        ("income_statement",),
        include_dimensions=False,
    ),
    produced_at=produced_at,
)

It supports full SEC SGML with embedded traditional XBRL schema, presentation, label, and instance documents. For inline-only submissions, retain a separate canonical sec-xbrl-package-v1 observation containing SEC-extracted traditional components and call produce_sec_financials_from_xbrl_package; that entry also requires explicitly bound SOURCE_DECLARED timestamp evidence. The parser does not fetch filings or parse inline XBRL. max_rows bounds emitted normalized rows; it does not claim to be a hard limit on memory used inside the third-party XBRL parser.

For a separately retained extracted package, the caller supplies the original component bytes and evidence of when that exact package became public:

from ohmydata.core import AvailabilityBasis, AvailabilityEvidence, AvailabilityPrecision
from ohmydata.providers.sec import (
    SecXbrlPackageAvailability,
    SecXbrlPackageComponents,
    produce_sec_financials_from_xbrl_package,
    serialize_sec_xbrl_package,
)

package_bytes = serialize_sec_xbrl_package(
    sgml_observation=raw_observation,
    cik=request.cik,
    accession_number=request.accession_number,
    form=request.form,
    source_available_at=package_public_at,
    components=SecXbrlPackageComponents(
        schema=retained_schema_bytes,
        presentation=retained_presentation_bytes,
        labels=retained_labels_bytes,
        instance=retained_instance_bytes,
    ),
)
package_observation = package_store.observe(
    RequestSpec(
        "sec",
        "company-filing-xbrl-package",
        {
            "cik": request.cik,
            "accession_number": request.accession_number,
            "form": request.form,
        },
    ),
    package_bytes,
    package_observed_at,
    "sec-xbrl-package-v1",
)
package_availability = SecXbrlPackageAvailability(
    package_observation,
    AvailabilityEvidence.from_observation(
        package_store,
        package_observation,
        source_available_at=package_public_at,
        availability_basis=AvailabilityBasis.SOURCE_DECLARED,
        availability_precision=AvailabilityPrecision.TIMESTAMP,
    ),
)
production = produce_sec_financials_from_xbrl_package(
    source_store=source_store,
    source_observation=raw_observation,
    package_store=package_store,
    package_observation=package_observation,
    package_availability=package_availability,
    projection_store=projection_store,
    request=request,
    produced_at=produced_at,
)

Here request is an explicit SecSgmlFinancialsRequest as above. The normalized versions and vintage availability anchor use the later of header acceptance and the declared package publication timestamp; accepted_at retains the original header timestamp. Fetch time cannot substitute for that declaration. Retain both source observations and the evidence for rebuilding: bundle replay alone still verifies only the typed projection. Origin and filing correspondence remain caller assertions checked against the retained binding and CIK; this is reproducible parsing, not independent cross-validation. See the package contract for limits.

Both offline producers now default to their own parser v2, preserving complete raw unit definitions before building projections. For exact historical source reconstruction, explicitly pass the corresponding v1 parser_version: sec-sgml-financial-parser-v1-edgartools-5.56.0 or sec-xbrl-package-financial-parser-v1-edgartools-5.56.0. Their v2 names replace parser-v1 with parser-v2; the returned production exposes the selected version. V2 has new adapter/configuration identities even when simple-unit projection bytes are unchanged. Old bundles retain their recorded versions, and old quality or commit records do not qualify corrected versions. SGML v2 remains an acceptance proxy. See the offline unit repair contract.

When first publication is unknown, use the separate observed-package path. It records when the complete local inputs were known, without requiring a claimed publication timestamp:

from ohmydata.providers.sec import (
    produce_sec_financials_from_observed_xbrl_package,
    serialize_sec_observed_xbrl_package,
)

observed_package_bytes = serialize_sec_observed_xbrl_package(
    sgml_observation=raw_observation,
    cik=request.cik,
    accession_number=request.accession_number,
    form=request.form,
    components=SecXbrlPackageComponents(
        schema=retained_schema_bytes,
        presentation=retained_presentation_bytes,
        labels=retained_labels_bytes,
        instance=retained_instance_bytes,
    ),
)
observed_package = package_store.observe(
    RequestSpec(
        "sec",
        "company-filing-observed-xbrl-package",
        {
            "cik": request.cik,
            "accession_number": request.accession_number,
            "form": request.form,
        },
    ),
    observed_package_bytes,
    package_observed_at,
    "sec-observed-xbrl-package-v1",
)
observed_production = produce_sec_financials_from_observed_xbrl_package(
    source_store=source_store,
    source_observation=raw_observation,
    package_store=package_store,
    package_observation=observed_package,
    output_store=output_store,
    request=request,
    produced_at=produced_at,
)
known_by_at = observed_production.evidence.known_by_at

package_observed_at records local observation of the complete assembled envelope. known_by_at is the later of that receipt and the selected SGML receipt; both must be no later than production. The new vintage and output serialization keep this time separate from acceptance. Retain all three observations and rerun the producer with the same inputs to reproduce the result. These results are not inputs to the legacy PIT selector or bundles. Only the producer creates validated production objects; direct construction and substitutions of their bound fields are rejected. Persisted observed-row bytes retain their existing schema. See the known-by contract. Callers that previously constructed production objects directly must now use produce_sec_financials_from_observed_xbrl_package with their retained inputs. There is no snapshot migration or promotion of old PIT evidence.

New observed productions default to parser_version="sec-observed-xbrl-financial-parser-v2-edgartools-5.56.0". V2 resolves units directly from the retained instance: simple USD/shares/pure units remain strings; compound units use compact JSON containing type and either measures or numerator/denominator lists. Compound currency is None; a USD numerator does not make USD/share a plain currency amount. To reproduce historical v1 output explicitly pass parser_version="sec-observed-xbrl-financial-parser-v1-edgartools-5.56.0". Observed bundle loading selects the version recorded in the retained output and rebuilds that complete source chain. V1 retains its known compound-unit defect for historical reconstruction. V2 has a distinct configuration and production identity and requires its own quality assessment and consumer commit. The SGML and declared-availability package v2 paths use the same correction; their explicit v1 paths retain the known unit limitations. Live v2 uses the separate raw-instance evidence contract described above. See the versioned repair contract.

evaluate_sec_observed_accounting checks explicitly selected equalities within one observed production. It returns immutable MATCH/MISMATCH/MISSING/INCOMPARABLE diagnostics with row evidence, exact residuals and an explicit tolerance policy. It does not create normalized findings, financial PASS or consumer commits. Selectors, accounting completeness and cash-change definitions are caller declarations; absent FX never becomes zero. Example for caller-verified selectors:

from ohmydata.providers.sec import (
    SecAccountingApplicability,
    SecAccountingRule,
    SecAccountingTerm,
    SecAccountingTolerance,
    evaluate_sec_observed_accounting,
)

equation = SecAccountingRule(
    "assets-reported-total",
    SecAccountingApplicability.SAME_CONTEXT,
    (
        SecAccountingTerm("balance_sheet", "us-gaap_Assets", "c1", 1),
        SecAccountingTerm("balance_sheet", "us-gaap_LiabilitiesAndStockholdersEquity", "c1", -1),
    ),
    "iso4217:USD",
    SecAccountingTolerance.EXACT,
    "evidence:caller-verified-same-consolidated-scope",
)
accounting = evaluate_sec_observed_accounting(
    observed_production,
    [equation],
    detected_at=produced_at,
    recorded_at=produced_at,
)

Exact tolerance is zero. ASSUME_NEAREST_REPORTED_DECIMALS instead explicitly assumes nearest rounding for every term and derives the summed half-unit bound; missing or invalid native precision makes the comparison INCOMPARABLE. This is not SEC-certified rounding. Rules also support explicit three-term cash rollforwards. Limits and applicability are in the accounting contract.

The separate in-memory observed system selector requires caller-attested quality and consumer-commit records. A production alone is insufficient. A later quarantine or revocation blocks selection from that time onward; a later PASS needs a commit referencing that exact quality record. Future records cannot change an earlier cutoff. This records the caller's decisions, without checking the truth of a financial assessment or performing a consumer publication.

from ohmydata.providers.sec import (
    SecObservedFinancialConsumerCommit,
    SecObservedFinancialQualityRecord,
    SecObservedFinancialReplayPolicy,
    SecQualityStatus,
    select_sec_observed_financial_productions,
)

# These records represent a PASS assessment and a commit already made by the caller.
quality = SecObservedFinancialQualityRecord(
    observed_production.production_identity,
    "example-quality-v1",
    SecQualityStatus.PASS,
    quality_recorded_at,
)
commit = SecObservedFinancialConsumerCommit(
    observed_production.production_identity,
    quality.quality_record_id,
    consumer_dataset_identity,
    committed_at,
)
replay_policy = SecObservedFinancialReplayPolicy(
    output_schema_version="sec-financial-observed-rows-v1",
    parser_version="sec-observed-xbrl-financial-parser-v2-edgartools-5.56.0",
    configuration_version="sec-observed-xbrl-financial-config-v1",
    configuration_identity=expected_configuration_identity,
    quality_policy_version="example-quality-v1",
    consumer_dataset_identity=consumer_dataset_identity,
    knowledge_cutoff=knowledge_cutoff,
)
selected = select_sec_observed_financial_productions(
    [observed_production],
    [quality],
    [commit],
    replay_policy,
)

Selection uses explicit schema/parser/configuration, quality policy and consumer dataset identities, and returns all eligible complete packages. It has no MARKET_KNOWN mode and performs no network, snapshot reads or parsing. Retain and reproduce source productions and supply lifecycle records yourself; durable observed lifecycle persistence uses the separate bundle API below. See the system replay contract. Queries are bounded to 100 production inputs, 10,000 quality records, 10,000 commits and 100,000 aggregate rows; optional caller limits can only tighten these bounds. Over-limit inputs fail explicitly, including iterators. When several quality policies share a consumer dataset, selection ignores commits bound to an included quality record for another policy of the same production. Missing references and cross-production bindings still fail; commits for the selected policy must satisfy its original time and PASS gates.

Persist observed productions and their complete lifecycle in a separate immutable bundle. The caller resolves observation identities to retained stores; bundle JSON contains no storage paths. Save and load both rebuild the original source/package parsing chain and compare its exact bytes with the retained output before accepting production objects. Loading performs no writes.

from ohmydata.providers.sec import (
    load_sec_observed_financial_bundle,
    write_sec_observed_financial_bundle,
)

observations = {
    raw_observation.observation_identity: (source_store, raw_observation),
    observed_package.observation_identity: (package_store, observed_package),
    observed_production.output_observation.observation_identity: (
        output_store,
        observed_production.output_observation,
    ),
}
bundle_ref = write_sec_observed_financial_bundle(
    store=bundle_store,
    batch_identity="example-observed-batch-v1",
    productions=[observed_production],
    quality_records=[quality],
    consumer_commits=[commit],
    captured_at=captured_at,
    resolve_observation=observations.__getitem__,
)
restored = load_sec_observed_financial_bundle(
    store=bundle_store,
    bundle_ref=bundle_ref,
    resolve_observation=observations.__getitem__,
)
replayed = select_sec_observed_financial_productions(
    restored.productions,
    restored.quality_records,
    restored.consumer_commits,
    replay_policy,
)

captured_at must be no earlier than every retained observation, production, quality record and commit in the bundle. It records persistence, not historical eligibility. Reusing a batch identity with changed content or capture time fails; exact repeats are idempotent. Save/load validate all included quality policies and consumer datasets, including records beyond a later query's cutoff. Keep every referenced source/package/output observation and the pinned parser available for restoration. Defaults allow 10 productions, 10,000 quality records, 10,000 commits, 100,000 rows, an 8 MiB bundle and 32 MiB of unique dependency payloads, with 8 MiB per dependency. Caller limits may only be stricter. Larger source admission remains deferred after resource experiments; SGML validation scans source spans to avoid copying unused document bodies. See the observed bundle contract.

SnapshotStore.replay and replay_observation also accept optional max_payload_bytes (a non-negative integer). Their default None preserves unlimited payload reads; exceeding an explicit limit raises SnapshotIntegrityError. This limit covers response bytes, not JSON manifest metadata. Retain source snapshots and observation receipts alongside bundles.

from datetime import UTC, datetime

from ohmydata.core import (
    AvailabilityBasis,
    AvailabilityEvidence,
    AvailabilityPrecision,
    RequestSpec,
)
from ohmydata.providers.sec import (
    SecNormalizedFinancialFactVersion,
    evaluate_sec_structural_quality,
    serialize_sec_typed_rows_projection,
    write_sec_pit_bundle,
)

# ``vintage`` and the source artifact digest are caller-supplied and retained with its source evidence.
source_at = datetime(2024, 5, 1, 21, tzinfo=UTC)
payload = serialize_sec_typed_rows_projection(
    vintage, source_artifact_identity=artifact_sha256, source_available_at=source_at
)
observation = store.observe(
    RequestSpec("sec", "financial-typed-rows", {"accession": vintage.accession_number}, ()),
    payload,
    datetime.now(UTC),
    "sec-financial-typed-rows-projection-v1",
)
evidence = AvailabilityEvidence.from_observation(
    store,
    observation,
    source_available_at=source_at,
    availability_basis=AvailabilityBasis.SOURCE_DECLARED,
    availability_precision=AvailabilityPrecision.TIMESTAMP,
)
version = SecNormalizedFinancialFactVersion.from_projection(
    store=store,
    observation=observation,
    availability=evidence,
    vintage=vintage,
    row_ordinal=0,
    schema_version="sec-financial-normalized-v1",
    adapter_version="adapter-v1",
    normalization_version="normalization-v1",
    configuration_identity=config_sha256,
    recorded_at=datetime.now(UTC),
)
report = evaluate_sec_structural_quality(
    [version], detected_at=datetime.now(UTC), recorded_at=datetime.now(UTC)
)
# ``bundle_store`` is caller-provided; source evidence remains in ``store``.
bundle_ref = write_sec_pit_bundle(
    store=bundle_store,
    batch_identity="structural-1",
    versions=[version],
    quality_records=[],
    quality_findings=report.findings,
    source_store=store,
    resolve_observation=lambda identity: observation,
    captured_at=datetime.now(UTC),
)

The SEC extra supports edgartools==5.56.0. See the financial period contract and v2 migration before rebuilding existing financial datasets.

yfinance (US & Global Market Data, Fundamentals, and Zero-Drift Audit)

Install ohmydata[yfinance] to access normalized market data, valuation ratios, and financial statements with strict version pinning (yfinance==1.7.0):

from ohmydata.providers.yfinance import (
    YFinanceAdjustmentMode,
    YFinanceBatchPolicy,
    YFinanceClient,
    YFinanceDailyBarsRequest,
    YFinanceFundamentalsRequest,
    YFinanceRepairPolicy,
)

client = YFinanceClient()

# 1. Fetch normalized daily bars (OHLCV) with repair isolation
bars_req = YFinanceDailyBarsRequest(
    symbols=("SPY", "QQQ", "^VIX"),
    start_date="2024-01-01",
    end_date_exclusive="2024-02-01",
    adjustment_mode=YFinanceAdjustmentMode.RAW_WITH_ADJ_CLOSE,
    batch_policy=YFinanceBatchPolicy.STRICT,
    repair_policy=YFinanceRepairPolicy.PER_SYMBOL,
)
bars_result = client.fetch_daily_bars(bars_req)
df = bars_result.dataframe

# 2. Fetch fundamentals with FY1 Forward P/E and source metadata
fund_req = YFinanceFundamentalsRequest(
    symbols=("NVDA", "GEV"),
    include_financials=True,
    include_valuation=True,
    include_estimates=True,
)
fund_result = client.fetch_fundamentals(fund_req)

nvda = fund_result.records["NVDA"]
# Forward P/E is calibrated to current year consensus (FY1 0y.avg) rather than out-year (+1y)
print("NVDA Calibrated FPE:", nvda.valuation.forward_pe, nvda.valuation.forward_pe_source)
print("NVDA Raw Yahoo FPE:", nvda.valuation.raw_forward_pe)

gev = fund_result.records["GEV"]
# Legacy flag measures EPS-source divergence; accounting basis remains unknown.
if gev.estimates.has_gaap_distortion:
    print(f"GEV EPS-source gap: {gev.estimates.gaap_diff_pct * 100:.1f}%")

Financial values bind to actual statement columns, with per-metric dates and coverage. FY1 calibration requires an actual quote and compatible currencies; otherwise raw values remain available. See the period selection, valuation provenance and migration guide.

Zero-Drift Audit CLI (omd audit-drift)

Audit 10+ years of historical data against the 13-ETF r10a0 benchmark universe before any provider upgrade:

# Strict unadjusted market bar zero-drift gate (must be bit-exact 0.0 error)
uv run omd audit-drift --universe r10a0 --baseline-dir <old_version_dir> --target-dir <new_version_dir> --raw-only

Dataframe Adapters (Polars & Pandas)

The optional ohmydata[polars] extra provides explicit, eager representation adapters:

from ohmydata.adapters.polars import pandas_to_polars, polars_to_pandas

polars_frame = pandas_to_polars(pandas_frame)
# For validated empty/all-null Pandas object columns, opt into String:
polars_frame = pandas_to_polars(pandas_frame, empty_object_policy="string")
pandas_frame = polars_to_pandas(polars_frame)

Conversions preserve columns and row order, provider-native values, nulls, NaN/infinities, and supported temporal timezones. They do not parse dates, rename or sort columns, scale units, deduplicate, impute, or apply consumer schemas. Unsupported or potentially lossy dtypes fail with SchemaMismatchError; the adapter never contacts a provider or reads credentials. The default empty_object_policy="error" rejects ambiguous empty object columns; the explicit "string" policy casts only empty/all-null object columns to nullable Pandas strings before conversion and never changes populated object columns or imputes missing values.

Local Checks and Verification

uv lock
uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run ty check
uv build
git diff --check

Behavioral evidence for the initial consumers is in docs/behavioral-inventory.md. The adjusted ETF characterization is test-only and uses synthetic JSON fixtures. The public-contract changes and consumer-owned migration boundaries are summarized in docs/v0.1.0-migration.md, docs/v0.1.1-migration.md, docs/v0.1.2-lookthrough-migration.md, and docs/v0.1.3-vintage-plane-migration.md.

SEC document-source closure

produce_sec_document_source_package validates and retains one explicit filing's submissions/index/primary/XBRL source graph. SecDocumentSource binds each role, filename (null for metadata), SnapshotStore and original observation receipt. The seven required roles are submissions, index, primary, schema, presentation, labels, instance; declared calculation/definition links require corresponding retained sources. All filenames and reference targets must agree within one CIK/accession directory. Missing, ambiguous or cross-filing sources and base URI overrides fail before package retention.

from ohmydata.providers.sec import (
    produce_sec_document_source_package,
    restore_sec_document_source_package,
)

source_package = produce_sec_document_source_package(
    store=package_store,
    cik="0000000001",
    accession_number="0000000001-24-000001",
    form="10-Q",
    sources=source_items,  # Caller-retained SecDocumentSource objects.
    captured_at=captured_at,
)
restored_package = restore_sec_document_source_package(
    store=package_store,
    observation=source_package.observation,
    resolve_observation=resolve_observation,  # identity -> (store, observation)
)
assert restored_package.package_identity == source_package.package_identity

The immutable canonical manifest uses a separate source schema; it does not claim to be full SGML. known_by_at includes the package observation and all required sources. It does not establish first market publication or financial quality. This API closes and restores provenance only: financial production, financial-bundle replay and consumer quality integration are separate work. Fixed source limits are 2 MiB each for metadata/XML, 4 MiB for primary HTML, 16 MiB in aggregate and 256 KiB for the manifest. Primary HTML is limited to 256 attributes per element, 200,000 elements and depth 128. Inputs with more attributes that the initial source-v1 implementation accepted now fail explicitly. Invalid source contracts raise ValueError/SchemaMismatchError; snapshot corruption or request mismatch raises SnapshotIntegrityError. No network calls or credential lookup occur here. See the exact source contract for request endpoints, serialization identifiers, reference grammar and limits.

produce_sec_financials_from_document_source builds native financial rows from that source package using pinned edgartools 5.56.0 and mandatory raw unit evidence. It returns a distinct SecDocumentFinancialProduction; old SGML productions and identities retain their behavior. The selected request CIK/accession/form must match the package; symbol remains a caller label. Missing requested statements, empty rows or unresolved raw units fail before output retention.

from ohmydata.providers.sec import (
    SecSgmlFinancialsRequest,
    produce_sec_financials_from_document_source,
    restore_sec_document_financial_production,
)

financial = produce_sec_financials_from_document_source(
    package_store=package_store,
    package_observation=source_package.observation,
    resolve_observation=resolve_observation,
    output_store=financial_store,
    request=SecSgmlFinancialsRequest(
        "FAKE",
        "0000000001",
        "0000000001-24-000001",
        "10-Q",
        ("income_statement",),
        False,
    ),
    produced_at=produced_at,
)
restored_financial = restore_sec_document_financial_production(
    output_store=financial_store,
    output_observation=financial.output_observation,
    resolve_observation=resolve_observation,
)
assert restored_financial.production_identity == financial.production_identity

The resolver must locate both the source-package observation and every original source observation. Restoration reruns source closure and financial parsing, compares exact retained output bytes and performs no writes. Output is bounded by 8 MiB and 10,000 rows. known_by_at is the package observation timestamp; produced_at equals the output observation timestamp and cannot precede the package. The request class is reused as a selection value; no SGML is synthesized. Real filing qualification and consumer publication remain separate gates. See the financial production contract.

Document productions have a separate in-memory system selector:

from ohmydata.providers.sec import select_sec_document_financial_productions

eligible = select_sec_document_financial_productions(
    productions=(restored_financial,),
    quality_records=caller_quality_records,
    consumer_commits=caller_consumer_commits,
    policy=explicit_replay_policy,
)

Use SecObservedFinancialReplayPolicy, SecObservedFinancialQualityRecord and SecObservedFinancialConsumerCommit with the exact document production ID. The policy must explicitly match its schema/parser/configuration, quality policy, consumer dataset and cutoff. A complete production is eligible only after known-by, production, latest visible PASS and its exact consumer commit; missing records yield no eligible result. Future revocation does not change old cutoffs. These are caller attestations, not automatic quality approval or market-first publication evidence. The old observed selector continues to reject document types.

Inputs are bounded before copying/hashing: at most 100 productions, 100,000 rows, 10,000 quality records and 10,000 commits, including duplicate occurrences. A conservative 32MiB serialization budget, 500,000 nodes and depth 16 can reject inputs below those count caps. Text costs twelve bytes per character plus overhead; Decimal admission also limits its in-memory coefficient storage to 8KiB and adjusted exponent to ±10,000, and integers to 64 bits. Limits can be tightened, not expanded. Queries perform no I/O. See the known-by replay contract.

Persist document productions and their caller-attested lifecycle with a distinct immutable bundle. The resolver must locate every original observation, source package and financial output by exact observation identity:

from ohmydata.providers.sec import (
    load_sec_document_financial_bundle,
    write_sec_document_financial_bundle,
)

bundle_ref = write_sec_document_financial_bundle(
    store=bundle_store,
    batch_identity="synthetic-document-batch-v1",
    productions=(restored_financial,),
    quality_records=caller_quality_records,
    consumer_commits=caller_consumer_commits,
    captured_at=bundle_captured_at,
    resolve_observation=resolve_observation,
)
loaded = load_sec_document_financial_bundle(
    store=bundle_store,
    bundle_ref=bundle_ref,
    resolve_observation=resolve_observation,
)

Write validates the full source/parser closure before its single FROZEN snapshot. Load repeats that reconstruction, checks canonical bytes and all quality/commit causality, and performs no writes. The bundle is a manifest over retained dependencies; callers must preserve those snapshots for restart. It does not copy them into the bundle or grant financial quality approval. Capture cannot precede any included production, quality record or consumer commit.

Each operation admits at most 10 productions, 100,000 rows, 10,000 quality records, 10,000 commits and 120 distinct dependencies. The envelope limit is 8MiB; the 32MiB unique dependency budget includes outputs, source packages and all raw sources. Repeated dependencies are counted once but remain subject to each role's integrity checks; payloads are not cached. Writer input admission also uses the conservative document-selector budget. max_bundle_bytes and max_dependency_bytes can tighten these caps. Identical writes are idempotent; conflicting content under the same frozen batch identity fails explicitly. Old observed bundles keep their own schema and reject document productions. See the document bundle contract.

evaluate_sec_document_accounting applies the same explicit SecAccountingRule definitions and tolerance policies to one sealed document financial production. Its arguments match evaluate_sec_observed_accounting; it has a separate exact-type entry and performs no I/O. The returned SecObservedAccountingReport binds the document production/output/parser and configuration identities. This reuse does not turn it into an old SGML product. A MATCH is an arithmetic diagnostic, never financial quality PASS. Missing rows remain MISSING, incompatible contexts/units/precision remain INCOMPARABLE, and detection cannot precede production. See the document accounting contract.

diagnose_sec_fy_ytd_ttm evaluates offline FY+YTD TTM arithmetic directly over sealed observed and document productions without fabricating normalized versions or claiming MARKET_KNOWN publication. See the known-by TTM contract.

Dated invalidation uses explicit target declarations instead of inferring securities from an issuer. SecDatedInvalidationIndex accepts existing SecDependencyEdge values and SecDatedInvalidationTarget declarations binding a derived output, issuer, instrument/binding, recipe, valuation time, knowledge cutoff and PIT mode. A SecDatedInputChange names the old version being superseded; its optional new version is context, not the dependency search seed.

from ohmydata.providers.sec import (
    SecDatedInvalidationIndex,
    plan_sec_dated_invalidation,
)

target_index = SecDatedInvalidationIndex(dependency_edges, declared_targets)
reconsider = plan_sec_dated_invalidation(target_index, declared_changes, known_at=processing_cutoff)

A change's half-open valuation window selects dated targets. Separately, its mode-specific declared availability must be no later than each target's knowledge cutoff. Changes, edges and targets must have been recorded by known_at. Related targets lacking their required availability declaration fail explicitly. Missing market availability does not prevent a system-only plan. The result retains exact targets and a bounded deterministic proof path; it does not prove source timing, coverage of intervening dates, or execute metric recalculation. See the dated invalidation contract.

Embedded-linkbase SEC filings use the explicit produce_sec_embedded_document_source_package and produce_sec_financials_from_embedded_document_source entries. The source contains exactly five raw receipts: submissions, index, primary HTML, intact schema and extracted instance. Embedded label/presentation links stay inside the retained schema; no synthetic standalone linkbase observations are created. The new source, parser and output domains are distinct. Existing producers continue to accept only their original source domains.

restore_sec_document_financial_production, document bundles, selectors and accounting diagnostics support both sealed variants with exact identity binding. Embedded limits are 8 MiB primary, 12 MiB instance, 2 MiB schema/metadata and 24 MiB total source; output stays at 8 MiB/10,000 rows and bundle dependencies at 32 MiB. Sources exceeding a cumulative budget fail explicitly. This path requires one embedded linkbase container using the pinned parser's link: prefix; mixed external filing linkbase references are rejected. Offline resource acceptance does not establish real filing quality or market availability. See the source contract and financial contract.