MIF-DAL — Data Abstraction Layer

mif-dal is the data assembly layer of the MIF ecosystem. It fetches OHLCV market data from external sources, certifies it through mif-dqf, and delivers an immutable, hash-anchored result ready for metric computation.

---

Why mif-dal?

Quantitative research fails silently when its data foundation is uncertain. A strategy that looks profitable on Yahoo Finance data may behave differently on Kraken data — same asset, different gaps, different fills, different hashes.

mif-dal makes data provenance explicit and mandatory:

• Every stream carries a assembly_hash (SHA-256 of raw bytes before any transformation). Two calls with the same parameters produce the same hash, or the difference is auditable.
• Physical validity is guaranteed by mif-dqf before any data reaches your computation layer.
• Source fallback is recorded, not hidden. If Kraken was unavailable and Yahoo was used instead, that decision is in the source_manifest.
• The output — DALHandoff — is frozen. It cannot be mutated after emission.

---

The MIF Ecosystem

External data sources
        ↓
[mif-dal]   assembles stream, computes assembly_hash, delegates to mif-dqf
        ↓ DALHandoff
[mif-dqf]   certifies physical validity (OHLCV physics, calendar, gaps)
        ↓ DQFReport embedded in DALHandoff
[mif-core]  certifies metric/signal integrity (planned)
        ↓ MIFCertification
[QAAF Studio]  research layer (QS-PAF, QS-MÉTIS)

mif-dal sits between your data sources and your computation. It does not evaluate strategies. It does not qualify asset pairs. It answers one question: "Can I assemble a reproducible, certified OHLCV stream from available sources for this asset and date range?"

---

Installation

pip install mif-dal

Requires Python 3.11+ and mif-dqf >= 1.2.0 (installed automatically).

Optional: Dukascopy adapter

The Dukascopy adapter requires Node.js and dukascopy-node:

npm install -g dukascopy-node

---

Quick Start

from dal import DAL, DALConfig
from dal.adapters import KrakenAdapter, YahooAdapter

config = DALConfig()
dal = DAL(config, sources=(KrakenAdapter(), YahooAdapter()))

# One call per asset — caller composes the pair (architectural decision D-DAL-007)
h_paxg = dal.get_certified_stream(
    asset_id="PAXG-USD",
    source_preference=["kraken", "yahoo"],
    start="2024-01-01",
    end="2024-12-31",
    calendar="CRYPTO_247",
    dqf_version_target="1.2.0",
)

h_btc = dal.get_certified_stream(
    asset_id="BTC-USD",
    source_preference=["kraken", "yahoo"],
    start="2024-01-01",
    end="2024-12-31",
    calendar="CRYPTO_247",
    dqf_version_target="1.2.0",
)

# Caller constructs the ratio
prices_pair = h_paxg.stream["close"] / h_btc.stream["close"]

print(f"DQF status : {h_paxg.dqf_status} / {h_btc.dqf_status}")
print(f"AQI        : {h_paxg.aqi:.0f} / {h_btc.aqi:.0f}")
print(f"Hash PAXG  : {h_paxg.assembly_hash[:16]}...")
print(f"Hash BTC   : {h_btc.assembly_hash[:16]}...")

Diagnostic mode (exploratory, no version enforcement)

h = dal.get_diagnostic_stream(
    asset_id="BTC-USD",
    source_preference=["yahoo"],
    start="2023-01-01",
    end="2023-12-31",
    calendar="CRYPTO_247",
)
print(h.dqf_report)

---

The DALHandoff Object

Every successful call returns a DALHandoff — a frozen dataclass with 15 fields:

Field	Type	Description
stream	pd.DataFrame	OHLCV, UTC DatetimeIndex, no NaN, sorted ascending
asset_id	str	e.g. "PAXG-USD"
calendar	str	e.g. "CRYPTO_247", "NYSE"
assembly_hash	str	SHA-256 of raw bytes before any transformation
handoff_timestamp	datetime	UTC emission time
dal_version	str	mif-dal package version
source_manifest	tuple	Immutable record of sources used and fallback chain
coverage	str	FULL / PARTIAL / DEGRADED
truncated_days	int	0 if FULL
dqf_status	str	PASS or WARNING
dqf_mpi	float	MIF Purity Index 0–100 (from mif-dqf)
dqf_version	str	mif-dqf version used
dqf_version_target	str	Minimum version declared by caller
dqf_report	DQFReport	Full mif-dqf report
aqi	float	Assembly Quality Index 0–100

DALHandoff is frozen=True. It cannot be mutated after emission.

---

Assembly Quality Index (AQI)

AQI measures how much intervention was required to assemble the stream. 100 = preferred source, full range, no retries. Lower = more intervention.

AQI	Label	Meaning
100	EXCELLENT	Preferred source, full range, no retries
80–99	GOOD	Minor fallback or single retry
60–79	ADVISORY	Fallback + partial range
< 60	REVIEW	Heavy intervention — investigate

AQI is on the D-SIG Standard v0.5 0–100 scale and can be used directly as a dimensions entry in a D-SIG signal.

---

Available Sources

Source	Adapter	Assets	Notes
Kraken	KrakenAdapter	Crypto pairs	Data available ~last 12 months
Yahoo Finance	YahooAdapter	Equities, ETFs, Crypto	Broad coverage, variable quality
Dukascopy	DukascopyAdapter	Forex, Crypto	Requires dukascopy-node
In-memory	InMemorySource	Any	Testing and research

---

Error Handling

from dal.exceptions import DALHandoffError, DALVersionError, DALConfigError

try:
    h = dal.get_certified_stream(...)
except DALConfigError as e:
    # Missing or invalid configuration (e.g. calendar omitted)
    print(e)
except DALVersionError as e:
    # Installed mif-dqf < dqf_version_target
    print(e)
except DALHandoffError as e:
    # DQF returned VOID, or all sources failed
    print(e.reason)          # "DQF_VOID" | "ALL_SOURCES_FAILED"
    print(e.source_failures) # list of per-source error details

Errors are atomic: either a complete DALHandoff is returned, or an exception is raised. There is no partial or degraded return value.

---

Development

git clone https://github.com/symbioticode/mif-dal.git
cd mif-dal
uv sync --extra dev
./scripts/dev.sh check   # Ruff + Mypy + Pytest

mif-dal does not ship a flake.nix at this stage, unlike mif-dqf — this is a deliberate scope decision, not an oversight.

Running tests

# Without network (fast, CI-suitable)
pytest tests/ -q

# With real network (Kraken, Yahoo, Dukascopy)
pytest tests/ --run-network

Validation

python scripts/adversarial_dal_check_p3.py    # Adversarial suite (65 checks)

---

Documentation

Document	Description
docs/API.md	Full public API reference
docs/ARCHITECTURE.md	Pipeline, decisions, component map
TROUBLESHOOTING.md	Common issues (Kraken limits, NixOS, Dukascopy)
docs/DAL_SPECIFICATION_v1.0.md	Formal specification — source of truth

---

Relationship to mif-dqf

mif-dal depends on mif-dqf for physical data validation. mif-dal calls mif-dqf internally — you do not need to call mif-dqf directly when using mif-dal.

The assembly_hash computed by mif-dal is passed to mif-dqf as raw_data_hash, so both layers anchor on the same SHA-256 value. The MIF-UID produced by mif-dqf is included in DALHandoff.dqf_report.

Revalidating a certification manually, outside mif-dal

If you hold a DALHandoff and want to revalidate it independently by calling mif-dqf directly (rather than going back through mif-dal), you must pass raw_data_hash=handoff.assembly_hash explicitly:

from dqf import DQFConfig, DQFMode, DQFValidator

validator = DQFValidator(DQFConfig(mode=DQFMode.CERTIFICATION))

report = validator.validate(
    handoff.stream,
    calendar=handoff.calendar,
    raw_data_hash=handoff.assembly_hash,  # required to reproduce the same MIF-UID
)

assert report.mif_uid == handoff.dqf_report.mif_uid

Warning: mif-dal's assembly_hash and mif-dqf's self-computed default hash use two different, incompatible algorithms (different serialization, different output format). If you omit raw_data_hash, mif-dqf silently computes its own hash instead of raising an error — the call succeeds, but it produces a different MIF-UID than the one in the original DALHandoff.dqf_report, with no warning that anything went wrong. Always pass raw_data_hash=handoff.assembly_hash when revalidating outside mif-dal.

---

License

MIT — see LICENSE.

Part of the MIF ecosystem by symbioticode.