QALITA Core
QALITA Core is a lightweight helper library used by QALITA packs to load data from multiple sources, materialize them to Parquet in deterministic chunks, and share common utilities (sanitization and aggregation helpers).
Key features
- Unified data access via a simple
DataSourceabstraction and factory - File, database, and object storage loaders with streaming to Parquet
- Deterministic, size-bounded Parquet chunking with stable filenames
- Safe Parquet writing for pandas DataFrames (automatic sanitization)
- Shared aggregators for completeness, outliers, duplicates, and timeliness
- Minimal pack runtime with JSON config loading and simple asset persistence
Supported sources
- Files: CSV (
.csv), Excel (.xlsx), JSON, Parquet (pass-through) - Databases: PostgreSQL, MySQL, Oracle, MS SQL Server, SQLite
- Object storage: Amazon S3, Google Cloud Storage, Azure Blob (via
abfs), HDFS
Notes:
- Folder, MongoDB classes exist as placeholders; MongoDB is not yet implemented.
- SQLite is supported through the generic
DatabaseSourcewhen selected viatype: "sqlite".
Installation
Prerequisites: Python 3.10–3.12 and uv.
Install dependencies and set your environment:
pip install uv
uv sync
Open a uv shell when developing:
uv shell
Quickstart
Use within a Pack
Pack loads four JSON files by default (overridable) and provides load_data() for source or target triggers.
from qalita_core.pack import Pack
pack = Pack(configs={
"pack_conf": "./pack_conf.json",
"source_conf": "./source_conf.json",
"target_conf": "./target_conf.json",
"agent_file": "~/.qalita/.worker",
})
# Ensure chunking/output are set (can be in pack_conf["job"] too)
pack.pack_config.setdefault("job", {})
pack.pack_config["job"]["parquet_output_dir"] = "./parquet"
pack.pack_config["job"]["chunk_rows"] = 100_000
# Load source
source_paths = pack.load_data("source")
# Load target (optional)
target_paths = pack.load_data("target")
# Persist custom metrics/recommendations/schemas to JSON files
pack.metrics.data.append({"key": "score", "value": "0.95", "scope": {"perimeter": "dataset", "value": "my_dataset"}})
pack.metrics.save() # writes metrics.json
pack.recommendations.save() # writes recommendations.json
pack.schemas.save() # writes schemas.json
Parquet chunking and filenames
- CSV/JSON/Excel are streamed with
chunksizeinto multiple parquet files. - Databases are read with chunked SQL via SQLAlchemy/
pandas.read_sql. - Filenames use a stable pattern:
<source>_<object>_part_<k>.parquetwhere:<source>is a slug of the source type (e.g.file,sqlite,postgresql).<object>is a slug of the table name, query label, or file stem.- Example:
file_testdata_part_1.parquet,sqlite_items_part_3.parquet,sqlite_query_part_2.parquet.
Configure output and size via pack_config:
parquet_output_dir(default:./parquet)chunk_rows(default:100000)- Optional
job.source.skiprowsapplied to CSV/Excel
Safe Parquet writing for pandas
On import, QALITA Core installs a small monkeypatch so DataFrame.to_parquet:
- Ensures column names are strings
- Decodes bytes to UTF‑8 strings when present
- Normalizes mixed-type object columns and categoricals
- Defaults to
engine="pyarrow"
You can also call the sanitizer explicitly:
from qalita_core import sanitize_dataframe_for_parquet
clean_df = sanitize_dataframe_for_parquet(df)
Aggregation helpers (for packs)
Helpers centralize common result/metric aggregation logic:
from qalita_core import (
detect_chunked_from_items,
normalize_and_dedupe_recommendations,
CompletenessAggregator,
OutlierAggregator,
DuplicateAggregator,
TimelinessAggregator,
)
CompletenessAggregator: column/dataset completeness and schema extractionOutlierAggregator: per-column and dataset outlier/normality metricsDuplicateAggregator: duplicate counts and dataset-level score using key columnsTimelinessAggregator: dates/years coverage and recency scoring
Development
- Tests:
uv run pytest - Formatting:
uv run black . - Linting:
uv run flake8anduv run pylint <module> - Editable install while debugging:
uv sync
uv pip install -e .
Documentation
Additional material can be found in the online documentation: https://doc.qalita.io/.
1 maintainer
Release files for qalita-core 1.6.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| qalita_core-1.6.7.tar.gz | 1.5 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| qalita_core-1.6.7-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 1.5 MB
Release files / qalita_core-1.6.7.tar.gz
| Download URL | qalita_core-1.6.7.tar.gz |
|---|---|
| Size | 1.5 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
70c12639bbbb8fc73cf6b2813b7fbe4d50eefd357b979cf749888e0a870b4071
|
|
BLAKE2b-256 checksum How to use checksums |
a0e786bed1424f98e66c0f2f8c5be547cab98600b78ad770620b2a06f839f440
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 8, 2026.
Transparency logRelease files / qalita_core-1.6.7-py3-none-any.whl
| Download URL | qalita_core-1.6.7-py3-none-any.whl |
|---|---|
| Size | 38.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
874b85897e2028fd8536a54640012c49e5a54421727361f5e3c9d8008d1e8491
|
|
BLAKE2b-256 checksum How to use checksums |
1a6f692215a45ad8aca26cd2c46f939daadbabc495bfbdf8764e6c0edbcd3290
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 8, 2026.
Transparency log
