Skip to main content

🌾 SQL Grain

SQLite databases as Grain data sources.

sql-grain lets you prototype ML data pipelines using SQL queries before committing to a production data format. Define your training examples with expressive SQL—joins, window functions, filtering—and iterate quickly without preprocessing. When you're ready to scale, convert to ArrayRecord or similar formats; sql-grain is not designed for large-scale training.

from sqlgrain import Sqlite3DataSource
import grain

source = Sqlite3DataSource(
    "data.db",
    key_query="SELECT id FROM users",
    record_query="SELECT item FROM purchases WHERE user_id = :id ORDER BY timestamp",
)
dataset = grain.MapDataset.source(source).shuffle().batch(32)

Converting to ArrayRecord

Once you're ready to run larger experiments, convert the dataset to ArrayRecords using the to_array_record function which serializes records in msgpack format with native support for NumPy arrays.

from sqlgrain import to_array_record

to_array_record(source, "output/", shard_every=1000)

Grain's ArrayRecordDataSource reads the raw bytes, and we need to integrate a decoder into the data pipeline. SQL Grain's decode_record takes care of that, matching the behavior of to_array_record.

from sqlgrain import from_array_record, decode_record

ar_source, metadata = from_array_record("output/")
dataset = grain.MapDataset.source(ar_source).map(decode_record).batch(32)

to_array_record is agnostic to the type of the records, and you can also serialize datasets, e.g., for pre-batching.

Metadata

Release files for sql-grain 0.2.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sql-grain 0.2.3
File Size Uploaded Provenance Actions
sql_grain-0.2.3.tar.gz 9.0 kB With provenance Details for sql_grain-0.2.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sql-grain 0.2.3
File Interpreter ABI Platform Actions
sql_grain-0.2.3-py3-none-any.whl Python 3 none any Details for sql_grain-0.2.3-py3-none-any.whl

Total release size: 16.8 kB

Release files / sql_grain-0.2.3.tar.gz

Download URL sql_grain-0.2.3.tar.gz
Size 9.0 kB
Tags Source
SHA-256 checksum
How to use checksums
a2b965b5a3d56106162ec8262397a34f44627221130282ee7baa8cf761babbf9
BLAKE2b-256 checksum
How to use checksums
da8857470eb68470c1778b27daa1c5d97d70122521b009fa1ad71d90c653628e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 9, 2026.

Transparency log

Release files / sql_grain-0.2.3-py3-none-any.whl

Download URL sql_grain-0.2.3-py3-none-any.whl
Size 7.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
48130e28fdb6e1fe564656b8a4f9eec6e8d0c4ffa0ce6477efd1364b029a9c95
BLAKE2b-256 checksum
How to use checksums
42624eb46786da7b9ac0656ba937a57d390906398edb98874202811b695e508f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 9, 2026.

Transparency log

Security for sql-grain 0.2.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for sql_grain-0.2.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jan 9, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page