Skip to main content
Daffodil logo

Python Daffodil

Daffodil (DAtaFrames For Optimized Data Inspection and Logical processing) gives Python fast, lightweight 2-D data tables. A table is a list of rows, and each row is a plain Python list. Use it where data arrives and is processed row by row: reading files, building records, cleaning, reshaping, and writing them out again.

Documentation: https://raylutz.github.io/daffodil/

Status: alpha, and used in production for several years. The changes of each release are in the changelog.

Install

pip install daffodil

Python 3.10 or later. The table code is pure Python. pandas, NumPy and a few other packages are needed only for the features that use them, such as to_pandas_df(), and are not installed with daffodil. The Install section of the Guide lists them.

The data model

The Daffodil data model: lol, hd, kd and dtypes

The rows are kept in lol, a list of lists. The column names are kept once, in the header dict hd, which maps each name to its position. A table can have a keyfield, a column whose values identify the rows, and then the key dict kd maps each key to its row. A cell can hold any Python object: text, a number, a list, a dict, or even another table.

Quick start

from daffodil.daf import Daf

daf = Daf(cols=['id', 'item', 'qty'], keyfield='id')
daf.append({'id': 'a1', 'item': 'pen', 'qty': 3})
daf.append(['b2', 'ink', 10])

daf['b2']                     # the row with key 'b2'
daf[0]                        # the first row
daf[:, 'qty']                 # the qty column
daf['b2', 'qty'].to_value()   # 10
daf.select_where(lambda row: row['qty'] > 5)
print(daf)                    # a Markdown table

Why Daffodil

  • Adding a row is cheap. It is appended to a Python list. pandas has no cheap way to add a row, so code that uses it often collects the rows first and converts them at the end.
  • No conversion. The data stays the Python objects you put in, so there is no cost to get it in or out.
  • A selection shares the rows. Selecting rows copies no data, and lookups by key are dict lookups.
  • Mixed data is fine. A column can mix text, numbers and empty cells, and nothing is coerced to NaN or to a common type.
  • Tables print as Markdown, ready for a terminal or a report, and can be read back.
  • Small and pure Python, so it starts fast, which matters in short-lived cloud functions.

For heavy math over whole numeric columns, pandas, NumPy or Polars are faster. Build and clean the data as rows in Daffodil, then hand the numeric columns over with to_pandas_df() or to_numpy(). The documentation compares the two, with measurements.

Tables larger than memory

SqlDaf stores a table in a SQLite database file, so its rows stay out of Python memory. It supports tables larger than the memory available, and many operations run directly in SQLite. A small selection comes into memory as a Daf with to_daf().

A Daf in memory is faster. Whole-table operations ran 2 to 6 times faster in our timings, and a lookup by key about 11 times faster. So use a Daf for a table that fits in memory.

SqlDaf has the most common Daf methods, under the same names. Some behave differently, because the data is in a database. Any other Daf method raises NotImplementedError. The guide page Tables larger than memory shows how to use it.

Learn more

Questions and ideas are welcome as GitHub issues. Daffodil is released under the MIT license.

Metadata

Release files for daffodil 0.9.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for daffodil 0.9.0
File Size Uploaded Provenance Actions
daffodil-0.9.0.tar.gz 540.8 kB Without provenance Details for daffodil-0.9.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for daffodil 0.9.0
File Interpreter ABI Platform Actions
daffodil-0.9.0-py3-none-any.whl Python 3 none any Details for daffodil-0.9.0-py3-none-any.whl

Total release size: 837.6 kB

Release files / daffodil-0.9.0.tar.gz

Download URL daffodil-0.9.0.tar.gz
Size 540.8 kB
Tags Source
SHA-256 checksum
How to use checksums
3680f4b230ed575f428222704032504271f4cd942c12bf07feb8fd1429006801
BLAKE2b-256 checksum
How to use checksums
add61ead07f47dbab3310a47d05f91b51bfa0862f9fa8d0cf587712610e5c559
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / daffodil-0.9.0-py3-none-any.whl

Download URL daffodil-0.9.0-py3-none-any.whl
Size 296.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
296d21f7d124188d23cd85dab98da57c8c1bb4e8e55eeeb8a54ad2319b392973
BLAKE2b-256 checksum
How to use checksums
66720d42a0c25180ca7acc3403889a0d03e860b6c9df83ca154ee970d59b6a96
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for daffodil 0.9.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page