Python Daffodil

Daffodil (DAtaFrames For Optimized Data Inspection and Logical processing) gives Python fast, lightweight 2-D data tables. A table is a list of rows, and each row is a plain Python list. Use it where data arrives and is processed row by row: reading files, building records, cleaning, reshaping, and writing them out again.

Documentation: https://raylutz.github.io/daffodil/

Status: alpha, and used in production for several years. The changes of each release are in the changelog.

Install

pip install daffodil

Python 3.10 or later. The table code is pure Python. pandas, NumPy and a few other packages are needed only for the features that use them, such as to_pandas_df(), and are not installed with daffodil. The Install section of the Guide lists them.

The data model

The rows are kept in lol, a list of lists. The column names are kept once, in the header dict hd, which maps each name to its position. A table can have a keyfield, a column whose values identify the rows, and then the key dict kd maps each key to its row. A cell can hold any Python object: text, a number, a list, a dict, or even another table.

Quick start

from daffodil.daf import Daf

daf = Daf(cols=['id', 'item', 'qty'], keyfield='id')
daf.append({'id': 'a1', 'item': 'pen', 'qty': 3})
daf.append(['b2', 'ink', 10])

daf['b2']                     # the row with key 'b2'
daf[0]                        # the first row
daf[:, 'qty']                 # the qty column
daf['b2', 'qty'].to_value()   # 10
daf.select_where(lambda row: row['qty'] > 5)
print(daf)                    # a Markdown table

Why Daffodil

• Adding a row is cheap. It is appended to a Python list. pandas has no cheap way to add a row, so code that uses it often collects the rows first and converts them at the end.
• No conversion. The data stays the Python objects you put in, so there is no cost to get it in or out.
• A selection shares the rows. Selecting rows copies no data, and lookups by key are dict lookups.
• Mixed data is fine. A column can mix text, numbers and empty cells, and nothing is coerced to NaN or to a common type.
• Tables print as Markdown, ready for a terminal or a report, and can be read back.
• Small and pure Python, so it starts fast, which matters in short-lived cloud functions.

For heavy math over whole numeric columns, pandas, NumPy or Polars are faster. Build and clean the data as rows in Daffodil, then hand the numeric columns over with to_pandas_df() or to_numpy(). The documentation compares the two, with measurements.

Tables larger than memory

SqlDaf stores a table in a SQLite database file, so its rows stay out of Python memory. It supports tables larger than the memory available, and many operations run directly in SQLite. A small selection comes into memory as a Daf with to_daf().

A Daf in memory is faster. Whole-table operations ran 2 to 6 times faster in our timings, and a lookup by key about 11 times faster. So use a Daf for a table that fits in memory.

SqlDaf has the most common Daf methods, under the same names. Some behave differently, because the data is in a database. Any other Daf method raises NotImplementedError. The guide page Tables larger than memory shows how to use it.

Learn more

• Cheatsheet, everything on one page
• Getting started
• Building tables, including the fast way to build rows in a loop
• Selecting and indexing
• Columns, keys and schemas
• Types and conversion
• Apply, reduce and formulas
• Joins
• Markdown reports
• Coming from pandas
• Tables larger than memory, with SqlDaf
• The Daf API, every method with examples
• The SqlDaf API

Questions and ideas are welcome as GitHub issues. Daffodil is released under the MIT license.