Skip to main content

Arrow, pydantic style

Project description

Welcome to arrowdantic

Arrowdantic is a small Python library backed by a Rust implementation of Apache Arrow to read Parquet and Arrow IPC files to Python.

Its main differences vs pyarrow are:

  • it is quite small (3Mb vs 90Mb)
  • faster
  • likely safer (no segfaults, core dumps, buffer overflows, etc.)
  • it is type-hinted
  • it has a much smaller subset of its functionality
    • basic arrays (integers, floats, boolean, string, binary)
    • read Apache Arrow IPC file
    • read Apache Parquet

Example

import io

import arrowdantic as ad
# pyarrow is not needed; we just use it here to write a parquet file for the example
import pyarrow as pa

def _write_a_parquet() -> io.BytesIO:
    arrays = [
        pa.array([True, None, False], type=pa.bool_()),
    ]

    schema = pa.schema([
        pa.field(f'c{i}', array.type)
        for i, array in enumerate(arrays)
    ])

    import io
    data = io.BytesIO()
    pa.parquet.write_table(pa.table(arrays, schema), data)
    data.seek(0)
    return data


parquet_file = _write_a_parquet()

reader = arrowdantic.ParquetFileReader(parquet_file)
chunk = next(reader)
assert len(chunk) == 3
arrays = chunk.arrays()
assert arrays[0] == arrowdantic.BooleanArray([True, None, False])
assert list(arrays[0]) == [True, None, False]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

arrowdantic-0.0.2.tar.gz (10.8 kB view hashes)

Uploaded Source

Built Distributions

arrowdantic-0.0.2-pp37-pypy37_pp73-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.1 MB view hashes)

Uploaded PyPy manylinux: glibc 2.12+ x86-64

arrowdantic-0.0.2-cp310-none-win_amd64.whl (1.9 MB view hashes)

Uploaded CPython 3.10 Windows x86-64

arrowdantic-0.0.2-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.1 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.12+ x86-64

arrowdantic-0.0.2-cp310-cp310-macosx_10_7_x86_64.whl (1.9 MB view hashes)

Uploaded CPython 3.10 macOS 10.7+ x86-64

arrowdantic-0.0.2-cp39-none-win_amd64.whl (1.9 MB view hashes)

Uploaded CPython 3.9 Windows x86-64

arrowdantic-0.0.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.1 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.12+ x86-64

arrowdantic-0.0.2-cp39-cp39-macosx_10_7_x86_64.whl (1.9 MB view hashes)

Uploaded CPython 3.9 macOS 10.7+ x86-64

arrowdantic-0.0.2-cp38-none-win_amd64.whl (1.9 MB view hashes)

Uploaded CPython 3.8 Windows x86-64

arrowdantic-0.0.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.1 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.12+ x86-64

arrowdantic-0.0.2-cp38-cp38-macosx_10_7_x86_64.whl (1.9 MB view hashes)

Uploaded CPython 3.8 macOS 10.7+ x86-64

arrowdantic-0.0.2-cp37-none-win_amd64.whl (1.9 MB view hashes)

Uploaded CPython 3.7 Windows x86-64

arrowdantic-0.0.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.1 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.12+ x86-64

arrowdantic-0.0.2-cp37-cp37m-macosx_10_7_x86_64.whl (1.9 MB view hashes)

Uploaded CPython 3.7m macOS 10.7+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page