Fast read/write of AVRO files

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.8
Classifiers

Project description

fastavro

Build Status Documentation Status codecov

Because the Apache Python avro package is written in pure Python, it is relatively slow. In one test case, it takes about 14 seconds to iterate through a file of 10,000 records. By comparison, the JAVA avro SDK reads the same file in 1.9 seconds.

The fastavro library was written to offer performance comparable to the Java library. With regular CPython, fastavro uses C extensions which allow it to iterate the same 10,000 record file in 1.7 seconds. With PyPy, this drops to 1.5 seconds (to be fair, the JAVA benchmark is doing some extra JSON encoding/decoding).

fastavro supports the following Python versions:

  • Python 3.8
  • Python 3.9
  • Python 3.10
  • Python 3.11
  • Python 3.12
  • PyPy3

Supported Features

  • File Writer
  • File Reader (iterating via records or blocks)
  • Schemaless Writer
  • Schemaless Reader
  • JSON Writer
  • JSON Reader
  • Codecs (Snappy, Deflate, Zstandard, Bzip2, LZ4, XZ)
  • Schema resolution
  • Aliases
  • Logical Types
  • Parsing schemas into the canonical form
  • Schema fingerprinting

Missing Features

  • Anything involving Avro's RPC features

Documentation

Documentation is available at http://fastavro.readthedocs.io/en/latest/

Installing

fastavro is available both on PyPI

pip install fastavro

and on conda-forge conda channel.

conda install -c conda-forge fastavro

Contributing

  • Bugs and new feature requests typically start as GitHub issues where they can be discussed. I try to resolve these as time affords, but PRs are welcome from all.
  • Get approval from discussing on the GitHub issue before opening the pull request
  • Tests must be passing for pull request to be considered

Developer requirements can be installed with pip install -r developer_requirements.txt. If those are installed, you can run the tests with ./run-tests.sh. If you have trouble installing those dependencies, you can run docker build . to run the tests inside a Docker container. This won't test on all versions of Python or on PyPy, so it's possible to still get CI failures after making a pull request, but we can work through those errors if/when they happen. .run-tests.sh only covers the Cython tests. In order to test the pure Python implementation, comment out python setup.py build_ext --inplace and re-run.

NOTE: Some tests might fail when running the tests locally. An example of this is this codec tests. If the supporting codec library is not available, the test will fail. These failures can be ignored since the tests will on pull requests and will be run in the correct environments with the correct dependencies set up.

Releasing

We release both to PyPI and to conda-forge.

We assume you have twine installed and that you've created your own fork of fastavro-feedstock.

  • Make sure the tests pass
  • Run make tag
  • Wait for all artifacts to be built and published the the Github release
  • Run make publish
  • The conda-forge PR should get created and merged automatically

Changes

See the ChangeLog

Contact

Project Home

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for scottbelden from gravatar.com scottbelden Avatar for tebeka from gravatar.com tebeka

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: Miki Tebeka
  • Requires: Python >=3.8
Classifiers

Release history Release notifications | RSS feed

This version

1.9.7

1.9.6

1.9.5

1.9.4

1.9.3

1.9.2

1.9.1

1.9.0

1.8.4

1.8.3

1.8.2

1.8.1

1.8.0

1.7.4

1.7.3

1.7.2

1.7.1

1.7.0

1.6.1

1.6.0

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.12

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.6

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.5

1.3.4

1.3.3

1.3.2

1.3.1

1.3.0

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.1

1.1.0

1.0.0.post1

1.0.0 yanked

Reason this release was yanked:

Missing python_requires to prevent Python 2 users from installing

0.24.2

0.24.1

0.24.0

0.23.6

0.23.5

0.23.4

0.23.3

0.23.2

0.23.1

0.23.0

0.22.13

0.22.12

0.22.11

0.22.10

0.22.9

0.22.8

0.22.7

0.22.6

0.22.5

0.22.4

0.22.3

0.22.2

0.22.1

0.22.0

0.21.24

0.21.23

0.21.22

0.21.21

0.21.20

0.21.19

0.21.18

0.21.17

0.21.16

0.21.15

0.21.14

0.21.13

0.21.12

0.21.11

0.21.10

0.21.9

0.21.8

0.21.7

0.21.6

0.21.5

0.21.4

0.21.3

0.21.2

0.21.1

0.21.0

0.21.0rc1 pre-release

0.20.0

0.19.9

0.19.8

0.19.7

0.19.6

0.19.5

0.19.4

0.19.3

0.19.2

0.19.1

0.19.0

0.18.2

0.18.1

0.18.0

0.17.10

0.17.9

0.17.8

0.17.7

0.17.6

0.17.5

0.17.4

0.17.3

0.17.2

0.17.1

0.17.0

0.16.7

0.16.6

0.16.5

0.16.4

0.16.3

0.16.1

0.14.11

0.14.10

0.14.9

0.14.8

0.14.7

0.14.6

0.14.5

0.14.3

0.14.2

0.14.1

0.14.0

0.13.0

0.12.2

0.12.1

0.12.0

0.11.1

0.11.0

0.10.2

0.10.1

0.9.11

0.9.10

0.9.9

0.9.8

0.9.7

0.9.6

0.9.5

0.9.4

0.9.3

0.9.2

0.9.1

0.9.0

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.10

0.7.9

0.7.8

0.7.7

0.7.6

0.7.5

0.7.4

0.7.3

0.7.2

0.7.1

0.7.0

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.0

0.4.2

0.4.1

0.4.0

0.3.2

0.3.1

0.3.0

0.2.2

0.2.1

0.2.0

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fastavro-1.9.7.tar.gz (987.8 kB view hashes)

Uploaded Source

Built Distributions

fastavro-1.9.7-cp312-cp312-win_amd64.whl (487.8 kB view hashes)

Uploaded CPython 3.12 Windows x86-64

fastavro-1.9.7-cp312-cp312-musllinux_1_2_x86_64.whl (3.4 MB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.2+ x86-64

fastavro-1.9.7-cp312-cp312-musllinux_1_2_aarch64.whl (3.3 MB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.2+ ARM64

fastavro-1.9.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

fastavro-1.9.7-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.3 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

fastavro-1.9.7-cp312-cp312-macosx_10_9_universal2.whl (1.0 MB view hashes)

Uploaded CPython 3.12 macOS 10.9+ universal2 (ARM64, x86-64)

fastavro-1.9.7-cp311-cp311-win_amd64.whl (500.1 kB view hashes)

Uploaded CPython 3.11 Windows x86-64

fastavro-1.9.7-cp311-cp311-musllinux_1_2_x86_64.whl (3.4 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.2+ x86-64

fastavro-1.9.7-cp311-cp311-musllinux_1_2_aarch64.whl (3.3 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.2+ ARM64

fastavro-1.9.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

fastavro-1.9.7-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.3 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

fastavro-1.9.7-cp311-cp311-macosx_10_9_universal2.whl (1.0 MB view hashes)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64)

fastavro-1.9.7-cp310-cp310-win_amd64.whl (497.2 kB view hashes)

Uploaded CPython 3.10 Windows x86-64

fastavro-1.9.7-cp310-cp310-musllinux_1_2_x86_64.whl (3.1 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.2+ x86-64

fastavro-1.9.7-cp310-cp310-musllinux_1_2_aarch64.whl (3.0 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.2+ ARM64

fastavro-1.9.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

fastavro-1.9.7-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.0 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

fastavro-1.9.7-cp310-cp310-macosx_10_9_universal2.whl (1.0 MB view hashes)

Uploaded CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64)

fastavro-1.9.7-cp39-cp39-win_amd64.whl (546.2 kB view hashes)

Uploaded CPython 3.9 Windows x86-64

fastavro-1.9.7-cp39-cp39-musllinux_1_2_x86_64.whl (3.1 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.2+ x86-64

fastavro-1.9.7-cp39-cp39-musllinux_1_2_aarch64.whl (3.0 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.2+ ARM64

fastavro-1.9.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

fastavro-1.9.7-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.0 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

fastavro-1.9.7-cp39-cp39-macosx_10_9_universal2.whl (1.0 MB view hashes)

Uploaded CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64)

fastavro-1.9.7-cp38-cp38-win_amd64.whl (550.3 kB view hashes)

Uploaded CPython 3.8 Windows x86-64

fastavro-1.9.7-cp38-cp38-musllinux_1_2_x86_64.whl (3.2 MB view hashes)

Uploaded CPython 3.8 musllinux: musl 1.2+ x86-64

fastavro-1.9.7-cp38-cp38-musllinux_1_2_aarch64.whl (3.1 MB view hashes)

Uploaded CPython 3.8 musllinux: musl 1.2+ ARM64

fastavro-1.9.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

fastavro-1.9.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.1 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

fastavro-1.9.7-cp38-cp38-macosx_11_0_universal2.whl (1.0 MB view hashes)

Uploaded CPython 3.8 macOS 11.0+ universal2 (ARM64, x86-64)

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page