Skip to main content

Manipulate JSON-like data with NumPy-like idioms.

Project description

Awkward Array is a library for nested, variable-sized data, including arbitrary-length lists, records, mixed types, and missing data, using NumPy-like idioms.

Arrays are dynamically typed, but operations on them are compiled and fast. Their behavior coincides with NumPy when array dimensions are regular and generalizes when they're not.

How-to tutorials

Python API reference

C++ API reference

Motivating example

Given an array of objects with x, y fields and variable-length nested lists like

array = ak.Array([
    [{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}, {"x": 3.3, "y": [1, 2, 3]}],
    [],
    [{"x": 4.4, "y": {1, 2, 3, 4]}, {"x": 5.5, "y": [1, 2, 3, 4, 5]}]
])

the following slices out the y values, drops the first element from each inner list, and runs NumPy's np.square function on everything that is left:

output = np.square(array["y", ..., 1:])

The result is

[
    [[], [4], [4, 9]],
    [],
    [[4, 9, 16], [4, 9, 16, 25]]
]

The equivalent using only Python is

output = []
for sublist in array:
    tmp1 = []
    for record in sublist:
        tmp2 = []
        for number in record["y"][1:]:
            tmp2.append(np.square(number))
        tmp1.append(tmp2)
    output.append(tmp1)

Not only is the expression using Awkward Arrays more concise, using idioms familiar from NumPy, but it's much faster and uses less memory.

For a similar problem 10 million times larger than the one above (on a single-threaded 2.2 GHz processor),

  • the Awkward Array one-liner takes 4.6 seconds to run and uses 2.1 GB of memory,
  • the equivalent using Python lists and dicts takes 138 seconds to run and uses 22 GB of memory.

Speed and memory factors in the double digits are common because we're replacing Python's dynamically typed, pointer-chasing virtual machine with type-specialized, precompiled routines on contiguous data. (In other words, for the same reasons as NumPy.) Even higher speedups are possible when Awkward Array is paired with Numba.

Our presentation at SciPy 2020 provides a good introduction, showing how to use these arrays in a real analysis.

Installation

Awkward Array can be installed from PyPI using pip:

pip install awkward1

Most users will get a precompiled binary (wheel) for your operating system and Python version. If not, the above attempts to compile from source.

  • Report bugs, request features, and ask for additional documentation on GitHub Issues. If you have a general "How do I…?" question, we'll answer it as a new example in the tutorial.
  • If you have a problem that's too specific to be new documentation or it isn't exclusively related to Awkward Array, it might be more appropriate to ask on StackOverflow with the [awkward-array] tag. Be sure to include tags for any other libraries that you use, such as Pandas or PyTorch.
  • The Gitter Scikit-HEP/community is a way to get in touch with all Scikit-HEP developers and users.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

awkward1-0.2.32.tar.gz (655.8 kB view hashes)

Uploaded Source

Built Distributions

awkward1-0.2.32-cp39-cp39-manylinux2010_x86_64.whl (9.5 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.12+ x86-64

awkward1-0.2.32-cp39-cp39-manylinux1_x86_64.whl (8.8 MB view hashes)

Uploaded CPython 3.9

awkward1-0.2.32-cp39-cp39-manylinux1_i686.whl (8.9 MB view hashes)

Uploaded CPython 3.9

awkward1-0.2.32-cp38-cp38-win_amd64.whl (12.2 MB view hashes)

Uploaded CPython 3.8 Windows x86-64

awkward1-0.2.32-cp38-cp38-win32.whl (9.3 MB view hashes)

Uploaded CPython 3.8 Windows x86

awkward1-0.2.32-cp38-cp38-manylinux2010_x86_64.whl (9.5 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.12+ x86-64

awkward1-0.2.32-cp38-cp38-manylinux1_x86_64.whl (8.8 MB view hashes)

Uploaded CPython 3.8

awkward1-0.2.32-cp38-cp38-manylinux1_i686.whl (8.9 MB view hashes)

Uploaded CPython 3.8

awkward1-0.2.32-cp38-cp38-macosx_10_9_x86_64.whl (9.1 MB view hashes)

Uploaded CPython 3.8 macOS 10.9+ x86-64

awkward1-0.2.32-cp37-cp37m-win_amd64.whl (12.2 MB view hashes)

Uploaded CPython 3.7m Windows x86-64

awkward1-0.2.32-cp37-cp37m-win32.whl (9.3 MB view hashes)

Uploaded CPython 3.7m Windows x86

awkward1-0.2.32-cp37-cp37m-manylinux2010_x86_64.whl (9.5 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.12+ x86-64

awkward1-0.2.32-cp37-cp37m-manylinux1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.7m

awkward1-0.2.32-cp37-cp37m-manylinux1_i686.whl (8.9 MB view hashes)

Uploaded CPython 3.7m

awkward1-0.2.32-cp37-cp37m-macosx_10_9_x86_64.whl (9.0 MB view hashes)

Uploaded CPython 3.7m macOS 10.9+ x86-64

awkward1-0.2.32-cp36-cp36m-win_amd64.whl (12.2 MB view hashes)

Uploaded CPython 3.6m Windows x86-64

awkward1-0.2.32-cp36-cp36m-win32.whl (9.3 MB view hashes)

Uploaded CPython 3.6m Windows x86

awkward1-0.2.32-cp36-cp36m-manylinux2010_x86_64.whl (9.5 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.12+ x86-64

awkward1-0.2.32-cp36-cp36m-manylinux1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.6m

awkward1-0.2.32-cp36-cp36m-manylinux1_i686.whl (8.9 MB view hashes)

Uploaded CPython 3.6m

awkward1-0.2.32-cp36-cp36m-macosx_10_9_x86_64.whl (9.0 MB view hashes)

Uploaded CPython 3.6m macOS 10.9+ x86-64

awkward1-0.2.32-cp35-cp35m-win_amd64.whl (12.2 MB view hashes)

Uploaded CPython 3.5m Windows x86-64

awkward1-0.2.32-cp35-cp35m-win32.whl (9.3 MB view hashes)

Uploaded CPython 3.5m Windows x86

awkward1-0.2.32-cp35-cp35m-manylinux2010_x86_64.whl (9.5 MB view hashes)

Uploaded CPython 3.5m manylinux: glibc 2.12+ x86-64

awkward1-0.2.32-cp35-cp35m-manylinux1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 3.5m

awkward1-0.2.32-cp35-cp35m-manylinux1_i686.whl (8.9 MB view hashes)

Uploaded CPython 3.5m

awkward1-0.2.32-cp27-cp27mu-manylinux2010_x86_64.whl (9.5 MB view hashes)

Uploaded CPython 2.7mu manylinux: glibc 2.12+ x86-64

awkward1-0.2.32-cp27-cp27mu-manylinux1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 2.7mu

awkward1-0.2.32-cp27-cp27mu-manylinux1_i686.whl (8.9 MB view hashes)

Uploaded CPython 2.7mu

awkward1-0.2.32-cp27-cp27m-win_amd64.whl (12.2 MB view hashes)

Uploaded CPython 2.7m Windows x86-64

awkward1-0.2.32-cp27-cp27m-win32.whl (9.3 MB view hashes)

Uploaded CPython 2.7m Windows x86

awkward1-0.2.32-cp27-cp27m-manylinux2010_x86_64.whl (9.5 MB view hashes)

Uploaded CPython 2.7m manylinux: glibc 2.12+ x86-64

awkward1-0.2.32-cp27-cp27m-manylinux1_x86_64.whl (8.9 MB view hashes)

Uploaded CPython 2.7m

awkward1-0.2.32-cp27-cp27m-manylinux1_i686.whl (8.9 MB view hashes)

Uploaded CPython 2.7m

awkward1-0.2.32-cp27-cp27m-macosx_10_9_x86_64.whl (9.0 MB view hashes)

Uploaded CPython 2.7m macOS 10.9+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page