atlas-python

Thousands of NetCDF datasets in one immutable file. It sits on local disk or on object storage: S3, GCS, Azure, or HTTP. A Rust core, five operations, and one command.

pip install atlas-python

atlas create /data/nc /data/collection
atlas ls     /data/collection
atlas show   /data/collection 2024-01.nc
atlas info   /data/collection
atlas rm     /data/collection 2024-02.nc 2024-03.nc

python -m atlas runs the same command without a PATH lookup, for a shell that cannot find atlas.

Extra Install Adds
cloud pip install "atlas-python[cloud]" S3 / GCS / Azure / HTTP via obstore

numpy, xarray, and dask install automatically.

Five operations

The same five as a library:

import atlas

atlas.create("/data/nc", "/data/collection")   # from a directory of NetCDF files
atlas.list_datasets("/data/collection")        # ['2024-01.nc', '2024-02.nc', '2024-03.nc']
atlas.describe("/data/collection", "2024-01.nc")  # types, shapes, attrs, statistics
atlas.info("/data/collection")                 # counts, size, codec, statistics
atlas.remove("/data/collection", ["2024-02.nc"])  # updates the mask

Every one takes a local path, a URL, or an obstore handle:

atlas ls s3://my-bucket/collections/2024 --region eu-west-1

atlas.list_datasets("s3://my-bucket/collections/2024", region="eu-west-1")

Two things to internalise

One write builds a collection. There is no append, no in-place update, and no flush. The file has a valid trailer, or it is no collection. To change a dataset, rebuild the collection. remove is the one exception. It writes a small mask file, and never touches the container. It therefore reclaims no space, and moves no ordinal.

Python writes. Rust reads array data. From Python you build a collection and read its metadata. That is the dataset names, the array types, the shapes, the chunk shapes, the fill values, the attributes, and the statistics of the write. There is no read_array. Array values come from the Rust API.

That split makes the read side free. The footer an open already fetched answers every metadata call. A catalogue of a thousand datasets is therefore one request.

What show gives you

$ atlas show /data/collection 2024-01.nc
dataset 2024-01.nc {
dimensions:
	lat = 4 ;
	lon = 6 ;
variables:
	float32 temperature(lat, lon) ;
		temperature:_FillValue = nan ;
		temperature:units = "celsius" ;
		// stats: count=24  min=1.0  max=24.0
	string station(lat) ;
		// stats: count=4  min="a"  max="d"

// global attributes:
		:month = 1 ;

// ordinal 0, segment bytes 8..1691
}

The shape follows ncdump -h. It adds the statistics of the write: the minimum, the maximum, and how many elements are missing. --json on any read command gives the same content as a structure.

Ingest

create scans a directory for .nc, .nc4, .cdf, and .netcdf, and descends into every subdirectory. It sorts them, and writes one dataset per file, named after the file. Each coordinate and data variable becomes an array. Each variable attribute becomes a per-array attribute. _FillValue becomes the fill of the array.

Each file opens with dask chunking. A file far larger than memory therefore streams block by block. --chunk-size sets the block budget, and defaults to 128 MiB. It is about the memory ceiling per variable. Those blocks also become the stored chunk shape.

Nothing at the destination is readable until every file lands. A failure part-way leaves no collection, and not a partial one. on_error="skip", or --skip-errors, trades that for progress.

Atlas cannot store every numpy dtype, and bool is the common case. One such variable fails the whole file by default. on_unsupported="skip", or --skip-unsupported, leaves out that one array and lands the rest.

--log-file PATH, or atlas.log_to_file(path), appends every error and warning to a file, each with its reason and the file it came from.

dtypes

numpy atlas
int / uint widths, float32, float64 the same
datetime64[ns] timestamp_nanoseconds
timedelta64[*] int64 nanoseconds, plus a unit marker
object / S / U string

bool, binary, and the list types work as an attribute value. No one of them works yet as an array element type. timestamp_nanoseconds goes the other way: it is an array element type, and no attribute takes it. Store the nanoseconds as int64 and name the unit beside it.

Documentation

The full docs sit at https://maris-development.github.io/atlas/. They hold the command reference, and guides for creating, inspecting, removing, dtypes, reading data, and cloud storage.

The format itself is documented in docs/.

License

Apache-2.0. See LICENSE.

Project links

Data verified by PyPI on Sep 8, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for robinkooyman from gravatar.com robinkooyman

Credits

Author: Robin Kooyman

GitHub Statistics

Data verified by PyPI on Sep 8, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.10

Provides Extra

cloud test

Tags

array ndarray xarray scientific storage object-store

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

atlas_python-0.17.2.tar.gz (210.9 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

atlas_python-0.17.2-cp310-abi3-win_amd64.whl (4.8 MB view details)

Uploaded CPython 3.10+Windows x86-64

atlas_python-0.17.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.6 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

atlas_python-0.17.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (5.8 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

atlas_python-0.17.2-cp310-abi3-macosx_11_0_arm64.whl (5.0 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

atlas_python-0.17.2-cp310-abi3-macosx_10_12_x86_64.whl (5.2 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file atlas_python-0.17.2.tar.gz.

File metadata

  • Download URL: atlas_python-0.17.2.tar.gz
  • Upload date:
  • Size: 210.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for atlas_python-0.17.2.tar.gz
Algorithm Hash digest
SHA256 b2e91a553bc80a5613044ae61042cd53f7c379805bf185933ee99ca1caed2f5c
MD5 45ab7daeaa9dd57a1be9f6082f2642d8
BLAKE2b-256 0f74dab5299942a3b9081eb8df08480a055e659ce9b29de0b9aaaedb9031b62a

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.17.2.tar.gz:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.17.2-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for atlas_python-0.17.2-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 5415fdca4180a468759ed3abb785232a03fe7c2181120bf3da770fefe0c4dbce
MD5 8afc39f1846847f43b88c8ea72b52e1e
BLAKE2b-256 f8c7b0de37ac5b14effff61087693a75c505c6928b911ff8713659be31189b1a

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.17.2-cp310-abi3-win_amd64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.17.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for atlas_python-0.17.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6b92f3762e6576224cd5430fa090c0a97ad56df093219556b9f94673b02a7eea
MD5 6f519cfe56b09c305f0c2af6db17183d
BLAKE2b-256 25ef5d500b58b68a926fa32c7d3021ff479fa32fc7ef56bf5b8a59c51afde790

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.17.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.17.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for atlas_python-0.17.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ff691a212cda74690126e2c579735d1a62f3e4ae4e24466447da075ce2d2fe3a
MD5 64a2604927020db53c9db7186984640d
BLAKE2b-256 c9b6a395410cc6c905580c97c71ac78862435180eaf9193e10b72d058f3d4e12

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.17.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.17.2-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for atlas_python-0.17.2-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 1041d50eafcba66306e24bf0241896ec5ffbd86f20f4d36fc548202ce94f1674
MD5 1b8231fc9622893cbcea487210cff83c
BLAKE2b-256 18cf59cfaecd56b56cf8ae99854559ede19e46a1e838cff96e6f03d7148d827c

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.17.2-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.17.2-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for atlas_python-0.17.2-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 877e07acadeca3ddeab8c27280cab17bf60f037c4695900305a3f02a42bbdf49
MD5 9e0060f1c2f8d660de837b581bc218f6
BLAKE2b-256 e4945961775d22139d8a2c8af8cf6b564ac46c6608ff0894f19b753b1299d0a6

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.17.2-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.17.2 This release

6 files

0.17.1

6 files

0.17.0

6 files

0.16.4

6 files

0.16.3

6 files

0.16.2

6 files

0.16.1

6 files

0.16.0

6 files

0.14.0

6 files

0.13.1

6 files

0.13.0

6 files

0.12.0

6 files

0.10.0

6 files

0.9.1

6 files

0.9.0

6 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page