atlas-python

Thousands of NetCDF datasets in one immutable file. It sits on local disk or on object storage: S3, GCS, Azure, or HTTP. A Rust core, five operations, and one command.

pip install atlas-python

atlas create /data/nc /data/collection
atlas ls     /data/collection
atlas show   /data/collection 2024-01.nc
atlas info   /data/collection
atlas rm     /data/collection 2024-02.nc 2024-03.nc
Extra Install Adds
cloud pip install "atlas-python[cloud]" S3 / GCS / Azure / HTTP via obstore

numpy, xarray, and dask install automatically.

Five operations

The same five as a library:

import atlas

atlas.create("/data/nc", "/data/collection")   # from a directory of NetCDF files
atlas.list_datasets("/data/collection")        # ['2024-01.nc', '2024-02.nc', '2024-03.nc']
atlas.describe("/data/collection", "2024-01.nc")  # types, shapes, attrs, statistics
atlas.info("/data/collection")                 # counts, size, codec, statistics
atlas.remove("/data/collection", ["2024-02.nc"])  # updates the mask

Every one takes a local path, a URL, or an obstore handle:

atlas ls s3://my-bucket/collections/2024 --region eu-west-1

atlas.list_datasets("s3://my-bucket/collections/2024", region="eu-west-1")

Two things to internalise

One write builds a collection. There is no append, no in-place update, and no flush. The file has a valid trailer, or it is no collection. To change a dataset, rebuild the collection. remove is the one exception. It writes a small mask file, and never touches the container. It therefore reclaims no space, and moves no ordinal.

Python writes. Rust reads array data. From Python you build a collection and read its metadata. That is the dataset names, the array types, the shapes, the chunk shapes, the fill values, the attributes, and the statistics of the write. There is no read_array. Array values come from the Rust API.

That split makes the read side free. The footer an open already fetched answers every metadata call. A catalogue of a thousand datasets is therefore one request.

What show gives you

$ atlas show /data/collection 2024-01.nc
dataset 2024-01.nc {
dimensions:
	lat = 4 ;
	lon = 6 ;
variables:
	float32 temperature(lat, lon) ;
		temperature:_FillValue = nan ;
		temperature:units = "celsius" ;
		// stats: count=24  min=1.0  max=24.0
	string station(lat) ;
		// stats: count=4  min="a"  max="d"

// global attributes:
		:month = 1 ;

// ordinal 0, segment bytes 8..1691
}

The shape follows ncdump -h. It adds the statistics of the write: the minimum, the maximum, and how many elements are missing. --json on any read command gives the same content as a structure.

Ingest

create scans a directory for .nc, .nc4, .cdf, and .netcdf. It sorts them, and writes one dataset per file, named after the file. Each coordinate and data variable becomes an array. Each variable attribute becomes a per-array attribute. _FillValue becomes the fill of the array.

Each file opens with dask chunking. A file far larger than memory therefore streams block by block. --chunk-size sets the block budget, and defaults to 128 MiB. It is about the memory ceiling per variable. Those blocks also become the stored chunk shape.

Nothing at the destination is readable until every file lands. A failure part-way leaves no collection, and not a partial one. on_error="skip", or --skip-errors, trades that for progress.

Atlas cannot store every numpy dtype, and bool is the common case. One such variable fails the whole file by default. on_unsupported="skip", or --skip-unsupported, leaves out that one array and lands the rest.

--log-file PATH, or atlas.log_to_file(path), appends every error and warning to a file, each with its reason and the file it came from.

dtypes

numpy atlas
int / uint widths, float32, float64 the same
datetime64[ns] timestamp_nanoseconds
timedelta64[*] int64 nanoseconds, plus a unit marker
object / S / U string

bool, binary, and the list types work as an attribute value. No one of them works yet as an array element type.

Documentation

The full docs sit at https://maris-development.github.io/atlas/. They hold the command reference, and guides for creating, inspecting, removing, dtypes, reading data, and cloud storage.

The format itself is documented in docs/.

License

Apache-2.0. See LICENSE.

Project links

Data verified by PyPI on Sep 1, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.17.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for robinkooyman from gravatar.com robinkooyman

Credits

Author: Robin Kooyman

GitHub Statistics

Data verified by PyPI on Sep 1, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.10

Provides Extra

cloud test

Tags

array ndarray xarray scientific storage object-store

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

atlas_python-0.16.1.tar.gz (174.1 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

atlas_python-0.16.1-cp310-abi3-win_amd64.whl (5.3 MB view details)

Uploaded CPython 3.10+Windows x86-64

atlas_python-0.16.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.0 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

atlas_python-0.16.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (6.2 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

atlas_python-0.16.1-cp310-abi3-macosx_11_0_arm64.whl (5.3 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

atlas_python-0.16.1-cp310-abi3-macosx_10_12_x86_64.whl (5.6 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file atlas_python-0.16.1.tar.gz.

File metadata

  • Download URL: atlas_python-0.16.1.tar.gz
  • Upload date:
  • Size: 174.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for atlas_python-0.16.1.tar.gz
Algorithm Hash digest
SHA256 69a3d0c3df24cd819a6b3b46fe708de342f5e9d046e078e4d7333b6a35dc0174
MD5 3ef9611d28c0cff84381900d69bedf3e
BLAKE2b-256 41ab38919b49f2f3fe2c3d2e2f7d7fdc7ca7455af893d831b259dc3f82cb51d9

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.1.tar.gz:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.16.1-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for atlas_python-0.16.1-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 7646390ae62a5e97823cc9b5067c66d184a8ba192d7fe1b34e31da6de79f3bc3
MD5 cabf0eb34a67737d294c83e4f0fea7cf
BLAKE2b-256 e76fc49ea923faec9f44cc8e6ccd08546172bdf3f8a734de717d72c5c34a1083

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.1-cp310-abi3-win_amd64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.16.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for atlas_python-0.16.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3054fc12052baf686ad45085c78ea4609948b8bdb1a19c1f0568943d940e6fac
MD5 94ad2c5b8013ba71ea69c1713b821fc4
BLAKE2b-256 f9d9d42794103e34b1d2c602b8c312f60077c82a87d32512e31f8738b164351c

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.16.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for atlas_python-0.16.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 bdc5a7ed8f928a580650bb0ebfb2821330e68a61c34f3e80f314f80854d5ac08
MD5 d936848c02ff0d260432762e006edb29
BLAKE2b-256 f5a606b728734d9a4b64330b201cb2938e7e7afbdd4e2e66d1042b38ee3054cd

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.16.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for atlas_python-0.16.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 0ccb1ef656d6d977e50ce46f51c9e7e5d0f6f1c9c0c1a8abaea540c0cdec64a6
MD5 29608c9f51e606ed937cb15085e1497f
BLAKE2b-256 feacaf09af309df3647f06a5f8e9e136760e4c77f618e34acff5e5ad068611a9

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.1-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.16.1-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for atlas_python-0.16.1-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 73838aaf47bdab9f9769f7cd94b749d4c30a029d9406da3ac18bd20b1d4cd349
MD5 b7bad932387cd8209c3ba82deb62c2e4
BLAKE2b-256 353a2c5703908766599718046ff4923a193eddd9931fa2939af44dedd6bab107

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.1-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.17.2

6 files

0.17.1

6 files

0.17.0

6 files

0.16.4

6 files

0.16.3

6 files

0.16.2

6 files

This release

0.16.1 This release

6 files

0.16.0

6 files

0.14.0

6 files

0.13.1

6 files

0.13.0

6 files

0.12.0

6 files

0.10.0

6 files

0.9.1

6 files

0.9.0

6 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page