atlas-python

Thousands of NetCDF datasets in one immutable file. It sits on local disk or on object storage: S3, GCS, Azure, or HTTP. A Rust core, five operations, and one command.

pip install atlas-python

atlas create /data/nc /data/collection
atlas ls     /data/collection
atlas show   /data/collection 2024-01
atlas info   /data/collection
atlas rm     /data/collection 2024-02 2024-03
Extra Install Adds
cloud pip install "atlas-python[cloud]" S3 / GCS / Azure / HTTP via obstore

numpy, xarray, and dask install automatically.

Five operations

The same five as a library:

import atlas

atlas.create("/data/nc", "/data/collection")   # from a directory of NetCDF files
atlas.list_datasets("/data/collection")        # ['2024-01', '2024-02', '2024-03']
atlas.describe("/data/collection", "2024-01")  # types, shapes, attrs, statistics
atlas.info("/data/collection")                 # counts, size, codec, statistics
atlas.remove("/data/collection", ["2024-02"])  # updates the mask

Every one takes a local path, a URL, or an obstore handle:

atlas ls s3://my-bucket/collections/2024 --region eu-west-1

atlas.list_datasets("s3://my-bucket/collections/2024", region="eu-west-1")

Two things to internalise

One write builds a collection. There is no append, no in-place update, and no flush. The file has a valid trailer, or it is no collection. To change a dataset, rebuild the collection. remove is the one exception. It writes a small mask file, and never touches the container. It therefore reclaims no space, and moves no ordinal.

Python writes. Rust reads array data. From Python you build a collection and read its metadata. That is the dataset names, the array types, the shapes, the chunk shapes, the fill values, the attributes, and the statistics of the write. There is no read_array. Array values come from the Rust API.

That split makes the read side free. The footer an open already fetched answers every metadata call. A catalogue of a thousand datasets is therefore one request.

What show gives you

$ atlas show /data/collection 2024-01
dataset 2024-01 {
dimensions:
	lat = 4 ;
	lon = 6 ;
variables:
	float32 temperature(lat, lon) ;
		temperature:_FillValue = nan ;
		temperature:units = "celsius" ;
		// stats: count=24  min=1.0  max=24.0
	string station(lat) ;
		// stats: count=4  min="a"  max="d"

// global attributes:
		:month = 1 ;

// ordinal 0, segment bytes 8..1691
}

The shape follows ncdump -h. It adds the statistics of the write: the minimum, the maximum, and how many elements are missing. --json on any read command gives the same content as a structure.

Ingest

create scans a directory for .nc, .nc4, .cdf, and .netcdf. It sorts them, and writes one dataset per file, named after the stem. Each coordinate and data variable becomes an array. Each variable attribute becomes a per-array attribute. _FillValue becomes the fill of the array.

Each file opens with dask chunking. A file far larger than memory therefore streams block by block. --chunk-size sets the block budget, and defaults to 128 MiB. It is about the memory ceiling per variable. Those blocks also become the stored chunk shape.

Nothing at the destination is readable until every file lands. A failure part-way leaves no collection, and not a partial one. on_error="skip", or --skip-errors, trades that for progress.

dtypes

numpy atlas
int / uint widths, float32, float64 the same
datetime64[ns] timestamp_nanoseconds
timedelta64[*] int64 nanoseconds, plus a unit marker
object / S / U string

bool, binary, and the list types work as an attribute value. No one of them works yet as an array element type.

Documentation

The full docs sit at https://maris-development.github.io/atlas/. They hold the command reference, and guides for creating, inspecting, removing, dtypes, reading data, and cloud storage.

The format itself is documented in docs/.

License

Apache-2.0. See LICENSE.

Project links

Data verified by PyPI on Sep 1, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.17.2)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for robinkooyman from gravatar.com robinkooyman

Credits

Author: Robin Kooyman

GitHub Statistics

Data verified by PyPI on Sep 1, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.10

Provides Extra

cloud test

Tags

array ndarray xarray scientific storage object-store

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

atlas_python-0.16.0.tar.gz (166.9 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

atlas_python-0.16.0-cp310-abi3-win_amd64.whl (5.3 MB view details)

Uploaded CPython 3.10+Windows x86-64

atlas_python-0.16.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.0 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

atlas_python-0.16.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (6.2 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

atlas_python-0.16.0-cp310-abi3-macosx_11_0_arm64.whl (5.3 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

atlas_python-0.16.0-cp310-abi3-macosx_10_12_x86_64.whl (5.6 MB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file atlas_python-0.16.0.tar.gz.

File metadata

  • Download URL: atlas_python-0.16.0.tar.gz
  • Upload date:
  • Size: 166.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for atlas_python-0.16.0.tar.gz
Algorithm Hash digest
SHA256 7091762a7edf0883256fdebebaada5afed2d333937a476f6c1edce63dd05fb92
MD5 f7547b9d475f9c3634937529fd0ecd27
BLAKE2b-256 f696553a19530aea6ad746c0b2fbed1087883f357fd81844e76b2c646c4900e2

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.0.tar.gz:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.16.0-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for atlas_python-0.16.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 6b3117c9f61bfb4ada8839670a6406901a14aa180ea2a90749a9348c8b6b8f79
MD5 4cbb8c657c2018318542c37d9ed7c6a8
BLAKE2b-256 708d96d9d790c0a8ac7f4d832184e01fd778e2860de4b4013c3890849c953ce7

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.0-cp310-abi3-win_amd64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.16.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for atlas_python-0.16.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 0589abd282c4b7127e2ecf195bc0346efead0064ec915246f06cefd0f519768f
MD5 e975fb954590ff8c9a0f2806803f9090
BLAKE2b-256 785cd314efe35133d4a7783796cc326d5aad7f3a4dfa730462fb67baa6ecdece

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.16.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for atlas_python-0.16.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ed8538d332315907391a82d130f0a32ef3188e23b3cb09af94f46df5f2c5ece3
MD5 e9898d3c944abcc73ee558952bfb4ead
BLAKE2b-256 07511803f3dc45ea25c4c4225c9364f4a877cc3ec4fadf71249b4abcad81bbe9

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.16.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for atlas_python-0.16.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 6b78acab32b8c73ed248aea8aabacb427d8a98d7def6c21bf81c163441df834a
MD5 9210d079264363e50e70253857047402
BLAKE2b-256 2dd1ec2bccb2ffcf9cd11352ff282e7ae833d5b814f33c307ed2c8dbd948cc02

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file atlas_python-0.16.0-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for atlas_python-0.16.0-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 ea1ef3203ad01d0a3f1f1386d80e9c989284cac02553162ed1fc087654740ae2
MD5 cd733685c164822c29d2cada7998ce5c
BLAKE2b-256 466e394f43ade635836e54b2f57959ce89b647d39ce50a1f4208dd5af62160d7

See more details on using hashes here.

Provenance

The following attestation bundles were made for atlas_python-0.16.0-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: atlas-python-release.yaml on maris-development/atlas

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.17.2

6 files

0.17.1

6 files

0.17.0

6 files

0.16.4

6 files

0.16.3

6 files

0.16.2

6 files

0.16.1

6 files

This release

0.16.0 This release

6 files

0.14.0

6 files

0.13.1

6 files

0.13.0

6 files

0.12.0

6 files

0.10.0

6 files

0.9.1

6 files

0.9.0

6 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page