Skip to main content

ArcticDB DataFrame Database

Project description



ArcticDB Website | ArcticDB Blog | Press Release | Press Release | Community


ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem. Launched in March 2023, it is the successor to Arctic.

ArcticDB offers an intuitive Python-centric API enabling you to read and write Pandas DataFrames to S3 or LMDB utilising a fast C++ data-processing and compression engine.

ArcticDB allows you to:

  • Pandas in, Pandas out: Read and write Pandas DataFrames, NumPy arrays and native types to S3 and LMDB without leaving Python.
  • Built for time-series data: Efficiently index and query time-series data across billions of rows
  • Time travel: Travel back in time to see previous versions of your data and create customizable snapshots of the database
  • Schemaless Database: Append, update and modify data without being constrained by the existing schema
  • Optimised for streaming data: Built in support for efficient sparse data storage
  • Powerful processing: Filter, aggregate and create new columns on-the-fly with a Pandas-like syntax
  • C++ efficiency: Accelerate analytics though concurrency in the C++ data-processing engine

ArcticDB handles data that is big in both row count and column count, so a 20-year history of more than 400,000 unique securities can be stored in a single symbol. Each symbol is maintained as a separate entity with no shared data which means ArcticDB can scale horizontally across symbols, maximising the performance potential of your compute, storage and network.

ArcticDB is designed from the outset to be resilient; there is no single point of failure, and persistent data structures in the storage mean that once a version of a symbol has been written, it can never be corrupted by subsequent updates. Pulling compressed data directly from storage to the client means that there is no server to overload, so your data is always available when you need it.

Quickstart

Prebuilt binary availability

PyPI (Python 3.6 - 3.11) conda-forge (Python 3.8 - 3.11)
Linux (Intel/AMD) ✔️ ✔️
Windows (Intel/AMD) ✔️
MacOS Beta️

For conda-forge see the release-info.

Storage compatibility

Linux Windows Mac
S3 ✔️ ✔️ ✔️
LMDB ✔️ ✔️ ✔️
Azure Blob Storage ✔️ ✔️

We have tested against the following S3 backends:

  • AWS S3
  • Ceph
  • MinIO on Linux
  • Pure Storage S3
  • Scality S3
  • VAST Data S3

Installation

Install ArcticDB:

$ pip install arcticdb

or using conda-forge

$ conda install -c conda-forge arcticdb

Import ArcticDB:

>>> import arcticdb as adb

Create an instance on your S3 storage (with or without explicit credentials):

# Leave AWS to derive credential information
>>> ac = adb.Arctic('s3://MY_ENDPOINT:MY_BUCKET?aws_auth=true')

# Manually specify creds
>>> ac = adb.Arctic('s3://MY_ENDPOINT:MY_BUCKET?region=YOUR_REGION&access=ABCD&secret=DCBA')

Or create an instance on your local disk:

>>> ac = adb.Arctic("lmdb:///<path>")

Create your first library and list the libraries in the instance:

>>> ac.create_library('travel_data')
>>> ac.list_libraries()

Create a test dataframe:

>>> import numpy as np
>>> import pandas as pd
>>> NUM_COLUMNS=10
>>> NUM_ROWS=100_000
>>> df = pd.DataFrame(np.random.randint(0,100,size=(NUM_ROWS, NUM_COLUMNS)), columns=[f"COL_{i}" for i in range(NUM_COLUMNS)], index=pd.date_range('2000', periods=NUM_ROWS, freq='h'))

Get the library, write some data to it, and read it back:

>>> lib = ac['travel_data']
>>> lib.write("my_data", df)
>>> data = lib.read("my_data")

To find out more about working with data, visit our docs


Documentation

The source code for the ArcticDB docs are located in the docs folder, and are hosted at docs.arcticdb.io.

License

ArcticDB is released under a Business Source License 1.1 (BSL)

BSL features are free to use and the source code is available, but users may not use ArcticDB for production use or for a Database Service, without agreement with Man Group Operations Limited.

Use of ArcticDB in production or for a Database Service requires a paid for license from Man Group Operations Limited and is licensed under the ArcticDB Software License Agreement. For more information please contact arcticdb@man.com.

The BSL is not certified as an open-source license, but most of the Open Source Initiative (OSI) criteria are met. Please see version conversion dates in the below table:

ArcticDB Version License Converts to Apache 2.0
1.0 Business Source License 1.1 Mar 16, 2025
1.2 Business Source License 1.1 May 22, 2025
1.3 Business Source License 1.1 Jun 9, 2025
1.4 Business Source License 1.1 Jun 23, 2025
1.5 Business Source License 1.1 Jul 11, 2025
1.6 Business Source License 1.1 Jul 25, 2025
2.0 Business Source License 1.1 Aug 29, 2025
3.0 Business Source License 1.1 Sep 13, 2025
4.0 Business Source License 1.1 Sep 27, 2025
4.1 Business Source License 1.1 Nov 1, 2025
4.2 Business Source License 1.1 Nov 12, 2025
4.3 Business Source License 1.1 Feb 7, 2026

Code of Conduct

Code of Conduct

This project has adopted a Code of Conduct. If you have any concerns about the Code, or behaviour that you have experienced in the project, please contact us at arcticdb@man.com.

Contributing/Building From Source

We welcome your contributions to help us improve and extend this project!

Please refer to the Contributing page and feel free to open issues on GitHub.

We are also always looking for feedback from our dedicated community! If you have used ArcticDB please let us know, we would love to hear about your experience!

Our release process is documented here.

Community

We would love to hear how your ArcticDB journey evolves, email us at arcticdb@man.com or come chat to us on Twitter!

Interested in learning more about ArcticDB? Head over to our blog!

Do you have any questions or issues? Chat to us and other users through our dedicated Slack Workspace - sign up for Slack access on our website.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

arcticdb-4.4.4rc1-cp311-cp311-win_amd64.whl (6.3 MB view details)

Uploaded CPython 3.11 Windows x86-64

arcticdb-4.4.4rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.4 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

arcticdb-4.4.4rc1-cp310-cp310-win_amd64.whl (6.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

arcticdb-4.4.4rc1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.4 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

arcticdb-4.4.4rc1-cp39-cp39-win_amd64.whl (6.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

arcticdb-4.4.4rc1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.4 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

arcticdb-4.4.4rc1-cp38-cp38-win_amd64.whl (6.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

arcticdb-4.4.4rc1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.4 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

arcticdb-4.4.4rc1-cp37-cp37m-win_amd64.whl (6.3 MB view details)

Uploaded CPython 3.7m Windows x86-64

arcticdb-4.4.4rc1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.5 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

arcticdb-4.4.4rc1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.5 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

File details

Details for the file arcticdb-4.4.4rc1-cp311-cp311-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 6fed3e97aceebd3eb396fee04e7d52ccabe9746f98d10b5f9aad6f9f50cc7925
MD5 cf9cb822c463f3600dc7a0575e372592
BLAKE2b-256 e90720f320093560f542a533da1f2c1a310642e8e47f3c3c68bfb71cc8d92261

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.4rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7238e5f1094e5f4bbba62bd5ed9387d007cb9b09c9947c0e360455b7371c6489
MD5 a7549084742927d2158191840e400b6b
BLAKE2b-256 5a6dffadf0463156c94b5f17094816068facdca1b5c68e0c22d5b906f55f382c

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.4rc1-cp310-cp310-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 a2a4fa024ea4f23e70193c6d48bb51a3131524b2b3e07bb47049fdbd396bde37
MD5 e19b62f868a31cd2b3f4ae205a0b394e
BLAKE2b-256 5a1085fc0a6d278cee6d1d9f30713c66a9b2908a7df7ca90bc66eb46c1e9b06e

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.4rc1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 fc234f2448c6d6aa67e5a448795a76ca8e141a790c279e7d4e76c92b682801aa
MD5 837bfa59f6e73515b5aa5e70129b0d0c
BLAKE2b-256 d15b5714df24bea9b8d35cfb304f242ba808f48e0f3f294dc72ce4f791e4ff0b

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.4rc1-cp39-cp39-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 617248b3a258b0a7a98a7dcb4340d1e65a0ec09936e60b2f1a7e53a76f39a284
MD5 84b5f39d11ab173c4818dc86a044a029
BLAKE2b-256 26a4f3a7ce829a644117b5881c5145de64a862026a8f3dedff91551aa03053a6

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.4rc1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 145c0ee0a56ffea81354ac904c78552ebd0e47784b33974af2ee4814521b2128
MD5 fa05e5dab24b74a8817664be2137d281
BLAKE2b-256 dc361717f3649dfa8c6b1fcd7d8bea2514851e5b719c6ccd333cc93af8ff578b

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.4rc1-cp38-cp38-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 5f7eb265839e9f1dc23dacebfdf1e05187f55fbd425f78397c337cf6d2302ddd
MD5 0e1f1f488f1c445dcfa941e7cf984fd6
BLAKE2b-256 50d88a8b042e200ab6a5b5da83e615803c908cc631e381e4f905107a485814d0

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.4rc1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 fa160b2234ce102cba8636d2b7267e824b6b9f423b7413bd19fb0267d5afba00
MD5 32ac9d418839f99e87a0ee85f5f20dfb
BLAKE2b-256 34b97d303a20b0eb589e9bec4a52d33dfafe096e3a74555bb96e7be3e06fd9bf

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.4rc1-cp37-cp37m-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 645bc6ac5ee55858ee42972bd38022f843a484948a75da32065b4831b625e0c0
MD5 dc8e1652835866b9c57f0a8693ccbc52
BLAKE2b-256 2d90906c846d8ba11fc8627350bf50adca7c45d2cffec31621cd18757ce1b4c8

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.4rc1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 03c820670e8d8f47c0ebb88f70a6e8a275afedf60141ec5ad7afe26ec061d732
MD5 41b48fbc55123fdfcf988b315b04cbe5
BLAKE2b-256 f2f36774a1ccea85b1225627963845f3a5e32d1ca0752f9f75ea3014e75b0791

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.4rc1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.4rc1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 fdb0d646db0cdc2446d4365b0ade74ef48b11e291017ce1dee8b6d2347b5cde3
MD5 88abcdba230e8c34ce9a04ff3b638b1c
BLAKE2b-256 153161fd3fa64bbaa4615bec94e87d4bcc23c78733a2f4e3a40f8b7252ff1866

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page