Skip to main content

ArcticDB DataFrame Database

Project description



ArcticDB Website | ArcticDB Blog | Press Release | Press Release | Community


ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem. Launched in March 2023, it is the successor to Arctic.

ArcticDB offers an intuitive Python-centric API enabling you to read and write Pandas DataFrames to S3 or LMDB utilising a fast C++ data-processing and compression engine.

ArcticDB allows you to:

  • Pandas in, Pandas out: Read and write Pandas DataFrames, NumPy arrays and native types to S3 and LMDB without leaving Python.
  • Built for time-series data: Efficiently index and query time-series data across billions of rows
  • Time travel: Travel back in time to see previous versions of your data and create customizable snapshots of the database
  • Schemaless Database: Append, update and modify data without being constrained by the existing schema
  • Optimised for streaming data: Built in support for efficient sparse data storage
  • Powerful processing: Filter, aggregate and create new columns on-the-fly with a Pandas-like syntax
  • C++ efficiency: Accelerate analytics though concurrency in the C++ data-processing engine

ArcticDB handles data that is big in both row count and column count, so a 20-year history of more than 400,000 unique securities can be stored in a single symbol. Each symbol is maintained as a separate entity with no shared data which means ArcticDB can scale horizontally across symbols, maximising the performance potential of your compute, storage and network.

ArcticDB is designed from the outset to be resilient; there is no single point of failure, and persistent data structures in the storage mean that once a version of a symbol has been written, it can never be corrupted by subsequent updates. Pulling compressed data directly from storage to the client means that there is no server to overload, so your data is always available when you need it.

Quickstart

Prebuilt binary availability

PyPI (Python 3.6 - 3.11) conda-forge (Python 3.8 - 3.11)
Linux (Intel/AMD) ✔️ ✔️
Windows (Intel/AMD) ✔️
MacOS Beta️

For conda-forge see the release-info.

Storage compatibility

Linux Windows Mac
S3 ✔️ ✔️ ✔️
LMDB ✔️ ✔️ ✔️
Azure Blob Storage ✔️ ✔️

We have tested against the following S3 backends:

  • AWS S3
  • Ceph
  • MinIO on Linux
  • Pure Storage S3
  • Scality S3
  • VAST Data S3

Installation

Install ArcticDB:

$ pip install arcticdb

or using conda-forge

$ conda install -c conda-forge arcticdb

Import ArcticDB:

>>> import arcticdb as adb

Create an instance on your S3 storage (with or without explicit credentials):

# Leave AWS to derive credential information
>>> ac = adb.Arctic('s3://MY_ENDPOINT:MY_BUCKET?aws_auth=true')

# Manually specify creds
>>> ac = adb.Arctic('s3://MY_ENDPOINT:MY_BUCKET?region=YOUR_REGION&access=ABCD&secret=DCBA')

Or create an instance on your local disk:

>>> ac = adb.Arctic("lmdb:///<path>")

Create your first library and list the libraries in the instance:

>>> ac.create_library('travel_data')
>>> ac.list_libraries()

Create a test dataframe:

>>> import numpy as np
>>> import pandas as pd
>>> NUM_COLUMNS=10
>>> NUM_ROWS=100_000
>>> df = pd.DataFrame(np.random.randint(0,100,size=(NUM_ROWS, NUM_COLUMNS)), columns=[f"COL_{i}" for i in range(NUM_COLUMNS)], index=pd.date_range('2000', periods=NUM_ROWS, freq='h'))

Get the library, write some data to it, and read it back:

>>> lib = ac['travel_data']
>>> lib.write("my_data", df)
>>> data = lib.read("my_data")

To find out more about working with data, visit our docs


Documentation

The source code for the ArcticDB docs are located in the docs folder, and are hosted at docs.arcticdb.io.

License

ArcticDB is released under a Business Source License 1.1 (BSL)

BSL features are free to use and the source code is available, but users may not use ArcticDB for production use or for a Database Service, without agreement with Man Group Operations Limited.

Use of ArcticDB in production or for a Database Service requires a paid for license from Man Group Operations Limited and is licensed under the ArcticDB Software License Agreement. For more information please contact arcticdb@man.com.

The BSL is not certified as an open-source license, but most of the Open Source Initiative (OSI) criteria are met. Please see version conversion dates in the below table:

ArcticDB Version License Converts to Apache 2.0
1.0 Business Source License 1.1 Mar 16, 2025
1.2 Business Source License 1.1 May 22, 2025
1.3 Business Source License 1.1 Jun 9, 2025
1.4 Business Source License 1.1 Jun 23, 2025
1.5 Business Source License 1.1 Jul 11, 2025
1.6 Business Source License 1.1 Jul 25, 2025
2.0 Business Source License 1.1 Aug 29, 2025
3.0 Business Source License 1.1 Sep 13, 2025
4.0 Business Source License 1.1 Sep 27, 2025
4.1 Business Source License 1.1 Nov 1, 2025
4.2 Business Source License 1.1 Nov 12, 2025
4.3 Business Source License 1.1 Feb 7, 2026

Code of Conduct

Code of Conduct

This project has adopted a Code of Conduct. If you have any concerns about the Code, or behaviour that you have experienced in the project, please contact us at arcticdb@man.com.

Contributing/Building From Source

We welcome your contributions to help us improve and extend this project!

Please refer to the Contributing page and feel free to open issues on GitHub.

We are also always looking for feedback from our dedicated community! If you have used ArcticDB please let us know, we would love to hear about your experience!

Our release process is documented here.

Community

We would love to hear how your ArcticDB journey evolves, email us at arcticdb@man.com or come chat to us on Twitter!

Interested in learning more about ArcticDB? Head over to our blog!

Do you have any questions or issues? Chat to us and other users through our dedicated Slack Workspace - sign up for Slack access on our website.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

arcticdb-4.4.3rc1-cp311-cp311-win_amd64.whl (6.3 MB view details)

Uploaded CPython 3.11 Windows x86-64

arcticdb-4.4.3rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.4 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

arcticdb-4.4.3rc1-cp310-cp310-win_amd64.whl (6.3 MB view details)

Uploaded CPython 3.10 Windows x86-64

arcticdb-4.4.3rc1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.4 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

arcticdb-4.4.3rc1-cp39-cp39-win_amd64.whl (6.3 MB view details)

Uploaded CPython 3.9 Windows x86-64

arcticdb-4.4.3rc1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.4 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

arcticdb-4.4.3rc1-cp38-cp38-win_amd64.whl (6.3 MB view details)

Uploaded CPython 3.8 Windows x86-64

arcticdb-4.4.3rc1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.4 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

arcticdb-4.4.3rc1-cp37-cp37m-win_amd64.whl (6.3 MB view details)

Uploaded CPython 3.7m Windows x86-64

arcticdb-4.4.3rc1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.5 MB view details)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

arcticdb-4.4.3rc1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.5 MB view details)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

File details

Details for the file arcticdb-4.4.3rc1-cp311-cp311-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 a688fa718b08a87e230e4a416909577e569c1750a131bb07005c6bfd621762fa
MD5 9c83e182f6b8d08bd6230ff8df18e6ff
BLAKE2b-256 dcb0b8657ca45dd6732c085384ad1d8a8301cbc14c526d3fdcd4b7ee98ecf9b8

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.3rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9f1fb64b9cc316958d427f8371e97c8fec21377d59720a2ee16b0052f46e358b
MD5 3f0a1e289847e5d2dd62137668618002
BLAKE2b-256 653b6274993cbc631667317895149633d06166423485a7b5f4a931bb3f52cbf5

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.3rc1-cp310-cp310-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp310-cp310-win_amd64.whl
Algorithm Hash digest
SHA256 a69486b48cca02b31d8a80ed40f9e0fbe463ac1d76351f4db6153d3510bc62d3
MD5 48a247320b2b855fa9986958110e3ab4
BLAKE2b-256 dd3e5639ea0b19800dc0ba6765957ee0ec2c36f9d0086fdcc310e65cfbb232b8

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.3rc1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f4b2b906eeeb739ccc05749a7a959b5119c00f99f5cc74ba6221ec0d20671ec9
MD5 ff05004fd5b7d69bc26e9b6d1bac131c
BLAKE2b-256 73249cf4dc8fef6143bf43277f7b8961c1503c7944daae5fa7e75f0270fa387f

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.3rc1-cp39-cp39-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp39-cp39-win_amd64.whl
Algorithm Hash digest
SHA256 04db934327be5d86e0ec212249871d2d73686865c8985f02ba1b3d589974da12
MD5 910a54cef3b1f12a83eaa052f2ec1305
BLAKE2b-256 c60cb4b39076ae8128cd4f522cdcb8cc04a556b327e60936f9b6cf3935dde999

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.3rc1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3814d52f3a09ceaa493ce35131ab111e785d8ea4fafa16a1ab21327bfdc27f07
MD5 49dd985c4ba0f508225fc36f07c4397e
BLAKE2b-256 f52d72621d958fff06f58d550d9cc011c29c0ff30d54350b5afc054a32b1ec9d

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.3rc1-cp38-cp38-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 de41e5442c345b44cc10a1c9e5c8a2f17d5719ab67ace39fa0b9181f245f46c1
MD5 8def61eef24f77e29dd04dc4fd15ed0d
BLAKE2b-256 d6d8a00a2222006862b6e0151ceb0d72111151684670d7af101935615918f837

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.3rc1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 fb6c997887e31df87461cde647dd9b5a6b3917b27b8f415302b74da7d759731c
MD5 3134017614c13998efbb8ff02a459c5b
BLAKE2b-256 fddc7251de3229912403def60eb7b04eb2e0a07b077a1e324fe3f8b394dc5610

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.3rc1-cp37-cp37m-win_amd64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 725531758e98e90847f1d829bed5c8480ee40fa96e37eb79bfc3ff9acb0b4446
MD5 99a64e4835ee8da801974a3640161a06
BLAKE2b-256 636740e5c0285528e8545ca22001e20c4cc77f70323c96f9d0c56ee269ffdf78

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.3rc1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4e1f6a279b9b19eb4545aad1500853847130e2b4a7d0f58362bc80570d6ae4b3
MD5 bfceef2491ac7cf8d4182b423f752bab
BLAKE2b-256 f477fac726be0fdcde0ce385606bc6a83f96a1c6a7e8e2beea98c15c56c99e60

See more details on using hashes here.

File details

Details for the file arcticdb-4.4.3rc1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for arcticdb-4.4.3rc1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 4241f92296433838206e39c0a63ef29db64d1147624a908b0c19cdcb262eb609
MD5 a944c03de76b8b50d5631ef87836fc79
BLAKE2b-256 a47c91bd81ec10178003d0e46a8f9f1a018e13d5927f02b558ea92aace8bce46

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page