Skip to main content


Deep Lake: Database for AI

PyPI version PyPI version

DocsGet StartedAPI ReferenceLangChain & VectorDBs CourseBlogWhitepaperSlackTwitter

What is Deep Lake?

Deep Lake is a Database for AI powered by a storage format optimized for deep-learning applications. Deep Lake can be used for:

  1. Storing and searching data plus vectors while building LLM applications
  2. Managing datasets while training deep learning models

Deep Lake simplifies the deployment of enterprise-grade LLM-based products by offering storage for all data types (embeddings, audio, text, videos, images, dicom, pdfs, annotations, and more), querying and vector search, data streaming while training models at scale, data versioning and lineage, and integrations with popular tools such as LangChain, LlamaIndex, Weights & Biases, and many more. Deep Lake works with data of any size, it is serverless, and it enables you to store all of your data in your own cloud and in one place. Deep Lake is used by Intel, Bayer Radiology, Matterport, ZERO Systems, Red Cross, Yale, & Oxford.

Deep Lake includes the following features:

Multi-Cloud Support (S3, GCP, Azure) Use one API to upload, download, and stream datasets to/from S3, Azure, GCP, Activeloop cloud, local storage, or in-memory storage. Compatible with any S3-compatible storage such as MinIO.
Native Compression with Lazy NumPy-like Indexing Store images, audio, and videos in their native compression. Slice, index, iterate, and interact with your data like a collection of NumPy arrays in your system's memory. Deep Lake lazily loads data only when needed, e.g., when training a model or running queries.
Dataloaders for Popular Deep Learning Frameworks Deep Lake comes with built-in dataloaders for Pytorch and TensorFlow. Train your model with a few lines of code - we even take care of dataset shuffling. :)
Integrations with Powerful Tools Deep Lake has integrations with Langchain and LLamaIndex as a vector store for LLM apps, Weights & Biases for data lineage during model training, MMDetection for training object detection models, and MMSegmentation for training semantic segmentation models.
100+ most-popular image, video, and audio datasets available in seconds Deep Lake community has uploaded 100+ image, video and audio datasets like MNIST, COCO, ImageNet, CIFAR, GTZAN and others.
Instant Visualization Support in the Deep Lake App Deep Lake datasets are instantly visualized with bounding boxes, masks, annotations, etc. in Deep Lake Visualizer (see below).

Visualizer

🚀 How to install Deep Lake

Deep Lake can be installed using pip:

pip install deeplake

To access all of Deep Lake's features, please register in the Deep Lake App.

🧠 Deep Lake Code Examples by Application

Vector Store Applications

Using Deep Lake as a Vector Store for building LLM applications:

- Vector Store Quickstart

- Vector Store Tutorials

- LangChain Integration

- LlamaIndex Integration

- Image Similarity Search with Deep Lake

Deep Learning Applications

Using Deep Lake for managing data while training Deep Learning models:

- Deep Learning Quickstart

- Tutorials for Training Models

⚙️ Integrations

Deep Lake offers integrations with other tools in order to streamline your deep learning workflows. Current integrations include:

📚 Documentation

Getting started guides, examples, tutorials, API reference, and other useful information can be found on our documentation page.

🎓 For Students and Educators

Deep Lake users can access and visualize a variety of popular datasets through a free integration with Deep Lake's App. Universities can get up to 1TB of data storage and 100,000 monthly queries on the Tensor Database for free per month. Chat in on our website: to claim the access!

👩‍💻 Comparisons to Familiar Tools

Deep Lake vs Chroma

Both Deep Lake & ChromaDB enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. ChromaDB is a Vector Database that can be deployed locally or on a server using Docker and will offer a hosted solution shortly. Deep Lake is a serverless Vector Store deployed on the user’s own cloud, locally, or in-memory. All computations run client-side, which enables users to support lightweight production apps in seconds. Unlike ChromaDB, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. ChromaDB is limited to light metadata on top of the embeddings and has no visualization. Deep Lake datasets can be visualized and version controlled. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

Deep Lake vs Pinecone

Both Deep Lake and Pinecone enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. Pinecone is a fully-managed Vector Database that is optimized for highly demanding applications requiring a search for billions of vectors. Deep Lake is serverless. All computations run client-side, which enables users to get started in seconds. Unlike Pinecone, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. Deep Lake datasets can be visualized and version controlled. Pinecone is limited to light metadata on top of the embeddings and has no visualization. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

Deep Lake vs Weaviate

Both Deep Lake and Weaviate enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. Weaviate is a Vector Database that can be deployed in a managed service or by the user via Kubernetes or Docker. Deep Lake is serverless. All computations run client-side, which enables users to support lightweight production apps in seconds. Unlike Weaviate, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. Deep Lake datasets can be visualized and version controlled. Weaviate is limited to light metadata on top of the embeddings and has no visualization. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

Deep Lake vs DVC

Deep Lake and DVC offer dataset version control similar to git for data, but their methods for storing data differ significantly. Deep Lake converts and stores data as chunked compressed arrays, which enables rapid streaming to ML models, whereas DVC operates on top of data stored in less efficient traditional file structures. The Deep Lake format makes dataset versioning significantly easier compared to traditional file structures by DVC when datasets are composed of many files (i.e., many images). An additional distinction is that DVC primarily uses a command-line interface, whereas Deep Lake is a Python package. Lastly, Deep Lake offers an API to easily connect datasets to ML frameworks and other common ML tools and enables instant dataset visualization through Activeloop's visualization tool.

Deep Lake vs MosaicML MDS format
  • Data Storage Format: Deep Lake operates on a columnar storage format, whereas MDS utilizes a row-wise storage approach. This fundamentally impacts how data is read, written, and organized in each system.
  • Compression: Deep Lake offers a more flexible compression scheme, allowing control over both chunk-level and sample-level compression for each column or tensor. This feature eliminates the need for additional compressions like zstd, which would otherwise demand more CPU cycles for decompressing on top of formats like jpeg.
  • Shuffling: MDS currently offers more advanced shuffling strategies.
  • Version Control & Visualization Support: A notable feature of Deep Lake is its native version control and in-browser data visualization, a feature not present for MosaicML data format. This can provide significant advantages in managing, understanding, and tracking different versions of the data.
Deep Lake vs TensorFlow Datasets (TFDS)

Deep Lake and TFDS seamlessly connect popular datasets to ML frameworks. Deep Lake datasets are compatible with both PyTorch and TensorFlow, whereas TFDS are only compatible with TensorFlow. A key difference between Deep Lake and TFDS is that Deep Lake datasets are designed for streaming from the cloud, whereas TFDS must be downloaded locally prior to use. As a result, with Deep Lake, one can import datasets directly from TensorFlow Datasets and stream them either to PyTorch or TensorFlow. In addition to providing access to popular publicly available datasets, Deep Lake also offers powerful tools for creating custom datasets, storing them on a variety of cloud storage providers, and collaborating with others via simple API. TFDS is primarily focused on giving the public easy access to commonly available datasets, and management of custom datasets is not the primary focus. A full comparison article can be found here.

Deep Lake vs HuggingFace Deep Lake and HuggingFace offer access to popular datasets, but Deep Lake primarily focuses on computer vision, whereas HuggingFace focuses on natural language processing. HuggingFace Transforms and other computational tools for NLP are not analogous to features offered by Deep Lake.
Deep Lake vs WebDatasets Deep Lake and WebDatasets both offer rapid data streaming across networks. They have nearly identical steaming speeds because the underlying network requests and data structures are very similar. However, Deep Lake offers superior random access and shuffling, its simple API is in python instead of command-line, and Deep Lake enables simple indexing and modification of the dataset without having to recreate it.
Deep Lake vs Zarr Deep Lake and Zarr both offer storage of data as chunked arrays. However, Deep Lake is primarily designed for returning data as arrays using a simple API, rather than actually storing raw arrays (even though that's also possible). Deep Lake stores data in use-case-optimized formats, such as jpeg or png for images, or mp4 for video, and the user treats the data as if it's an array, because Deep Lake handles all the data processing in between. Deep Lake offers more flexibility for storing arrays with dynamic shape (ragged tensors), and it provides several features that are not naively available in Zarr such as version control, data streaming, and connecting data to ML Frameworks.

Community

Join our Slack community to learn more about unstructured dataset management using Deep Lake and to get help from the Activeloop team and other users.

We'd love your feedback by completing our 3-minute survey.

As always, thanks to our amazing contributors!

Made with contributors-img.

Please read CONTRIBUTING.md to get started with making contributions to Deep Lake.

README Badge

Using Deep Lake? Add a README badge to let everyone know:

deeplake

[![deeplake](https://img.shields.io/badge/powered%20by-Deep%20Lake%20-ff5a1f.svg)](https://github.com/activeloopai/deeplake)

Disclaimers

Dataset Licenses

Deep Lake users may have access to a variety of publicly available datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have a license to use the datasets. It is your responsibility to determine whether you have permission to use the datasets under their license.

If you're a dataset owner and do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thank you for your contribution to the ML community!

Citation

If you use Deep Lake in your research, please cite Activeloop using:

@article{deeplake,
  title = {Deep Lake: a Lakehouse for Deep Learning},
  author = {Hambardzumyan, Sasun and Tuli, Abhinav and Ghukasyan, Levon and Rahman, Fariz and Topchyan, Hrant and Isayan, David and Harutyunyan, Mikayel and Hakobyan, Tatevik and Stranic, Ivo and Buniatyan, Davit},
  url = {https://www.cidrdb.org/cidr2023/papers/p69-buniatyan.pdf},
  booktitle={Proceedings of CIDR},
  year = {2023},
}

Acknowledgment

This technology was inspired by our research work at Princeton University. We would like to thank William Silversmith @SeungLab for his awesome cloud-volume tool.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

deeplake-4.7.3-cp313-cp313-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.13

deeplake-4.7.3-cp313-cp313-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.13

deeplake-4.7.3-cp313-cp313-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

deeplake-4.7.3-cp312-cp312-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.12

deeplake-4.7.3-cp312-cp312-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.12

deeplake-4.7.3-cp312-cp312-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

deeplake-4.7.3-cp311-cp311-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.11

deeplake-4.7.3-cp311-cp311-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.11

deeplake-4.7.3-cp311-cp311-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

deeplake-4.7.3-cp310-cp310-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.10

deeplake-4.7.3-cp310-cp310-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.10

deeplake-4.7.3-cp310-cp310-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.10macOS 11.0+ ARM64

deeplake-4.7.3-cp39-cp39-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.9

deeplake-4.7.3-cp39-cp39-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.9

deeplake-4.7.3-cp39-cp39-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.9macOS 11.0+ ARM64

File details

Details for the file deeplake-4.7.3-cp313-cp313-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp313-cp313-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a9253961c01530f9ee95be5bb7ed72da60a824f1ca26b38c62d2aa7b56aadebe
MD5 e660137efff43023286bbe1c7457e36f
BLAKE2b-256 ff1b653d175823db327cc81866a4def10c6826bcdddd16e9baf56e6939e4cf28

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp313-cp313-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp313-cp313-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 f1d41e47ac4635f57a349fe627f52655960501866838898b8d0ee9ebfb9e96ff
MD5 81ca0a12b767e8f1c5435b859cb58963
BLAKE2b-256 c513d2a9c4864739800a502319bab0216b148c80e86fece6d93ce8a1c07d52b7

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 17ae61576f6698d9676cde311caa98f59ba6730036e5727208cd41c5e479a2ec
MD5 2c97df3cc618b7cc172401c5e44b0aca
BLAKE2b-256 8cb1a557c262602fb6230b341fcc855f63b20ae7c3e63f52791a0489912cb510

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp312-cp312-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp312-cp312-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d985276f495be33b0a0fc0d76140d7923a694461e7ccfe1bfaab52809eb53856
MD5 68e5662c180ae9296ba8661340beab65
BLAKE2b-256 bc363662912fd87b43d57bab83a96d50f6330c427a712f4d5b30e49c4bc06fc4

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp312-cp312-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp312-cp312-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 927f9cde8228f2daf3a489277e125b24a2e2876e2fbfe087a5dbc77b7e7c15f1
MD5 93516fa843dd45d5d6b0c43466f54403
BLAKE2b-256 fa68216531038a1094220800b9a8b7158b292de4b779f7a5b9298454a8b1afc1

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 abc028dc8359bca0b0585d9c76bcf2edd0155a96104bb802fe31c5c58f445c89
MD5 448263a04815c613e31f2cb797e97be8
BLAKE2b-256 28814ae7eb1cb0e641f32fc8e90a202f0f8d5986d000f181d2d75c47a6676839

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp311-cp311-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp311-cp311-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 b12c83f94b92efb8e082727815a88635ea465ba1322f628a3d4ed583ecbbc90d
MD5 ca62d363c295e02f5fe241fb23081d55
BLAKE2b-256 460abb57e0d977fa5f8163ee393e2a8b00e00d2ba2993fb80c7288adef2321c5

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp311-cp311-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp311-cp311-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b75f6826e6bc2beaa04f513cb23e4a24c7fdbf5b400864058602975e8e7d573b
MD5 6db2520b587f16b6cbb30acfbf6c1d26
BLAKE2b-256 ff9e329c3e9f4aed87c0d32a36335cd889f920efc7a3c3316e75f68166e6c37b

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 5c6ae3da245f6cd616394df4db271776f647abafe78cd03624fa534397f19b4c
MD5 506c7303c7eff3120ad8a4e2be1839f8
BLAKE2b-256 92bbf11dce1fddae9b26aede5b71cf5371e6585095c17cf2e3d4c28d3e9a288b

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp310-cp310-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp310-cp310-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 25cebfa8354d511bef0d65ffeadb9f97bccdb96ae4a1e3343276a3a35fd5c3e4
MD5 5d62cfd71d9f3acf59f2bdef59f5f29c
BLAKE2b-256 ec6036fa0bc0aee656c142f1b37cf6f5c6048bad6fb1a84dcc48794b8067571c

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp310-cp310-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp310-cp310-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 e05aed1fafaa6cb423db879d8401be6ab35f60e20bff984286867a1d41d6c0c4
MD5 b9ae2c85b99f047f8596696f655f8fc8
BLAKE2b-256 8a3f23adc78215ad6a6beb49c8682b19336fc86a50b48f50f9b231224b8179b3

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 922cf564d6e1d2bff039866653ea982d6923086e0630d52b74002ab8bcf401b3
MD5 eeba7f2ee2ac0f5ce57f8e5dd9ba2c12
BLAKE2b-256 cafdb3e46debbede9888ac72801f9a16a2a79394eed3190e0d7504dc64a394f0

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 294d7cbb7f17d9a3dbacc459f2b81fd79e115d292c4d0e98373778cfabc13c9f
MD5 69dddc745e490dc0b0539d253c6e3a6e
BLAKE2b-256 534c94dbd8a019cceb736d8e51780986cd9cbca72c59695733e7434cd83c1870

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp39-cp39-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp39-cp39-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 31b2737e04d6ebdd60c1f0849ba45f1b008bff8e3bfec78e7e48ddae54402995
MD5 6b6e2bc885d4c243eba66c320eb964ee
BLAKE2b-256 12d407dc53093645f811925ca0cd0b41288dc2b5dc9e6c1bff0bccca94eaeacb

See more details on using hashes here.

File details

Details for the file deeplake-4.7.3-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.3-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 a184c7e57798e23573dd1d21409cc3abb5f77ed4d95516ee582f6912d546cb2f
MD5 71efd5ec034f7fd63642c358db65ec0a
BLAKE2b-256 1706b054462cf0e55b54050caf73ff37e522d15d4754461af5a406ccde3e76a3

See more details on using hashes here.

Release history Release notifications | RSS feed

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page