Skip to main content


Deep Lake: Database for AI

PyPI version PyPI version

DocsGet StartedAPI ReferenceLangChain & VectorDBs CourseBlogWhitepaperSlackTwitter

What is Deep Lake?

Deep Lake is a Database for AI powered by a storage format optimized for deep-learning applications. Deep Lake can be used for:

  1. Storing and searching data plus vectors while building LLM applications
  2. Managing datasets while training deep learning models

Deep Lake simplifies the deployment of enterprise-grade LLM-based products by offering storage for all data types (embeddings, audio, text, videos, images, dicom, pdfs, annotations, and more), querying and vector search, data streaming while training models at scale, data versioning and lineage, and integrations with popular tools such as LangChain, LlamaIndex, Weights & Biases, and many more. Deep Lake works with data of any size, it is serverless, and it enables you to store all of your data in your own cloud and in one place. Deep Lake is used by Intel, Bayer Radiology, Matterport, ZERO Systems, Red Cross, Yale, & Oxford.

Deep Lake includes the following features:

Multi-Cloud Support (S3, GCP, Azure) Use one API to upload, download, and stream datasets to/from S3, Azure, GCP, Activeloop cloud, local storage, or in-memory storage. Compatible with any S3-compatible storage such as MinIO.
Native Compression with Lazy NumPy-like Indexing Store images, audio, and videos in their native compression. Slice, index, iterate, and interact with your data like a collection of NumPy arrays in your system's memory. Deep Lake lazily loads data only when needed, e.g., when training a model or running queries.
Dataloaders for Popular Deep Learning Frameworks Deep Lake comes with built-in dataloaders for Pytorch and TensorFlow. Train your model with a few lines of code - we even take care of dataset shuffling. :)
Integrations with Powerful Tools Deep Lake has integrations with Langchain and LLamaIndex as a vector store for LLM apps, Weights & Biases for data lineage during model training, MMDetection for training object detection models, and MMSegmentation for training semantic segmentation models.
100+ most-popular image, video, and audio datasets available in seconds Deep Lake community has uploaded 100+ image, video and audio datasets like MNIST, COCO, ImageNet, CIFAR, GTZAN and others.
Instant Visualization Support in the Deep Lake App Deep Lake datasets are instantly visualized with bounding boxes, masks, annotations, etc. in Deep Lake Visualizer (see below).

Visualizer

🚀 How to install Deep Lake

Deep Lake can be installed using pip:

pip install deeplake

To access all of Deep Lake's features, please register in the Deep Lake App.

🧠 Deep Lake Code Examples by Application

Vector Store Applications

Using Deep Lake as a Vector Store for building LLM applications:

- Vector Store Quickstart

- Vector Store Tutorials

- LangChain Integration

- LlamaIndex Integration

- Image Similarity Search with Deep Lake

Deep Learning Applications

Using Deep Lake for managing data while training Deep Learning models:

- Deep Learning Quickstart

- Tutorials for Training Models

⚙️ Integrations

Deep Lake offers integrations with other tools in order to streamline your deep learning workflows. Current integrations include:

📚 Documentation

Getting started guides, examples, tutorials, API reference, and other useful information can be found on our documentation page.

🎓 For Students and Educators

Deep Lake users can access and visualize a variety of popular datasets through a free integration with Deep Lake's App. Universities can get up to 1TB of data storage and 100,000 monthly queries on the Tensor Database for free per month. Chat in on our website: to claim the access!

👩‍💻 Comparisons to Familiar Tools

Deep Lake vs Chroma

Both Deep Lake & ChromaDB enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. ChromaDB is a Vector Database that can be deployed locally or on a server using Docker and will offer a hosted solution shortly. Deep Lake is a serverless Vector Store deployed on the user’s own cloud, locally, or in-memory. All computations run client-side, which enables users to support lightweight production apps in seconds. Unlike ChromaDB, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. ChromaDB is limited to light metadata on top of the embeddings and has no visualization. Deep Lake datasets can be visualized and version controlled. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

Deep Lake vs Pinecone

Both Deep Lake and Pinecone enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. Pinecone is a fully-managed Vector Database that is optimized for highly demanding applications requiring a search for billions of vectors. Deep Lake is serverless. All computations run client-side, which enables users to get started in seconds. Unlike Pinecone, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. Deep Lake datasets can be visualized and version controlled. Pinecone is limited to light metadata on top of the embeddings and has no visualization. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

Deep Lake vs Weaviate

Both Deep Lake and Weaviate enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. Weaviate is a Vector Database that can be deployed in a managed service or by the user via Kubernetes or Docker. Deep Lake is serverless. All computations run client-side, which enables users to support lightweight production apps in seconds. Unlike Weaviate, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. Deep Lake datasets can be visualized and version controlled. Weaviate is limited to light metadata on top of the embeddings and has no visualization. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

Deep Lake vs DVC

Deep Lake and DVC offer dataset version control similar to git for data, but their methods for storing data differ significantly. Deep Lake converts and stores data as chunked compressed arrays, which enables rapid streaming to ML models, whereas DVC operates on top of data stored in less efficient traditional file structures. The Deep Lake format makes dataset versioning significantly easier compared to traditional file structures by DVC when datasets are composed of many files (i.e., many images). An additional distinction is that DVC primarily uses a command-line interface, whereas Deep Lake is a Python package. Lastly, Deep Lake offers an API to easily connect datasets to ML frameworks and other common ML tools and enables instant dataset visualization through Activeloop's visualization tool.

Deep Lake vs MosaicML MDS format
  • Data Storage Format: Deep Lake operates on a columnar storage format, whereas MDS utilizes a row-wise storage approach. This fundamentally impacts how data is read, written, and organized in each system.
  • Compression: Deep Lake offers a more flexible compression scheme, allowing control over both chunk-level and sample-level compression for each column or tensor. This feature eliminates the need for additional compressions like zstd, which would otherwise demand more CPU cycles for decompressing on top of formats like jpeg.
  • Shuffling: MDS currently offers more advanced shuffling strategies.
  • Version Control & Visualization Support: A notable feature of Deep Lake is its native version control and in-browser data visualization, a feature not present for MosaicML data format. This can provide significant advantages in managing, understanding, and tracking different versions of the data.
Deep Lake vs TensorFlow Datasets (TFDS)

Deep Lake and TFDS seamlessly connect popular datasets to ML frameworks. Deep Lake datasets are compatible with both PyTorch and TensorFlow, whereas TFDS are only compatible with TensorFlow. A key difference between Deep Lake and TFDS is that Deep Lake datasets are designed for streaming from the cloud, whereas TFDS must be downloaded locally prior to use. As a result, with Deep Lake, one can import datasets directly from TensorFlow Datasets and stream them either to PyTorch or TensorFlow. In addition to providing access to popular publicly available datasets, Deep Lake also offers powerful tools for creating custom datasets, storing them on a variety of cloud storage providers, and collaborating with others via simple API. TFDS is primarily focused on giving the public easy access to commonly available datasets, and management of custom datasets is not the primary focus. A full comparison article can be found here.

Deep Lake vs HuggingFace Deep Lake and HuggingFace offer access to popular datasets, but Deep Lake primarily focuses on computer vision, whereas HuggingFace focuses on natural language processing. HuggingFace Transforms and other computational tools for NLP are not analogous to features offered by Deep Lake.
Deep Lake vs WebDatasets Deep Lake and WebDatasets both offer rapid data streaming across networks. They have nearly identical steaming speeds because the underlying network requests and data structures are very similar. However, Deep Lake offers superior random access and shuffling, its simple API is in python instead of command-line, and Deep Lake enables simple indexing and modification of the dataset without having to recreate it.
Deep Lake vs Zarr Deep Lake and Zarr both offer storage of data as chunked arrays. However, Deep Lake is primarily designed for returning data as arrays using a simple API, rather than actually storing raw arrays (even though that's also possible). Deep Lake stores data in use-case-optimized formats, such as jpeg or png for images, or mp4 for video, and the user treats the data as if it's an array, because Deep Lake handles all the data processing in between. Deep Lake offers more flexibility for storing arrays with dynamic shape (ragged tensors), and it provides several features that are not naively available in Zarr such as version control, data streaming, and connecting data to ML Frameworks.

Community

Join our Slack community to learn more about unstructured dataset management using Deep Lake and to get help from the Activeloop team and other users.

We'd love your feedback by completing our 3-minute survey.

As always, thanks to our amazing contributors!

Made with contributors-img.

Please read CONTRIBUTING.md to get started with making contributions to Deep Lake.

README Badge

Using Deep Lake? Add a README badge to let everyone know:

deeplake

[![deeplake](https://img.shields.io/badge/powered%20by-Deep%20Lake%20-ff5a1f.svg)](https://github.com/activeloopai/deeplake)

Disclaimers

Dataset Licenses

Deep Lake users may have access to a variety of publicly available datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have a license to use the datasets. It is your responsibility to determine whether you have permission to use the datasets under their license.

If you're a dataset owner and do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thank you for your contribution to the ML community!

Citation

If you use Deep Lake in your research, please cite Activeloop using:

@article{deeplake,
  title = {Deep Lake: a Lakehouse for Deep Learning},
  author = {Hambardzumyan, Sasun and Tuli, Abhinav and Ghukasyan, Levon and Rahman, Fariz and Topchyan, Hrant and Isayan, David and Harutyunyan, Mikayel and Hakobyan, Tatevik and Stranic, Ivo and Buniatyan, Davit},
  url = {https://www.cidrdb.org/cidr2023/papers/p69-buniatyan.pdf},
  booktitle={Proceedings of CIDR},
  year = {2023},
}

Acknowledgment

This technology was inspired by our research work at Princeton University. We would like to thank William Silversmith @SeungLab for his awesome cloud-volume tool.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

deeplake-4.7.2-cp313-cp313-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.13

deeplake-4.7.2-cp313-cp313-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.13

deeplake-4.7.2-cp313-cp313-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

deeplake-4.7.2-cp312-cp312-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.12

deeplake-4.7.2-cp312-cp312-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.12

deeplake-4.7.2-cp312-cp312-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

deeplake-4.7.2-cp311-cp311-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.11

deeplake-4.7.2-cp311-cp311-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.11

deeplake-4.7.2-cp311-cp311-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

deeplake-4.7.2-cp310-cp310-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.10

deeplake-4.7.2-cp310-cp310-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.10

deeplake-4.7.2-cp310-cp310-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.10macOS 11.0+ ARM64

deeplake-4.7.2-cp39-cp39-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.9

deeplake-4.7.2-cp39-cp39-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.9

deeplake-4.7.2-cp39-cp39-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.9macOS 11.0+ ARM64

File details

Details for the file deeplake-4.7.2-cp313-cp313-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp313-cp313-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 5ac43f9a1a2b1b94883d65fd32330b485d86ffe0855af894b6ab2d7d04d27613
MD5 e46332bd90ad3224becc8958b9a0848f
BLAKE2b-256 81314a44f4f61f5172cc078ac57393674831683a37a6481ee6580dd74a2c6749

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp313-cp313-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp313-cp313-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 53ee1c255ee42508d5f5ef4a36af9cc96f9cdbd0b1e6071624e76ab9e866bd7f
MD5 f5ec34829f2ffce5bbb004eeb3cc98cc
BLAKE2b-256 9cdfdbf2eb52360c8a389f1879230a4c853c6206bb62e230d07d02812c04537e

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 c040d12c4c1179f2be57c727bf678e7e7e85ed5396bcca72bd766d7188ce2370
MD5 d408249e7abbdcafe9f3478bbf6b0d42
BLAKE2b-256 b03c3392e71577930d30a3a1f7e4561489841be8d88eeb597060aa0e257c12b6

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp312-cp312-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp312-cp312-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 cf566de6e237dee606a233c0d6f8d2f45aa6b63f6b0ffa1bf4d78a09ed742a60
MD5 ce649b792c25785582975799e2757827
BLAKE2b-256 5d18b4b4900c398bc760daae249ee479bb854ac43697c8e98b30897b0dc623af

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp312-cp312-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp312-cp312-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 57018c293d3f0ab2636fcbfe28f4e8c93944ec6d27965634f38efc60e112a662
MD5 9eeed372e4b5a8c13e65d3bea04ab2a4
BLAKE2b-256 8eed13c30a567484cda54218747a88fcbcf610c6d4356b158992411306c95759

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 7e1b22842fae076a36dfbeadfa63086c47e36b12ceaac35a9dfe2a17e51bc6de
MD5 81a406ca11ee8fefa7dc90a03eb26fec
BLAKE2b-256 a1fbaa04607db9b71501e86ebe136ef22fe043f4b231452ca81931bd2e6d4fa2

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp311-cp311-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp311-cp311-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c4ccf5e069943146524904933724cd517a2db75584976f0e800b6854edff7e61
MD5 6920e713e5c352da6aee5a3f328de58a
BLAKE2b-256 d98ed85479cb0ffa3ec17c9177f36895940d36c97004ed4f2382a9845bf79490

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp311-cp311-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp311-cp311-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 255b7db2e6c3d6bda2d92993e39e3cf47cb1bb7ee59a17a15b3daf8fe778a631
MD5 73f5363d2fd863fcfce313083fdf3a0e
BLAKE2b-256 39c32f15886fa5a03181a4917365ef9e4051279bb869683108e6cd0e2e080881

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 b2239b7d5f6bb72219606a48d546fe97ecdc4240f01f96666cd86069e2438747
MD5 60b44bd048849c9e0b4f093b76b464af
BLAKE2b-256 588fabefb25e8b67be081a76b4d090264a3317106dc39f9d0d57519fccadf96f

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp310-cp310-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp310-cp310-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c475503d38bd30ba59c4ebb5175767c3e00c07c770bd74504a401e6affb97ecf
MD5 a3c59997ac1071535768c32095261596
BLAKE2b-256 ef57f462674b1f3e2359f4dbaf798c441d864d82f1df766975dc0b6b2741002d

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp310-cp310-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp310-cp310-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9cfd5547a8a9caf3fbe16716c56d270a69173b640f08b95b7289284fb346d0c2
MD5 f07cfc0f639b6419f76b538afd354993
BLAKE2b-256 b05cda05d97d058eef3fa11e8384c9e9e4df2cb357432c200ef0ccc8c9bf8fb8

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 19f11f1eca8a16995f690a403c14b81ad541e9e669c8cef17bfceb962c085c17
MD5 43e358fc866110f5de4e39e8ea4e992b
BLAKE2b-256 6a528a3ec84d248a2530ac3c1179d8e333fcf6cff25f6b0eec5af9a485e26ea6

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6d160d5f9ed1c22389f6e1c067ca0e1d4e7a36e5d2198acf5d57b86e244696ee
MD5 95c4aca929746739d36ee63ae05bc51e
BLAKE2b-256 41717e7044e7a2ee6215338842722604d40d1a1fd1885f9d8e6d39a510748137

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp39-cp39-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp39-cp39-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 06662a068fde56e319bdbb683f09c39d8712077db604d95414fd0a2ce3bf3310
MD5 f580446c3a28da362564becba5fd34f5
BLAKE2b-256 7c4b786486d09bc8107f406c5868f0c449031dc7b746ed969703849acb23525c

See more details on using hashes here.

File details

Details for the file deeplake-4.7.2-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.2-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 f5c18966eeb9252555c9fa0453a1c0193391ba55fbb584e7e51fcf18a1264356
MD5 099937ae531bb2abfb47a7faa456da4f
BLAKE2b-256 41a332db1a036dfc72b630af7b1f54551612e162c29b446e71d27c8811fcc584

See more details on using hashes here.

Release history Release notifications | RSS feed

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page