Skip to main content


Deep Lake: Database for AI

PyPI version PyPI version

DocsGet StartedAPI ReferenceLangChain & VectorDBs CourseBlogWhitepaperSlackTwitter

What is Deep Lake?

Deep Lake is a Database for AI powered by a storage format optimized for deep-learning applications. Deep Lake can be used for:

  1. Storing and searching data plus vectors while building LLM applications
  2. Managing datasets while training deep learning models

Deep Lake simplifies the deployment of enterprise-grade LLM-based products by offering storage for all data types (embeddings, audio, text, videos, images, dicom, pdfs, annotations, and more), querying and vector search, data streaming while training models at scale, data versioning and lineage, and integrations with popular tools such as LangChain, LlamaIndex, Weights & Biases, and many more. Deep Lake works with data of any size, it is serverless, and it enables you to store all of your data in your own cloud and in one place. Deep Lake is used by Intel, Bayer Radiology, Matterport, ZERO Systems, Red Cross, Yale, & Oxford.

Deep Lake includes the following features:

Multi-Cloud Support (S3, GCP, Azure) Use one API to upload, download, and stream datasets to/from S3, Azure, GCP, Activeloop cloud, local storage, or in-memory storage. Compatible with any S3-compatible storage such as MinIO.
Native Compression with Lazy NumPy-like Indexing Store images, audio, and videos in their native compression. Slice, index, iterate, and interact with your data like a collection of NumPy arrays in your system's memory. Deep Lake lazily loads data only when needed, e.g., when training a model or running queries.
Dataloaders for Popular Deep Learning Frameworks Deep Lake comes with built-in dataloaders for Pytorch and TensorFlow. Train your model with a few lines of code - we even take care of dataset shuffling. :)
Integrations with Powerful Tools Deep Lake has integrations with Langchain and LLamaIndex as a vector store for LLM apps, Weights & Biases for data lineage during model training, MMDetection for training object detection models, and MMSegmentation for training semantic segmentation models.
100+ most-popular image, video, and audio datasets available in seconds Deep Lake community has uploaded 100+ image, video and audio datasets like MNIST, COCO, ImageNet, CIFAR, GTZAN and others.
Instant Visualization Support in the Deep Lake App Deep Lake datasets are instantly visualized with bounding boxes, masks, annotations, etc. in Deep Lake Visualizer (see below).

Visualizer

🚀 How to install Deep Lake

Deep Lake can be installed using pip:

pip install deeplake

To access all of Deep Lake's features, please register in the Deep Lake App.

🧠 Deep Lake Code Examples by Application

Vector Store Applications

Using Deep Lake as a Vector Store for building LLM applications:

- Vector Store Quickstart

- Vector Store Tutorials

- LangChain Integration

- LlamaIndex Integration

- Image Similarity Search with Deep Lake

Deep Learning Applications

Using Deep Lake for managing data while training Deep Learning models:

- Deep Learning Quickstart

- Tutorials for Training Models

⚙️ Integrations

Deep Lake offers integrations with other tools in order to streamline your deep learning workflows. Current integrations include:

📚 Documentation

Getting started guides, examples, tutorials, API reference, and other useful information can be found on our documentation page.

🎓 For Students and Educators

Deep Lake users can access and visualize a variety of popular datasets through a free integration with Deep Lake's App. Universities can get up to 1TB of data storage and 100,000 monthly queries on the Tensor Database for free per month. Chat in on our website: to claim the access!

👩‍💻 Comparisons to Familiar Tools

Deep Lake vs Chroma

Both Deep Lake & ChromaDB enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. ChromaDB is a Vector Database that can be deployed locally or on a server using Docker and will offer a hosted solution shortly. Deep Lake is a serverless Vector Store deployed on the user’s own cloud, locally, or in-memory. All computations run client-side, which enables users to support lightweight production apps in seconds. Unlike ChromaDB, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. ChromaDB is limited to light metadata on top of the embeddings and has no visualization. Deep Lake datasets can be visualized and version controlled. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

Deep Lake vs Pinecone

Both Deep Lake and Pinecone enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. Pinecone is a fully-managed Vector Database that is optimized for highly demanding applications requiring a search for billions of vectors. Deep Lake is serverless. All computations run client-side, which enables users to get started in seconds. Unlike Pinecone, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. Deep Lake datasets can be visualized and version controlled. Pinecone is limited to light metadata on top of the embeddings and has no visualization. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

Deep Lake vs Weaviate

Both Deep Lake and Weaviate enable users to store and search vectors (embeddings) and offer integrations with LangChain and LlamaIndex. However, they are architecturally very different. Weaviate is a Vector Database that can be deployed in a managed service or by the user via Kubernetes or Docker. Deep Lake is serverless. All computations run client-side, which enables users to support lightweight production apps in seconds. Unlike Weaviate, Deep Lake’s data format can store raw data such as images, videos, and text, in addition to embeddings. Deep Lake datasets can be visualized and version controlled. Weaviate is limited to light metadata on top of the embeddings and has no visualization. Deep Lake also has a performant dataloader for fine-tuning your Large Language Models.

Deep Lake vs DVC

Deep Lake and DVC offer dataset version control similar to git for data, but their methods for storing data differ significantly. Deep Lake converts and stores data as chunked compressed arrays, which enables rapid streaming to ML models, whereas DVC operates on top of data stored in less efficient traditional file structures. The Deep Lake format makes dataset versioning significantly easier compared to traditional file structures by DVC when datasets are composed of many files (i.e., many images). An additional distinction is that DVC primarily uses a command-line interface, whereas Deep Lake is a Python package. Lastly, Deep Lake offers an API to easily connect datasets to ML frameworks and other common ML tools and enables instant dataset visualization through Activeloop's visualization tool.

Deep Lake vs MosaicML MDS format
  • Data Storage Format: Deep Lake operates on a columnar storage format, whereas MDS utilizes a row-wise storage approach. This fundamentally impacts how data is read, written, and organized in each system.
  • Compression: Deep Lake offers a more flexible compression scheme, allowing control over both chunk-level and sample-level compression for each column or tensor. This feature eliminates the need for additional compressions like zstd, which would otherwise demand more CPU cycles for decompressing on top of formats like jpeg.
  • Shuffling: MDS currently offers more advanced shuffling strategies.
  • Version Control & Visualization Support: A notable feature of Deep Lake is its native version control and in-browser data visualization, a feature not present for MosaicML data format. This can provide significant advantages in managing, understanding, and tracking different versions of the data.
Deep Lake vs TensorFlow Datasets (TFDS)

Deep Lake and TFDS seamlessly connect popular datasets to ML frameworks. Deep Lake datasets are compatible with both PyTorch and TensorFlow, whereas TFDS are only compatible with TensorFlow. A key difference between Deep Lake and TFDS is that Deep Lake datasets are designed for streaming from the cloud, whereas TFDS must be downloaded locally prior to use. As a result, with Deep Lake, one can import datasets directly from TensorFlow Datasets and stream them either to PyTorch or TensorFlow. In addition to providing access to popular publicly available datasets, Deep Lake also offers powerful tools for creating custom datasets, storing them on a variety of cloud storage providers, and collaborating with others via simple API. TFDS is primarily focused on giving the public easy access to commonly available datasets, and management of custom datasets is not the primary focus. A full comparison article can be found here.

Deep Lake vs HuggingFace Deep Lake and HuggingFace offer access to popular datasets, but Deep Lake primarily focuses on computer vision, whereas HuggingFace focuses on natural language processing. HuggingFace Transforms and other computational tools for NLP are not analogous to features offered by Deep Lake.
Deep Lake vs WebDatasets Deep Lake and WebDatasets both offer rapid data streaming across networks. They have nearly identical steaming speeds because the underlying network requests and data structures are very similar. However, Deep Lake offers superior random access and shuffling, its simple API is in python instead of command-line, and Deep Lake enables simple indexing and modification of the dataset without having to recreate it.
Deep Lake vs Zarr Deep Lake and Zarr both offer storage of data as chunked arrays. However, Deep Lake is primarily designed for returning data as arrays using a simple API, rather than actually storing raw arrays (even though that's also possible). Deep Lake stores data in use-case-optimized formats, such as jpeg or png for images, or mp4 for video, and the user treats the data as if it's an array, because Deep Lake handles all the data processing in between. Deep Lake offers more flexibility for storing arrays with dynamic shape (ragged tensors), and it provides several features that are not naively available in Zarr such as version control, data streaming, and connecting data to ML Frameworks.

Community

Join our Slack community to learn more about unstructured dataset management using Deep Lake and to get help from the Activeloop team and other users.

We'd love your feedback by completing our 3-minute survey.

As always, thanks to our amazing contributors!

Made with contributors-img.

Please read CONTRIBUTING.md to get started with making contributions to Deep Lake.

README Badge

Using Deep Lake? Add a README badge to let everyone know:

deeplake

[![deeplake](https://img.shields.io/badge/powered%20by-Deep%20Lake%20-ff5a1f.svg)](https://github.com/activeloopai/deeplake)

Disclaimers

Dataset Licenses

Deep Lake users may have access to a variety of publicly available datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have a license to use the datasets. It is your responsibility to determine whether you have permission to use the datasets under their license.

If you're a dataset owner and do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thank you for your contribution to the ML community!

Citation

If you use Deep Lake in your research, please cite Activeloop using:

@article{deeplake,
  title = {Deep Lake: a Lakehouse for Deep Learning},
  author = {Hambardzumyan, Sasun and Tuli, Abhinav and Ghukasyan, Levon and Rahman, Fariz and Topchyan, Hrant and Isayan, David and Harutyunyan, Mikayel and Hakobyan, Tatevik and Stranic, Ivo and Buniatyan, Davit},
  url = {https://www.cidrdb.org/cidr2023/papers/p69-buniatyan.pdf},
  booktitle={Proceedings of CIDR},
  year = {2023},
}

Acknowledgment

This technology was inspired by our research work at Princeton University. We would like to thank William Silversmith @SeungLab for his awesome cloud-volume tool.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

deeplake-4.7.1-cp313-cp313-manylinux2014_x86_64.whl (46.1 MB view details)

Uploaded CPython 3.13

deeplake-4.7.1-cp313-cp313-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.13

deeplake-4.7.1-cp313-cp313-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.13macOS 11.0+ ARM64

deeplake-4.7.1-cp312-cp312-manylinux2014_x86_64.whl (46.1 MB view details)

Uploaded CPython 3.12

deeplake-4.7.1-cp312-cp312-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.12

deeplake-4.7.1-cp312-cp312-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.12macOS 11.0+ ARM64

deeplake-4.7.1-cp311-cp311-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.11

deeplake-4.7.1-cp311-cp311-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.11

deeplake-4.7.1-cp311-cp311-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

deeplake-4.7.1-cp310-cp310-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.10

deeplake-4.7.1-cp310-cp310-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.10

deeplake-4.7.1-cp310-cp310-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.10macOS 11.0+ ARM64

deeplake-4.7.1-cp39-cp39-manylinux2014_x86_64.whl (46.0 MB view details)

Uploaded CPython 3.9

deeplake-4.7.1-cp39-cp39-manylinux2014_aarch64.whl (42.2 MB view details)

Uploaded CPython 3.9

deeplake-4.7.1-cp39-cp39-macosx_11_0_arm64.whl (33.9 MB view details)

Uploaded CPython 3.9macOS 11.0+ ARM64

File details

Details for the file deeplake-4.7.1-cp313-cp313-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp313-cp313-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7ad40ccf90cbf202be799b039936120321ea51acd0dd481b20c02c2fd8ddb233
MD5 351411e2c02848855a7a700b6ae90903
BLAKE2b-256 c4ef5eeca961563b4b7141a6ac0e9493f2767cad3d29ebc7188a6fb04837c5ec

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp313-cp313-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp313-cp313-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 1a2aa99205fb3c8beb37d4eb0e3c3d2a3a7e8daf5ed14867f46638d2b9cf0e2c
MD5 687128334c1277236969832cfd2577c0
BLAKE2b-256 8c8b4564a83042c9c7e0fdea55453590a44d181f6a6c8d55169d835264c1361e

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp313-cp313-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp313-cp313-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 bc1489a50043e95e247e153fedf649ccd6b3a2c224f9b51b483e7b49f5736868
MD5 941bce7acf44a152dcb0486a36990e4b
BLAKE2b-256 a158b952370576bb52e9da6e3219dfe907e3f8ef623def46b0bfbfe644527468

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp312-cp312-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp312-cp312-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 87d2a5d05da0efcbed093bf09b58ade94436ec14b5876a4d1acee894c315f253
MD5 db1ba6a87b35d6fb335629ebee662df9
BLAKE2b-256 8ff432e491c54e4e959e9b0462c6c21df577a91772fd8bdd61a7de02aeda23aa

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp312-cp312-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp312-cp312-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 7b6cc4ff91ec3137379f32896c6d2cabfc10abc763fe5a2626443b8cd4f0b986
MD5 1e0572de83e991ede4f8b48edc194f95
BLAKE2b-256 dede360b127c82be4afac2328566432c251200c2ed1e904de18e8d4962dbd89d

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp312-cp312-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp312-cp312-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 2810ad65446bd065b800cec72cf2d0a61c9bd9682a675b2a16122d69bb001c91
MD5 0bbfc3968bd2a459c440b4fe14201ac3
BLAKE2b-256 7490a4571d5eac29a7e1166b19722fdf7ac18b9aea992d2757fe481b10d3801f

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp311-cp311-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp311-cp311-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 19a1f01d253071605e37729736e867ccb04ccbe03bacfdcc86ea33c5153aeddd
MD5 8719d6c4bc3beb48663d2a72750a9733
BLAKE2b-256 c6099142f81e4d875bcf4cd513694c8a5508c7172835448b2b9708ca2c503606

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp311-cp311-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp311-cp311-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 08aa8ff4d96641ac497dcb92cb1b4b52cb2c9a23dc1ee01a0ad705d50b494f3e
MD5 d12a6d69189e1208a4344168d3583ed5
BLAKE2b-256 061da8589c8cadc2544da58941b7ecf6853d24d1f864c92769f12d1264378e36

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 5adf5f522a69b8cc15943e4b4ee4455c5dbbcbd87fdee92b9b24d08af78aa1a1
MD5 2dd9b20bf14f8b9087519734dee0d14c
BLAKE2b-256 e3f5857fc971fc844d0f126629fe0e69c385ba8e23fee4257e60416f2e10df3a

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp310-cp310-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp310-cp310-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 98adb00b481639d8eab90816efa75724413a94d2784135ff43db44a8c36d4763
MD5 8730a062b83da02945b7e6aec6101d3f
BLAKE2b-256 1d5fe4a4d15a659686238abec999830482b1396d2f0f40adf1aedbca4d17cd03

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp310-cp310-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp310-cp310-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 9ee79cf0584f6349f15769fd22ea223262436fce50e589e7db7c9fd9f3b2a14d
MD5 741f2236cec9acad38794791e35fc6db
BLAKE2b-256 039fd373173641ce3f41896cf80776d995a93e41273147d5d9eaa3646b532c55

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp310-cp310-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 fdd9c54d15f879cef959d864ce5bbf0d7952ff12be829718a3eb241878ada004
MD5 2198dc7664f7faa80411de1fc5a6fc72
BLAKE2b-256 97fd73e52d1a3684b36ea53a9302cbbcd0313de34a7db83426e73bfa07a134ea

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp39-cp39-manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp39-cp39-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d5fe0e4409305264109ec0454f2e84b1d063a2f116fdb100675486179101f171
MD5 d3d0daa3dfbf9a608e7e5f8f94830876
BLAKE2b-256 fcac1d1b511013d6f24decfde1a99d2a1a6fe31c19e669b0622fb37fd328c701

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp39-cp39-manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp39-cp39-manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 577eeaa345fb4aee9f929fda237e837867675c76a0e4842ee672c8dc7cf7c2b9
MD5 3b780ce5d434cdfcdafc21834efe16ee
BLAKE2b-256 67c5c0e8d8965ec178dcdde2586e5f53ee4297801cd1cf625323fc372e2cd6f9

See more details on using hashes here.

File details

Details for the file deeplake-4.7.1-cp39-cp39-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for deeplake-4.7.1-cp39-cp39-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 52a2bfa0cf0e152fe69c0c64e9ca59f345cdad5160dbe66021e396d55c73afca
MD5 aeb5c4e4044864af338116b580209051
BLAKE2b-256 2052fb9619bd0fd6bd5a3379b78da2c10aacab4fdf9d1a9254774654b1a8092c

See more details on using hashes here.

Release history Release notifications | RSS feed

This release

4.7.1

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page