Skip to main content

python wrapper for Lance columnar format

Project description

Python bindings for Lance Data Format

:warning: Under heavy development

Lance Logo

Lance is a new columnar data format for data science and machine learning

Why you should use Lance

  1. Is order of magnitude faster than parquet for point queries and nested data structures common to DS/ML
  2. Comes with a fast vector index that delivers sub-millisecond nearest neighbors search performance
  3. Is automatically versioned and supports lineage and time-travel for full reproducibility
  4. Integrated with duckdb/pandas/polars already. Easily convert from/to parquet in 2 lines of code

Quick start

Installation

pip install pylance

Make sure you have a recent version of pandas (1.5+), pyarrow (10.0+), and DuckDB (0.7.0+)

Converting to Lance

import lance

import pandas as pd
import pyarrow as pa
import pyarrow.dataset

df = pd.DataFrame({"a": [5], "b": [10]})
uri = "/tmp/test.parquet"
tbl = pa.Table.from_pandas(df)
pa.dataset.write_dataset(tbl, uri, format='parquet')

parquet = pa.dataset.dataset(uri, format='parquet')
lance.write_dataset(parquet, "/tmp/test.lance")

Reading Lance data

dataset = lance.dataset("/tmp/test.lance")
assert isinstance(dataset, pa.dataset.Dataset)

Pandas

df = dataset.to_table().to_pandas()

DuckDB

import duckdb

# If this segfaults, make sure you have duckdb v0.7+ installed
duckdb.query("SELECT * FROM dataset LIMIT 10").to_df()

Vector search

Download the sift1m subset

wget ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz
tar -xzf sift.tar.gz

Convert it to Lance

import lance
from lance.vector import vec_to_table
import numpy as np
import struct

nvecs = 1000000
ndims = 128
with open("sift/sift_base.fvecs", mode="rb") as fobj:
    buf = fobj.read()
    data = np.array(struct.unpack("<128000000f", buf[4 : 4 + 4 * nvecs * ndims])).reshape((nvecs, ndims))
    dd = dict(zip(range(nvecs), data))

table = vec_to_table(dd)
uri = "vec_data.lance"
sift1m = lance.write_dataset(table, uri, max_rows_per_group=8192, max_rows_per_file=1024*1024)

Build the index

sift1m.create_index("vector",
                    index_type="IVF_PQ", 
                    num_partitions=256,  # IVF
                    num_sub_vectors=16)  # PQ

Search the dataset

# Get top 10 similar vectors
import duckdb

dataset = lance.dataset(uri)

# Sample 100 query vectors. If this segfaults, make sure you have duckdb v0.7+ installed
sample = duckdb.query("SELECT vector FROM dataset USING SAMPLE 100").to_df()
query_vectors = np.array([np.array(x) for x in sample.vector])

# Get nearest neighbors for all of them
rs = [dataset.to_table(nearest={"column": "vector", "k": 10, "q": q})      
      for q in query_vectors]

*More distance metrics, HNSW, and distributed support is on the roadmap

Python package details

Install from PyPI: pip install pylance # >=0.3.0 is the new rust-based implementation Install from source: maturin develop (under the /python directory) Run unit tests: make test Run integration tests: make integtest

Import via: import lance

The python integration is done via pyo3 + custom python code:

  1. We make wrapper classes in Rust for Dataset/Scanner/RecordBatchReader that's exposed to python.
  2. These are then used by LanceDataset / LanceScanner implementations that extend pyarrow Dataset/Scanner for duckdb compat.
  3. Data is delivered via the Arrow C Data Interface

Motivation

Why do we need a new format for data science and machine learning?

1. Reproducibility is a must-have

Versioning and experimentation support should be built into the dataset instead of requiring multiple tools.
It should also be efficient and not require expensive copying everytime you want to create a new version.
We call this "Zero copy versioning" in Lance. It makes versioning data easy without increasing storage costs.

2. Cloud storage is now the default

Remote object storage is the default now for data science and machine learning and the performance characteristics of cloud are fundamentally different.
Lance format is optimized to be cloud native. Common operations like filter-then-take can be order of magnitude faster using Lance than Parquet, especially for ML data.

3. Vectors must be a first class citizen, not a separate thing

The majority of reasonable scale workflows should not require the added complexity and cost of a specialized database just to compute vector similarity. Lance integrates optimized vector indices into a columnar format so no additional infrastructure is required to get low latency top-K similarity search.

4. Open standards is a requirement

The DS/ML ecosystem is incredibly rich and data must be easily accessible across different languages, tools, and environments. Lance makes Apache Arrow integration its primary interface, which means conversions to/from is 2 lines of code, your code does not need to change after conversion, and nothing is locked-up to force you to pay for vendor compute. We need open-source not fauxpen-source.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

pylance-0.30.0-cp39-abi3-win_amd64.whl (39.7 MB view details)

Uploaded CPython 3.9+Windows x86-64

pylance-0.30.0-cp39-abi3-manylinux_2_28_x86_64.whl (38.5 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.28+ x86-64

pylance-0.30.0-cp39-abi3-manylinux_2_28_aarch64.whl (35.5 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.28+ ARM64

pylance-0.30.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.6 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

pylance-0.30.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (35.5 MB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64

pylance-0.30.0-cp39-abi3-macosx_11_0_arm64.whl (33.6 MB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

pylance-0.30.0-cp39-abi3-macosx_10_15_x86_64.whl (36.7 MB view details)

Uploaded CPython 3.9+macOS 10.15+ x86-64

File details

Details for the file pylance-0.30.0-cp39-abi3-win_amd64.whl.

File metadata

  • Download URL: pylance-0.30.0-cp39-abi3-win_amd64.whl
  • Upload date:
  • Size: 39.7 MB
  • Tags: CPython 3.9+, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.9.13

File hashes

Hashes for pylance-0.30.0-cp39-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 72b4fdb24152148970e4dbbcca2ab3e1a02d63afcc9bceef14474c536c5b978d
MD5 96ce1a1f9a1d5b0f686cc0db97c0d106
BLAKE2b-256 16baddc813f2be56bab55d8c570aea1cd1a74741aca8b95f4e92d1913775f47f

See more details on using hashes here.

File details

Details for the file pylance-0.30.0-cp39-abi3-manylinux_2_28_x86_64.whl.

File metadata

File hashes

Hashes for pylance-0.30.0-cp39-abi3-manylinux_2_28_x86_64.whl
Algorithm Hash digest
SHA256 8644ac7f02c8ff2d73db084c670bbd5de7e9e59fea566fd71bd4e1a98f889cd0
MD5 3676d02e561e1d3ed18b0e777b40c28e
BLAKE2b-256 d51ab7a340aeabdb55c9e9632c8c00401fca5a6b640839afaac6e960f154664c

See more details on using hashes here.

File details

Details for the file pylance-0.30.0-cp39-abi3-manylinux_2_28_aarch64.whl.

File metadata

File hashes

Hashes for pylance-0.30.0-cp39-abi3-manylinux_2_28_aarch64.whl
Algorithm Hash digest
SHA256 f5a946bb96a3b3219e4f04bc4d544dd74c97a9d060b7236574d30cc3e44b502a
MD5 8f9d7a22ac04e0baa1613f05b62d801d
BLAKE2b-256 acd9b9f8b295d5409873393a79eab5131898fcdbc54c21b5929a4e1105f5e828

See more details on using hashes here.

File details

Details for the file pylance-0.30.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for pylance-0.30.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3adbde0b2b7a02e2d1fbf751e38d614d77527f670bcf81e79ba5db3fb59312f0
MD5 7b672626903d384637ccb7a3878e7730
BLAKE2b-256 816ad7bd49e1ce736dabc433bbab34a254031c45d7205ed2ac21d07fa4b82e75

See more details on using hashes here.

File details

Details for the file pylance-0.30.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for pylance-0.30.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 e60dcdb62fc8d2cc130f89a37f989c2af43e007f6d652044d64ff913efa73619
MD5 50eabc88516552c2ed462019664b1d95
BLAKE2b-256 4cb80995986cd48a0dd5324af8f9104cfada4dc1b34ea42dffe9d18167b4e101

See more details on using hashes here.

File details

Details for the file pylance-0.30.0-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for pylance-0.30.0-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 b3378e9146ece409b6073ea1524fd45b8dcde396b8dfeca63a9909817dd4179d
MD5 873f79cdfe55ebd4521a12c3d6e384ba
BLAKE2b-256 3b553e0c978c1ae20e33f3aa08a31c8467c4d96d1a8339c187ee190acb4604a2

See more details on using hashes here.

File details

Details for the file pylance-0.30.0-cp39-abi3-macosx_10_15_x86_64.whl.

File metadata

File hashes

Hashes for pylance-0.30.0-cp39-abi3-macosx_10_15_x86_64.whl
Algorithm Hash digest
SHA256 b64ae00a69a1d463d121198220df46c3dcf5672c0c02d81a9fc74c73b8725636
MD5 0f47e30488e4b7a21af68aeb1a83a808
BLAKE2b-256 799c1f94c6f30bd6cea63cd009ad447af3603ab03d21984096936bd14d3eeedf

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page