Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

OssTables Lance Connector

Connect Lance to the Alibaba Cloud OssTables catalog.

OssTables exposes a Lance REST Namespace endpoint secured with AWS Signature Version 4. This connector implements that namespace and signs every request, so you can use OssTables from Lance, Spark, Trino and Ray with your normal Alibaba Cloud credentials.

Requirements

Component Version
Python 3.9 or later
Java 8 or later (tested on 17, 21 and 24)

Installation

Python

pip install --pre osstables-lance-connector

To read and write table data you also need Lance itself:

pip install pylance

Java

<dependency>
    <groupId>com.aliyun.osstables</groupId>
    <artifactId>osstables-lance-connector</artifactId>
    <version>1.0.0-rc2</version>
</dependency>

Arrow is a provided dependency, so add an Arrow allocator to your application:

<dependency>
    <groupId>org.apache.arrow</groupId>
    <artifactId>arrow-memory-netty</artifactId>
    <version>15.0.0</version>
    <scope>runtime</scope>
</dependency>

Configuration

Every option uses the osstables. prefix.

Option Required Default Description
osstables.uri yes Catalog endpoint, e.g. https://my-bucket.cn-hangzhou.oss-tables.aliyuncs.com/lance
osstables.region yes Region used for request signing, e.g. cn-hangzhou
osstables.access_key_id no Access key ID. Omit to use environment variables
osstables.secret_access_key no Access key secret
osstables.session_token no Security token, when using STS credentials
osstables.delimiter no $ Separator for multi-level table identifiers
osstables.verify_ssl no true Verify TLS certificates; accepts only true or false
osstables.service no osstables SigV4 service name; only osstables is accepted
osstables.double_uri_encode no true Double-encode URI paths for SigV4; only true is accepted

osstables.service and osstables.double_uri_encode are retained for compatibility with older configurations, but OssTables requires their fixed values. The connector fails fast if either option is set to another value. Setting osstables.verify_ssl=false disables both certificate-chain and hostname verification; use it only for isolated testing, never in production.

Credentials

Credentials are taken from the first source that provides them:

  1. The osstables.access_key_id and osstables.secret_access_key options above.
  2. ALIBABA_CLOUD_ACCESS_KEY_ID, ALIBABA_CLOUD_ACCESS_KEY_SECRET and, optionally, ALIBABA_CLOUD_SECURITY_TOKEN.
  3. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and, optionally, AWS_SESSION_TOKEN.

Explicit access key ID and secret access key options must be provided together. Supplying only one is an error and does not fall back to environment variables. When using temporary credentials, supply the matching session token from the same source.

These credentials authenticate you to the catalog. Reading and writing table data goes straight to OSS and is authorized separately, using the storage options shown under "Reading and writing data" below.

Getting started

Python

import lance_namespace
import osstables_lance_connector  # registers the "osstables" implementation
from lance_namespace import CreateNamespaceRequest, ListTablesRequest

namespace = lance_namespace.connect(
    "osstables",
    {
        "osstables.uri": "https://my-bucket.cn-hangzhou.oss-tables.aliyuncs.com/lance",
        "osstables.region": "cn-hangzhou",
        "osstables.access_key_id": "...",
        "osstables.secret_access_key": "...",
        # "osstables.session_token": "...",  # required for STS credentials
    },
)

namespace.create_namespace(CreateNamespaceRequest(id=["sales"]))
print(namespace.list_tables(ListTablesRequest(id=["sales"])).tables)

If you would rather not import the package, pass the class path to connect instead:

namespace = lance_namespace.connect(
    "osstables_lance_connector.OssTablesNamespace", {...}
)

Java

import com.aliyun.osstables.lance.OssTablesNamespace;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import org.apache.arrow.memory.RootAllocator;
import org.lance.namespace.LanceNamespace;
import org.lance.namespace.model.CreateNamespaceRequest;

Map<String, String> options = new HashMap<>();
options.put("osstables.uri", "https://my-bucket.cn-hangzhou.oss-tables.aliyuncs.com/lance");
options.put("osstables.region", "cn-hangzhou");
options.put("osstables.access_key_id", "...");
options.put("osstables.secret_access_key", "...");
// options.put("osstables.session_token", "..."); // required for STS credentials

try (RootAllocator allocator = new RootAllocator();
    OssTablesNamespace namespace =
        (OssTablesNamespace)
            LanceNamespace.connect(OssTablesNamespace.class.getName(), options, allocator)) {
    namespace.createNamespace(
        new CreateNamespaceRequest().id(Collections.singletonList("sales")));
}

Reading and writing data

The catalog stores table metadata; the table data lives in OSS. Pass your OSS credentials as storage options so Lance can reach the data files:

import lance
import pyarrow as pa

storage_options = {
    "access_key_id": "...",
    "access_key_secret": "...",
    # "security_token": "...",  # required for STS credentials
    "endpoint": "https://oss-cn-hangzhou.aliyuncs.com",
    "region": "cn-hangzhou",
}

table = pa.table({"id": [1, 2, 3], "city": ["Hangzhou", "Beijing", "Shanghai"]})

lance.write_dataset(
    table,
    namespace_client=namespace,
    table_id=["sales", "orders"],
    mode="create",
    storage_options=storage_options,
)

dataset = lance.dataset(
    namespace_client=namespace,
    table_id=["sales", "orders"],
    storage_options=storage_options,
)
print(dataset.to_table())

Instead of passing storage_options, Lance can read OSS credentials and connection settings from OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET, optional OSS_SECURITY_TOKEN, OSS_ENDPOINT and OSS_REGION. These data-plane variables are independent of the catalog credential variables described above.

Query engines

Add the connector jar to the engine's class path and point the Lance catalog at this implementation. Catalog credentials use the osstables. prefix; data access uses storage.. With temporary credentials, configure both the catalog session token and the OSS storage session token. If you use environment variables instead, make them available to every process that accesses the catalog or data, including Spark drivers and executors or Trino servers and workers.

Spark

Every key is spark.sql.catalog.<catalog-name>.<option>, so an osstables. option appears after the catalog name. my_catalog below is the name used in SQL and can be anything.

spark.sql.catalog.my_catalog                             org.lance.spark.LanceNamespaceSparkCatalog
spark.sql.catalog.my_catalog.impl                        com.aliyun.osstables.lance.OssTablesNamespace
spark.sql.catalog.my_catalog.osstables.uri               https://my-bucket.cn-hangzhou.oss-tables.aliyuncs.com/lance
spark.sql.catalog.my_catalog.osstables.region            cn-hangzhou
spark.sql.catalog.my_catalog.osstables.access_key_id     ...
spark.sql.catalog.my_catalog.osstables.secret_access_key ...
spark.sql.catalog.my_catalog.osstables.session_token     ...
spark.sql.catalog.my_catalog.storage.access_key_id       ...
spark.sql.catalog.my_catalog.storage.access_key_secret   ...
spark.sql.catalog.my_catalog.storage.security_token      ...
spark.sql.catalog.my_catalog.storage.endpoint            https://oss-cn-hangzhou.aliyuncs.com
spark.sql.catalog.my_catalog.storage.region              cn-hangzhou
CREATE NAMESPACE my_catalog.sales;
CREATE TABLE my_catalog.sales.orders (id INT) USING lance;
SELECT * FROM my_catalog.sales.orders;

Trino

etc/catalog/my_catalog.properties, where the file name is the catalog name in SQL:

connector.name=lance
lance.impl=com.aliyun.osstables.lance.OssTablesNamespace
lance.osstables.uri=https://my-bucket.cn-hangzhou.oss-tables.aliyuncs.com/lance
lance.osstables.region=cn-hangzhou
lance.osstables.access_key_id=...
lance.osstables.secret_access_key=...
lance.osstables.session_token=...
lance.storage.access_key_id=...
lance.storage.access_key_secret=...
lance.storage.security_token=...
lance.storage.endpoint=https://oss-cn-hangzhou.aliyuncs.com
lance.storage.region=cn-hangzhou

Ray

from lance_ray import read_lance

dataset = read_lance(
    table_id=["sales", "orders"],
    namespace_impl="osstables_lance_connector.OssTablesNamespace",
    namespace_properties={
        "osstables.uri": "https://my-bucket.cn-hangzhou.oss-tables.aliyuncs.com/lance",
        "osstables.region": "cn-hangzhou",
    },
    storage_options=storage_options,
)

License

Released under the MIT License.

Metadata

Release files for osstables-lance-connector 1.0.0rc2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for osstables-lance-connector 1.0.0rc2
File Size Uploaded Provenance Actions
osstables_lance_connector-1.0.0rc2.tar.gz 15.4 kB Without provenance Details for osstables_lance_connector-1.0.0rc2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for osstables-lance-connector 1.0.0rc2
File Interpreter ABI Platform Actions
osstables_lance_connector-1.0.0rc2-py3-none-any.whl Python 3 none any Details for osstables_lance_connector-1.0.0rc2-py3-none-any.whl

Total release size: 28.5 kB

Release files / osstables_lance_connector-1.0.0rc2.tar.gz

Download URL osstables_lance_connector-1.0.0rc2.tar.gz
Size 15.4 kB
Tags Source
SHA-256 checksum
How to use checksums
a35b4a702eed238ad213f6744c411a3655c31ad2a74307fbb57ff0644c54d83d
BLAKE2b-256 checksum
How to use checksums
c13cdaba0d079ed33fe93ae87555bbe3154daff28aab04cf4f40b3423ae72072
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.2

Release files / osstables_lance_connector-1.0.0rc2-py3-none-any.whl

Download URL osstables_lance_connector-1.0.0rc2-py3-none-any.whl
Size 13.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
c99d3b8f38775657d84b82218195f68b61696ba47177998c25af3b1b4c691384
BLAKE2b-256 checksum
How to use checksums
3dbd638000664da7a3b7c273e196971e6a3c166d1b6b59532bd942618eabf6cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.11.2

Security for osstables-lance-connector 1.0.0rc2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page