lucidlink

The LucidLink Python SDK lets you programmatically access LucidLink filespaces: reading and writing files, streaming data, integrating with data libraries such as pandas and PyArrow, and managing external cloud storage via LucidLink Connect (Enterprise add-on).

Using a version before 1.0.0 (beta)? See the 1.0 migration guide and the beta documentation.

Key Concepts

• Workspace — your organization's LucidLink account.
• Filespace — a named storage volume within a workspace. Most SDK operations are scoped to a filespace.
• Client — the SDK's main entry point. Authenticates with a service account and manages the SDK's lifecycle.
• Service Account — a non-human identity used to authenticate SDK calls. Tokens look like sa_live:your_key_id:your_secret.
• LucidLink Connect — a feature (Enterprise add-on) that surfaces files from external S3 storage inside a filespace without copying them.

Authentication

All SDK operations require a service account token. Create one in the LucidLink web client (requires admin permissions). Service accounts are available on Business and Enterprise plans, as well as during Trial.

credentials = lucidlink.ServiceAccountCredentials(
    token="sa_live:your_key_id:your_secret"
)

Features

• Service Account Authentication — authenticate using service account tokens
• File Operations — read, write, create, delete files and directories
• Streaming I/O — full io.RawIOBase interface for standard Python I/O compatibility
• fsspec Integration — read and write lucidlink:// URLs from libraries that work with fsspec, such as pandas
• LucidLink Connect — manage external S3 data stores and link files into filespaces (Enterprise)

Installation

Requires Python 3.11 or later.

pip install lucidlink

For fsspec integration:

pip install "lucidlink[fsspec]"

Quick Start

import lucidlink

# Create a Client (ephemeral temp storage by default)
with lucidlink.Client() as client:
    # Authenticate with service account
    credentials = lucidlink.ServiceAccountCredentials(
        token="sa_live:your_key_id:your_secret"
    )
    client.login(credentials)

    # Service-account tokens are workspace-scoped
    workspace = client.workspace

    # Link to a filespace by id (see list_filespaces())
    filespace = workspace.link_filespace("fs-uuid-12345")

    # List directory
    for entry in filespace.file_system.read_dir("/"):
        print(f"{entry.name}: {entry.size} bytes")

    # Write a file
    lucidlink.FsUtils(filespace.file_system).write_bytes("/example.txt", b"Hello from LucidLink!")

    # Read a file
    data = lucidlink.FsUtils(filespace.file_system).read_bytes("/example.txt")
    print(data)

    # Save the changes: block until they have reached LucidLink services
    filespace.sync()

# client.close() runs on __exit__ — unlinks filespaces (with a best-effort
# final sync) and stops the runtime; a failed final sync is raised afterwards.

Streaming File Access

Full io.RawIOBase compatibility for streaming:

# Binary streaming
with filespace.file_system.open("/large_file.dat", "rb", buffering=8192) as f:
    for chunk in iter(lambda: f.read(4096), b""):
        process(chunk)

# Text streaming with encoding
with filespace.file_system.open("/document.txt", "rt", encoding="utf-8") as f:
    for line in f:
        print(line.strip())

# Byte range reads
with filespace.file_system.open("/data.bin", "rb") as f:
    f.seek(1000)
    data = f.read(100)  # Read 100 bytes from offset 1000

fsspec Integration

Access LucidLink files from libraries that work with fsspec, such as pandas (requires pip install "lucidlink[fsspec]"):

import pandas as pd

# Read CSV directly from LucidLink
df = pd.read_csv(
    "lucidlink://workspace/filespace/data.csv",
    storage_options={"token": "sa_live:..."}
)

# Write Parquet to LucidLink
df.to_parquet(
    "lucidlink://workspace/filespace/output.parquet",
    storage_options={"token": "sa_live:..."}
)

PyArrow does not open lucidlink:// URLs: give it a LucidLinkFileSystem as filesystem:

import pyarrow.parquet as pq
from lucidlink.fsspec import LucidLinkFileSystem

with LucidLinkFileSystem(token="sa_live:...") as fs:
    table = pq.read_table("lucidlink://workspace/filespace/data.parquet", filesystem=fs)

Direct fsspec usage:

from lucidlink.fsspec import LucidLinkFileSystem

# Closed, with a final sync, at the end of the with block
with LucidLinkFileSystem(token="sa_live:...") as fs:
    # List, download, upload
    entries = fs.ls("lucidlink://workspace/filespace/", detail=True)
    fs.get("lucidlink://workspace/filespace/file.txt", "local_file.txt")
    fs.put("local_file.txt", "lucidlink://workspace/filespace/uploaded.txt")

    # Move/rename (native operation, faster than copy+delete)
    fs.mv("lucidlink://workspace/filespace/old.txt",
          "lucidlink://workspace/filespace/new.txt")

Each LucidLinkFileSystem instance is a separate client, and fsspec creates one per thread, per process and per set of storage options; see Limitations.

Storage Modes

Ephemeral Mode (Default)

Uses a temporary directory that is automatically cleaned up:

client = lucidlink.Client()  # ephemeral by default

Persistent Mode

Uses a .lucid folder under the given directory, kept after the client closes. Not available with LucidLinkFileSystem, which always uses ephemeral storage:

from pathlib import Path

from lucidlink import Client, StorageConfig, StorageMode

client = Client(storage=StorageConfig(
    mode=StorageMode.PERSISTENT,
    root_path=Path.home() / "lucid_data",
))

Multiple clients must not share persistent storage: give each client its own root_path. Use ephemeral storage whenever more than one client is involved.

Limitations

• A client is not thread safe. Do not use one Client, or the objects obtained from it, from several threads at the same time. The same holds for one LucidLinkFileSystem instance, which owns one client.
• Multiple clients must not share persistent storage. With StorageMode.PERSISTENT, each client needs its own root_path. Use ephemeral storage (the default) whenever more than one client is involved.
• Multiple clients are only eventually consistent. A client sees changes made through another client — in the same process, in another process or on another machine — only after a delay. For example, after one client closes a file, another client can find it missing, empty or truncated for a few seconds; two clients creating the same new directory conflict, and one of the directories ends up as <name>_conflicted. filespace.sync() (or sync_all() on a LucidLinkFileSystem) returns once the changes have reached LucidLink services; it does not by itself make them visible to another client at once.
• A process forked while a client runs cannot use LucidLink. Every LucidLink call there raises ClientError. fork is the default multiprocessing start method on Linux before Python 3.14, e.g. for PyTorch DataLoader workers: use spawn or forkserver instead, and give each worker its own client.

fsspec creates clients implicitly

Each LucidLinkFileSystem instance owns its own client. fsspec reuses a cached instance only for the same storage options in the same thread and process, so fsspec.filesystem("lucidlink", ...), fsspec.core.url_to_fs(...), fsspec.open("lucidlink://...") and every library call that takes a lucidlink:// URL create another instance, and so another client:

• in another thread, including the threads a library starts for its own I/O;
• in another process, including worker processes that receive an instance by pickling;
• for other storage options, even in the same thread: files written through an instance created with auto_mkdir=True and read with pd.read_csv("lucidlink://...") without it go through two clients, and the reader can miss files or get them truncated;
• with skip_instance_cache=True.

These clients are eventually consistent with each other. LucidLinkFileSystem always uses ephemeral storage, since its per-thread and per-process instances would otherwise all get the same root_path.

The first and third limitations meet here: sharing one instance between threads uses one client from several threads, while an instance per thread means several clients.

Libraries that do not cope with eventual consistency

Libraries that reach lucidlink:// data through several clients, from worker processes or from their own I/O threads, can miss files that were just written or read them incomplete, often without an error. In our tests this happened with:

• dask with every scheduler but synchronous
• Ray Data
• zarr 3, and xarray and anndata zarr stores, opened by lucidlink:// URL
• kerchunk (reference:// with remote_protocol="lucidlink")
• Hugging Face datasets with lucidlink:// data files
• fsspec.generic.rsync and GenericFileSystem

The same applies to any code that runs work in other processes, such as PyTorch DataLoader workers, multiprocessing or concurrent.futures.ProcessPoolExecutor.

Telemetry

The SDK sends usage telemetry to LucidLink. Each event records the user or service account, the workspace, the filespace and the client node it concerns, together with the operating system and its version, the SDK version and the time of the event. LucidLink collects this data under the Developer Platform License Agreement (see "License and usage") and handles personal data under its privacy policy: https://www.lucidlink.com/privacy.

License and usage

This Software Development Kit (the “SDK”) is proprietary to LucidLink Corp. and is part of the LucidLink Developer Platform.

Use of this SDK is governed by the LucidLink Developer Platform License Agreement, available at https://developer.lucidlink.com/LucidLink%20Developer%20Platform%20License%20Agreement.pdf (the “Agreement”), or, if you or your organization have entered into a separate written agreement with LucidLink that governs your use of the Developer Platform, by that agreement. See the LICENSE file for details.

Under the Agreement you may modify the sample code and examples in the documentation for use in your own applications, and you may distribute the SDK, unmodified, as a component of your applications solely for use by their end users.

This SDK is not licensed under an open‑source license. All rights are reserved by LucidLink except as expressly granted in the Agreement.

This SDK includes third-party open-source components, each licensed under its own terms. See the NOTICE file for the list of components and their licenses.

Support

• Support: support@lucidlink.com