lucidlink 0.8.5

Python SDK for LucidLink — programmatic access to LucidLink filespaces

lucidlink

The LucidLink Python SDK lets you programmatically access LucidLink filespaces: reading and writing files, streaming data, integrating with data libraries like Pandas and Dask, and managing external cloud storage via LucidLink Connect (Enterprise add-on).

Key Concepts

• Workspace — your organization's LucidLink account.
• Filespace — a named storage volume within a workspace. Most SDK operations are scoped to a filespace.
• Daemon — the SDK's main entry point. Initializes the LucidLink client and manages its lifecycle.
• Service Account — a non-human identity used to authenticate SDK calls. Tokens look like sa_live:your_key.
• LucidLink Connect — a feature (Enterprise add-on) that surfaces files from external S3 storage inside a filespace without copying them.

Authentication

All SDK operations require a service account token. Create one in the LucidLink web client (requires admin permissions). Service accounts are available on Business and Enterprise plans, as well as during Trial.

credentials = lucidlink.ServiceAccountCredentials(
    token="sa_live:your_key"
)

Features

• Service Account Authentication — authenticate using service account tokens
• File Operations — read, write, create, delete files and directories
• Streaming I/O — full io.RawIOBase interface for standard Python I/O compatibility
• fsspec Integration — compatible with Pandas, Dask, PyArrow, and other data libraries
• LucidLink Connect — manage external S3 data stores and link files into filespaces (Enterprise)

Installation

Requires Python 3.10 or later.

pip install lucidlink

For fsspec integration:

pip install "lucidlink[fsspec]"

Quick Start

import lucidlink

# Create and start daemon
with lucidlink.Daemon() as daemon:
    # Authenticate with service account
    credentials = lucidlink.ServiceAccountCredentials(
        token="sa_live:your_key"
    )
    workspace = daemon.authenticate(credentials)

    # Link to a filespace
    filespace = workspace.link_filespace(name="production-data")

    # List directory
    for entry in filespace.fs.read_dir("/"):
        print(f"{entry.name}: {entry.size} bytes")

    # Write a file
    filespace.fs.write_file("/example.txt", b"Hello from LucidLink!")

    # Read a file
    data = filespace.fs.read_file("/example.txt")
    print(data)

    # Cleanup
    workspace.unlink_filespace()

Streaming File Access

Full io.RawIOBase compatibility for streaming:

# Binary streaming
with filespace.open("/large_file.dat", "rb", buffering=8192) as f:
    for chunk in iter(lambda: f.read(4096), b""):
        process(chunk)

# Text streaming with encoding
with filespace.open("/document.txt", "rt", encoding="utf-8") as f:
    for line in f:
        print(line.strip())

# Byte range reads
with filespace.open("/data.bin", "rb") as f:
    f.seek(1000)
    data = f.read(100)  # Read 100 bytes from offset 1000

fsspec Integration

Access LucidLink files using standard data libraries (requires pip install "lucidlink[fsspec]"):

import pandas as pd

# Read CSV directly from LucidLink
df = pd.read_csv(
    "lucidlink://workspace/filespace/data.csv",
    storage_options={"token": "sa_live:..."}
)

# Write Parquet to LucidLink
df.to_parquet(
    "lucidlink://workspace/filespace/output.parquet",
    storage_options={"token": "sa_live:..."}
)

With Dask for distributed computing:

import dask.dataframe as dd

ddf = dd.read_parquet(
    "lucidlink://workspace/filespace/dataset/*.parquet",
    storage_options={"token": "sa_live:..."}
)
result = ddf.groupby("category").agg({"value": "sum"}).compute()

Direct fsspec usage:

from lucidlink.fsspec import LucidLinkFileSystem

fs = LucidLinkFileSystem(token="sa_live:...", sandboxed=True)

# List, download, upload
entries = fs.ls("lucidlink://workspace/filespace/", detail=True)
fs.get("lucidlink://workspace/filespace/file.txt", "local_file.txt")
fs.put("local_file.txt", "lucidlink://workspace/filespace/uploaded.txt")

# Move/rename (native operation, faster than copy+delete)
fs.mv("lucidlink://workspace/filespace/old.txt",
      "lucidlink://workspace/filespace/new.txt")

fs.close()

Storage Modes

Sandboxed Mode (Default)

Uses a temporary directory that is automatically cleaned up:

daemon = lucidlink.Daemon()  # sandboxed by default

Physical Mode

Uses a persistent .lucid folder:

from lucidlink import Daemon, StorageConfig, StorageMode

# With cleanup on exit
daemon = Daemon(storage=StorageConfig(mode=StorageMode.PHYSICAL))

# Keep files after exit
daemon = Daemon(storage=StorageConfig(
    mode=StorageMode.PHYSICAL,
    persist_on_exit=True,
))

# Custom storage location
daemon = Daemon(storage=StorageConfig(
    mode=StorageMode.PHYSICAL,
    persist_on_exit=True,
    root_path=Path("D:/lucid_data"),
))

API Reference

Core Classes

Class	Description
Daemon	Daemon lifecycle management
Workspace	Workspace operations and filespace discovery
Filespace	Filespace context — provides access to filesystem and connect interfaces
Filesystem	File and directory operations
LucidFileStream	Streaming file I/O (io.RawIOBase)
LucidLinkFileSystem	fsspec integration (fsspec.AbstractFileSystem)
ServiceAccountCredentials	Authentication credentials
StorageConfig / StorageMode	Storage configuration
ConnectManager	External files (S3 data store) management

Exception Hierarchy

File operations raise standard Python exceptions (FileNotFoundError, FileExistsError, PermissionError, IsADirectoryError, etc.). SDK-specific errors use the following hierarchy:

LucidLinkError
├── DaemonError          # Daemon lifecycle errors
├── FilespaceError       # Filespace operation errors
├── AuthenticationError  # Authentication failures
└── ConfigurationError   # Configuration errors

License

Copyright 2026 LucidLink Corp. All rights reserved.

By using this software, you agree to the LucidLink Beta Agreement.

Support

• Support: [e-mail]