altastata 0.1.43

A Python package for Altastata data processing and machine learning integration

Altastata Python Package

A powerful Python package for secure, encrypted cloud storage with seamless integration for data processing, AI, machine learning, and RAG applications.

Installation

pip install altastata

Features

• fsspec filesystem interface - Use standard Python file operations with encrypted cloud storage
• S3-compatible API - Drive boto3 / aws CLI / s3fs / pyarrow against the bundled S3 gateway with one helper call
• Real-time Event Notifications - Listen for file share, delete, and create events
• LangChain Integration - Native support for document loaders and vector stores
• PyTorch & TensorFlow Support - Custom datasets for machine learning workflows
• Multi-cloud Support - Works with AWS, Azure, GCP, and more
• End-to-end Encryption - AES-256 encryption with zero-trust architecture

Quick Start

from altastata import AltaStataFunctions, AltaStataPyTorchDataset, AltaStataTensorFlowDataset
from altastata.altastata_tensorflow_dataset import register_altastata_functions_for_tensorflow
from altastata.altastata_pytorch_dataset import register_altastata_functions_for_pytorch

# Configuration parameters
user_properties = """#My Properties
#Sun Jan 05 12:10:23 EST 2025
AWSSecretKey=*****
AWSAccessKeyId=*****
myuser=bob123
accounttype=amazon-s3-secure
................................................................
region=us-east-1"""

private_key = """-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3,F26EBECE6DDAEC52

poe21ejZGZQ0GOe+EJjDdJpNvJcq/Yig9aYXY2rCGyxXLGVFeYJFg7z6gMCjIpSd
................................................................
wV5BUmp5CEmbeB4r/+BlFttRZBLBXT1sq80YyQIVLumq0Livao9mOg==
-----END RSA PRIVATE KEY-----"""

# Create an instance of AltaStataFunctions
altastata_functions = AltaStataFunctions.from_credentials(user_properties, private_key)
altastata_functions.set_password("my_password")

gRPC transport (recommended for Python + browser integration)

from altastata import AltaStataFunctions

f = AltaStataFunctions.from_account_dir(
    "/path/to/account",
    transport="grpc",
    password="123",
)

transport="grpc" auto-starts altastata-grpc when needed.

To run the gRPC server explicitly (for browser JS or local testing), use either:

altastata-grpc-server
# or (same entry point via Python module):
python -m altastata.grpc_server

The launcher binds gRPC to 127.0.0.1:9877 and, when the wheel ships with a bundled AltaStata Console SPA at altastata/lib/altastata-console-static/, serves the SPA from the same port. Open http://127.0.0.1:9877 in a browser to get a Finder-style file manager with auto-refresh on SHARE / DELETE events from other users (see mycloud/altastata-grpc/EventsService and altastata-console). The launcher exports ALTASTATA_WEB_UI_DIR automatically; set ALTASTATA_WEB_UI_DIR= to disable the UI and keep the port gRPC-only.

PyTorch & TensorFlow Integration

# Register the altastata functions for PyTorch or TensorFlow as a custom dataset
register_altastata_functions_for_pytorch(altastata_functions, "my_account")
register_altastata_functions_for_tensorflow(altastata_functions, "my_account")

# For PyTorch application use
torch_dataset = AltaStataPyTorchDataset(
    "my_account",
    root_dir=root_dir,
    file_pattern=pattern,
    transform=transform
)

# For TensorFlow application use
tensorflow_dataset = AltaStataTensorFlowDataset(
    "my_account",
    root_dir=root_dir,
    file_pattern=pattern,
    preprocess_fn=preprocess_fn
)

fsspec Integration

Altastata implements the fsspec interface, making it compatible with any Python library that uses standard file operations:

from altastata import AltaStataFunctions
from altastata.fsspec import create_filesystem

# Create AltaStata connection
altastata_functions = AltaStataFunctions.from_account_dir('/path/to/account')
altastata_functions.set_password("your_password")

# Create fsspec filesystem
fs = create_filesystem(altastata_functions, "my_account")

# Use it like any Python file system
files = fs.ls("Public/")
with fs.open("Public/Documents/file.txt", "r") as f:
    content = f.read()
    print(content)

This means you can use Altastata with pandas, dask, xarray, and hundreds of other libraries without any special configuration.

boto3 / aws CLI / s3fs (S3-compatible API)

The bundled altastata-services JVM exposes an S3-compatible REST API on port 9876 from inside the same process that backs the Python API, so boto3 and the Python API see the same files. Three helpers on AltaStataFunctions drive the admin bootstrap and surface the access/secret pair the gateway generates:

from altastata import AltaStataFunctions

alt = AltaStataFunctions.from_account_dir("/path/to/account")
alt.set_password("your_password")

# One-liner: ready-to-use boto3 S3 client.
s3 = alt.boto3_s3()
print(s3.list_buckets())
# `altastata-bucket` is the virtual bucket the gateway exposes for this
# account; substitute your own name if your deployment uses a different one.
s3.put_object(Bucket="altastata-bucket", Key="hello.txt", Body=b"hi")

# Or: just the kwargs — pass to any AWS SDK / s3fs / pyarrow / awswrangler.
creds = alt.s3_credentials()
# {'endpoint_url': 'http://127.0.0.1:9876',
#  'aws_access_key_id': 'AKIA...',
#  'aws_secret_access_key': '...',
#  'region_name': 'us-east-1'}

import s3fs
fs = s3fs.S3FileSystem(
    key=creds["aws_access_key_id"],
    secret=creds["aws_secret_access_key"],
    client_kwargs={"endpoint_url": creds["endpoint_url"]},
)

# Or: install AWS_* env vars so `!aws s3 ls`, `!s3cmd ls`, and any SDK
# that reads the ambient environment all "just work" — handy from Jupyter
# notebook shell cells.
alt.install_aws_env()

Requirements:

• The S3 gateway must be enabled on the JVM (ALTASTATA_SERVICES_S3GATEWAY_ENABLED=true, which is the default in the bundled jupyter-datascience docker compose).
• boto3 is not in the wheel's install_requires — install it separately (pip install boto3) only if you want the alt.boto3_s3() convenience. s3_credentials() and install_aws_env() use stdlib only.

Event Listener

Get real-time notifications when file operations occur:

from altastata import AltaStataFunctions

# Event handler
def event_handler(event_name, data):
    print(f"📢 Event: {event_name}, Data: {data}")
    if event_name == "SHARE":
        print("File was shared!")
    elif event_name == "DELETE":
        print("File was deleted!")

# Initialize with callback server
altastata = AltaStataFunctions.from_account_dir(
    '/path/to/account',
    enable_callback_server=True,
    callback_server_port=25334
)
altastata.set_password("your_password")

# Register listener
listener = altastata.add_event_listener(event_handler)

# Events will now be delivered in real-time!
# See examples/event-listener-example/ for complete demos

Perfect for:

• Data sharing among the users
• Audit logging and compliance
• Workflow automation

See examples/event-listener-example/ for complete documentation and working examples.

LangChain Integration

Use Altastata as a document source for LangChain applications:

from langchain.document_loaders import DirectoryLoader
from altastata.fsspec import create_filesystem
from altastata import AltaStataFunctions

# Create AltaStata connection
altastata_functions = AltaStataFunctions.from_account_dir('/path/to/account')
altastata_functions.set_password("your_password")

# Create fsspec filesystem
fs = create_filesystem(altastata_functions, "my_account")

# Use with LangChain document loaders
loader = DirectoryLoader("Public/Documents/", filesystem=fs)
documents = loader.load()

# Use with vector stores
from langchain.vectorstores import FAISS
from langchain.embeddings import OpenAIEmbeddings

vectorstore = FAISS.from_documents(documents, OpenAIEmbeddings())

Perfect for:

• RAG (Retrieval-Augmented Generation) applications
• Document processing pipelines
• Knowledge base construction
• Multi-modal AI applications

See the full documentation for more examples and advanced usage.

This project is licensed under the MIT License.