altastata 0.1.44

A Python package for Altastata data processing and machine learning integration

Altastata Python Package

Secure, encrypted cloud storage for Python — with fsspec, PyTorch/TensorFlow, boto3/S3, gRPC, and a bundled Web Console file manager.

pip install altastata

What you get

Capability	How
Encrypted files in S3, Azure, IBM COS, etc.	AltaStataFunctions + account folder
Standard Python file APIs	fsspec (create_filesystem)
ML datasets	AltaStataPyTorchDataset, AltaStataTensorFlowDataset
S3 tools (boto3, aws CLI, s3fs)	Bundled S3 gateway on port 9876
gRPC API + browser UI	Bundled gateway on port 9877
Real-time share/delete events	gRPC EventsService or Py4J callback server

---

Account folder

Each user keeps one directory under ~/.altastata/accounts/<display-name>/, for example:

amazon.rsa.bob123/
  altastata-myorg-bob123.user.properties   # cloud credentials (from your admin)
  private.key                              # RSA (password-encrypted PEM)
  public.key

Account type	Key files in folder	Password at login
RSA	private.key, public.key	Yes — decrypts private.key
PQC	kyber_private.key, dilithium_private.key, …	Yes
HPCS	hpcs-privkey.blob, public.key, hpcs.marker	No — GREP11 on gateway
HSM	*user.properties only (keys in cloud KMS/HSM)	No

Your org admin creates *user.properties after you send them public.key (RSA/PQC/HPCS).

---

Quick start (recommended: gRPC)

transport="grpc" starts the bundled Java gateway if it is not already running.

from altastata import AltaStataFunctions

# RSA / PQC — use your real password
f = AltaStataFunctions.from_account_dir(
    "/path/to/.altastata/accounts/amazon.rsa.bob123",
    transport="grpc",
    password="your_password",
)

# HPCS / HSM — empty password
f = AltaStataFunctions.from_account_dir(
    "/path/to/.altastata/accounts/amazon.rsa.hpcs.myuser",
    transport="grpc",
    password="",
)

versions = f.list_cloud_versions("Public/", True)
print(versions)

Ports (local gateway)

Port	Service
9877	gRPC + Web Console (open http://127.0.0.1:9877)
9876	S3-compatible REST API
25333	Py4J (legacy in-process bridge)

Start the gateway manually:

altastata-grpc-server
# same as: python -m altastata.grpc_server

When the wheel includes altastata/lib/altastata-console-static/, the launcher serves the AltaStata Console SPA on 9877. In the browser: Settings → choose your account folder → Sign in (HSM/HPCS: leave password blank).

Set ALTASTATA_WEB_UI_DIR= (empty) to disable the UI and keep gRPC-only.

HPCS in Docker / Jupyter

The gateway needs a populated grep11client.yaml (mount at /etc/ep11client/grep11client.yaml) and access to your hpcs-privkey.blob. See containers/jupyter/README-Docker.md and .cursor/rules in the mycloud repo for compose overrides.

---

Legacy Py4J transport (default)

from altastata import AltaStataFunctions

f = AltaStataFunctions.from_account_dir("/path/to/account")
f.set_password("your_password")   # omit for HSM; HPCS uses GREP11 on gateway

Or inline credentials:

f = AltaStataFunctions.from_credentials(user_properties_text, private_key_pem)
f.set_password("your_password")

---

fsspec

from altastata import AltaStataFunctions
from altastata.fsspec import create_filesystem

f = AltaStataFunctions.from_account_dir("/path/to/account", transport="grpc", password="secret")
fs = create_filesystem(f, "my_account")

for name in fs.ls("Public/"):
    print(name)

with fs.open("Public/readme.txt", "r") as fh:
    print(fh.read())

Works with pandas, dask, LangChain DirectoryLoader, and other fsspec consumers.

---

S3-compatible API (boto3, aws CLI, s3fs)

The bundled JVM exposes S3 on 9876 (same account as the Python API).

f = AltaStataFunctions.from_account_dir("/path/to/account", transport="grpc", password="secret")

# Convenience wrapper (requires: pip install boto3)
s3 = f.boto3_s3()
s3.put_object(Bucket="altastata-bucket", Key="hello.txt", Body=b"hi")

# Or pass creds to any S3 client
creds = f.s3_credentials()
# endpoint_url http://127.0.0.1:9876, aws_access_key_id, aws_secret_access_key, ...

f.install_aws_env()   # sets AWS_* for shell / !aws s3 ls in Jupyter

S3 gateway is enabled by default in the Jupyter Docker image (ALTASTATA_SERVICES_S3GATEWAY_ENABLED=true).

---

PyTorch & TensorFlow

from altastata import AltaStataFunctions, AltaStataPyTorchDataset, AltaStataTensorFlowDataset
from altastata.altastata_pytorch_dataset import register_altastata_functions_for_pytorch
from altastata.altastata_tensorflow_dataset import register_altastata_functions_for_tensorflow

f = AltaStataFunctions.from_account_dir("/path/to/account", transport="grpc", password="secret")
register_altastata_functions_for_pytorch(f, "my_account")

dataset = AltaStataPyTorchDataset("my_account", root_dir="Public/", file_pattern="*.jpg")

See examples/pytorch-example/ and examples/tensorflow-example/.

---

Event notifications

def on_event(name, data):
    print(name, data)

f = AltaStataFunctions.from_account_dir(
    "/path/to/account",
    enable_callback_server=True,
    callback_server_port=25334,
)
f.set_password("secret")
f.add_event_listener(on_event)

With transport="grpc", use the Web Console or gRPC EventsService.Watch for cross-user SHARE/DELETE notifications.

See examples/event-listener-example/.

---

Docker Jupyter (optional)

Pre-built images: ghcr.io/sergevil/altastata/jupyter-datascience-{arm64,amd64}:latest

cd containers/jupyter
docker compose -f docker-compose.yml -f docker-compose-ghcr.yml up -d

• JupyterLab: http://127.0.0.1:8888
• Web Console / gRPC: http://127.0.0.1:9877

Full build/run guide: containers/jupyter/README-Docker.md

---

More documentation

• Developers (build wheel, bundle JAR + Console SPA, PyPI): README-developer.md
• gRPC design (LoginV2, account setup): mycloud/altastata-grpc/CONSOLE_ACCOUNT_SETUP_DESIGN.md
• Examples: examples/

License

MIT License — see LICENSE.