PoreStore Python SDK

Python client for accessing PoreStore from scripts and notebooks. The distribution and import name are both porestore; this repository is named porestore-sdk to distinguish it from the PoreStore application.

The SDK supports two-way NumPy image transfer and remote transport, structure, statistical, REV porosity, mercury porosimetry, and network-extraction jobs. The SDK is in alpha. The broader proposal is described in PoreStore discussion #1477. Scientific calculations run through PoreStore; the SDK handles image transfer, job submission, polling, retrieval, and local result caching.

Supported calculations

The metrics namespace also exports the Structure and Statistics measurements, plus metrics.rev_porosity. These are the same functions and reuse the same jobs. image.metrics displays measurements already included in image metadata, without starting a calculation or fetching data. Call image.refresh() to retrieve updated metadata.

See the analysis tool reference for settings, API names, result retrieval, and an example of each additional measurement. The service controls tool and optional-parameter availability for your account. Grain size, REV diffusivity, and REV permeability have no implemented backend analysis yet.

Installation

The SDK requires Python 3.12 or later. Until a release is published, install from your local checkout into the environment used by your script or notebook:

uv add --editable /path/to/porestore_sdk
# Alternatively, in an activated Python environment:
python -m pip install -e /path/to/porestore_sdk

Select that environment as the Jupyter kernel. PoreSpy, Matplotlib, and OpenPNM are optional dependencies used in some examples below; install them separately in the same environment when needed. Keep the SDK environment separate from the PoreStore application environment, since both use the name porestore.

Notebook workflow

Configure a connection

Create a team API key in PoreStore and set PORESTORE_API_KEY and PORESTORE_TEAM_ID in your environment or a .env file in the current working directory. Use the immutable Team ID displayed in team settings. PORESTORE_URL optionally selects another service, such as http://localhost:8000; the default is https://porestore.com.

For example, a .env file beside the notebook can contain:

PORESTORE_API_KEY=your-team-api-key
PORESTORE_TEAM_ID=your-immutable-team-id
PORESTORE_URL=https://porestore.com

Replace the placeholders with your team's values and keep the key out of version control. Jupyter's current working directory determines which .env is read; Path.cwd() from Python's pathlib module shows that directory.

Upload a pore image and run transport calculations

This example starts with a segmented image saved as voids.npy, where True denotes pore space and False denotes solid. The voxel edge length is 70 nm, as in a nanoscale reconstruction. PoreStore takes this length in micrometres; the conversion from metres is explicit below.

import numpy as np
import porestore

# Reads credentials from the environment or .env in the current working directory.
client = porestore.configure()

local_array = np.load("voids.npy", allow_pickle=False)
voxel_size_m = 70e-9
im = porestore.upload(
    local_array,
    name="Multiscale voids",
    voxel_size=voxel_size_m * 1e6,  # 0.070 micrometres
    wait=True,
    timeout=1800,
    poll_interval=5,
)
print(im.id, im.status)  # Save the ID. The image is ready for analysis.

diffusivity = porestore.tools.transport.diffusivity(
    im=im,
    direction="x",
    target_relative_error=0.001,
)
permeability = porestore.tools.transport.permeability(
    im=im,
    direction="x",
    max_iterations=200_000,
)
print("Diffusivity:", diffusivity.id, "reused:", diffusivity.reused)
print("Permeability:", permeability.id, "reused:", permeability.reused)

# Both jobs have been submitted before either wait begins.
diffusivity_result = diffusivity.get_result(
    wait=True,
    timeout=86400,
    poll_interval=10,
)
permeability_result = permeability.get_result(
    wait=True,
    timeout=86400,
    poll_interval=10,
)
display(diffusivity_result.solution)
display(permeability_result.solution)

result = diffusivity_result  # Used in the result-inspection examples below.

The tool calls return Job objects without waiting for calculations to finish. The service schedules both jobs; available resources determine whether they run concurrently. Timeouts above are in seconds and limit the local wait. The examples use Jupyter's display(); in a script, use print() instead.

For an image already in PoreStore, replace the load and upload with im = porestore.get_image("your-saved-image-uuid"). This retrieves metadata without downloading the array. Keep that image ID for subsequent sessions: rerunning the upload cell creates another image record.

If starting with a local PoreSpy calculation, pass its Boolean pore array directly to upload(). For example, the following generates a small test image that can replace local_array in the upload above:

import porespy as ps

local_array = ps.generators.blobs(
    shape=[128, 128, 128],
    porosity=0.6,
    blobiness=2,
    seed=0,
)

PoreSpy generation runs locally; transport calculations submitted through the SDK run remotely. A synthetic image is useful for exercising the workflow, but its geometry and voxel size should be chosen for the scientific problem.

Inspect results and their units

result.solution is a dictionary with a compact table display, similar to PoreSpy's Results object. When the API supplies scientific summary tables, the SDK displays those first, using the same labels and formatting as PoreStore's results UI. Solver-specific outputs, aliases, diagnostics, and provenance such as input_sha256 remain in a collapsed Details table in notebooks.

Every dictionary, list, tuple, and array inside a notebook table has its own expandable subtable. Lists use indexed rows; NumPy arrays show coordinate indices. Scientific summaries start open, while details and nested collections start closed. Direct dictionary/list lookups also produce table-aware values, so display(result.solution["flow_percolation"]) or display(result.solution["slice_fluxes_lu"]) is inspectable on its own. Dictionary/list behaviour, mutation, and JSON serialization remain available. Unusually large or deep structures have bounded notebook previews with an explicit notice; their complete data remains available in Python.

print(result.solution) prints the summary and a hint for inspecting the raw data. Use print(result.solution.details) or display(result.solution.details) to inspect the complete raw table explicitly. The details dictionary is a copy of the raw fields, including their original names. The exact diagnostics and provenance fields depend on what each solver reports.

Without a valid API summary, the raw property/value table remains visible. Raw floating-point values are displayed with six significant figures. Text output summarizes collections; notebook tables let you expand their contents. Original values, precision, and units remain available through normal dictionary access such as result.solution["effective_diffusivity"] or result.solution["input_sha256"]. JSON serialization preserves the complete data. Cached results have the same display.

For numerical work, use the underlying fields directly. The available keys depend on the analysis and solver, so inspect them before selecting a quantity:

print(list(result.solution))
display(result.solution.details)
print(result.solution.get("units"))
print(diffusivity.metadata["parameters"])

Retain the API's unit labels when plotting or exporting values. With a known image voxel size, the service converts scientific results into the reported physical units; without one, length-dependent quantities may remain in voxel units. Do not multiply a returned transport result by the voxel size again. Downloaded pore networks have a separate stored-unit convention, described under Network extraction.

Connection settings and available analyses

The SDK reads .env automatically when a connection is created; notebooks and scripts do not need to call load_dotenv. Explicit api_key=, team_id=, and base_url= arguments take precedence over environment variables, which take precedence over .env. File values remain local to the connection and are not exported into os.environ. Only the current directory is checked; parent directories are not searched. If .env is absent, explicit arguments and environment variables continue to work.

Explicit configuration is also available, for example when selecting a local service or a different cache directory:

client = porestore.configure(
    base_url="http://localhost:8000",
    cache_dir="./porestore-cache",
    timeout=60,
)  # Credentials still come from the environment or .env.

Use this as an alternative to the initial configure() call. Tool arguments follow the connected service's schemas. Inspect those schemas before selecting solver-specific settings:

analyses = {item["slug"]: item for item in client.analysis_types()}
print(list(analyses))
display(analyses["permeability"]["parameters_schema"])

Reuse existing calculations or request a new run

Tools reuse existing jobs by default. PoreStore computes a SHA-256 fingerprint from the image ID, analysis type, and exact server-normalized arguments, including defaults, backend, and requested compute tier. Supported legacy aliases normalize to their current names. Matching includes jobs launched on the website; there is no approximate comparison of numerical settings.

The newest completed match with stored results is preferred. Its JSON results are fetched from the API and the returned Job has reused=True. If no completed result exists, a matching active job is returned for explicit waiting. Otherwise, a matching failed or cancelled job is returned with its original error; rerunning requires refresh=True. A completed result that did not converge is retained with its convergence information. Historical arguments that cannot be established exactly are not guessed from current defaults.

If there is no match, a new job is submitted with the normal access, readiness, validation, resource, and credit checks. refresh=True always requests a new run:

permeability = porestore.tools.transport.permeability(
    im=im,
    direction="x",
    max_iterations=200_000,
    refresh=True,
)

Reuse is scoped to the same image ID. Uploading the same array again creates a new image ID; recover the existing image with porestore.get_image(image_id) to reuse its jobs. Lookup and result-access errors are raised without submitting a replacement calculation. The server must support analysis-jobs/matches/; older deployments return an API error rather than silently creating another job.

Keyword arguments are normalized using the same forms as website and API submission. Unknown arguments are rejected. Use client.analysis_types() to inspect currently available analyses and their parameter schemas. Porosimetry follows the UI's Multiphase category and maps to API analysis hg_porosimetry.

Recover images and jobs in a later session

To bring an image back into a notebook, including in a later session:

im = porestore.Image.from_id("saved-image-uuid", client=client)
local_array = im.download()  # NumPy ndarray; explicit full-image download
np.save("downloaded.npy", local_array, allow_pickle=False)

job = porestore.Job.from_id("saved-job-uuid", client=client)
result = job.get_result(wait=True, timeout=1800, poll_interval=5)

# A saved image ID can also be submitted directly.
job = porestore.tools.transport.diffusivity(
    im=im.id,
    client=client,
    direction="y",
)

For a reproducible notebook, save the IDs and submitted parameters after the tool calls return, before waiting for completion:

import json
from pathlib import Path

run = {
    "image_id": im.id,
    "jobs": {
        "diffusivity": diffusivity.id,
        "permeability": permeability.id,
    },
    "parameters": {
        "diffusivity": diffusivity.metadata["parameters"],
        "permeability": permeability.metadata["parameters"],
    },
}
Path("porestore-run.json").write_text(json.dumps(run, indent=2), encoding="utf-8")

After restarting the kernel, configure the connection and recover the same records. No upload or new calculation is needed:

import json
from pathlib import Path

import porestore

client = porestore.configure()
run = json.loads(Path("porestore-run.json").read_text(encoding="utf-8"))
im = porestore.get_image(run["image_id"])
diffusivity = porestore.get_job(run["jobs"]["diffusivity"])
permeability = porestore.get_job(run["jobs"]["permeability"])
result = diffusivity.get_result(wait=True, timeout=86400, poll_interval=10)
display(result.solution)

# Download only if the local array is needed for further processing.
local_array = im.download()

Results can also be exported with their complete metrics, unit labels, and job metadata. This JSON export does not include the image or a network artifact:

export = {
    "job_id": result.job_id,
    "job_metadata": diffusivity.metadata,
    "metrics": result.metrics,
}
Path("diffusivity-result.json").write_text(
    json.dumps(export, indent=2),
    encoding="utf-8",
)

Image transfer and readiness

porestore.upload(data, ...) accepts a NumPy array or a local filename and returns an Image. For example, porestore.upload("sample.npy", wait=True) uploads the file directly; the API selects the decoder from its extension. RAW files also need raw_metadata with their dimensions and dtype, following the API upload contract. The configured connection is used automatically; pass client= only when selecting a different connection.

Image transfer and remote preparation are separate stages. After receiving an upload, PoreStore validates the image and prepares its storage and metadata. With wait=True, upload() waits for that preparation before returning a ready image, so the next line can submit an analysis. Without it, the returned image may still be processing; use image.wait_until_ready() when needed. Both waiting forms accept timeout and poll_interval, and report ingestion failures or timeouts rather than returning an apparently ready image.

Image(array) is equivalent to Image.from_numpy(array); both remain available when a local Image object is useful before uploading. Arrays must be nonempty 2-D or 3-D boolean, integer, or floating-point NumPy arrays. Uploads use NPY without pickle, pass directly to the signed storage target, and obey the server's file-size limit. Calling upload() again on an already uploaded Image reuses its remote record. To upload local edits as a separate image, construct a new Image from the edited array.

download() retrieves the current stored sample array, preserving its shape, dtype, and axis order; this may differ from the original uploaded data after server-side ingestion or processing. Decompression happens locally, and the complete array must fit in notebook memory. The downloaded array is retained on im.array; repeated downloads reuse it until download(refresh=True) or an explicit metadata refresh reports a changed image version. This first version retrieves chunks sequentially through the normal API rate limit.

Image download requires the companion v1 image-data endpoint in PoreStore issue #1485 to be deployed. Tool calls also require the read-only matching endpoint in PoreStore issue #1489. Both are included in API PR #1486. Upload uses existing v1 endpoints. The SDK uses no browser-only or private worker routes.

Compare transport in three directions

For a 3-D image, submit each direction before collecting results. Each distinct direction is a separate calculation, subject to the usual credit and resource checks when no reusable job exists:

directional_jobs = {
    direction: porestore.tools.transport.diffusivity(
        im=im,
        direction=direction,
        target_relative_error=0.001,
    )
    for direction in ("x", "y", "z")
}
for direction, job in directional_jobs.items():
    print(direction, job.id, job.status, "reused:", job.reused)

directional_results = {}
for direction, job in directional_jobs.items():
    directional_results[direction] = job.get_result(
        wait=True,
        timeout=86400,
        poll_interval=10,
    )
    print("Direction:", direction)
    display(directional_results[direction].solution)

Use porestore.tools.transport.permeability in the same pattern to compare permeability, supplying that solver's settings such as max_iterations. The dictionary keys retain the association between direction and result.

Mercury porosimetry and a local intrusion plot

Porosimetry uses the same uploaded pore image. Set the number of pressure steps through the service's steps parameter:

porosimetry = porestore.tools.multiphase.porosimetry(im=im, steps=25)
porosimetry_result = porosimetry.get_result(
    wait=True,
    timeout=86400,
    poll_interval=10,
)
display(porosimetry_result.solution)

With Matplotlib installed, plot the returned intrusion curve. Filter missing or non-finite values and nonpositive pressures before using a logarithmic axis, and take the pressure unit from the result:

import matplotlib.pyplot as plt
import numpy as np

solution = porosimetry_result.solution
pressure = np.asarray(solution["pc_intrusion"], dtype=float)
saturation = np.asarray(solution["snwp_intrusion"], dtype=float)
valid = np.isfinite(pressure) & np.isfinite(saturation) & (pressure > 0)

fig, ax = plt.subplots(figsize=(4, 4))
ax.semilogx(pressure[valid], saturation[valid], "o-")
ax.set_xlabel(f"Intrusion pressure [{solution['units']['pressure']}]")
ax.set_ylabel("Mercury saturation")
fig.tight_layout()
plt.show()

Network extraction

Network extraction follows the UI's Networks category and the same Job/Result workflow, exact matching, refresh=True, readiness and access rules as the other tools. The service's existing network-extraction audience applies; adding the SDK entry point does not grant access to a staff-only analysis.

extraction = porestore.tools.networks.extraction(
    im=im,
    method="snow2",  # or "magnet", subject to service availability
    boundary_width=3,
    sigma=0.4,
    r_max=4,
)
result = extraction.get_result(wait=True, timeout=86400, poll_interval=10)
display(result.solution)  # Small scientific summary; no network download yet.
network = result.download_network()
display(network)  # Dictionary with NumPy arrays and expandable tables.
print("Pores:", network["pore.coords"].shape[0])
print("Throats:", network["throat.conns"].shape[0])
print(result.network_metadata["length_unit"])

# Optional: OpenPNM is installed separately, not required by the SDK.
import openpnm as op

pn = op.io.network_from_porespy(network)

download_network() returns a dictionary of NumPy arrays, including pore.coords, integer throat.conns, and pore/throat labels and properties. The returned dictionary uses the same table display as solutions. The network must fit in notebook memory. After download it is available as result.network; beforehand this is None, and reading it performs no network request.

Values retain their stored units. Current extractions store coordinates and lengths in voxels. For 3D images, areas are in voxel squared and volumes are in voxel cubed. OpenPNM uses the numbers supplied; importing does not convert them to metres. result.network_metadata reports length_unit (voxel, um, or None for an unknown historical basis), the artifact version, and the original extraction spacing for legacy physical networks. Editing the image's current voxel size does not rescale the stored network.

Decoded arrays are cached as NPZ without object arrays under the configured cache directory. A new Result checks the API artifact version before reusing a cache entry; repeated calls on the same Result reuse its in-memory dictionary. download_network(refresh=True) fetches that artifact again and never starts a new extraction. Corrupt or invalid cache entries are downloaded again.

The current service stores network dictionaries as a pickle of JSON-compatible values; legacy JSON and numeric NPZ artifacts are also supported. The SDK rejects pickle constructors/globals and object arrays, so historical pickles containing NumPy objects require a fresh extraction or a data-only JSON/NPZ export. It does not execute downloaded Python objects.

Network retrieval requires the companion API endpoints in PoreStore issue #1490, included in API PR #1486. Ordinary result retrieval remains JSON-only; no network file is fetched during submission, waiting, cache matching, or get_result().

Lifecycle, retries, and caching

Reading id, status, or metadata never contacts the service. Call refresh() to update metadata. wait_until_ready() and wait() poll only when requested and have finite timeouts. Each HTTP request also has its own timeout (Client(timeout=60) by default). Unknown future status values remain pending.

get_result() raises ResultNotReadyError for pending work, JobFailedError for failures, and its subclass JobCancelledError for cancellations. Use get_result(wait=True) to wait explicitly. Image download raises ImageNotReadyError while ingestion runs and ImageProcessingError if it fails. API errors retain status_code, code, and details from the service.

A local wait timeout does not cancel the remote job. Retain its ID and resume waiting on the same job. For example, when waiting on the diffusivity job:

try:
    result = diffusivity.get_result(wait=True, timeout=600, poll_interval=5)
except porestore.WaitTimeoutError:
    print("Local wait timed out; resume with job ID:", diffusivity.id)
except porestore.JobFailedError as error:  # Also catches JobCancelledError.
    print("Job did not complete:", diffusivity.id, str(error))
except porestore.APIError as error:
    print(error.status_code, error.code, error.details)
else:
    display(result.solution)

# In a later cell, refresh status without submitting another calculation.
diffusivity.refresh()
print(diffusivity.id, diffusivity.status)

The refresh operations act on different parts of the workflow:

On a cache miss, submissions use the normalized request fingerprint as their Idempotency-Key, so concurrent identical SDK misses replay one job. A forced run uses a fresh random key. The client retries transient responses and connection errors up to twice, retaining the same key and request. Supply idempotency_key="your-stable-key" to retain the key yourself. If retries are exhausted, the raised TransportError or submission APIError carries idempotency_key and the resolved request_parameters; reuse that key with **error.request_parameters after an uncertain response. For a forced run, retain refresh=True and the same key to recover that particular attempt; using the key again intentionally replays it. Upload creation and completion are not automatically retried.

Completed JSON results are cached under ~/.cache/porestore, or the Client(cache_dir=...) directory. Cache identity includes service, team, credential identity, job, completion time, worker provenance, and current image metadata and voxel size. A recovered job can reuse a valid cache entry in a new Python session. Corrupt cache entries are downloaded again; get_result(refresh=True) explicitly downloads the existing result again. This differs from a tool's refresh=True, which starts a new calculation. Job.refresh() checks updated job and image metadata before a cached result is reused; a result already held in memory remains a snapshot until refreshed. Automatic tool reuse always fetches the result from the API. Results retain the service's units, parameters, and provenance in the result metrics and job metadata. Large field exports and local visualization are not implemented in this version. Images are currently cached only in memory.

API requests and storage transfers use separate HTTP clients so API credentials are never sent to storage. Explicit connections can be managed with with porestore.Client(...) as client:; call client.close() when finished with a configured notebook connection.

Development

Install uv, then run:

cd /path/to/porestore_sdk
make init
uv run python -c "import porestore; print(porestore.__file__)"

The project requires Python 3.12 or later. The .python-version file selects Python 3.12 for development. uv sync --locked creates an isolated .venv, installs the package in editable mode with its development tools, and uses the committed uv.lock. Runtime dependencies are NumPy, HTTPX, python-dotenv, and Zarr 2, matching the service's current image storage format. The SDK does not require the adjacent PoreStore application to be installed.

make check   # Ruff lint and formatting checks
make format  # Format Python source
make test    # Test the client against mocked public API and storage requests
make build   # Build a wheel and source distribution in dist/
make check-dist  # Validate and install both built distributions in clean environments
make docs    # Build Zensical documentation and generated API reference
make docs-serve  # Preview the documentation locally

Tests use small arrays and mocked HTTP transports. They do not contact a live service, upload customer data, or submit paid calculations.

From another project, install the SDK locally with:

uv add /path/to/porestore_sdk

The application already uses the distribution and module name porestore. Keep SDK development and consumer environments separate from the application environment to avoid import and package-name collisions.

Releases and documentation

GitHub Actions checks tests, formatting, distribution contents and documentation. Release Please prepares version updates and release notes in a pull request; merging that PR publishes the alpha package to PyPI with Trusted Publishing. Once the first alpha is published, install it with python -m pip install --pre porestore. The SDK uses the MIT license.

The documentation site uses Zensical, includes this notebook guide, and generates API reference pages from public Python signatures and docstrings. Actions builds the site on each PR and deploys main to GitHub Pages once Pages is configured. See Maintaining the SDK for one-time GitHub and PyPI setup, release conventions, stable-release policy, and publication recovery. The intended documentation address is https://porolytics.github.io/porestore-sdk/.