Skip to main content

pyamselect

Python client library and CLI for AirMettle Select. Run SQL queries directly against data in your Azure Blob Storage — CSV, JSON lines, and Parquet, including gzip-compressed CSV and JSON lines — without moving or ingesting it.

Installation

pip install pyamselect

Requires Python 3.10+. Dependencies (httpx[http2], crc32c) are installed automatically.

Verify the install:

amselect --version

Quick Start

You need the following, provided with your AirMettle Select subscription:

  • Your subscription ID and API key
  • The query endpoint (host and port) and the metadata service endpoint (URL)
  • Data in an Azure storage account that is registered under your subscription

1. Prepare Your Data

Before an object can be queried, the service generates query metadata for it. Run this once per object:

amselect prepare \
    --endpoint "https://<metadata-endpoint>" \
    --blob-url "https://<account>.blob.core.windows.net/<container>/<blob>" \
    --subscription-id "<subscription-id>" \
    --api-key "<api-key>"

The service accesses the blob with the access key stored in the storage account's bucket registration, so the storage account must be registered under your subscription first.

The command prints a JSON result and waits (up to 5 minutes) for the operation to complete. Pass --overwrite to regenerate metadata after an object has changed, or --no-wait to return as soon as an asynchronous prepare is accepted.

2. Run a Query

amselect query -H "<query-host>" -r '{
  "select_request": {
    "expression": "SELECT * FROM Object WHERE age > 30",
    "container": "<container>",
    "blob": "<blob>",
    "storage_account": "<account>",
    "subscription_id": "<subscription-id>",
    "api_key": "<api-key>",
    "input_options": {"type": "csv", "value": {"csv_header_config": "use"}},
    "output_options": {"type": "csv", "value": {"recordDelimiter": "\n", "fieldDelimiter": ","}}
  }
}'

Results stream to stdout; pass -o results.csv to write them to a file instead. Larger requests can be kept in a file and passed with --request-file request.json.

3. Query from Python

from pyamselect import AMSelectClient, SelectRequest, CSVInputOptions, CSVOutputOptions

request = SelectRequest(
    expression="SELECT * FROM Object WHERE age > 30",
    container="<container>",
    blob="<blob>",
    storage_account="<account>",
    subscription_id="<subscription-id>",
    api_key="<api-key>",
    input_options=CSVInputOptions(csv_header_config="use", csv_field_delimiter=","),
    output_options=CSVOutputOptions(),
)

with AMSelectClient("<query-host>") as client:
    result = client.select_to_string(request)
    print(result)

An async client with the same API is also available — see Async Client.

Building Requests

Dataclasses vs Raw Dicts

Input/output options can be passed as typed dataclasses (IDE autocomplete, validation) or raw dicts (the same JSON structure sent on the wire).

# Dataclass
from pyamselect import CSVInputOptions
input_options = CSVInputOptions(csv_header_config="use", csv_field_delimiter=",")

# Equivalent raw dict
input_options = {"type": "csv", "value": {"csv_header_config": "use", "csv_field_delimiter": ","}}

CSV Convenience Factory

request = SelectRequest.csv_query(
    expression="SELECT name, age FROM Object",
    container="mycontainer",
    blob="people.csv",
    storage_account="myaccount",
    subscription_id="sub-123",
    api_key="key-456",
)

JSON and Parquet Input

from pyamselect import JSONInputOptions, JSONOutputOptions, ParquetInputOptions, CSVOutputOptions

# JSON lines (one JSON record per line)
request = SelectRequest(
    expression="""SELECT t."Timestamp", t.devname FROM Object as t WHERE t.windows_event_id='4624'""",
    container="mycontainer",
    blob="events.jsonl",
    storage_account="myaccount",
    input_options=JSONInputOptions(json_type="lines"),
    output_options=JSONOutputOptions(record_delimiter="\n"),
)

# Parquet
request = SelectRequest(
    expression="SELECT col1, col2 FROM Object",
    container="mycontainer",
    blob="data.parquet",
    storage_account="myaccount",
    input_options=ParquetInputOptions(),
    output_options=CSVOutputOptions(),
)

Compressed Input

CSV and JSON lines objects can be gzip-compressed. Set the compression on the input options:

from pyamselect import CSVInputOptions, JSONInputOptions

# gzip-compressed CSV, e.g. orders.csv.gz
input_options = CSVInputOptions(csv_header_config="use", csv_compression_type="gzip")

# gzip-compressed JSON lines, e.g. events.jsonl.gz
input_options = JSONInputOptions(json_type="lines", json_compression_type="gzip")

In request JSON these are the csv_compression_type and json_compression_type fields of the input options value.

Request JSON Format

The request JSON sent to the service has the following format:

{
  "select_request": {
    "expression": "SELECT * FROM Object",
    "expression_type": "sql",
    "container": "mycontainer",
    "blob": "myfile.csv",
    "storage_account": "myaccount",
    "subscription_id": "sub-123",
    "api_key": "key-456",
    "input_options": {
      "type": "csv",
      "value": { "csv_header_config": "use", "csv_field_delimiter": "," }
    },
    "output_options": {
      "type": "csv",
      "value": { "recordDelimiter": "\n", "fieldDelimiter": "," }
    }
  }
}

The select_request envelope is optional when using SelectRequest.from_json() -- bare inner objects are accepted too.

Client API

Both clients share the same methods: select(), select_to_string(), select_to_file(), and select_to_stream().

Sync Client

with AMSelectClient("<query-host>") as client:
    # Collect all data as a string
    result = client.select_to_string(request)

    # Write directly to a file
    client.select_to_file(request, "output.csv")

    # Write to any writable binary stream
    client.select_to_stream(request, stream)

Streaming Events

For full control over the event stream (data, stats, continue events):

from pyamselect import AMSelectClient, EventType

with AMSelectClient("<query-host>") as client:
    for event in client.select(request):
        if event.event_type == EventType.DATA:
            print(event.payload.decode("utf-8"), end="")
        elif event.event_type == EventType.STATS:
            print(f"Stats: {event.payload}")
        elif event.event_type == EventType.END:
            print("Done.")

Async Client

import asyncio
from pyamselect import AsyncAMSelectClient

async def main():
    async with AsyncAMSelectClient("<query-host>") as client:
        result = await client.select_to_string(request)

        await client.select_to_file(request, "output.csv")

        await client.select_to_stream(request, stream)

asyncio.run(main())

Async Streaming

import asyncio
from pyamselect import AsyncAMSelectClient, EventType

async def main():
    async with AsyncAMSelectClient("<query-host>") as client:
        async for event in client.select(request):
            if event.event_type == EventType.DATA:
                print(event.payload.decode("utf-8"), end="")

asyncio.run(main())

Preparing Objects

MetadataClient generates query metadata for an object from Python, the same as amselect prepare:

from pyamselect import MetadataClient, AMSelectPrepareError

with MetadataClient("https://<metadata-endpoint>") as client:
    try:
        result = client.prepare({
            "blob_url": "https://<account>.blob.core.windows.net/<container>/<blob>",
            "subscription_id": "<subscription-id>",
            "api_key": "<api-key>",
        })
        print(result)
    except AMSelectPrepareError as e:
        print(f"{e}: {e.result}")

prepare() submits the request asynchronously and by default waits (up to 5 minutes) for it to finish; pass wait=False to return once it is accepted, or timeout= to wait for a different length of time. Add "force": True to the request to regenerate metadata after an object has changed. A prepare that is still running when the wait ends raises AMSelectPrepareTimeoutError, a subclass of AMSelectPrepareError.

Configuration

These options apply to both AMSelectClient and AsyncAMSelectClient.

Timeouts

with AMSelectClient("<query-host>", connect_timeout=10.0, read_timeout=300.0) as client:
    result = client.select_to_string(request)

Error Handling

from pyamselect import AMSelectHTTPError, AMSelectStreamError, AMSelectConnectionError

with AMSelectClient("<query-host>") as client:
    try:
        result = client.select_to_string(request)
    except AMSelectConnectionError as e:
        print(f"Connection failed: {e}")
    except AMSelectHTTPError as e:
        print(f"HTTP {e.status_code}: {e.body}")
    except AMSelectStreamError as e:
        print(f"Query error {e.error_code}: {e.error_message}")

CLI Reference

Installing the package provides the amselect command with two subcommands: query and prepare. Query options can also be passed directly to amselect without the query subcommand. All commands exit non-zero on failure.

amselect --version     # Show version
amselect --buildinfo   # Show detailed build information

query

Runs a select request against the query endpoint and streams the results.

# Inline request (results to stdout)
amselect query -H "<query-host>" -r '{"select_request": {...}}'

# Request from a file
amselect query -H "<query-host>" --request-file request.json

# Write results to a file
amselect query -H "<query-host>" --request-file request.json -o results.csv
-H, --host              Query endpoint host (required)
-P, --port              Query endpoint port (default: 443)
-r, --request           JSON request string
    --request-file      Path to JSON request file
-o, --output            Output file path (default: stdout)

prepare

Generates the query metadata for an object via the metadata service. An object must be prepared before it can be queried, and re-prepared (with --overwrite) after its contents change.

amselect prepare \
    --endpoint "https://<metadata-endpoint>" \
    --blob-url "https://<account>.blob.core.windows.net/<container>/<blob>" \
    --subscription-id "<subscription-id>" \
    --api-key "<api-key>"

The service accesses the blob with the access key stored in the storage account's bucket registration; requests for storage accounts not registered under the subscription are rejected. The result is printed as JSON. By default the command waits (up to 5 minutes) for an asynchronous prepare to finish.

    --endpoint          Metadata service base URL (required)
    --blob-url          Full URL of the blob to prepare
    --subscription-id   Subscription ID (required with --blob-url)
    --api-key           Subscription API key (required with --blob-url)
-r, --request           JSON request string (alternative to --blob-url)
    --request-file      Path to JSON request file (alternative to --blob-url)
    --overwrite         Regenerate metadata if it already exists
    --no-wait           Return as soon as an asynchronous prepare is accepted

Common Options

Available on both subcommands:

    --ca-cert           Path to CA certificate bundle
    --insecure-skip-tls-verify
                        Disable TLS certificate verification
    --connect-timeout   Connection timeout in seconds (default: 60)
    --read-timeout      Read timeout in seconds (default: 60)
-v, --verbose           Increase verbosity (-v=INFO, -vv=DEBUG)

License

MIT — see the LICENSE file. Use of the AirMettle Select service itself is governed by your service agreement.

Metadata

Release files for pyamselect 1.2.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyamselect 1.2.7
File Size Uploaded Provenance Actions
pyamselect-1.2.7.tar.gz 14.6 kB Without provenance Details for pyamselect-1.2.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pyamselect 1.2.7
File Interpreter ABI Platform Actions
pyamselect-1.2.7-py3-none-any.whl Python 3 none any Details for pyamselect-1.2.7-py3-none-any.whl

Total release size: 33.1 kB

Release files / pyamselect-1.2.7.tar.gz

Download URL pyamselect-1.2.7.tar.gz
Size 14.6 kB
Tags Source
SHA-256 checksum
How to use checksums
e6878cc21e198365f2241bb11873b1af541efbcf9e36188a152dfab2bfd84781
BLAKE2b-256 checksum
How to use checksums
955669830f8700a7f6ec38141d28c7eb1710ff3aa839fd1c47167c65ebcbd9c3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / pyamselect-1.2.7-py3-none-any.whl

Download URL pyamselect-1.2.7-py3-none-any.whl
Size 18.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
24d99d29b4a0cfe0a449ea47ba9f066066f0926c9c8f8c1e27f5b18d4750e083
BLAKE2b-256 checksum
How to use checksums
75b3a345536e34e7c6c0fbe27af21ebdabb99304528372ce0dec0fb1a5390338
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for pyamselect 1.2.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.2.7 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page