interloper-aws
AWS integration for interloper: an S3Destination that stores asset data in
an Amazon S3 bucket, and the AWSConnection that holds the credentials.
S3Destination is an ObjectStoreDestination, the same base as the Cloud
Storage destination in interloper-google-cloud, so the two behave
identically: one object per partition in a hive-partitioned layout, Parquet
(the default), JSONL or CSV, the partition column in the path only, and each
object's row count stored as row_count user metadata.
s3://{bucket}/{prefix}/{dataset}/{table}/data.{ext}
s3://{bucket}/{prefix}/{dataset}/{table}/{column}={partition}/data.{ext}
Usage
import interloper as il
from interloper_aws import AWSConnection, S3Destination
destination = S3Destination(
connection=AWSConnection(region="eu-central-1"),
bucket="acme-lake",
prefix="raw",
format="parquet",
)
source = Shop(destinations=destination)
Credentials
The connection takes an access key pair (access_key_id,
secret_access_key, plus session_token for temporary credentials) and a
region, defaulting to eu-central-1. Each field also loads from the
standard environment variables: AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN and AWS_REGION.
Leave the key pair empty and boto3's default credential chain applies: environment, shared config files, then the instance or pod role (an EC2 instance profile, EKS IRSA or Pod Identity). In a cluster, that means no stored secret at all: grant the workload's role access to the bucket.
The connection's check calls STS GetCallerIdentity, which needs no IAM
permission, so a failing check means the credentials themselves are wrong.
The bucket picker calls ListBuckets.
Permissions
The destination needs, on the bucket and its objects:
s3:PutObjectto writes3:GetObjectto read, and to count objects written withoutrow_countmetadatas3:ListBucketfor partition row countss3:ListAllMyBucketsfor the bucket picker only
Notes
ListObjectsV2 returns no user metadata, so partition row counts issue one
HeadObject per listed object. That is still no download: the count comes
from the row_count metadata stamped at write time. Objects written by other
tools, without that metadata, are downloaded and counted.
Metadata
Release files for interloper-aws 0.114.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| interloper_aws-0.114.0.tar.gz | 4.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| interloper_aws-0.114.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 11.2 kB
Release files / interloper_aws-0.114.0.tar.gz
| Download URL | interloper_aws-0.114.0.tar.gz |
|---|---|
| Size | 4.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6ceee754c564a8cba6a0766e274736e4218a0196cd8c928b83c92e36034cd172
|
|
BLAKE2b-256 checksum How to use checksums |
d31ca4c8922c87e8c447129f6061a55431af893ef636760b8d2840165e9d2cf4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.1 {"installer":{"name":"uv","version":"0.13.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / interloper_aws-0.114.0-py3-none-any.whl
| Download URL | interloper_aws-0.114.0-py3-none-any.whl |
|---|---|
| Size | 6.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2212474488a1e625d160615494f870982aefda891734b63a67772be98ef6cb46
|
|
BLAKE2b-256 checksum How to use checksums |
77ea2845d6e360ac99c9cf1320047b767f9ba0019226f2b7da1d0b59be51d3f2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.1 {"installer":{"name":"uv","version":"0.13.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
