Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

Build and Publish codecov

dynamicio — Wrap your I/O The dynamicio panda mascot

Keep your business logic independent of I/O. dynamicio is the seam between your pandas code and the places your data lives: describe each dataset once in YAML, and your code just calls read() and write().

The same code reads a parquet file on your laptop, a CSV prefix on S3, a PostgreSQL table or an Athena query. You pick which one by changing a single identifier, so developers stop worrying about connection strings, file-format keyword arguments and which bucket a task reads this week.

Documentation: dynamicio.readthedocs.io: guides, examples and API reference · Changelog: CHANGELOG.md

Why dynamicio

  • Logic that does not know about storage. Resource definitions are YAML, per environment (LOCAL, DEV, PROD, or your own). Moving a pipeline between environments never touches Python.
  • Fast, safe feedback loops. Run the real pipeline against sample files on your machine, with no cloud credentials, no deploy and no mocks of external services. Compare the outputs with expected files using the built-in test harness (dynamicio.testharness).
  • Made for the LLM era. A coding agent that changes a pipeline needs somewhere safe to run it and a check that says whether the change was right. A LOCAL environment and expected outputs give it both, and production credentials never enter the loop.
  • Contracts for your dataframes. A Pandera model declares columns, dtypes and validations. Bad data fails at the boundary with every violation listed, not three steps later, and the same model can emit metrics.
  • Pandas behaviour, not a new dialect. Options in the YAML go to pandas / awswrangler, so what you know still applies.
  • Sync and async. Read and write from plain code or from asyncio services.

Supported sources

Source Read Write Formats
Local files yes yes parquet, hdf, json, csv
S3 (single object or prefix) yes yes parquet, hdf, json, csv
PostgreSQL yes yes tables
AWS Athena yes no SQL query results
Kafka no yes messages

Dataframe execution currently supports pandas only; Polars is not yet supported.

Install

pip install dynamicio
# or, in a uv project
uv add dynamicio

Python 3.10 to 3.14.

See it work

Describe the resource in orders.yaml:

ORDERS:
  LOCAL:
    type: local
    local:
      file_path: "[[ DATA_DIR ]]/orders.parquet"
      file_type: parquet

Bind it to a schema, then read and write:

import pandas as pd
import pandera.pandas as pa
from pandera.typing import Series

from dynamicio import UnifiedIO
from dynamicio.config import IOConfig


class Settings:
    DATA_DIR = "."


class Orders(pa.DataFrameModel):
    order_id: Series[int] = pa.Field(nullable=False, unique=True)
    amount: Series[float] = pa.Field(nullable=False, ge=0)


class OrdersIO(UnifiedIO):
    schema = Orders


config = IOConfig("orders.yaml", env_identifier="LOCAL", dynamic_vars=Settings)
orders = OrdersIO(source_config=config.get(source_key="ORDERS"), apply_schema_validations=True)

orders.write(pd.DataFrame({"order_id": [1, 2, 3], "amount": [9.5, 20.0, 3.25]}))
df = orders.read()

To run the same code against S3, add a DEV block to the YAML and set env_identifier="DEV". The Python does not change.

Where next

  1. The idea: why dynamicio.
  2. New here? The quickstart explains each step above, and its code is run by the test suite.
  3. Core ideas: concepts: resource definitions, IO classes, schemas.
  4. Task guides: reading, writing, testing pipelines.
  5. Reference: API reference.

Community and contributing

Original logo illustrated by Nick Loucas; the refreshed logo retains its circular I/O motif.

Changelog

v11.0.0

Pandera DataFrameModel classes are now the only way to declare a schema. Upgrading from 8, 9 or 10: see Upgrading; dynamicio migrate converts the YAML schemas of version 8 and earlier.

  • breaking: dict schemas (schema = {...}, schema_validations, schema_metrics) and YAML schemas (schema = SCHEMA_FROM_FILE, and the schema: block of a resource definition) are removed. An IO class whose schema is anything but a DataFrameModel or SCHEMA_FROM_ARGUMENT raises a TypeError when the class is defined, and a resource definition with a schema: key raises a ValueError when it loads. SCHEMA_FROM_FILE is removed from dynamicio.core, SafeDynamicSchemaLoader from dynamicio.config.io_config, and IOBinding.dynamicio_schema / IOEnvironment.dynamicio_schema with it. SCHEMA_FROM_ARGUMENT and the schema= argument are unchanged
  • feat: a model names its dataset with a class attribute, __dataset_name__ = "foo", instead of class Config: name = "foo". The name is resolved from __dataset_name__, then Config.name (still honoured, and treated as unset when equal to the class name), then the class name in snake case
  • breaking: a model with neither __dataset_name__ nor Config.name is now labelled with its class name in snake case, upper-cased in logs and METRIC lines: FinalBar is FINAL_BAR, where it was FINALBAR. Dashboards that parse METRIC lines for such models need the new label, or set __dataset_name__ to the old one
  • breaking: the Postgres table name is the new class attribute __table_name__, the exact name of the table in the database, and no longer the dataset name. A model used with a Postgres resource without it raises TableNameNotFoundError when the IO object is created, and a read of a missing table logs the __table_name__ to correct. DataframeSchema gains table_name
  • feat: typed metadata for the parts of a field that Pandera has no argument for: from dynamicio.schema import Metric, metrics, null_fraction_below, exactly_values, strictly_between, combined with |, as in pa.Field(nullable=True, lt=1000, metadata=metrics(Metric.MIN, Metric.MAX) | null_fraction_below(0.2)). exactly_values is is_in with match_all=False and strictly_between is is_between for bounds that are not known on definition. Validator names, error payloads and metric lines are unchanged. dynamicio.schema.metrics is now the helper function on the package; import the metric classes with from dynamicio.schema.metrics import ... (or dynamicio.metrics), which keep working, not through the dynamicio.schema.metrics attribute
  • breaking: the raw metadata={"dynamicio": {"metrics": [...], "validations": {...}}} dict is rejected with a ValueError that names the helpers above
  • feat: dynamicio migrate is the path from version 8: it converts YAML schemas to models for version 11 with __dataset_name__, the typed helpers, and __table_name__ for every schema that a resource definition under --scan binds to a Postgres environment. The generated models keep class Config: strict = "filter" (without name), so Model.validate(df) still drops unlisted columns. The prompt it writes lists the call sites for version 11 (SCHEMA_FROM_FILE, SafeDynamicSchemaLoader, schema: blocks, raw metadata dicts, Config)
  • feat: the dynamicio command line (dynamicio -s|-b -p PATH -o DIR) writes a Pandera model module per dataset (<dataset>.py) instead of a YAML schema
  • docs: the schemas guide, quickstart, examples and README describe models only; new ADR 0009. The demo project's models use __dataset_name__ and the typed helpers

v10.4.1

  • docs: the Documentation project link of the package, and the README, point to the documentation on Read the Docs (https://dynamicio.readthedocs.io/en/latest/), and the documentation home page shows the panda mascot next to the logo
  • docs: remove the changelog sections of the v8.1.0 release candidates.

v10.4.0

  • feat: the constructor of an IO class takes missing_ok: bool = False. With missing_ok=True, read() and async_read() return None and log a warning when the reader raises a FileNotFoundError. The caller handles the None, so the setting is in the code of the caller
  • chore: remove missing_ok from the resource definition. Release v10.2.0 added it, and it has no known user. A resource definition with a missing_ok key now fails to load, because dynamicio reads the key as an environment name. IOBinding.missing_ok and IOEnvironment.missing_ok are removed, and the dict() output of a binding has no missing_ok entry. Move the setting to the constructor of the IO class
  • docs: new "Why dynamicio" page and a rewritten home page and README that explain the purpose: business logic independent of I/O, fast local feedback loops, and what that gives coding agents. The documentation is restructured into getting-started, guides, examples, API reference, architecture decisions and development pages (docs/), built with MkDocs for Read the Docs (.readthedocs.yaml; versioned latest, stable and per-release documentation). The README is now a concise overview and the long walkthrough moved into the guides, with its content unchanged. The Python examples of the quickstart and the basic I/O page are executed by the test suite. Polars is still not supported.
  • docs: docstrings that the API reference could not parse (Returns:, Args: and Raises: entries of DynamicDataIO.async_read, IOConfig, WithS3, WithPostgres, WithKafka, allow_options, args_of and OutputSandbox) are corrected. args_of gains a Callable annotation on *funcs and OutputSandbox annotates directory; behaviour is unchanged.
  • chore: the package metadata gains Documentation, Source and Issues project URLs next to Changelog. The Documentation URL points at the docs/ directory until the Read the Docs site is confirmed.
  • ci: the "Build and Publish" workflow builds the documentation in strict mode on every push, builds and validates the distributions once (twine check, tag against version) and publishes exactly those files. A release tag must be vX.Y.Z or vX.Y.Z-rc.N, a stable tag must be on master and have a changelog section, and the GitHub release notes are the changelog section plus links (.github/tools/release_tools.py, tested). Re-running a publish is safe: files already on PyPI are skipped and an existing GitHub release is updated.
  • chore: the docs dependency group no longer includes mkdocs-techdocs-core and stays on mkdocs<2; docs/requirements.txt pins the documentation tooling from uv.lock.

v10.3.0

  • chore: remove the magic-logger dependency. Each dynamicio module logs through logging.getLogger(__name__), with the same logger names as before. magic-logger has only a source distribution, so an install with uv sync --no-build stopped with an error. magic-logger also called inspect.stack() on each log call. dynamicio.logger is a logging.Logger for the logger dynamicio, and it does not have the dict_config, file_config, listen and stop_listening methods. The logging config of dynamicio/__init__.py names the logger dynamicio, so disable_existing_loggers does not disable the module loggers. If you set DYNAMICIO_DO_NOT_CONFIGURE_LOGGING=1 and call logging.config.dictConfig after the dynamicio import, name the logger dynamicio in your config or set disable_existing_loggers to false. If you do not, your config disables the dynamicio loggers

v10.2.0

  • feat: a resource definition can set missing_ok: true on a source, beside its environments. When the reader of that source raises a FileNotFoundError (a missing local file, ObjectNotFoundError, NoObjectsFoundError), read() and async_read() return None and log a warning. All other errors raise as before. The default is false. IOBinding.missing_ok and IOEnvironment.missing_ok expose the flag, which adds a missing_ok: False entry to the dict() output of a binding. See "Optional sources: missing_ok" in the README
  • fix: an S3 read of a csv or json file_path raises dynamicio.errors.ObjectNotFoundError("No object found at s3://...") when the key does not exist. Before, it raised the awswrangler NoFilesFound, which is not a FileNotFoundError. ObjectNotFoundError is a FileNotFoundError and a NoFilesFound, so a caller can treat the key as a missing source, and code that catches NoFilesFound works as before. The errors of the parquet and hdf reads for a missing key do not change

v10.1.0

  • docs: refresh the standalone logo, clarify pandas-only support, remove unused and superseded images, and expose architecture decisions in the documentation navigation; the S3 benchmark page is replaced by ADR 0007, which keeps the headline figures (the full tables stay in PR #88).
  • feat: resource definitions accept an optional expected: {path, file_type, options} key per environment, the reference file a test compares the generated output against. Reading and writing ignore it, so existing definitions and their behaviour are unchanged. (IOEnvironment.expected is None when absent, which adds an expected: None entry to the model's dict() output.)
  • feat: the dynamicio.testharness package, to test a task end to end: run it in a LOCAL environment, then lazily compare each generated output with its expected file. ExpectedOutputs collects all mismatches and raises one OutputMismatchError; generated files of matching outputs are removed (the others are kept); JSON is read with the resource's orient (or expected.options); paths can carry {placeholders} filled from params; DYNAMICIO_UPDATE_EXPECTED=1 regenerates the expected files. Expectations curates how strict a comparison is (row/column order, dtypes, tolerances, sort_by, only_columns, ignore_columns, list_columns_to_sort, custom assertions; strict() / lenient() presets, with_() to derive variants), per session or per check; dict columns are compared by content. FrameComparison and OutputFiles are usable on their own. See "Testing Locally" in the README.
  • feat: dynamicio.testharness additions: ExpectedOutputs.check_each(config, key, grid) checks a resource whose path has several {placeholders} for every combination of their values (clear_each removes their leftovers); Expectations(check_parquet_schema=True) also compares the arrow schema and stored pandas index of parquet outputs; OutputSandbox(directory, *configs) redirects the local outputs of a task into a scratch directory so tests that write the same files can run in parallel; ExpectedOutputs(environment=...) makes every IOConfig that defines that environment select it inside the with block (and restores them on exit), so the task under test runs and is checked in an environment the test states, without patching env_identifier; check(..., environment=...) overrides it for one check, configs=[...] narrows the switch, EnvironmentSwitch(environment, *configs) does the same explicitly and rejects an environment a config does not define. Without environment=, the default is the first environment the resource definitions list, whatever the process runs in. IOConfig.instances() and IOConfig.default_environment support this. The README gains "Choosing the environment" and "Recommended testing practices".
  • fix: dynamicio.testharness review fixes: JSON Lines outputs (options.lines: true) are read back with lines (it is passed with orient to the JSON reader only); a column named in sort_by, only_columns or list_columns_to_sort that is missing from the output or the reference is reported as a failed check, not a KeyError that aborts the remaining checks; update mode (DYNAMICIO_UPDATE_EXPECTED=1) refuses to copy an output over a reference declared in another file type, leaving both files untouched.
  • feat: ExpectedOutputs.check_all(config, skip=, rules_for=, params=) checks every resource of a config that declares expected (a new output is covered without touching the test; resources with unfilled {placeholders} are listed in skipped), and ExpectedOutputs.clear_all(config) removes the stale generated files of those resources.
  • feat: a passing test run now says what it verified. ExpectedOutputs prints a SessionReport at exit (green ✓ per matched output with its rows x columns and expected file, red ✗ per failure, ↻ updated, - skipped), outputs.report returns it, OutputMismatchError also lists the outputs that matched, and a pytest plugin (pytest11 entry point dynamicio; -p no:dynamicio turns it off) prints every report in the terminal summary, coloured. Nothing is printed when a session checked nothing.
  • docs: the README walkthrough follows the demo/ project: four resource definitions (staging_input, staging_output, transform_input, transform_output) with LOCAL, DEV and PROD environments, expected paths on the outputs' LOCAL environment, and the staging_*/transform_* configs, instead of sample/actual and input/raw/processed.
  • docs: the demo/ project now has one input and one output definition per runner (staging_input, staging_output, transform_input, transform_output) with LOCAL, DEV and PROD environments, and its end-to-end test uses ExpectedOutputs. The definitions use YAML anchors and merge keys (&, *, <<:) so PROD only restates what differs from DEV.

v10.0.1

  • fix: an S3 path-prefix read with no objects under the prefix raises dynamicio.errors.NoObjectsFoundError("No objects found under s3://..."). Before, it raised the pandas error ValueError: No objects to concatenate, which a caller cannot tell apart from other ValueErrors. The change applies to all the file types (parquet, csv, json, hdf) and to the hdf no_disk_space read. NoObjectsFoundError is a FileNotFoundError and a ValueError, so a caller can treat the prefix as a missing source, and code that catches ValueError works as before. An S3 parquet file_path that ends in / and has no objects raises the same error. It raised a plain FileNotFoundError before. Objects that are all empty do not change: a parquet prefix returns an empty frame, and a csv, json or hdf prefix raises the pandas ValueError as before

v10.0.0

  • fix: dynamicio migrate writes a YAML object column as Series[object], not Series[str]. object meant "any values", and list columns declared str fail plain Pandera validation (and would be recast under pandas 3). Models already generated with Series[str] for list columns should be changed to Series[object]; the generated module now says where str is safe
  • docs: the README diagrams are now text-driven SVGs (supported sources with Athena next to S3, Postgres and Kafka; the sample pipeline as an Airflow DAG; how a read works; the file formats), they render on GitHub and on PyPI, and the Pandera logo is shown in the Pandera models step. The table of contents anchors and the CONTRIBUTING.md link are fixed, and the dependency list names what the package really uses. The demo/ example datasets are generic (orders).
  • fix: the dtype check compares dtypes instead of their names. A column already at string[pyarrow] (whose name is string) was recast on every read and logged a casting warning; a bare category still accepts any categorical dtype and a bare string any string storage (string[python] or string[pyarrow] ask for one). A declared type pandas does not know is reported by the cast as before. Columns that were cast before are cast as before
  • feat: Pandera constructs that dynamicio has no validator for now run instead of being ignored or raising a bare KeyError. Column checks such as str_matches, str_length and custom @pa.check run as Pandera checks; @pa.dataframe_check, Config.unique, Config.ordered and index declarations run on the whole frame. Failures are raised in SchemaValidationError under the check's name
  • feat: Pandera options dynamicio cannot honour (Config.strict = True, add_missing_columns, on_missing_columns, unique_column_names, drop_invalid_rows, parsers, dtype, and per column regex, required=False, default, on_missing) and non-pandas (for example polars) models raise ValueError when the schema is compiled, instead of being dropped. Use strict = "filter". coerce is honoured through the compiled dtype (a nullable int becomes Int64)
  • fix: Series[str] compiles to the pandas default string dtype (object on pandas 2, str on pandas 3) instead of a hard-coded object. Existing pandas 2 behaviour is unchanged
  • feat: models written by dynamicio migrate set strict = "filter", so Model.validate(df) on its own drops columns that are not in the schema, as dynamicio does. coerce stays off: Pandera would turn a nullable int column into Int64, where dynamicio casts to int64. Dynamicio's own validation is unchanged
  • fix: no more UserWarning: Valid config keys have changed in V2: 'underscore_attrs_are_private' has been removed on import under pydantic 2. The key is now only set under pydantic 1, where it is needed to keep IOEnvironment._parent private
  • fix: WithPostgres reads and writes work when the table is declared only by a Pandera model. The table name and columns come from the resource YAML's schema when it has one (unchanged), otherwise from the model (Config.name is the table name). Before, a postgres resource without a YAML schema failed with AssertionError: The schema must be specified for SQL tables
  • feat: dynamicio migrate -o <package dir> writes one models module per sub-directory of the YAML schemas (the grouping by runner and input/output carries over); -o file.py still writes a single module. The README recommends one models module per runner stage or definitions file, and the demo/ project now follows it (demo/src/resources/schemas/{input,processed}.py)
  • fix: dynamicio migrate handles real-world schemas: mixed-type is_in values (incl. null), is_between bounds that are [[ VARIABLES ]], dtype aliases such as double/int, and explicit default options; a failed parity check now lists the differing columns.
  • refactor: schema modules now live in a dynamicio.schema subpackage (schema.validations, schema.metrics, schema.models, schema.migrate). dynamicio.validations and dynamicio.metrics keep working as aliases of the same module objects, and the dynamicio.metrics logger name is unchanged. dynamicio.pandera_schema and dynamicio.migrate (new in 10.0.0-rc.1 only) moved to dynamicio.schema.models and dynamicio.schema.migrate without aliases.
  • refactor: the helpers of dynamicio.schema are grouped into classes (ModelCompiler, and for migrate: SchemaComparison, ModelRenderer, LegacyDeclarations, MigrationPrompt, Migration), and _is_declared_dtype is a static method of DynamicDataIO. The public function names (to_dataframe_schema, dataframe_validator, migrate, render_model, render_module, compile_module, schemas_equivalent, ...) remain as aliases, so imports keep working.
  • chore: raise the numpy floor to 1.24.4, the oldest release Pandera 0.34 accepts (found by the dependency-ranges (lowest) job).
  • Docs: README review. Fixed stale schema wording, the is_between include_right default, the casting-failure exception (ColumnsDataTypeError), unique Step 4.3 anchors, and added a summary of validation/metrics behaviour to the closing notes.
  • test: tests/test_contracts/ pins the validation, metric, lifecycle-order and dtype behaviour that the schema work must preserve, for every schema declaration style
  • feat: schemas can be declared as Pandera DataFrameModel classes (class MyIO(UnifiedIO): schema = MyModel). Dynamicio's validators and metric logging are unchanged, so error payloads and METRIC log lines are identical to the YAML and dict styles. pandera is now a required dependency. Anything Pandera cannot express natively goes in Field(metadata={"dynamicio": {"validations": {...}, "metrics": [...]}}). Note Pandera's default is nullable=False; dynamicio's YAML allowed nulls unless has_no_null_values was declared
  • fix: when the same validation fails on several columns, SchemaValidationError.message now holds every column's message (joined with ; under the one validation-name key). Previously only the last column's message survived
  • feat: schema= constructor argument and SCHEMA_FROM_ARGUMENT placeholder, so one IO class can take a different Pandera model per instance (the Pandera counterpart of SCHEMA_FROM_FILE). The argument takes precedence over the class attribute
  • feat: dynamicio migrate <schemas dir> -o schemas.py converts YAML schemas to Pandera models (dataset names, validations and metrics preserved, verified by a parity check) and writes MIGRATION_PROMPT.md listing the call sites to update by hand or with an LLM. The existing -b/-s schema generator is unchanged
  • docs: README Step 4.1 (Pandera models) and migration guide, ADR 0005; the demo/ project now declares its schemas as Pandera models and keeps its resource definitions, models and IO classes together under demo/src/resources (with environment.py and constants.py under demo/src/config), with demo/tests mirroring demo/src, documented in the README as the recommended pipeline project structure

v9.2.3

  • fix: WithPostgres builds one SQLAlchemy model per table definition and reuses it, instead of a new same-named class on every read or write. The classes piled up in the declarative registry, and SQLAlchemy's cleanup of them could raise a RecursionError during garbage collection (seen as a failing test on the lowest-dependency CI job). Each definition has its own declarative base, so two definitions of one table cannot alter each other's columns, and the cache is thread-safe for async_read / async_write. Queries and tables are unchanged.

v9.2.2

  • fix: an IO object can be read or written repeatedly. A read or write removed values from self.options, so a second call on the same object failed or read the wrong data. The second call of a path with template fields (for example {runner_id}) raised ValueError: Expected values for all fields. Three cases gave no error. The S3 path-prefix read removed no_disk_space. The Athena read removed query, and the Postgres read removed sql_query, so the second read ran the default query and returned wrong data. Each call now works on a copy of the options, and self.options does not change. The two private methods WithS3._read_parquet_object and WithS3._concat_parquet_objects are removed. The module-level functions with the same names are unchanged

v9.2.1

  • fix: transient S3 connection errors no longer fail a read or write. The shared boto3 client now uses the standard retry mode with 10 attempts, connect_timeout=10 and read_timeout=60. The in-memory and file transfers (put_object, upload_fileobj, get_object, download_fileobj) are repeated, up to 5 attempts with backoff, on ConnectionClosedError, ConnectTimeoutError, EndpointConnectionError, ReadTimeoutError, IncompleteReadError, ResponseStreamingError and the RetriesExceededError that the transfer manager raises (s3transfer's and boto3's) when a download above 16 MiB gave up on a broken part. botocore does not retry these once the body is being sent or read. The buffer is rewound before each attempt (bytes written are identical), each retry is logged at WARNING with the bucket and key, and the shared client is rebuilt so the next attempt opens fresh connections. Other errors (for example ClientError) are raised as before, and nothing changes when no error occurs

v9.2.0

  • feat: a single-file S3 parquet read with filters reads only the row groups whose footer statistics can match, over ranged GetObject requests through pyarrow.fs.S3FileSystem (one for each process and bucket). The rows, index and dtypes are the same as before. Reads without filters or with filters=[], fastparquet reads, reads with a non-pyarrow filesystem, prefix reads and reads with wrangler-only kwargs do not change. On the ranged read, a missing key or a permission problem raises pyarrow FileNotFoundError or OSError in place of botocore ClientError, and a custom S3 endpoint of boto3 (AWS_ENDPOINT_URL) does not apply. See ADR 0004
  • feat: new postgres write option create_table (default False). When True, the writer creates a missing table from the schema columns, with no primary key and with unbounded string columns, in the same transaction as the write. The postgres writer now lists its options (truncate_and_append, create_table), so it drops other options with a warning, as the other writers do. A create_table value that is not a bool (for example the YAML string "false") raises TypeError
  • fix: the cache of pyarrow S3 file systems is cleared at exit, before the pyarrow S3 finalizer runs. With pyarrow 24 and later, a process that read with filters on a pod with web identity credentials did not exit (https://github.com/apache/arrow/issues/50188)
  • chore: dev dependency moto[server]>=5 for the tests of the ranged read

v9.1.0

  • chore(deps): dependency audit, ranges widened to what the code needs and CI proves. Removed packages the code never imports (fsspec, libcst, logzero, no_implicit_optional); dropped exact pins (confluent-kafka>=2.4.0,<3, psycopg2-binary>=2.9.3,<3, python-json-logger>=2.0.1, simplejson>=3.17.2); lifted the numpy<2 ceiling (numpy 2.x passes the suite) and the python<3.13 ceiling (3.13 and 3.14 pass; added to CI and classifiers). Raised floors so every allowed combination works with numpy 2 as well as numpy 1: pandas>=2.2.2 (JSON writes with index=False for records/values, relied on since 8.1.1, need pandas 2.1, and pandas before 2.2.2 breaks with numpy 2), pyarrow>=16.0.0, fastparquet>=2024.5.0, awswrangler>=3.9.0 (older ones cannot import against numpy 2 or pyarrow 16), PyYAML>=6.0 (5.4.1 does not build with current toolchains). New CI job dependency-ranges tests the minimums (Python 3.10), the minimums with numpy 2 (Python 3.11) and the newest releases (Python 3.14); publishing waits for it
  • fix: python-json-logger 3.1+ import path (pythonjsonlogger.json) with fallback to the old module, avoiding a DeprecationWarning on import

v9.0.0

Breaking: Athena resources must rename s3_staging_dir to s3_output (see the first Athena entry below).

  • docs: add Backstage catalog-info.yaml to enable TechDocs
  • docs: README documents Athena I/O (resource options, ctas_approach, dtypes, s3_staging_dir migration); ADR 0003
  • chore: migrate project and dependency management from Poetry to uv (PEP 621 pyproject.toml, uv.lock, hatchling + hatch-vcs build backend). The version is still derived from the git tag and the published wheel is unchanged apart from standard metadata. Contributors now use make dev-env-setup / uv sync; see CONTRIBUTING.md and AGENTS.md
  • feat(RND-13624)!: WithAthena reads through AWS Data Wrangler (wr.athena.read_sql_query) instead of PyAthena, and the pyathena dependency is dropped. Breaking (resource definitions): the s3_staging_dir key of an athena: section is removed. Loading such a definition now fails with an error saying to rename it to s3_output (same value); nothing else changes in the YAML. region_name and query stay optional, and database (default default, Athena's own default) and workgroup are new optional keys. Reads use the shared per-process boto3 session unless region_name is set. Extra options are filtered to what read_sql_query accepts (e.g. ctas_approach, unload_approach, athena_cache_settings); chunksize is rejected with an error, because a chunked read returns an iterator rather than a DataFrame
  • ctas_approach defaults to False (as PyAthena, which read the query's result file): column names keep the case used in the query (CTAS lower-cases them) and no temporary Glue table is created. Pass ctas_approach: true in options to opt in for large results
  • note: results now come back with pandas nullable dtypes (Int64, string, boolean) and date as Python dates, as returned by awswrangler; declare the dtypes in the resource schema to cast them on read, as for any other source

v8.1.1

  • fix: S3 JSON reads and writes use pandas' defaults again: lines=False (a JSON array for orient="records", as written by 7.x and earlier) unless the caller passes lines. 8.1.0 defaulted to JSON Lines for orient="records", which could not read existing files and wrote files that 7.x readers cannot parse. Set lines: true in the resource options to opt in to JSON Lines
  • fix: S3 JSON writes default index per orient as pandas does (True for index/columns/split/table, False for records/values) instead of always False, which made orient="index" and orient="columns" writes fail or drop the index

Metadata

Release files for dynamicio 11.0.0rc2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dynamicio 11.0.0rc2
File Size Uploaded Provenance Actions
dynamicio-11.0.0rc2.tar.gz 3.1 MB Without provenance Details for dynamicio-11.0.0rc2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dynamicio 11.0.0rc2
File Interpreter ABI Platform Actions
dynamicio-11.0.0rc2-py3-none-any.whl Python 3 none any Details for dynamicio-11.0.0rc2-py3-none-any.whl

Total release size: 3.3 MB

Release files / dynamicio-11.0.0rc2.tar.gz

Download URL dynamicio-11.0.0rc2.tar.gz
Size 3.1 MB
Tags Source
SHA-256 checksum
How to use checksums
db25dd137822152a9acfac0ee22e2b7ad5134583a6c4b704623dee804782d60d
BLAKE2b-256 checksum
How to use checksums
14f017a676c9ad774f96af8f67b773d89c2bdfe97177215f52f31eef98ccb6ef
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / dynamicio-11.0.0rc2-py3-none-any.whl

Download URL dynamicio-11.0.0rc2-py3-none-any.whl
Size 112.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
93e6ce7e5494310b8e05204394c0568d57778a1fab62c0c4d16d23e5e4bf0e56
BLAKE2b-256 checksum
How to use checksums
53ac69ca6502478b5b751421752ed0988c74153544cc3d2ffdca113d4a11f714
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for dynamicio 11.0.0rc2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release
Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page