Skip to main content

Build and Publish codecov

dynamicio — Wrap your I/O The dynamicio panda mascot

Keep your business logic independent of I/O. dynamicio is the seam between your pandas code and the places your data lives: describe each dataset once in YAML, and your code just calls read() and write().

The same code reads a parquet file on your laptop, a CSV prefix on S3, a PostgreSQL table or an Athena query. You pick which one by changing a single identifier, so developers stop worrying about connection strings, file-format keyword arguments and which bucket a task reads this week.

Documentation: dynamicio.readthedocs.io: guides, examples and API reference · Changelog: CHANGELOG.md

Why dynamicio

  • Logic that does not know about storage. Resource definitions are YAML, per environment (LOCAL, DEV, PROD, or your own). Moving a pipeline between environments never touches Python.
  • Fast, safe feedback loops. Run the real pipeline against sample files on your machine, with no cloud credentials, no deploy and no mocks of external services. Compare the outputs with expected files using the built-in test harness (dynamicio.testharness).
  • Made for the LLM era. A coding agent that changes a pipeline needs somewhere safe to run it and a check that says whether the change was right. A LOCAL environment and expected outputs give it both, and production credentials never enter the loop.
  • Contracts for your dataframes. A Pandera model declares columns, dtypes and validations. Bad data fails at the boundary with every violation listed, not three steps later, and the same model can emit metrics.
  • Pandas behaviour, not a new dialect. Options in the YAML go to pandas / awswrangler, so what you know still applies.
  • Sync and async. Read and write from plain code or from asyncio services.

Supported sources

Source Read Write Formats
Local files yes yes parquet, hdf, json, csv
S3 (single object or prefix) yes yes parquet, hdf, json, csv
PostgreSQL yes yes tables
AWS Athena yes no SQL query results
Kafka no yes messages

Dataframe execution currently supports pandas only; Polars is not yet supported.

Install

pip install dynamicio
# or, in a uv project
uv add dynamicio

Python 3.10 to 3.14.

See it work

Describe the resource in orders.yaml:

ORDERS:
  LOCAL:
    type: local
    local:
      file_path: "[[ DATA_DIR ]]/orders.parquet"
      file_type: parquet

Bind it to a schema, then read and write:

import pandas as pd
from pandera.typing import Series

from dynamicio import UnifiedIO
from dynamicio.config import IOConfig
from dynamicio.schema import DataModel, Field


class Settings:
    DATA_DIR = "."


class Orders(DataModel, dataset="orders"):
    order_id: Series[int] = Field(nullable=False, unique=True)
    amount: Series[float] = Field(nullable=False, ge=0)


class OrdersIO(UnifiedIO):
    schema = Orders


config = IOConfig("orders.yaml", env_identifier="LOCAL", dynamic_vars=Settings)
orders = OrdersIO(source_config=config.get(source_key="ORDERS"), apply_schema_validations=True)

orders.write(pd.DataFrame({"order_id": [1, 2, 3], "amount": [9.5, 20.0, 3.25]}))
df = orders.read()

To run the same code against S3, add a DEV block to the YAML and set env_identifier="DEV". The Python does not change.

Where next

  1. The idea: why dynamicio.
  2. New here? The quickstart explains each step above, and its code is run by the test suite.
  3. Core ideas: concepts: resource definitions, IO classes, schemas.
  4. Task guides: reading, writing, testing pipelines.
  5. Reference: API reference.

Community and contributing

Original logo illustrated by Nick Loucas; the refreshed logo retains its circular I/O motif.

Changelog

v11.0.0

Pandera DataFrameModel classes are now the only way to declare a schema. Upgrading from 8, 9 or 10: see Upgrading; dynamicio migrate converts the YAML schemas of version 8 and earlier.

  • breaking: dict schemas (schema = {...}, schema_validations, schema_metrics) and YAML schemas (schema = SCHEMA_FROM_FILE, and the schema: block of a resource definition) are removed. An IO class whose schema is anything but a DataFrameModel or SCHEMA_FROM_ARGUMENT raises a TypeError when the class is defined, and so does a class that still sets schema_validations or schema_metrics next to a model (the model's own would replace them). A resource definition with a schema: key raises a ValueError when it loads. SCHEMA_FROM_FILE is removed from dynamicio.core and SafeDynamicSchemaLoader from dynamicio.config.io_config (importing either raises an ImportError that names the replacement), and IOBinding.dynamicio_schema / IOEnvironment.dynamicio_schema with it. SCHEMA_FROM_ARGUMENT and the schema= argument are unchanged
  • feat: DataModel (from dynamicio.schema import DataModel, Field, Metric) is the base class of a model: a Pandera DataFrameModel that names its dataset and table in class keywords, class Orders(DataModel, dataset="orders", table="orders_v2"), so the body holds only columns and the editor completes the keywords. Any other class keyword raises a TypeError that names it. A plain pa.DataFrameModel declares the same with the dunders __dataset_name__ and __table_name__. Validation, metrics and casting checks stay arguments of the IO call (apply_schema_validations, log_schema_metrics, ...), unchanged from 10.x, so a call site always shows what it does
  • feat: Field declares a column: pa.Field with dynamicio's validations and metrics as named, documented arguments (nullable= required, unique, gt/ge/lt/le, isin, exactly, max_null_fraction, metrics=[Metric.MIN, ...]); any other pa.Field argument passes through, metadata= is refused. A range is its two bounds (ge=1941, le=NEXT_YEAR): bounds are not compared on definition, so they may come from settings, where Pandera's in_range would raise. Field returns a pa.Field, so it also declares the columns of a plain pa.DataFrameModel; it is the one public way to declare dynamicio's metrics and validations, and from dynamicio.schema import metrics is the metrics module again (as in 10.x), not a helper
  • feat: Metric is a plain Enum, no longer a str subclass, so an editor offers only its eight members after Metric.; its .value is still the name the METRIC log line reports
  • breaking: every model must name its dataset (dataset= or __dataset_name__), and nothing is inferred: a model without one raises DatasetNameNotFoundError when the IO object is created, where 10.x labelled it with its class name. The names are not inherited. Config.name is no longer read, and a model that relied on the class-name label must now write it, upper-cased in logs and METRIC lines as before
  • breaking: the Postgres table name is table= (or __table_name__), the exact name of the table in the database, and no longer the dataset name. A model used with a Postgres resource without it raises TableNameNotFoundError when the IO object is created (a read with sql_query, which names its own tables, needs none), and a read of a table that does not exist logs which one to correct. A dotted name is a table of a Postgres schema (table="sales.orders"), and every statement quotes the name as Postgres needs it, so a mixed-case table="Orders" is truncated, deleted from and copied into as "Orders"; writes load the rows with COPY ... FROM STDIN instead of copy_from, which quoted a schema-qualified name as one identifier
  • breaking: the raw metadata={"dynamicio": {"metrics": [...], "validations": {...}}} dict is rejected with a ValueError that names Field
  • breaking: the match_all option of the is_in validator, which read the wrong way round (match_all=False was the check that every value is present), is now exact with the opposite meaning: is_in(..., exact=True) requires exactly the allowed values, the default exact=False only a subset of them. Field(exactly=[...]) compiles to exact=True and isin=[...] to the default; dynamicio migrate translates a YAML match_all. Code that calls dynamicio.schema.validations.is_in directly with match_all must switch
  • feat: dynamicio migrate is the path from version 8: it converts YAML schemas to DataModel classes declared with Field, dataset=, and table= for every schema that a resource definition under --scan binds to a Postgres environment (the old schema name was the table name), checks each converted model compiles back to an equivalent schema, and writes MIGRATION_PROMPT.md listing the remaining call sites. Dtypes are declared as types (np.uint32, pd.UInt32Dtype), not calls. A YAML is_between becomes its two bounds, merged with the other bounds of the column into the tightest one per side: such a column now reports is_greater_than/is_lower_than (or the _or_equal forms) instead of one is_between in SchemaValidationError. A [[ VARIABLE ]] null threshold migrates like any other bound (Field(max_null_fraction=...) checks a number on definition and keeps a placeholder as it is). Class and module names are valid Python whatever the file is called (20261113.yaml becomes Dataset20261113), and a column named like a name the module uses (Field, Metric, Series, a dynamic variable) gets a column_ attribute with an alias. The prompt lists the name = lines of class Config blocks (which 11 no longer reads) instead of every class Config, and the search skips hidden, node_modules, __pycache__, venv and site-packages directories
  • feat: the dynamicio command line (dynamicio -s|-b -p PATH -o DIR) writes a Pandera model module per dataset (<dataset>.py, prefixed dataset_ where the name would not be importable) instead of a YAML schema
  • docs: the schemas guide, quickstart, examples and README describe models only; new ADR 0009. The demo project's models are DataModel classes declared with Field; Bar, bound to Postgres in DEV and PROD, names its table, which a test now checks for every environment

v10.4.1

  • docs: the Documentation project link of the package, and the README, point to the documentation on Read the Docs (https://dynamicio.readthedocs.io/en/latest/), and the documentation home page shows the panda mascot next to the logo
  • docs: remove the changelog sections of the v8.1.0 release candidates.

v10.4.0

  • feat: the constructor of an IO class takes missing_ok: bool = False. With missing_ok=True, read() and async_read() return None and log a warning when the reader raises a FileNotFoundError. The caller handles the None, so the setting is in the code of the caller
  • chore: remove missing_ok from the resource definition. Release v10.2.0 added it, and it has no known user. A resource definition with a missing_ok key now fails to load, because dynamicio reads the key as an environment name. IOBinding.missing_ok and IOEnvironment.missing_ok are removed, and the dict() output of a binding has no missing_ok entry. Move the setting to the constructor of the IO class
  • docs: new "Why dynamicio" page and a rewritten home page and README that explain the purpose: business logic independent of I/O, fast local feedback loops, and what that gives coding agents. The documentation is restructured into getting-started, guides, examples, API reference, architecture decisions and development pages (docs/), built with MkDocs for Read the Docs (.readthedocs.yaml; versioned latest, stable and per-release documentation). The README is now a concise overview and the long walkthrough moved into the guides, with its content unchanged. The Python examples of the quickstart and the basic I/O page are executed by the test suite. Polars is still not supported.
  • docs: docstrings that the API reference could not parse (Returns:, Args: and Raises: entries of DynamicDataIO.async_read, IOConfig, WithS3, WithPostgres, WithKafka, allow_options, args_of and OutputSandbox) are corrected. args_of gains a Callable annotation on *funcs and OutputSandbox annotates directory; behaviour is unchanged.
  • chore: the package metadata gains Documentation, Source and Issues project URLs next to Changelog. The Documentation URL points at the docs/ directory until the Read the Docs site is confirmed.
  • ci: the "Build and Publish" workflow builds the documentation in strict mode on every push, builds and validates the distributions once (twine check, tag against version) and publishes exactly those files. A release tag must be vX.Y.Z or vX.Y.Z-rc.N, a stable tag must be on master and have a changelog section, and the GitHub release notes are the changelog section plus links (.github/tools/release_tools.py, tested). Re-running a publish is safe: files already on PyPI are skipped and an existing GitHub release is updated.
  • chore: the docs dependency group no longer includes mkdocs-techdocs-core and stays on mkdocs<2; docs/requirements.txt pins the documentation tooling from uv.lock.

v10.3.0

  • chore: remove the magic-logger dependency. Each dynamicio module logs through logging.getLogger(__name__), with the same logger names as before. magic-logger has only a source distribution, so an install with uv sync --no-build stopped with an error. magic-logger also called inspect.stack() on each log call. dynamicio.logger is a logging.Logger for the logger dynamicio, and it does not have the dict_config, file_config, listen and stop_listening methods. The logging config of dynamicio/__init__.py names the logger dynamicio, so disable_existing_loggers does not disable the module loggers. If you set DYNAMICIO_DO_NOT_CONFIGURE_LOGGING=1 and call logging.config.dictConfig after the dynamicio import, name the logger dynamicio in your config or set disable_existing_loggers to false. If you do not, your config disables the dynamicio loggers

v10.2.0

  • feat: a resource definition can set missing_ok: true on a source, beside its environments. When the reader of that source raises a FileNotFoundError (a missing local file, ObjectNotFoundError, NoObjectsFoundError), read() and async_read() return None and log a warning. All other errors raise as before. The default is false. IOBinding.missing_ok and IOEnvironment.missing_ok expose the flag, which adds a missing_ok: False entry to the dict() output of a binding. See "Optional sources: missing_ok" in the README
  • fix: an S3 read of a csv or json file_path raises dynamicio.errors.ObjectNotFoundError("No object found at s3://...") when the key does not exist. Before, it raised the awswrangler NoFilesFound, which is not a FileNotFoundError. ObjectNotFoundError is a FileNotFoundError and a NoFilesFound, so a caller can treat the key as a missing source, and code that catches NoFilesFound works as before. The errors of the parquet and hdf reads for a missing key do not change

v10.1.0

  • docs: refresh the standalone logo, clarify pandas-only support, remove unused and superseded images, and expose architecture decisions in the documentation navigation; the S3 benchmark page is replaced by ADR 0007, which keeps the headline figures (the full tables stay in PR #88).
  • feat: resource definitions accept an optional expected: {path, file_type, options} key per environment, the reference file a test compares the generated output against. Reading and writing ignore it, so existing definitions and their behaviour are unchanged. (IOEnvironment.expected is None when absent, which adds an expected: None entry to the model's dict() output.)
  • feat: the dynamicio.testharness package, to test a task end to end: run it in a LOCAL environment, then lazily compare each generated output with its expected file. ExpectedOutputs collects all mismatches and raises one OutputMismatchError; generated files of matching outputs are removed (the others are kept); JSON is read with the resource's orient (or expected.options); paths can carry {placeholders} filled from params; DYNAMICIO_UPDATE_EXPECTED=1 regenerates the expected files. Expectations curates how strict a comparison is (row/column order, dtypes, tolerances, sort_by, only_columns, ignore_columns, list_columns_to_sort, custom assertions; strict() / lenient() presets, with_() to derive variants), per session or per check; dict columns are compared by content. FrameComparison and OutputFiles are usable on their own. See "Testing Locally" in the README.
  • feat: dynamicio.testharness additions: ExpectedOutputs.check_each(config, key, grid) checks a resource whose path has several {placeholders} for every combination of their values (clear_each removes their leftovers); Expectations(check_parquet_schema=True) also compares the arrow schema and stored pandas index of parquet outputs; OutputSandbox(directory, *configs) redirects the local outputs of a task into a scratch directory so tests that write the same files can run in parallel; ExpectedOutputs(environment=...) makes every IOConfig that defines that environment select it inside the with block (and restores them on exit), so the task under test runs and is checked in an environment the test states, without patching env_identifier; check(..., environment=...) overrides it for one check, configs=[...] narrows the switch, EnvironmentSwitch(environment, *configs) does the same explicitly and rejects an environment a config does not define. Without environment=, the default is the first environment the resource definitions list, whatever the process runs in. IOConfig.instances() and IOConfig.default_environment support this. The README gains "Choosing the environment" and "Recommended testing practices".
  • fix: dynamicio.testharness review fixes: JSON Lines outputs (options.lines: true) are read back with lines (it is passed with orient to the JSON reader only); a column named in sort_by, only_columns or list_columns_to_sort that is missing from the output or the reference is reported as a failed check, not a KeyError that aborts the remaining checks; update mode (DYNAMICIO_UPDATE_EXPECTED=1) refuses to copy an output over a reference declared in another file type, leaving both files untouched.
  • feat: ExpectedOutputs.check_all(config, skip=, rules_for=, params=) checks every resource of a config that declares expected (a new output is covered without touching the test; resources with unfilled {placeholders} are listed in skipped), and ExpectedOutputs.clear_all(config) removes the stale generated files of those resources.
  • feat: a passing test run now says what it verified. ExpectedOutputs prints a SessionReport at exit (green ✓ per matched output with its rows x columns and expected file, red ✗ per failure, ↻ updated, - skipped), outputs.report returns it, OutputMismatchError also lists the outputs that matched, and a pytest plugin (pytest11 entry point dynamicio; -p no:dynamicio turns it off) prints every report in the terminal summary, coloured. Nothing is printed when a session checked nothing.
  • docs: the README walkthrough follows the demo/ project: four resource definitions (staging_input, staging_output, transform_input, transform_output) with LOCAL, DEV and PROD environments, expected paths on the outputs' LOCAL environment, and the staging_*/transform_* configs, instead of sample/actual and input/raw/processed.
  • docs: the demo/ project now has one input and one output definition per runner (staging_input, staging_output, transform_input, transform_output) with LOCAL, DEV and PROD environments, and its end-to-end test uses ExpectedOutputs. The definitions use YAML anchors and merge keys (&, *, <<:) so PROD only restates what differs from DEV.

v10.0.1

  • fix: an S3 path-prefix read with no objects under the prefix raises dynamicio.errors.NoObjectsFoundError("No objects found under s3://..."). Before, it raised the pandas error ValueError: No objects to concatenate, which a caller cannot tell apart from other ValueErrors. The change applies to all the file types (parquet, csv, json, hdf) and to the hdf no_disk_space read. NoObjectsFoundError is a FileNotFoundError and a ValueError, so a caller can treat the prefix as a missing source, and code that catches ValueError works as before. An S3 parquet file_path that ends in / and has no objects raises the same error. It raised a plain FileNotFoundError before. Objects that are all empty do not change: a parquet prefix returns an empty frame, and a csv, json or hdf prefix raises the pandas ValueError as before

v10.0.0

  • fix: dynamicio migrate writes a YAML object column as Series[object], not Series[str]. object meant "any values", and list columns declared str fail plain Pandera validation (and would be recast under pandas 3). Models already generated with Series[str] for list columns should be changed to Series[object]; the generated module now says where str is safe
  • docs: the README diagrams are now text-driven SVGs (supported sources with Athena next to S3, Postgres and Kafka; the sample pipeline as an Airflow DAG; how a read works; the file formats), they render on GitHub and on PyPI, and the Pandera logo is shown in the Pandera models step. The table of contents anchors and the CONTRIBUTING.md link are fixed, and the dependency list names what the package really uses. The demo/ example datasets are generic (orders).
  • fix: the dtype check compares dtypes instead of their names. A column already at string[pyarrow] (whose name is string) was recast on every read and logged a casting warning; a bare category still accepts any categorical dtype and a bare string any string storage (string[python] or string[pyarrow] ask for one). A declared type pandas does not know is reported by the cast as before. Columns that were cast before are cast as before
  • feat: Pandera constructs that dynamicio has no validator for now run instead of being ignored or raising a bare KeyError. Column checks such as str_matches, str_length and custom @pa.check run as Pandera checks; @pa.dataframe_check, Config.unique, Config.ordered and index declarations run on the whole frame. Failures are raised in SchemaValidationError under the check's name
  • feat: Pandera options dynamicio cannot honour (Config.strict = True, add_missing_columns, on_missing_columns, unique_column_names, drop_invalid_rows, parsers, dtype, and per column regex, required=False, default, on_missing) and non-pandas (for example polars) models raise ValueError when the schema is compiled, instead of being dropped. Use strict = "filter". coerce is honoured through the compiled dtype (a nullable int becomes Int64)
  • fix: Series[str] compiles to the pandas default string dtype (object on pandas 2, str on pandas 3) instead of a hard-coded object. Existing pandas 2 behaviour is unchanged
  • feat: models written by dynamicio migrate set strict = "filter", so Model.validate(df) on its own drops columns that are not in the schema, as dynamicio does. coerce stays off: Pandera would turn a nullable int column into Int64, where dynamicio casts to int64. Dynamicio's own validation is unchanged
  • fix: no more UserWarning: Valid config keys have changed in V2: 'underscore_attrs_are_private' has been removed on import under pydantic 2. The key is now only set under pydantic 1, where it is needed to keep IOEnvironment._parent private
  • fix: WithPostgres reads and writes work when the table is declared only by a Pandera model. The table name and columns come from the resource YAML's schema when it has one (unchanged), otherwise from the model (Config.name is the table name). Before, a postgres resource without a YAML schema failed with AssertionError: The schema must be specified for SQL tables
  • feat: dynamicio migrate -o <package dir> writes one models module per sub-directory of the YAML schemas (the grouping by runner and input/output carries over); -o file.py still writes a single module. The README recommends one models module per runner stage or definitions file, and the demo/ project now follows it (demo/src/resources/schemas/{input,processed}.py)
  • fix: dynamicio migrate handles real-world schemas: mixed-type is_in values (incl. null), is_between bounds that are [[ VARIABLES ]], dtype aliases such as double/int, and explicit default options; a failed parity check now lists the differing columns.
  • refactor: schema modules now live in a dynamicio.schema subpackage (schema.validations, schema.metrics, schema.models, schema.migrate). dynamicio.validations and dynamicio.metrics keep working as aliases of the same module objects, and the dynamicio.metrics logger name is unchanged. dynamicio.pandera_schema and dynamicio.migrate (new in 10.0.0-rc.1 only) moved to dynamicio.schema.models and dynamicio.schema.migrate without aliases.
  • refactor: the helpers of dynamicio.schema are grouped into classes (ModelCompiler, and for migrate: SchemaComparison, ModelRenderer, LegacyDeclarations, MigrationPrompt, Migration), and _is_declared_dtype is a static method of DynamicDataIO. The public function names (to_dataframe_schema, dataframe_validator, migrate, render_model, render_module, compile_module, schemas_equivalent, ...) remain as aliases, so imports keep working.
  • chore: raise the numpy floor to 1.24.4, the oldest release Pandera 0.34 accepts (found by the dependency-ranges (lowest) job).
  • Docs: README review. Fixed stale schema wording, the is_between include_right default, the casting-failure exception (ColumnsDataTypeError), unique Step 4.3 anchors, and added a summary of validation/metrics behaviour to the closing notes.
  • test: tests/test_contracts/ pins the validation, metric, lifecycle-order and dtype behaviour that the schema work must preserve, for every schema declaration style
  • feat: schemas can be declared as Pandera DataFrameModel classes (class MyIO(UnifiedIO): schema = MyModel). Dynamicio's validators and metric logging are unchanged, so error payloads and METRIC log lines are identical to the YAML and dict styles. pandera is now a required dependency. Anything Pandera cannot express natively goes in Field(metadata={"dynamicio": {"validations": {...}, "metrics": [...]}}). Note Pandera's default is nullable=False; dynamicio's YAML allowed nulls unless has_no_null_values was declared
  • fix: when the same validation fails on several columns, SchemaValidationError.message now holds every column's message (joined with ; under the one validation-name key). Previously only the last column's message survived
  • feat: schema= constructor argument and SCHEMA_FROM_ARGUMENT placeholder, so one IO class can take a different Pandera model per instance (the Pandera counterpart of SCHEMA_FROM_FILE). The argument takes precedence over the class attribute
  • feat: dynamicio migrate <schemas dir> -o schemas.py converts YAML schemas to Pandera models (dataset names, validations and metrics preserved, verified by a parity check) and writes MIGRATION_PROMPT.md listing the call sites to update by hand or with an LLM. The existing -b/-s schema generator is unchanged
  • docs: README Step 4.1 (Pandera models) and migration guide, ADR 0005; the demo/ project now declares its schemas as Pandera models and keeps its resource definitions, models and IO classes together under demo/src/resources (with environment.py and constants.py under demo/src/config), with demo/tests mirroring demo/src, documented in the README as the recommended pipeline project structure

v9.2.3

  • fix: WithPostgres builds one SQLAlchemy model per table definition and reuses it, instead of a new same-named class on every read or write. The classes piled up in the declarative registry, and SQLAlchemy's cleanup of them could raise a RecursionError during garbage collection (seen as a failing test on the lowest-dependency CI job). Each definition has its own declarative base, so two definitions of one table cannot alter each other's columns, and the cache is thread-safe for async_read / async_write. Queries and tables are unchanged.

v9.2.2

  • fix: an IO object can be read or written repeatedly. A read or write removed values from self.options, so a second call on the same object failed or read the wrong data. The second call of a path with template fields (for example {runner_id}) raised ValueError: Expected values for all fields. Three cases gave no error. The S3 path-prefix read removed no_disk_space. The Athena read removed query, and the Postgres read removed sql_query, so the second read ran the default query and returned wrong data. Each call now works on a copy of the options, and self.options does not change. The two private methods WithS3._read_parquet_object and WithS3._concat_parquet_objects are removed. The module-level functions with the same names are unchanged

v9.2.1

  • fix: transient S3 connection errors no longer fail a read or write. The shared boto3 client now uses the standard retry mode with 10 attempts, connect_timeout=10 and read_timeout=60. The in-memory and file transfers (put_object, upload_fileobj, get_object, download_fileobj) are repeated, up to 5 attempts with backoff, on ConnectionClosedError, ConnectTimeoutError, EndpointConnectionError, ReadTimeoutError, IncompleteReadError, ResponseStreamingError and the RetriesExceededError that the transfer manager raises (s3transfer's and boto3's) when a download above 16 MiB gave up on a broken part. botocore does not retry these once the body is being sent or read. The buffer is rewound before each attempt (bytes written are identical), each retry is logged at WARNING with the bucket and key, and the shared client is rebuilt so the next attempt opens fresh connections. Other errors (for example ClientError) are raised as before, and nothing changes when no error occurs

v9.2.0

  • feat: a single-file S3 parquet read with filters reads only the row groups whose footer statistics can match, over ranged GetObject requests through pyarrow.fs.S3FileSystem (one for each process and bucket). The rows, index and dtypes are the same as before. Reads without filters or with filters=[], fastparquet reads, reads with a non-pyarrow filesystem, prefix reads and reads with wrangler-only kwargs do not change. On the ranged read, a missing key or a permission problem raises pyarrow FileNotFoundError or OSError in place of botocore ClientError, and a custom S3 endpoint of boto3 (AWS_ENDPOINT_URL) does not apply. See ADR 0004
  • feat: new postgres write option create_table (default False). When True, the writer creates a missing table from the schema columns, with no primary key and with unbounded string columns, in the same transaction as the write. The postgres writer now lists its options (truncate_and_append, create_table), so it drops other options with a warning, as the other writers do. A create_table value that is not a bool (for example the YAML string "false") raises TypeError
  • fix: the cache of pyarrow S3 file systems is cleared at exit, before the pyarrow S3 finalizer runs. With pyarrow 24 and later, a process that read with filters on a pod with web identity credentials did not exit (https://github.com/apache/arrow/issues/50188)
  • chore: dev dependency moto[server]>=5 for the tests of the ranged read

v9.1.0

  • chore(deps): dependency audit, ranges widened to what the code needs and CI proves. Removed packages the code never imports (fsspec, libcst, logzero, no_implicit_optional); dropped exact pins (confluent-kafka>=2.4.0,<3, psycopg2-binary>=2.9.3,<3, python-json-logger>=2.0.1, simplejson>=3.17.2); lifted the numpy<2 ceiling (numpy 2.x passes the suite) and the python<3.13 ceiling (3.13 and 3.14 pass; added to CI and classifiers). Raised floors so every allowed combination works with numpy 2 as well as numpy 1: pandas>=2.2.2 (JSON writes with index=False for records/values, relied on since 8.1.1, need pandas 2.1, and pandas before 2.2.2 breaks with numpy 2), pyarrow>=16.0.0, fastparquet>=2024.5.0, awswrangler>=3.9.0 (older ones cannot import against numpy 2 or pyarrow 16), PyYAML>=6.0 (5.4.1 does not build with current toolchains). New CI job dependency-ranges tests the minimums (Python 3.10), the minimums with numpy 2 (Python 3.11) and the newest releases (Python 3.14); publishing waits for it
  • fix: python-json-logger 3.1+ import path (pythonjsonlogger.json) with fallback to the old module, avoiding a DeprecationWarning on import

v9.0.0

Breaking: Athena resources must rename s3_staging_dir to s3_output (see the first Athena entry below).

  • docs: add Backstage catalog-info.yaml to enable TechDocs
  • docs: README documents Athena I/O (resource options, ctas_approach, dtypes, s3_staging_dir migration); ADR 0003
  • chore: migrate project and dependency management from Poetry to uv (PEP 621 pyproject.toml, uv.lock, hatchling + hatch-vcs build backend). The version is still derived from the git tag and the published wheel is unchanged apart from standard metadata. Contributors now use make dev-env-setup / uv sync; see CONTRIBUTING.md and AGENTS.md
  • feat(RND-13624)!: WithAthena reads through AWS Data Wrangler (wr.athena.read_sql_query) instead of PyAthena, and the pyathena dependency is dropped. Breaking (resource definitions): the s3_staging_dir key of an athena: section is removed. Loading such a definition now fails with an error saying to rename it to s3_output (same value); nothing else changes in the YAML. region_name and query stay optional, and database (default default, Athena's own default) and workgroup are new optional keys. Reads use the shared per-process boto3 session unless region_name is set. Extra options are filtered to what read_sql_query accepts (e.g. ctas_approach, unload_approach, athena_cache_settings); chunksize is rejected with an error, because a chunked read returns an iterator rather than a DataFrame
  • ctas_approach defaults to False (as PyAthena, which read the query's result file): column names keep the case used in the query (CTAS lower-cases them) and no temporary Glue table is created. Pass ctas_approach: true in options to opt in for large results
  • note: results now come back with pandas nullable dtypes (Int64, string, boolean) and date as Python dates, as returned by awswrangler; declare the dtypes in the resource schema to cast them on read, as for any other source

v8.1.1

  • fix: S3 JSON reads and writes use pandas' defaults again: lines=False (a JSON array for orient="records", as written by 7.x and earlier) unless the caller passes lines. 8.1.0 defaulted to JSON Lines for orient="records", which could not read existing files and wrote files that 7.x readers cannot parse. Set lines: true in the resource options to opt in to JSON Lines
  • fix: S3 JSON writes default index per orient as pandas does (True for index/columns/split/table, False for records/values) instead of always False, which made orient="index" and orient="columns" writes fail or drop the index

Metadata

Release files for dynamicio 11.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dynamicio 11.0.0
File Size Uploaded Provenance Actions
dynamicio-11.0.0.tar.gz 3.2 MB Without provenance Details for dynamicio-11.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dynamicio 11.0.0
File Interpreter ABI Platform Actions
dynamicio-11.0.0-py3-none-any.whl Python 3 none any Details for dynamicio-11.0.0-py3-none-any.whl

Total release size: 3.3 MB

Release files / dynamicio-11.0.0.tar.gz

Download URL dynamicio-11.0.0.tar.gz
Size 3.2 MB
Tags Source
SHA-256 checksum
How to use checksums
b241269be72b0cbae1cfe8d51186b3edd5ee28cbeef765609f168b09c9e7f96b
BLAKE2b-256 checksum
How to use checksums
4f8b2500b31a94055df3129bc1c98721fa7d8686f3f361a987847794d0e1394c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / dynamicio-11.0.0-py3-none-any.whl

Download URL dynamicio-11.0.0-py3-none-any.whl
Size 117.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f7b7654f21b0d50cb24dda14a6c058d6103df4f4d8d3fae905411fe24e8ec6da
BLAKE2b-256 checksum
How to use checksums
d79198a183fcb17ff0a1ee5434ab484bb46faa7d713ab292ab90925d10e955e8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for dynamicio 11.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

11.0.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page