This release is a pre-release and may not be stable for production use.
Keep your business logic independent of I/O. dynamicio is the seam between your pandas code and the places your
data lives: describe each dataset once in YAML, and your code just calls read() and write().
The same code reads a parquet file on your laptop, a CSV prefix on S3, a PostgreSQL table or an Athena query. You pick which one by changing a single identifier, so developers stop worrying about connection strings, file-format keyword arguments and which bucket a task reads this week.
Documentation: dynamicio.readthedocs.io: guides, examples and API reference · Changelog: CHANGELOG.md
Why dynamicio
- Logic that does not know about storage. Resource definitions are YAML, per environment (
LOCAL,DEV,PROD, or your own). Moving a pipeline between environments never touches Python. - Fast, safe feedback loops. Run the real pipeline against sample files on your machine, with no cloud credentials, no deploy and
no mocks of external services. Compare the outputs with expected files using the built-in test harness (
dynamicio.testharness). - Made for the LLM era. A coding agent that changes a pipeline needs somewhere safe to run it and a check that says whether the
change was right. A
LOCALenvironment and expected outputs give it both, and production credentials never enter the loop. - Contracts for your dataframes. A Pandera model declares columns, dtypes and validations. Bad data fails at the boundary with every violation listed, not three steps later, and the same model can emit metrics.
- Pandas behaviour, not a new dialect. Options in the YAML go to
pandas/awswrangler, so what you know still applies. - Sync and async. Read and write from plain code or from
asyncioservices.
Supported sources
| Source | Read | Write | Formats |
|---|---|---|---|
| Local files | yes | yes | parquet, hdf, json, csv |
| S3 (single object or prefix) | yes | yes | parquet, hdf, json, csv |
| PostgreSQL | yes | yes | tables |
| AWS Athena | yes | no | SQL query results |
| Kafka | no | yes | messages |
Dataframe execution currently supports pandas only; Polars is not yet supported.
Install
pip install dynamicio
# or, in a uv project
uv add dynamicio
Python 3.10 to 3.14.
See it work
Describe the resource in orders.yaml:
ORDERS:
LOCAL:
type: local
local:
file_path: "[[ DATA_DIR ]]/orders.parquet"
file_type: parquet
Bind it to a schema, then read and write:
import pandas as pd
import pandera.pandas as pa
from pandera.typing import Series
from dynamicio import UnifiedIO
from dynamicio.config import IOConfig
class Settings:
DATA_DIR = "."
class Orders(pa.DataFrameModel):
order_id: Series[int] = pa.Field(nullable=False, unique=True)
amount: Series[float] = pa.Field(nullable=False, ge=0)
class OrdersIO(UnifiedIO):
schema = Orders
config = IOConfig("orders.yaml", env_identifier="LOCAL", dynamic_vars=Settings)
orders = OrdersIO(source_config=config.get(source_key="ORDERS"), apply_schema_validations=True)
orders.write(pd.DataFrame({"order_id": [1, 2, 3], "amount": [9.5, 20.0, 3.25]}))
df = orders.read()
To run the same code against S3, add a DEV block to the YAML and set env_identifier="DEV". The Python does not change.
Where next
- The idea: why dynamicio.
- New here? The quickstart explains each step above, and its code is run by the test suite.
- Core ideas: concepts: resource definitions, IO classes, schemas.
- Task guides: reading, writing, testing pipelines.
- Reference: API reference.
Community and contributing
- Questions and bugs: GitHub issues
- Contributions are welcome: CONTRIBUTING.md
- Source: https://github.com/VorTECHsa/dynamicio · PyPI: https://pypi.org/project/dynamicio/
- License: Apache-2.0 (LICENSE)
Original logo illustrated by Nick Loucas; the refreshed logo retains its circular I/O motif.
Changelog
v11.0.0
Pandera DataFrameModel classes are now the only way to declare a schema. Upgrading from 8, 9 or 10: see Upgrading; dynamicio migrate converts the YAML schemas of version 8 and earlier.
- breaking: dict schemas (
schema = {...},schema_validations,schema_metrics) and YAML schemas (schema = SCHEMA_FROM_FILE, and theschema:block of a resource definition) are removed. An IO class whoseschemais anything but aDataFrameModelorSCHEMA_FROM_ARGUMENTraises aTypeErrorwhen the class is defined, and a resource definition with aschema:key raises aValueErrorwhen it loads.SCHEMA_FROM_FILEis removed fromdynamicio.core,SafeDynamicSchemaLoaderfromdynamicio.config.io_config, andIOBinding.dynamicio_schema/IOEnvironment.dynamicio_schemawith it.SCHEMA_FROM_ARGUMENTand theschema=argument are unchanged - feat: a model names its dataset with a class attribute,
__dataset_name__ = "foo", instead ofclass Config: name = "foo". The name is resolved from__dataset_name__, thenConfig.name(still honoured, and treated as unset when equal to the class name), then the class name in snake case - breaking: a model with neither
__dataset_name__norConfig.nameis now labelled with its class name in snake case, upper-cased in logs andMETRIClines:FinalBarisFINAL_BAR, where it wasFINALBAR. Dashboards that parseMETRIClines for such models need the new label, or set__dataset_name__to the old one - breaking: the Postgres table name is the new class attribute
__table_name__, the exact name of the table in the database, and no longer the dataset name. A model used with a Postgres resource without it raisesTableNameNotFoundErrorwhen the IO object is created, and a read of a missing table logs the__table_name__to correct.DataframeSchemagainstable_name - feat: typed metadata for the parts of a field that Pandera has no argument for:
from dynamicio.schema import Metric, metrics, null_fraction_below, exactly_values, strictly_between, combined with|, as inpa.Field(nullable=True, lt=1000, metadata=metrics(Metric.MIN, Metric.MAX) | null_fraction_below(0.2)).exactly_valuesisis_inwithmatch_all=Falseandstrictly_betweenisis_betweenfor bounds that are not known on definition. Validator names, error payloads and metric lines are unchanged.dynamicio.schema.metricsis now the helper function on the package; import the metric classes withfrom dynamicio.schema.metrics import ...(ordynamicio.metrics), which keep working, not through thedynamicio.schema.metricsattribute - breaking: the raw
metadata={"dynamicio": {"metrics": [...], "validations": {...}}}dict is rejected with aValueErrorthat names the helpers above - feat:
dynamicio migrateis the path from version 8: it converts YAML schemas to models for version 11 with__dataset_name__, the typed helpers, and__table_name__for every schema that a resource definition under--scanbinds to a Postgres environment. The generated models keepclass Config: strict = "filter"(withoutname), soModel.validate(df)still drops unlisted columns. The prompt it writes lists the call sites for version 11 (SCHEMA_FROM_FILE,SafeDynamicSchemaLoader,schema:blocks, raw metadata dicts,Config) - feat: the
dynamiciocommand line (dynamicio -s|-b -p PATH -o DIR) writes a Pandera model module per dataset (<dataset>.py) instead of a YAML schema - docs: the schemas guide, quickstart, examples and README describe models only; new ADR 0009. The demo project's models use
__dataset_name__and the typed helpers
v10.4.1
- docs: the
Documentationproject link of the package, and the README, point to the documentation on Read the Docs (https://dynamicio.readthedocs.io/en/latest/), and the documentation home page shows the panda mascot next to the logo - docs: remove the changelog sections of the
v8.1.0release candidates.
v10.4.0
- feat: the constructor of an IO class takes
missing_ok: bool = False. Withmissing_ok=True,read()andasync_read()returnNoneand log a warning when the reader raises aFileNotFoundError. The caller handles theNone, so the setting is in the code of the caller - chore: remove
missing_okfrom the resource definition. Release v10.2.0 added it, and it has no known user. A resource definition with amissing_okkey now fails to load, because dynamicio reads the key as an environment name.IOBinding.missing_okandIOEnvironment.missing_okare removed, and thedict()output of a binding has nomissing_okentry. Move the setting to the constructor of the IO class - docs: new "Why dynamicio" page and a rewritten home page and README that explain the purpose: business logic independent of I/O, fast local feedback loops, and what that gives coding agents. The documentation is restructured into getting-started, guides, examples, API reference, architecture decisions and development pages (
docs/), built with MkDocs for Read the Docs (.readthedocs.yaml; versionedlatest,stableand per-release documentation). The README is now a concise overview and the long walkthrough moved into the guides, with its content unchanged. The Python examples of the quickstart and the basic I/O page are executed by the test suite. Polars is still not supported. - docs: docstrings that the API reference could not parse (
Returns:,Args:andRaises:entries ofDynamicDataIO.async_read,IOConfig,WithS3,WithPostgres,WithKafka,allow_options,args_ofandOutputSandbox) are corrected.args_ofgains aCallableannotation on*funcsandOutputSandboxannotatesdirectory; behaviour is unchanged. - chore: the package metadata gains
Documentation,SourceandIssuesproject URLs next toChangelog. TheDocumentationURL points at thedocs/directory until the Read the Docs site is confirmed. - ci: the "Build and Publish" workflow builds the documentation in strict mode on every push, builds and validates the distributions once (
twine check, tag against version) and publishes exactly those files. A release tag must bevX.Y.ZorvX.Y.Z-rc.N, a stable tag must be onmasterand have a changelog section, and the GitHub release notes are the changelog section plus links (.github/tools/release_tools.py, tested). Re-running a publish is safe: files already on PyPI are skipped and an existing GitHub release is updated. - chore: the
docsdependency group no longer includesmkdocs-techdocs-coreand stays onmkdocs<2;docs/requirements.txtpins the documentation tooling fromuv.lock.
v10.3.0
- chore: remove the
magic-loggerdependency. Each dynamicio module logs throughlogging.getLogger(__name__), with the same logger names as before.magic-loggerhas only a source distribution, so an install withuv sync --no-buildstopped with an error.magic-loggeralso calledinspect.stack()on each log call.dynamicio.loggeris alogging.Loggerfor the loggerdynamicio, and it does not have thedict_config,file_config,listenandstop_listeningmethods. The logging config ofdynamicio/__init__.pynames the loggerdynamicio, sodisable_existing_loggersdoes not disable the module loggers. If you setDYNAMICIO_DO_NOT_CONFIGURE_LOGGING=1and calllogging.config.dictConfigafter the dynamicio import, name the loggerdynamicioin your config or setdisable_existing_loggerstofalse. If you do not, your config disables the dynamicio loggers
v10.2.0
- feat: a resource definition can set
missing_ok: trueon a source, beside its environments. When the reader of that source raises aFileNotFoundError(a missing local file,ObjectNotFoundError,NoObjectsFoundError),read()andasync_read()returnNoneand log a warning. All other errors raise as before. The default isfalse.IOBinding.missing_okandIOEnvironment.missing_okexpose the flag, which adds amissing_ok: Falseentry to thedict()output of a binding. See "Optional sources:missing_ok" in the README - fix: an S3 read of a csv or json
file_pathraisesdynamicio.errors.ObjectNotFoundError("No object found at s3://...")when the key does not exist. Before, it raised the awswranglerNoFilesFound, which is not aFileNotFoundError.ObjectNotFoundErroris aFileNotFoundErrorand aNoFilesFound, so a caller can treat the key as a missing source, and code that catchesNoFilesFoundworks as before. The errors of the parquet and hdf reads for a missing key do not change
v10.1.0
- docs: refresh the standalone logo, clarify pandas-only support, remove unused and superseded images, and expose architecture decisions in the documentation navigation; the S3 benchmark page is replaced by ADR 0007, which keeps the headline figures (the full tables stay in PR #88).
- feat: resource definitions accept an optional
expected: {path, file_type, options}key per environment, the reference file a test compares the generated output against. Reading and writing ignore it, so existing definitions and their behaviour are unchanged. (IOEnvironment.expectedisNonewhen absent, which adds anexpected: Noneentry to the model'sdict()output.) - feat: the
dynamicio.testharnesspackage, to test a task end to end: run it in aLOCALenvironment, then lazily compare each generated output with itsexpectedfile.ExpectedOutputscollects all mismatches and raises oneOutputMismatchError; generated files of matching outputs are removed (the others are kept); JSON is read with the resource'sorient(orexpected.options); paths can carry{placeholders}filled fromparams;DYNAMICIO_UPDATE_EXPECTED=1regenerates the expected files.Expectationscurates how strict a comparison is (row/column order, dtypes, tolerances,sort_by,only_columns,ignore_columns,list_columns_to_sort, customassertions;strict()/lenient()presets,with_()to derive variants), per session or per check; dict columns are compared by content.FrameComparisonandOutputFilesare usable on their own. See "Testing Locally" in the README. - feat:
dynamicio.testharnessadditions:ExpectedOutputs.check_each(config, key, grid)checks a resource whose path has several{placeholders}for every combination of their values (clear_eachremoves their leftovers);Expectations(check_parquet_schema=True)also compares the arrow schema and stored pandas index of parquet outputs;OutputSandbox(directory, *configs)redirects the local outputs of a task into a scratch directory so tests that write the same files can run in parallel;ExpectedOutputs(environment=...)makes everyIOConfigthat defines that environment select it inside thewithblock (and restores them on exit), so the task under test runs and is checked in an environment the test states, without patchingenv_identifier;check(..., environment=...)overrides it for one check,configs=[...]narrows the switch,EnvironmentSwitch(environment, *configs)does the same explicitly and rejects an environment a config does not define. Withoutenvironment=, the default is the first environment the resource definitions list, whatever the process runs in.IOConfig.instances()andIOConfig.default_environmentsupport this. The README gains "Choosing the environment" and "Recommended testing practices". - fix:
dynamicio.testharnessreview fixes: JSON Lines outputs (options.lines: true) are read back withlines(it is passed withorientto the JSON reader only); a column named insort_by,only_columnsorlist_columns_to_sortthat is missing from the output or the reference is reported as a failed check, not aKeyErrorthat aborts the remaining checks; update mode (DYNAMICIO_UPDATE_EXPECTED=1) refuses to copy an output over a reference declared in another file type, leaving both files untouched. - feat:
ExpectedOutputs.check_all(config, skip=, rules_for=, params=)checks every resource of a config that declaresexpected(a new output is covered without touching the test; resources with unfilled{placeholders}are listed inskipped), andExpectedOutputs.clear_all(config)removes the stale generated files of those resources. - feat: a passing test run now says what it verified.
ExpectedOutputsprints aSessionReportat exit (green✓per matched output with its rows x columns and expected file, red✗per failure,↻updated,-skipped),outputs.reportreturns it,OutputMismatchErroralso lists the outputs that matched, and a pytest plugin (pytest11entry pointdynamicio;-p no:dynamicioturns it off) prints every report in the terminal summary, coloured. Nothing is printed when a session checked nothing. - docs: the README walkthrough follows the
demo/project: four resource definitions (staging_input,staging_output,transform_input,transform_output) withLOCAL,DEVandPRODenvironments,expectedpaths on the outputs'LOCALenvironment, and thestaging_*/transform_*configs, instead ofsample/actualandinput/raw/processed. - docs: the
demo/project now has one input and one output definition per runner (staging_input,staging_output,transform_input,transform_output) withLOCAL,DEVandPRODenvironments, and its end-to-end test usesExpectedOutputs. The definitions use YAML anchors and merge keys (&,*,<<:) soPRODonly restates what differs fromDEV.
v10.0.1
- fix: an S3 path-prefix read with no objects under the prefix raises
dynamicio.errors.NoObjectsFoundError("No objects found under s3://..."). Before, it raised the pandas errorValueError: No objects to concatenate, which a caller cannot tell apart from otherValueErrors. The change applies to all the file types (parquet, csv, json, hdf) and to the hdfno_disk_spaceread.NoObjectsFoundErroris aFileNotFoundErrorand aValueError, so a caller can treat the prefix as a missing source, and code that catchesValueErrorworks as before. An S3 parquetfile_paththat ends in/and has no objects raises the same error. It raised a plainFileNotFoundErrorbefore. Objects that are all empty do not change: a parquet prefix returns an empty frame, and a csv, json or hdf prefix raises the pandasValueErroras before
v10.0.0
- fix:
dynamicio migratewrites a YAMLobjectcolumn asSeries[object], notSeries[str].objectmeant "any values", and list columns declaredstrfail plain Pandera validation (and would be recast under pandas 3). Models already generated withSeries[str]for list columns should be changed toSeries[object]; the generated module now says wherestris safe - docs: the README diagrams are now text-driven SVGs (supported sources with Athena next to S3, Postgres and Kafka; the sample pipeline as an Airflow DAG; how a read works; the file formats), they render on GitHub and on PyPI, and the Pandera logo is shown in the Pandera models step. The table of contents anchors and the
CONTRIBUTING.mdlink are fixed, and the dependency list names what the package really uses. Thedemo/example datasets are generic (orders). - fix: the dtype check compares dtypes instead of their names. A column already at
string[pyarrow](whose name isstring) was recast on every read and logged a casting warning; a barecategorystill accepts any categorical dtype and a barestringany string storage (string[python]orstring[pyarrow]ask for one). A declared type pandas does not know is reported by the cast as before. Columns that were cast before are cast as before - feat: Pandera constructs that dynamicio has no validator for now run instead of being ignored or raising a bare
KeyError. Column checks such asstr_matches,str_lengthand custom@pa.checkrun as Pandera checks;@pa.dataframe_check,Config.unique,Config.orderedand index declarations run on the whole frame. Failures are raised inSchemaValidationErrorunder the check's name - feat: Pandera options dynamicio cannot honour (
Config.strict = True,add_missing_columns,on_missing_columns,unique_column_names,drop_invalid_rows,parsers,dtype, and per columnregex,required=False,default,on_missing) and non-pandas (for example polars) models raiseValueErrorwhen the schema is compiled, instead of being dropped. Usestrict = "filter".coerceis honoured through the compiled dtype (a nullable int becomesInt64) - fix:
Series[str]compiles to the pandas default string dtype (objecton pandas 2,stron pandas 3) instead of a hard-codedobject. Existing pandas 2 behaviour is unchanged - feat: models written by
dynamicio migratesetstrict = "filter", soModel.validate(df)on its own drops columns that are not in the schema, as dynamicio does.coercestays off: Pandera would turn a nullable int column intoInt64, where dynamicio casts toint64. Dynamicio's own validation is unchanged - fix: no more
UserWarning: Valid config keys have changed in V2: 'underscore_attrs_are_private' has been removedon import under pydantic 2. The key is now only set under pydantic 1, where it is needed to keepIOEnvironment._parentprivate - fix:
WithPostgresreads and writes work when the table is declared only by a Pandera model. The table name and columns come from the resource YAML'sschemawhen it has one (unchanged), otherwise from the model (Config.nameis the table name). Before, a postgres resource without a YAML schema failed withAssertionError: The schema must be specified for SQL tables - feat:
dynamicio migrate -o <package dir>writes one models module per sub-directory of the YAML schemas (the grouping by runner and input/output carries over);-o file.pystill writes a single module. The README recommends one models module per runner stage or definitions file, and thedemo/project now follows it (demo/src/resources/schemas/{input,processed}.py) - fix:
dynamicio migratehandles real-world schemas: mixed-typeis_invalues (incl.null),is_betweenbounds that are[[ VARIABLES ]], dtype aliases such asdouble/int, and explicit default options; a failed parity check now lists the differing columns. - refactor: schema modules now live in a
dynamicio.schemasubpackage (schema.validations,schema.metrics,schema.models,schema.migrate).dynamicio.validationsanddynamicio.metricskeep working as aliases of the same module objects, and thedynamicio.metricslogger name is unchanged.dynamicio.pandera_schemaanddynamicio.migrate(new in 10.0.0-rc.1 only) moved todynamicio.schema.modelsanddynamicio.schema.migratewithout aliases. - refactor: the helpers of
dynamicio.schemaare grouped into classes (ModelCompiler, and formigrate:SchemaComparison,ModelRenderer,LegacyDeclarations,MigrationPrompt,Migration), and_is_declared_dtypeis a static method ofDynamicDataIO. The public function names (to_dataframe_schema,dataframe_validator,migrate,render_model,render_module,compile_module,schemas_equivalent, ...) remain as aliases, so imports keep working. - chore: raise the
numpyfloor to 1.24.4, the oldest release Pandera 0.34 accepts (found by thedependency-ranges (lowest)job). - Docs: README review. Fixed stale schema wording, the
is_betweeninclude_rightdefault, the casting-failure exception (ColumnsDataTypeError), unique Step 4.3 anchors, and added a summary of validation/metrics behaviour to the closing notes. - test:
tests/test_contracts/pins the validation, metric, lifecycle-order and dtype behaviour that the schema work must preserve, for every schema declaration style - feat: schemas can be declared as Pandera
DataFrameModelclasses (class MyIO(UnifiedIO): schema = MyModel). Dynamicio's validators and metric logging are unchanged, so error payloads andMETRIClog lines are identical to the YAML and dict styles.panderais now a required dependency. Anything Pandera cannot express natively goes inField(metadata={"dynamicio": {"validations": {...}, "metrics": [...]}}). Note Pandera's default isnullable=False; dynamicio's YAML allowed nulls unlesshas_no_null_valueswas declared - fix: when the same validation fails on several columns,
SchemaValidationError.messagenow holds every column's message (joined with;under the one validation-name key). Previously only the last column's message survived - feat:
schema=constructor argument andSCHEMA_FROM_ARGUMENTplaceholder, so one IO class can take a different Pandera model per instance (the Pandera counterpart ofSCHEMA_FROM_FILE). The argument takes precedence over the class attribute - feat:
dynamicio migrate <schemas dir> -o schemas.pyconverts YAML schemas to Pandera models (dataset names, validations and metrics preserved, verified by a parity check) and writesMIGRATION_PROMPT.mdlisting the call sites to update by hand or with an LLM. The existing-b/-sschema generator is unchanged - docs: README Step 4.1 (Pandera models) and migration guide, ADR 0005; the
demo/project now declares its schemas as Pandera models and keeps its resource definitions, models and IO classes together underdemo/src/resources(withenvironment.pyandconstants.pyunderdemo/src/config), withdemo/testsmirroringdemo/src, documented in the README as the recommended pipeline project structure
v9.2.3
- fix:
WithPostgresbuilds one SQLAlchemy model per table definition and reuses it, instead of a new same-named class on every read or write. The classes piled up in the declarative registry, and SQLAlchemy's cleanup of them could raise aRecursionErrorduring garbage collection (seen as a failing test on the lowest-dependency CI job). Each definition has its own declarative base, so two definitions of one table cannot alter each other's columns, and the cache is thread-safe forasync_read/async_write. Queries and tables are unchanged.
v9.2.2
- fix: an IO object can be read or written repeatedly. A read or write removed values from
self.options, so a second call on the same object failed or read the wrong data. The second call of a path with template fields (for example{runner_id}) raisedValueError: Expected values for all fields. Three cases gave no error. The S3 path-prefix read removedno_disk_space. The Athena read removedquery, and the Postgres read removedsql_query, so the second read ran the default query and returned wrong data. Each call now works on a copy of the options, andself.optionsdoes not change. The two private methodsWithS3._read_parquet_objectandWithS3._concat_parquet_objectsare removed. The module-level functions with the same names are unchanged
v9.2.1
- fix: transient S3 connection errors no longer fail a read or write. The shared boto3 client now uses the
standardretry mode with 10 attempts,connect_timeout=10andread_timeout=60. The in-memory and file transfers (put_object,upload_fileobj,get_object,download_fileobj) are repeated, up to 5 attempts with backoff, onConnectionClosedError,ConnectTimeoutError,EndpointConnectionError,ReadTimeoutError,IncompleteReadError,ResponseStreamingErrorand theRetriesExceededErrorthat the transfer manager raises (s3transfer's and boto3's) when a download above 16 MiB gave up on a broken part. botocore does not retry these once the body is being sent or read. The buffer is rewound before each attempt (bytes written are identical), each retry is logged at WARNING with the bucket and key, and the shared client is rebuilt so the next attempt opens fresh connections. Other errors (for exampleClientError) are raised as before, and nothing changes when no error occurs
v9.2.0
- feat: a single-file S3 parquet read with
filtersreads only the row groups whose footer statistics can match, over rangedGetObjectrequests throughpyarrow.fs.S3FileSystem(one for each process and bucket). The rows, index and dtypes are the same as before. Reads withoutfiltersor withfilters=[], fastparquet reads, reads with a non-pyarrowfilesystem, prefix reads and reads with wrangler-only kwargs do not change. On the ranged read, a missing key or a permission problem raises pyarrowFileNotFoundErrororOSErrorin place of botocoreClientError, and a custom S3 endpoint of boto3 (AWS_ENDPOINT_URL) does not apply. See ADR 0004 - feat: new postgres write option
create_table(defaultFalse). WhenTrue, the writer creates a missing table from the schema columns, with no primary key and with unbounded string columns, in the same transaction as the write. The postgres writer now lists its options (truncate_and_append,create_table), so it drops other options with a warning, as the other writers do. Acreate_tablevalue that is not a bool (for example the YAML string"false") raisesTypeError - fix: the cache of pyarrow S3 file systems is cleared at exit, before the pyarrow S3 finalizer runs. With pyarrow 24 and later, a process that read with
filterson a pod with web identity credentials did not exit (https://github.com/apache/arrow/issues/50188) - chore: dev dependency
moto[server]>=5for the tests of the ranged read
v9.1.0
- chore(deps): dependency audit, ranges widened to what the code needs and CI proves. Removed packages the code never imports (
fsspec,libcst,logzero,no_implicit_optional); dropped exact pins (confluent-kafka>=2.4.0,<3,psycopg2-binary>=2.9.3,<3,python-json-logger>=2.0.1,simplejson>=3.17.2); lifted thenumpy<2ceiling (numpy 2.x passes the suite) and thepython<3.13ceiling (3.13 and 3.14 pass; added to CI and classifiers). Raised floors so every allowed combination works with numpy 2 as well as numpy 1:pandas>=2.2.2(JSON writes withindex=Falseforrecords/values, relied on since 8.1.1, need pandas 2.1, and pandas before 2.2.2 breaks with numpy 2),pyarrow>=16.0.0,fastparquet>=2024.5.0,awswrangler>=3.9.0(older ones cannot import against numpy 2 or pyarrow 16),PyYAML>=6.0(5.4.1 does not build with current toolchains). New CI jobdependency-rangestests the minimums (Python 3.10), the minimums with numpy 2 (Python 3.11) and the newest releases (Python 3.14); publishing waits for it - fix:
python-json-logger3.1+ import path (pythonjsonlogger.json) with fallback to the old module, avoiding aDeprecationWarningon import
v9.0.0
Breaking: Athena resources must rename s3_staging_dir to s3_output (see the first Athena entry below).
- docs: add Backstage
catalog-info.yamlto enable TechDocs - docs: README documents Athena I/O (resource options,
ctas_approach, dtypes,s3_staging_dirmigration); ADR 0003 - chore: migrate project and dependency management from Poetry to uv (PEP 621
pyproject.toml,uv.lock,hatchling+hatch-vcsbuild backend). The version is still derived from the git tag and the published wheel is unchanged apart from standard metadata. Contributors now usemake dev-env-setup/uv sync; seeCONTRIBUTING.mdandAGENTS.md - feat(RND-13624)!:
WithAthenareads through AWS Data Wrangler (wr.athena.read_sql_query) instead of PyAthena, and thepyathenadependency is dropped. Breaking (resource definitions): thes3_staging_dirkey of anathena:section is removed. Loading such a definition now fails with an error saying to rename it tos3_output(same value); nothing else changes in the YAML.region_nameandquerystay optional, anddatabase(defaultdefault, Athena's own default) andworkgroupare new optional keys. Reads use the shared per-process boto3 session unlessregion_nameis set. Extraoptionsare filtered to whatread_sql_queryaccepts (e.g.ctas_approach,unload_approach,athena_cache_settings);chunksizeis rejected with an error, because a chunked read returns an iterator rather than a DataFrame ctas_approachdefaults toFalse(as PyAthena, which read the query's result file): column names keep the case used in the query (CTAS lower-cases them) and no temporary Glue table is created. Passctas_approach: trueinoptionsto opt in for large results- note: results now come back with pandas nullable dtypes (
Int64,string,boolean) anddateas Python dates, as returned by awswrangler; declare the dtypes in the resource schema to cast them on read, as for any other source
v8.1.1
- fix: S3 JSON reads and writes use pandas' defaults again:
lines=False(a JSON array fororient="records", as written by 7.x and earlier) unless the caller passeslines. 8.1.0 defaulted to JSON Lines fororient="records", which could not read existing files and wrote files that 7.x readers cannot parse. Setlines: truein the resourceoptionsto opt in to JSON Lines - fix: S3 JSON writes default
indexper orient as pandas does (Trueforindex/columns/split/table,Falseforrecords/values) instead of alwaysFalse, which madeorient="index"andorient="columns"writes fail or drop the index
Metadata
Release files for dynamicio 11.0.0rc2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dynamicio-11.0.0rc2.tar.gz | 3.1 MB | Without provenance | Details for dynamicio-11.0.0rc2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dynamicio-11.0.0rc2-py3-none-any.whl | Python 3 | none | any | Details for dynamicio-11.0.0rc2-py3-none-any.whl |
Total release size: 3.3 MB
Release files / dynamicio-11.0.0rc2.tar.gz
| Download URL | dynamicio-11.0.0rc2.tar.gz |
|---|---|
| Size | 3.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
db25dd137822152a9acfac0ee22e2b7ad5134583a6c4b704623dee804782d60d
|
|
BLAKE2b-256 checksum How to use checksums |
14f017a676c9ad774f96af8f67b773d89c2bdfe97177215f52f31eef98ccb6ef
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / dynamicio-11.0.0rc2-py3-none-any.whl
| Download URL | dynamicio-11.0.0rc2-py3-none-any.whl |
|---|---|
| Size | 112.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
93e6ce7e5494310b8e05204394c0568d57778a1fab62c0c4d16d23e5e4bf0e56
|
|
BLAKE2b-256 checksum How to use checksums |
53ac69ca6502478b5b751421752ed0988c74153544cc3d2ffdca113d4a11f714
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for dynamicio 11.0.0rc2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.