This release is a pre-release and may not be stable for production use.
Keep your business logic independent of I/O. dynamicio is the seam between your pandas or polars code and the places your
data lives: describe each dataset once in YAML, and your code just calls read() and write().
The same code reads a parquet file on your laptop, a CSV prefix on S3, a PostgreSQL table or an Athena query. You pick which one by changing a single identifier, so developers stop worrying about connection strings, file-format keyword arguments and which bucket a task reads this week.
Documentation: dynamicio.readthedocs.io: guides, examples and API reference · Changelog: CHANGELOG.md
Why dynamicio
- Logic that does not know about storage. Resource definitions are YAML, per environment (
LOCAL,DEV,PROD, or your own). Moving a pipeline between environments never touches Python. - Fast, safe feedback loops. Run the real pipeline against sample files on your machine, with no cloud credentials, no deploy and
no mocks of external services. Compare the outputs with expected files using the built-in test harness (
dynamicio.testharness). - Made for the LLM era. A coding agent that changes a pipeline needs somewhere safe to run it and a check that says whether the
change was right. A
LOCALenvironment and expected outputs give it both, and production credentials never enter the loop. - Contracts for your dataframes. A Pandera model declares columns, dtypes and validations. Bad data fails at the boundary with every violation listed, not three steps later, and the same model can emit metrics.
- Library behavior, not a new dialect. Options in the YAML go to
pandas/awswrangleron the pandas engine, and to the polars readers and writers on the polars engine, so what you know still applies. - Sync and async. Read and write from plain code or from
asyncioservices.
Supported sources
| Source | Read | Write | Formats |
|---|---|---|---|
| Local files | yes | yes | parquet, hdf, json, csv |
| S3 (single object or prefix) | yes | yes | parquet, hdf, json, csv |
| PostgreSQL | yes | yes | tables |
| AWS Athena | yes | no | SQL query results |
| Kafka | no | yes | messages |
The pandas engine is the default. The polars engine runs the same IO classes on polars frames. It reads and writes parquet, CSV and JSON files on local and S3 resources, and it reads and writes Postgres. It reads Athena and writes Kafka. One process uses one engine. See the polars engine.
Install
pip install dynamicio
# or, in a uv project
uv add dynamicio
# with the polars engine
pip install "dynamicio[polars]"
Python 3.11 to 3.14.
See it work
Describe the resource in orders.yaml:
ORDERS:
LOCAL:
type: local
local:
file_path: "[[ DATA_DIR ]]/orders.parquet"
file_type: parquet
Bind it to a schema, then read and write:
import pandas as pd
from pandera.typing import Series
from dynamicio import UnifiedIO
from dynamicio.config import IOConfig
from dynamicio.schema import DataModel, Field
class Settings:
DATA_DIR = "."
class Orders(DataModel, dataset="orders"):
order_id: Series[int] = Field(nullable=False, unique=True)
amount: Series[float] = Field(nullable=False, ge=0)
class OrdersIO(UnifiedIO):
schema = Orders
config = IOConfig("orders.yaml", env_identifier="LOCAL", dynamic_vars=Settings)
orders = OrdersIO(source_config=config.get(source_key="ORDERS"), apply_schema_validations=True)
orders.write(pd.DataFrame({"order_id": [1, 2, 3], "amount": [9.5, 20.0, 3.25]}))
df = orders.read()
To run the same code against S3, add a DEV block to the YAML and set env_identifier="DEV". The Python does not change.
Where next
- The idea: why dynamicio.
- New here? The quickstart explains each step above, and its code is run by the test suite.
- Core ideas: concepts: resource definitions, IO classes, schemas.
- Task guides: reading, writing, testing pipelines.
- Reference: API reference.
Community and contributing
- Questions and bugs: GitHub issues
- Contributions are welcome: CONTRIBUTING.md
- Source: https://github.com/VorTECHsa/dynamicio · PyPI: https://pypi.org/project/dynamicio/
- License: Apache-2.0 (LICENSE)
Original logo illustrated by Nick Loucas; the refreshed logo retains its circular I/O motif.
Changelog
v12.0.0
- breaking:
write()on the pandas engine casts the frame before its validations and metrics, in the order of the polars engine. A validator and a metric thus see the columns of the schema with the dtypes of the schema. In 11.x they saw the frame of the caller, and the cast came after them - breaking: a schema column that is not in the frame raises
ColumnsDataTypeErroron the pandas engine, as on the polars engine. 11.x raisedKeyError. This applies to reads and writes. For example, a pandas read of a JSON file without a schema column, or without records, raisesColumnsDataTypeError. A CSV or parquet read still raises the error of its reader - breaking: dynamicio needs Python 3.11 or later. polars 2.1 needs Python 3.11, and Python 3.10 reaches its end of life in October 2026. CI runs Python 3.11 to 3.14, and the
lowestdependency-range job runs on Python 3.11 - feat: an engine is the dataframe library of a process.
DynamicDataIOsends its read, write, cast, validations and metrics to the engine, in the same order as before. The pandas engine is the default and runs the code of 11.x, so a pandas process does not change and does not import polars. The engine classes are indynamicio.engines - feat:
dynamicio.configure(engine="pandas" | "polars")sets the engine. The first read or write fixes it and logs it once. After that, another engine raisesEngineConfigurationError. An engine that dynamicio does not have also raisesEngineConfigurationError, and the polars engine without polars installed raisesImportError - feat: the polars engine takes
pandera.polarsmodels and checks frames. It reads and writes thelocal,s3_file,s3_path_prefixandpostgresresources, readslocal_batchandathena, and writeskafka. A Kafka read and an Athena write raise theNotImplementedErrorof the pandas engine - feat: the polars engine reads and writes parquet, CSV and JSON files on a
localor ans3_fileresource, and reads alocal_batchresource. A file that it writes has the bytes of the pandas engine of 11.1.0 for CSV and JSON. A parquet file has the arrow types, the pandas metadata and the compression ofDataFrame.to_parquet, so the pandas engine reads it back with the pandas dtypes of the model. A nullable integer column keeps integers above 2**53 in a CSV or JSON write,decimal: ","reads back the floats that it writes, and a parquet write withindex: falsehas the pandas metadata ofto_parquet(index=False) - feat: the polars engine reads a file of the pandas engine. A CSV column that is not a number or a Boolean reads as a string, so
007keeps its text, and the cast then parses the datetimes, the durations and the Booleans. A JSON integer under a temporal column is an epoch indate_unit. A JSON file without records reads as a frame without rows, with the columns of the model - feat:
EngineResourceErrorfor a file type or an option that the polars engine cannot read or write, for example anhdffile, adtypeoption, a JSON orient other than records, a CSVna_valuesmap for each column, or a CSV read without a header line and withoutnames. The polars engine raises it when the IO object is built, and again at each read and write. It is also aValueError - feat:
scan()on the polars engine gives aLazyFrameof alocal,local_batch,s3_fileors3_path_prefixresource, with the casts of the model in its plan. Parquet and CSV usescan_parquetandscan_csv, so a filter or a selection reads only the row groups and the columns that it needs. A JSON scan reads the file eagerly and then callslazy(). A failed cast raises when the frame is collected. A scan does not validate or log metrics, and logs a warning if the IO object asks for them. A missing file or key raises the error ofread(), andmissing_ok=TruegivesNone - feat: the polars engine reads an
s3_path_prefixresource withpath_prefixordynamic_file_path, for parquet, CSV and JSON.read()collects the scan, so both give the same frame. The parquet scan reads the footer of each object: a column has the widest type of the footers, aspd.concatgives it, and a type pair that does not upcast raisesColumnsDataTypeError.Durationcolumns give the finest unit. The objects without rows do not count when the types do not upcast, as on the pandas engine. An object that lacks a column gets nulls and the warning of the pandas engine, and a column that no object has raises theValueErrorof the pandas engine. A prefix without objects raisesNoObjectsFoundError - feat: the polars engine reads S3 with the credentials and the region of the boto3 session of dynamicio, and refreshes them when they expire. A boto3 endpoint that is not an AWS endpoint, for example a local S3 stack, also goes to polars
- feat: the options
no_disk_space,partition_cols,modeandallow_deleteare accepted on ans3_path_prefixresource of the polars engine, and refused on the other resources - feat: the CSV
namesoption on the polars engine names the columns of the file, as on pandas. Withoutheader, the first line is data, andheader: 0replaces the names of the header line. A file with another number of columns raises theSchemaErrorof polars - feat: a missing S3 key on the polars engine raises the error of the pandas engine:
S3KeyNotFoundErrorfor parquet, andObjectNotFoundErrorfor CSV and JSON.missing_ok=TruegivesNone - feat: the polars engine casts a frame to the dtypes of its model, and changes float NaN to null. A
LazyFramegets the casts in its plan. A write changes float NaN to null before its validations and metrics, so a NaN failshas_no_null_valuesas on the pandas engine. A column that does not cast, a schema column that is not in the frame, and a column that is not in the schema raiseColumnsDataTypeError. A string column parses to aDatetime. The cast messages go to thedynamicio.corelogger. The engine ignoresstrict_dtype_check - feat: the polars engine runs the 9 validators of the pandas engine, with the same names, messages and values, and the Pandera column checks (
str_matches, a custom@pa.check). It also runsConfig.unique,Config.orderedand@pa.dataframe_check, with the keys and the message template of the pandas engine. For a dataframe check, the count is the number of failing rows, or 1 for a check that returns one bool.Config.uniquetakes a list of column groups.is_inaccepts a null when the pandasisinaccepts it. All of them run in onepl.collect_all, and the failures raise oneSchemaValidationError.PolarsValidations.validatorregisters a polars validator, as@validatordoes for the pandas engine - feat: the polars engine logs the 8 metrics of the pandas engine, with the same
METRIClines. It computes all of them in oneselect, after the validations. For an empty or all-null column,Min,Max,Mean,StdandVariancegive NaN, andCountsandUniqueCountsgive the counts of the pandas engine, such as 0 for an empty column.CountsPerLabellogs 0 for anEnumcategory that is not in the column.PolarsMetrics.metric("Name")registers a polars metric, as aMetricsubclass does for the pandas engine - feat:
dynamicio.schema.polars_dtypes.DtypeMapis the map between the pandas and the polars dtypes. It has the signed and unsigned integers, the floats,Boolean,String,Categorical,Enum,Datetime(ms,us,ns, with a time zone) andDuration.datetime64[s],Date,List,Decimaland the other dtypes have no pair, and a model with one raisesValueErrorat compile - feat:
Field(pandas_dtype="Int64")sets the pandas dtype of a column of a polars model, for example a nullable integer orstring[pyarrow]. It must be a pair of the polars dtype of the column. A column without it takes the pandas dtype of the reverse map, for exampleint64forInt64andobjectforString. A pandas model withpandas_dtyperaisesValueError - feat:
dynamicio.schema.polars.DataModelis the base class of a polars model, with the samedatasetandtableclass keywords asdynamicio.schema.DataModel.ModelCompilercompiles apandera.polarsmodel. Thedata_typeof a compiled column is a pandas dtype name, from the dtype map - feat:
DynamicDataIO.scan()andsink()read and write a lazy frame. On the pandas engine they raiseNotImplementedError, with a message that names the polars engine. The refusal does not fix the engine - feat:
ModelEngineMismatchErrorfor a model of the other engine, at the creation of an IO object and at each read and write.WrongFrameErrorfor a frame of the other engine atwrite()andsink(), before any I/O. Both are also aTypeError, andModelEngineMismatchErroris also aValueError, as the 11.x error of a polars model. A refused read or write does not fix the engine - feat:
sink()on the polars engine writes aLazyFrameto alocal, ans3_fileor ans3_path_prefixresource. It runs the write, the validations and the metrics in onepl.collect_all. A parquet sink streams the rows to a temporary file. It then copies the row groups to a file with the arrow types, the pandas metadata and the options ofwrite(). That file thus has the footer and the row groups ofwrite(). A CSV, a JSON or a prefix sink collects the frame and calls the writer ofwrite(), so the bytes are equal - feat: a failed validation of
sink()leaves the old file or object. A local parquet sink over a file keeps the mode of the old file, aswrite()does.sink()logs the metrics only after the validations pass. A failed cast in the plan raisesColumnsDataTypeError - feat: the polars engine writes an
s3_path_prefixresource with the prefix writer of the pandas engine. Thuspartition_cols,modeandallow_deletegive the same objects and delete the same old objects. A prefix read on either engine does not read thecol=valuedirectories back as a column - feat:
dynamicio.engines.s3_upload.S3Uploaduploads a parquet file of the polars engine to S3 with thebucket-owner-full-controlACL. An object above 64 MiB goes up in parts of 64 MiB, 4 at a time. A failed part aborts the upload, so S3 keeps no part and the old object does not change. A broken connection starts the upload again from the start - feat:
DynamicDataIO.footer()reads the arrow schema, the metadata and the row count of alocalors3_fileparquet file, without its rows. It is the same on the two engines and does not fix the engine. On S3 it reads the end of the object with rangedGetObjectcalls.missing_ok=TruegivesNonefor a missing file - feat:
sink()on the polars engine takes a list or a tuple of lazy frames, the parts of one frame in their order. Each part gets the schema columns and the casts of the model, so the parts can have other types of a column and another column order. An empty list raisesValueError - feat: the polars engine reads a
postgresresource with the query of the pandas engine, sosql_queryandparams(in the style of the driver) give the same rows. Apd.read_sqloption that changes the frame, for exampledtype, raisesEngineResourceError - feat: the polars engine writes a
postgresresource with the COPY of the pandas engine, in chunks of 100,000 rows and in one transaction. Each chunk changes to pandas with the pandas dtypes of the model, so the NULL rule and the text of each value do not change. A failed chunk rolls back the write - feat: the pandas engine maps
datetime64[us]anddatetime64[ms]to a Postgrestimestamp, asdatetime64[ns] - feat: the Kafka option
key_columnnames the column of the message keys on the two engines. The column is not in the message values. On the pandas engine it gives the bytes ofset_index(key_column). A value that is not a column name, or a column that the frame does not have, raisesValueErrorbefore the first message - feat: the polars engine writes a
kafkaresource with the producer of the pandas engine. The resource needskey_column, and a missing or wrongkey_columnraisesEngineResourceErrorwhen the IO object is built. The frame goes to pandas in chunks of 100,000 rows, with the dtypes of the model. The user callables thus get the values and types of the pandas engine. The producer flushes once - feat: the polars engine reads an
athenaresource with the awswrangler query of the pandas engine, and casts the result to the model - feat: a fork of a process that imported polars refuses the polars engine, because polars can deadlock in a fork. Its first polars read or write and
configure(engine="polars")raiseEngineConfigurationError, with a message that namesspawnandforkserver. The fork gets a new engine lock and can still use the pandas engine - feat: dynamicio does not set the number of polars threads, and
import dynamiciodoes not import polars.POLARS_MAX_THREADSset afterimport dynamiciothus applies - feat: the
dynamicio[polars]extra installspolars>=2.1 - feat:
dynamicio.testharnessreads and compares the outputs with the engine of the process. On the polars engine it reads parquet, CSV and JSON records with the polars readers, and compares withpolars.testing.assert_frame_equalunder the sameExpectations. A custom assertion gets polars frames. Withoutsort_by,check_row_order=Falsecompares the rows of a polars frame as a multiset.list_columns_to_sortsorts the lists of aListorArraycolumn and keeps the values of another column, as on pandas. A polars error of a comparison, for example a sort of anObjectcolumn, fails that comparison, and the harness checks the other outputs - feat: the pytest plugin adds the option
--dynamicio-engine=pandas|polars, which callsdynamicio.configure()before the first test - feat:
dynamicio migrate --engine polarswrites a polars model for each pandas model of a module or a package, checks the parity of each model, and writes a report. The report lists the checks to port, a model with an index, the resource definitions that the polars engine refuses, and the CSV definitions without anindexoption. The command does not change the resource definitions - test: the demo pipeline on the polars engine, in
demo/polars_pipeline. It reads the inputs of the pandas demo, and its outputs are equal to the expected outputs of the pandas demo. A CI job runs it with--dynamicio-engine=polars. - feat:
dynamicio.configure(engine="polars")anddynamicio migrate --engine polarsraiseImportErrorif polars is older than 2.1, the minimum of thepolarsextra. The check reads the version of the installed package and does not import polars. The message namespip install "dynamicio[polars]". - test: an install without polars. A test in a new process blocks the polars import, and checks that the modules of dynamicio, the pandas engine and the CLI work. The dependency-ranges CI jobs install dynamicio without the
polarsextra, and a step checks that polars is not installed. - fix: on the polars engine, an integer, unsigned integer or Boolean column with a null writes the nullable pandas dtype (
Int64,UInt8,boolean) to the pandas metadata of a parquet file, also when the model declares a numpy dtype. The pandas engine of 11.x then reads the exact values. Before, pandas readfloat64orobject, and an integer above 2**53 lost digits. A CSV or JSON write of such a column no longer raisescannot convert NA to integer. - fix: the polars engine gives polars the S3 endpoint of the boto3 client for each endpoint other than the public endpoint of a region. A VPC interface endpoint (
*.vpce.amazonaws.com), a FIPS endpoint and a dual-stack endpoint thus reach polars. Before, polars used the public endpoint for eachamazonaws.comhost. - fix:
footer()on S3 sends theETagof its first rangedGetObjectinIfMatchon each later call. If another writer replaces the object between two calls, the read raises thePreconditionFailedClientErrorof S3. The footer thus never mixes the bytes of two versions of the object - fix:
dynamicio.errors.__all__listsEngineResourceError, sofrom dynamicio.errors import *imports it. A test checks that__all__lists each exception of the module - fix: the resource type
s3is a deprecated alias ofs3_file. dynamicio reads it ass3_file, on the two engines, and gives aDeprecationWarning. Up to 11.x, a read or a write of ans3resource raisedMissingMixinError. dynamicio 13 removess3 - docs: the polars guide has runnable examples of a configuration, a model, a read, a write, a scan and a sink. It also has examples of a footer, a cast error, a validation error, a validator and a metric. The docstrings of
configure,Field, the polarsDataModel,scan,sinkandfooterhaveExamplessections. The test suite runs these examples. The S3, Postgres, Kafka and Athena examples show the YAML and the call, and do not run. The README, the home page and the installation, why-dynamicio, sources and testing pages describe the polars engine. The API reference addsField, the pandas and polarsDataModelclasses and the CLI - docs: the guides have a section for an upgrade from version 11 to version 12. The examples use
s3_filein place ofs3. The error table listsEngineResourceError,DatasetNameNotFoundErrorandTableNameNotFoundError. The polars guide lists the options, scans and sinks that it refuses. The reference adds the pandas engine, the schema modules and the pytest plugin - chore: the floci integration tests upload their coverage to Codecov with the
integrationflag.
v11.1.0
- fix: the checks for an IO class without
schema, an IO class without a mixin for its resource type, and a URL that is nots3://raiseSchemaNotDeclaredError,MissingMixinErrorandInvalidS3UrlError. They usedassert, whichpython -Oremoves. Each error is also anAssertionError, so code that catchesAssertionErrordoes not change - fix:
session_fordisposes its engine after use, also when the close of the session raises, so the connection pool of each Postgres read or write closes its connections at once - fix:
pickle_protocolholds a lock while it changes the globalpickle.HIGHEST_PROTOCOL. Two hdf writes on different threads could restore the value of each other, so that the protocol of one stayed in place for the rest of the process - fix: a missing S3 key of a parquet or hdf
file_pathraisesS3KeyNotFoundError, somissing_ok=TruereturnsNonefor it as it does for csv and json. For hdf, and for a parquet read that awswrangler does, S3 answers the same bare 404 for a missing bucket, so the read checks the bucket and raises itsClientErrorfor a missing bucket. The error survivespickleandcopy, so a read in a process pool worker gives it back to the caller. The error is aFileNotFoundErrorand also the botocoreClientErrorthat S3 returned, with itsresponse, so code that catchesClientErrordoes not change. Code that catches the narrowerNoSuchKeyclass of a boto3 client must catchClientErrororS3KeyNotFoundError. A parquet read withfiltersstill raises the pyarrowFileNotFoundError. A parquet read with an option that only awswrangler takes, such ass3_additional_kwargs, also raisesS3KeyNotFoundErrorfor a missing key - feat: a parquet
s3_path_prefixwrite takes themodeoption ofawswrangler.s3.to_parquet:append(the default),overwriteoroverwrite_partitions. With pyarrow, each write gives its files new names, so a second write of the same rows withappendadds them again. fastparquet names the files of each partitionpart.0.parquetand onward, so a write replaces the files of the same name, as in 11.0.0.overwritedeletes the objects that were under the prefix,overwrite_partitionsonly those in the partitions that the write writes to. These 2 modes delete objects only when the resource also setsallow_delete: true. Anallow_deletethat is not a bool, such as the YAML string"false", raisesTypeError. Up to 11.0.0, a prefix write droppedmodewith a warning and appended, so a resource that already setsmodeand notallow_deletestill appends, with a warning, also for a value that is not one of the 3. Withallow_delete: true, a value that is not one of the 3 raisesValueError. A delete mode raisesValueErrorwhen the key prefix is empty, because the prefix is then the whole bucket.overwriteof a DataFrame with no rows raisesValueErrorand deletes nothing: pyarrow writes no file for it, so the write would delete every object under the prefix. The old objects are deleted after all new objects are uploaded.overwrite_partitionsfinds the written partitions from thecol=valuedirectories ofpartition_cols, also whenpartition_colsis one column name and not a list, so a file directly under the prefix does not make it delete the other partitions.overwrite_partitionswith an emptypartition_colsraisesValueError, because every key would then be in one partition.overwrite_partitionswith the fastparquet engine, also throughengine: autoand the pandas optionio.parquet.engine, raisesValueError: fastparquet does not encode the partition values in the keys, so the valuea/bwrites to the directory of the valuea, and dynamicio would delete the old objects ofa. The listing of the old keys repeats after a broken connection, as the uploads and deletes do.s3_sync_upreturns the keys of the objects it uploads, where it returnedNone. dynamicio lists the old keys before the upload: if another writer replaces one of them during the write, dynamicio deletes that object, so let only one writer write to a prefix at a time - fix: a csv, json or hdf
path_prefixordynamic_file_pathreads the objects of its sub-prefixes, as a parquet one does. The download keeps the key layout, so a sub-prefix was a directory and the read raisedIsADirectoryError: adynamic_file_pathwith**/always failed for these types. The objects are now read in key order, the order of the parquet read. Before, the order of the rows was the order of the files in the temporary directory, which the file system chose - fix: a
local_batchsource with no matching file raisesNoObjectsFoundError, as an S3 prefix does, somissing_ok=TruereturnsNonefor it. The error is aFileNotFoundErrorand also aValueError, so code that catches the pandasValueError: No objects to concatenatedoes not change. A relativepath_prefixordynamic_file_pathreads its files: the read joined the prefix to the paths a second time and raisedFileNotFoundError - fix: a Kafka write polls the producer after each message, so a failed delivery raises during the write. When the local queue of the producer is full, the write waits for deliveries and sends the message again, where it raised
BufferError. The newflush_timeoutoption is the most seconds that a write waits for Kafka with no progress, and a smaller local queue is progress. An error event of the producer, such as a refused connection, is not progress. A full queue that has no room for a message in that time, or a final flush that delivers no message in that time, raisesTimeoutError. Without it a write waits until every message is delivered, as before. Aflush_timeoutthat is not a number of 0 or more, such as-1, NaN or the YAML string"30", raisesValueErrorbefore the first message. A falsy callable in one of these 4 options is used, as in 11.0.0, and not replaced by the default. The write no longer removeskey_generator,document_transformer,key_serializerandvalue_serializerfrom the options of the IO object. A message that does not fit in the empty queue raisesBufferErrorwith the name of the setting to change, and does not retry without end - fix: a Postgres write keeps a carriage return (
\r) in a text value. The temporary file of theCOPYread it back as a line feed (\n), so the table got a different value. The file is now UTF-8 for every locale - fix:
is_in(..., exact=True)accepts a list of values. It compared a set with the list and raisedTypeError. A model compilesField(exactly=...)to a set, so models were not affected - perf:
write()no longer copies the data of a frame that has only the schema columns. A check compared a set with a tuple, was always true, and selected the columns on each write. The output does not change, and the caller's frame keeps its dtypes
v11.0.0
Pandera DataFrameModel classes are now the only way to declare a schema. Upgrading from 8, 9 or 10: see Upgrading; dynamicio migrate converts the YAML schemas of version 8 and earlier.
- breaking: dict schemas (
schema = {...},schema_validations,schema_metrics) and YAML schemas (schema = SCHEMA_FROM_FILE, and theschema:block of a resource definition) are removed. An IO class whoseschemais anything but aDataFrameModelorSCHEMA_FROM_ARGUMENTraises aTypeErrorwhen the class is defined, and so does a class that still setsschema_validationsorschema_metricsnext to a model (the model's own would replace them). A resource definition with aschema:key raises aValueErrorwhen it loads.SCHEMA_FROM_FILEis removed fromdynamicio.coreandSafeDynamicSchemaLoaderfromdynamicio.config.io_config(importing either raises anImportErrorthat names the replacement), andIOBinding.dynamicio_schema/IOEnvironment.dynamicio_schemawith it.SCHEMA_FROM_ARGUMENTand theschema=argument are unchanged - feat:
DataModel(from dynamicio.schema import DataModel, Field, Metric) is the base class of a model: a PanderaDataFrameModelthat names its dataset and table in class keywords,class Orders(DataModel, dataset="orders", table="orders_v2"), so the body holds only columns and the editor completes the keywords. Any other class keyword raises aTypeErrorthat names it. A plainpa.DataFrameModeldeclares the same with the dunders__dataset_name__and__table_name__. Validation, metrics and casting checks stay arguments of the IO call (apply_schema_validations,log_schema_metrics, ...), unchanged from 10.x, so a call site always shows what it does - feat:
Fielddeclares a column:pa.Fieldwith dynamicio's validations and metrics as named, documented arguments (nullable=required,unique,gt/ge/lt/le,isin,exactly,max_null_fraction,metrics=[Metric.MIN, ...]); any otherpa.Fieldargument passes through,metadata=is refused. A range is its two bounds (ge=1941, le=NEXT_YEAR): bounds are not compared on definition, so they may come from settings, where Pandera'sin_rangewould raise.Fieldreturns apa.Field, so it also declares the columns of a plainpa.DataFrameModel; it is the one public way to declare dynamicio's metrics and validations, andfrom dynamicio.schema import metricsis the metrics module again (as in 10.x), not a helper - feat:
Metricis a plainEnum, no longer astrsubclass, so an editor offers only its eight members afterMetric.; its.valueis still the name theMETRIClog line reports - breaking: every model must name its dataset (
dataset=or__dataset_name__), and nothing is inferred: a model without one raisesDatasetNameNotFoundErrorwhen the IO object is created, where 10.x labelled it with its class name. The names are not inherited.Config.nameis no longer read, and a model that relied on the class-name label must now write it, upper-cased in logs andMETRIClines as before - breaking: the Postgres table name is
table=(or__table_name__), the exact name of the table in the database, and no longer the dataset name. A model used with a Postgres resource without it raisesTableNameNotFoundErrorwhen the IO object is created (a read withsql_query, which names its own tables, needs none), and a read of a table that does not exist logs which one to correct. A dotted name is a table of a Postgres schema (table="sales.orders"), and every statement quotes the name as Postgres needs it, so a mixed-casetable="Orders"is truncated, deleted from and copied into as"Orders"; writes load the rows withCOPY ... FROM STDINinstead ofcopy_from, which quoted a schema-qualified name as one identifier - breaking: the raw
metadata={"dynamicio": {"metrics": [...], "validations": {...}}}dict is rejected with aValueErrorthat namesField - breaking: the
match_alloption of theis_invalidator, which read the wrong way round (match_all=Falsewas the check that every value is present), is nowexactwith the opposite meaning:is_in(..., exact=True)requires exactly the allowed values, the defaultexact=Falseonly a subset of them.Field(exactly=[...])compiles toexact=Trueandisin=[...]to the default;dynamicio migratetranslates a YAMLmatch_all. Code that callsdynamicio.schema.validations.is_indirectly withmatch_allmust switch - feat:
dynamicio migrateis the path from version 8: it converts YAML schemas toDataModelclasses declared withField,dataset=, andtable=for every schema that a resource definition under--scanbinds to a Postgres environment (the old schema name was the table name), checks each converted model compiles back to an equivalent schema, and writesMIGRATION_PROMPT.mdlisting the remaining call sites. Dtypes are declared as types (np.uint32,pd.UInt32Dtype), not calls. A YAMLis_betweenbecomes its two bounds, merged with the other bounds of the column into the tightest one per side: such a column now reportsis_greater_than/is_lower_than(or the_or_equalforms) instead of oneis_betweeninSchemaValidationError. A[[ VARIABLE ]]null threshold migrates like any other bound (Field(max_null_fraction=...)checks a number on definition and keeps a placeholder as it is). Class and module names are valid Python whatever the file is called (20261113.yamlbecomesDataset20261113), and a column named like a name the module uses (Field,Metric,Series, a dynamic variable) gets acolumn_attribute with analias. The prompt lists thename =lines ofclass Configblocks (which 11 no longer reads) instead of everyclass Config, and the search skips hidden,node_modules,__pycache__,venvandsite-packagesdirectories - feat: the
dynamiciocommand line (dynamicio -s|-b -p PATH -o DIR) writes a Pandera model module per dataset (<dataset>.py, prefixeddataset_where the name would not be importable) instead of a YAML schema - docs: the schemas guide, quickstart, examples and README describe models only; new ADR 0009. The demo project's models are
DataModelclasses declared withField;Bar, bound to Postgres in DEV and PROD, names its table, which a test now checks for every environment
v10.4.1
- docs: the
Documentationproject link of the package, and the README, point to the documentation on Read the Docs (https://dynamicio.readthedocs.io/en/latest/), and the documentation home page shows the panda mascot next to the logo - docs: remove the changelog sections of the
v8.1.0release candidates.
v10.4.0
- feat: the constructor of an IO class takes
missing_ok: bool = False. Withmissing_ok=True,read()andasync_read()returnNoneand log a warning when the reader raises aFileNotFoundError. The caller handles theNone, so the setting is in the code of the caller - chore: remove
missing_okfrom the resource definition. Release v10.2.0 added it, and it has no known user. A resource definition with amissing_okkey now fails to load, because dynamicio reads the key as an environment name.IOBinding.missing_okandIOEnvironment.missing_okare removed, and thedict()output of a binding has nomissing_okentry. Move the setting to the constructor of the IO class - docs: new "Why dynamicio" page and a rewritten home page and README that explain the purpose: business logic independent of I/O, fast local feedback loops, and what that gives coding agents. The documentation is restructured into getting-started, guides, examples, API reference, architecture decisions and development pages (
docs/), built with MkDocs for Read the Docs (.readthedocs.yaml; versionedlatest,stableand per-release documentation). The README is now a concise overview and the long walkthrough moved into the guides, with its content unchanged. The Python examples of the quickstart and the basic I/O page are executed by the test suite. Polars is still not supported. - docs: docstrings that the API reference could not parse (
Returns:,Args:andRaises:entries ofDynamicDataIO.async_read,IOConfig,WithS3,WithPostgres,WithKafka,allow_options,args_ofandOutputSandbox) are corrected.args_ofgains aCallableannotation on*funcsandOutputSandboxannotatesdirectory; behaviour is unchanged. - chore: the package metadata gains
Documentation,SourceandIssuesproject URLs next toChangelog. TheDocumentationURL points at thedocs/directory until the Read the Docs site is confirmed. - ci: the "Build and Publish" workflow builds the documentation in strict mode on every push, builds and validates the distributions once (
twine check, tag against version) and publishes exactly those files. A release tag must bevX.Y.ZorvX.Y.Z-rc.N, a stable tag must be onmasterand have a changelog section, and the GitHub release notes are the changelog section plus links (.github/tools/release_tools.py, tested). Re-running a publish is safe: files already on PyPI are skipped and an existing GitHub release is updated. - chore: the
docsdependency group no longer includesmkdocs-techdocs-coreand stays onmkdocs<2;docs/requirements.txtpins the documentation tooling fromuv.lock.
v10.3.0
- chore: remove the
magic-loggerdependency. Each dynamicio module logs throughlogging.getLogger(__name__), with the same logger names as before.magic-loggerhas only a source distribution, so an install withuv sync --no-buildstopped with an error.magic-loggeralso calledinspect.stack()on each log call.dynamicio.loggeris alogging.Loggerfor the loggerdynamicio, and it does not have thedict_config,file_config,listenandstop_listeningmethods. The logging config ofdynamicio/__init__.pynames the loggerdynamicio, sodisable_existing_loggersdoes not disable the module loggers. If you setDYNAMICIO_DO_NOT_CONFIGURE_LOGGING=1and calllogging.config.dictConfigafter the dynamicio import, name the loggerdynamicioin your config or setdisable_existing_loggerstofalse. If you do not, your config disables the dynamicio loggers
v10.2.0
- feat: a resource definition can set
missing_ok: trueon a source, beside its environments. When the reader of that source raises aFileNotFoundError(a missing local file,ObjectNotFoundError,NoObjectsFoundError),read()andasync_read()returnNoneand log a warning. All other errors raise as before. The default isfalse.IOBinding.missing_okandIOEnvironment.missing_okexpose the flag, which adds amissing_ok: Falseentry to thedict()output of a binding. See "Optional sources:missing_ok" in the README - fix: an S3 read of a csv or json
file_pathraisesdynamicio.errors.ObjectNotFoundError("No object found at s3://...")when the key does not exist. Before, it raised the awswranglerNoFilesFound, which is not aFileNotFoundError.ObjectNotFoundErroris aFileNotFoundErrorand aNoFilesFound, so a caller can treat the key as a missing source, and code that catchesNoFilesFoundworks as before. The errors of the parquet and hdf reads for a missing key do not change
v10.1.0
- docs: refresh the standalone logo, clarify pandas-only support, remove unused and superseded images, and expose architecture decisions in the documentation navigation; the S3 benchmark page is replaced by ADR 0007, which keeps the headline figures (the full tables stay in PR #88).
- feat: resource definitions accept an optional
expected: {path, file_type, options}key per environment, the reference file a test compares the generated output against. Reading and writing ignore it, so existing definitions and their behaviour are unchanged. (IOEnvironment.expectedisNonewhen absent, which adds anexpected: Noneentry to the model'sdict()output.) - feat: the
dynamicio.testharnesspackage, to test a task end to end: run it in aLOCALenvironment, then lazily compare each generated output with itsexpectedfile.ExpectedOutputscollects all mismatches and raises oneOutputMismatchError; generated files of matching outputs are removed (the others are kept); JSON is read with the resource'sorient(orexpected.options); paths can carry{placeholders}filled fromparams;DYNAMICIO_UPDATE_EXPECTED=1regenerates the expected files.Expectationscurates how strict a comparison is (row/column order, dtypes, tolerances,sort_by,only_columns,ignore_columns,list_columns_to_sort, customassertions;strict()/lenient()presets,with_()to derive variants), per session or per check; dict columns are compared by content.FrameComparisonandOutputFilesare usable on their own. See "Testing Locally" in the README. - feat:
dynamicio.testharnessadditions:ExpectedOutputs.check_each(config, key, grid)checks a resource whose path has several{placeholders}for every combination of their values (clear_eachremoves their leftovers);Expectations(check_parquet_schema=True)also compares the arrow schema and stored pandas index of parquet outputs;OutputSandbox(directory, *configs)redirects the local outputs of a task into a scratch directory so tests that write the same files can run in parallel;ExpectedOutputs(environment=...)makes everyIOConfigthat defines that environment select it inside thewithblock (and restores them on exit), so the task under test runs and is checked in an environment the test states, without patchingenv_identifier;check(..., environment=...)overrides it for one check,configs=[...]narrows the switch,EnvironmentSwitch(environment, *configs)does the same explicitly and rejects an environment a config does not define. Withoutenvironment=, the default is the first environment the resource definitions list, whatever the process runs in.IOConfig.instances()andIOConfig.default_environmentsupport this. The README gains "Choosing the environment" and "Recommended testing practices". - fix:
dynamicio.testharnessreview fixes: JSON Lines outputs (options.lines: true) are read back withlines(it is passed withorientto the JSON reader only); a column named insort_by,only_columnsorlist_columns_to_sortthat is missing from the output or the reference is reported as a failed check, not aKeyErrorthat aborts the remaining checks; update mode (DYNAMICIO_UPDATE_EXPECTED=1) refuses to copy an output over a reference declared in another file type, leaving both files untouched. - feat:
ExpectedOutputs.check_all(config, skip=, rules_for=, params=)checks every resource of a config that declaresexpected(a new output is covered without touching the test; resources with unfilled{placeholders}are listed inskipped), andExpectedOutputs.clear_all(config)removes the stale generated files of those resources. - feat: a passing test run now says what it verified.
ExpectedOutputsprints aSessionReportat exit (green✓per matched output with its rows x columns and expected file, red✗per failure,↻updated,-skipped),outputs.reportreturns it,OutputMismatchErroralso lists the outputs that matched, and a pytest plugin (pytest11entry pointdynamicio;-p no:dynamicioturns it off) prints every report in the terminal summary, coloured. Nothing is printed when a session checked nothing. - docs: the README walkthrough follows the
demo/project: four resource definitions (staging_input,staging_output,transform_input,transform_output) withLOCAL,DEVandPRODenvironments,expectedpaths on the outputs'LOCALenvironment, and thestaging_*/transform_*configs, instead ofsample/actualandinput/raw/processed. - docs: the
demo/project now has one input and one output definition per runner (staging_input,staging_output,transform_input,transform_output) withLOCAL,DEVandPRODenvironments, and its end-to-end test usesExpectedOutputs. The definitions use YAML anchors and merge keys (&,*,<<:) soPRODonly restates what differs fromDEV.
v10.0.1
- fix: an S3 path-prefix read with no objects under the prefix raises
dynamicio.errors.NoObjectsFoundError("No objects found under s3://..."). Before, it raised the pandas errorValueError: No objects to concatenate, which a caller cannot tell apart from otherValueErrors. The change applies to all the file types (parquet, csv, json, hdf) and to the hdfno_disk_spaceread.NoObjectsFoundErroris aFileNotFoundErrorand aValueError, so a caller can treat the prefix as a missing source, and code that catchesValueErrorworks as before. An S3 parquetfile_paththat ends in/and has no objects raises the same error. It raised a plainFileNotFoundErrorbefore. Objects that are all empty do not change: a parquet prefix returns an empty frame, and a csv, json or hdf prefix raises the pandasValueErroras before
v10.0.0
- fix:
dynamicio migratewrites a YAMLobjectcolumn asSeries[object], notSeries[str].objectmeant "any values", and list columns declaredstrfail plain Pandera validation (and would be recast under pandas 3). Models already generated withSeries[str]for list columns should be changed toSeries[object]; the generated module now says wherestris safe - docs: the README diagrams are now text-driven SVGs (supported sources with Athena next to S3, Postgres and Kafka; the sample pipeline as an Airflow DAG; how a read works; the file formats), they render on GitHub and on PyPI, and the Pandera logo is shown in the Pandera models step. The table of contents anchors and the
CONTRIBUTING.mdlink are fixed, and the dependency list names what the package really uses. Thedemo/example datasets are generic (orders). - fix: the dtype check compares dtypes instead of their names. A column already at
string[pyarrow](whose name isstring) was recast on every read and logged a casting warning; a barecategorystill accepts any categorical dtype and a barestringany string storage (string[python]orstring[pyarrow]ask for one). A declared type pandas does not know is reported by the cast as before. Columns that were cast before are cast as before - feat: Pandera constructs that dynamicio has no validator for now run instead of being ignored or raising a bare
KeyError. Column checks such asstr_matches,str_lengthand custom@pa.checkrun as Pandera checks;@pa.dataframe_check,Config.unique,Config.orderedand index declarations run on the whole frame. Failures are raised inSchemaValidationErrorunder the check's name - feat: Pandera options dynamicio cannot honour (
Config.strict = True,add_missing_columns,on_missing_columns,unique_column_names,drop_invalid_rows,parsers,dtype, and per columnregex,required=False,default,on_missing) and non-pandas (for example polars) models raiseValueErrorwhen the schema is compiled, instead of being dropped. Usestrict = "filter".coerceis honoured through the compiled dtype (a nullable int becomesInt64) - fix:
Series[str]compiles to the pandas default string dtype (objecton pandas 2,stron pandas 3) instead of a hard-codedobject. Existing pandas 2 behaviour is unchanged - feat: models written by
dynamicio migratesetstrict = "filter", soModel.validate(df)on its own drops columns that are not in the schema, as dynamicio does.coercestays off: Pandera would turn a nullable int column intoInt64, where dynamicio casts toint64. Dynamicio's own validation is unchanged - fix: no more
UserWarning: Valid config keys have changed in V2: 'underscore_attrs_are_private' has been removedon import under pydantic 2. The key is now only set under pydantic 1, where it is needed to keepIOEnvironment._parentprivate - fix:
WithPostgresreads and writes work when the table is declared only by a Pandera model. The table name and columns come from the resource YAML'sschemawhen it has one (unchanged), otherwise from the model (Config.nameis the table name). Before, a postgres resource without a YAML schema failed withAssertionError: The schema must be specified for SQL tables - feat:
dynamicio migrate -o <package dir>writes one models module per sub-directory of the YAML schemas (the grouping by runner and input/output carries over);-o file.pystill writes a single module. The README recommends one models module per runner stage or definitions file, and thedemo/project now follows it (demo/src/resources/schemas/{input,processed}.py) - fix:
dynamicio migratehandles real-world schemas: mixed-typeis_invalues (incl.null),is_betweenbounds that are[[ VARIABLES ]], dtype aliases such asdouble/int, and explicit default options; a failed parity check now lists the differing columns. - refactor: schema modules now live in a
dynamicio.schemasubpackage (schema.validations,schema.metrics,schema.models,schema.migrate).dynamicio.validationsanddynamicio.metricskeep working as aliases of the same module objects, and thedynamicio.metricslogger name is unchanged.dynamicio.pandera_schemaanddynamicio.migrate(new in 10.0.0-rc.1 only) moved todynamicio.schema.modelsanddynamicio.schema.migratewithout aliases. - refactor: the helpers of
dynamicio.schemaare grouped into classes (ModelCompiler, and formigrate:SchemaComparison,ModelRenderer,LegacyDeclarations,MigrationPrompt,Migration), and_is_declared_dtypeis a static method ofDynamicDataIO. The public function names (to_dataframe_schema,dataframe_validator,migrate,render_model,render_module,compile_module,schemas_equivalent, ...) remain as aliases, so imports keep working. - chore: raise the
numpyfloor to 1.24.4, the oldest release Pandera 0.34 accepts (found by thedependency-ranges (lowest)job). - Docs: README review. Fixed stale schema wording, the
is_betweeninclude_rightdefault, the casting-failure exception (ColumnsDataTypeError), unique Step 4.3 anchors, and added a summary of validation/metrics behaviour to the closing notes. - test:
tests/test_contracts/pins the validation, metric, lifecycle-order and dtype behaviour that the schema work must preserve, for every schema declaration style - feat: schemas can be declared as Pandera
DataFrameModelclasses (class MyIO(UnifiedIO): schema = MyModel). Dynamicio's validators and metric logging are unchanged, so error payloads andMETRIClog lines are identical to the YAML and dict styles.panderais now a required dependency. Anything Pandera cannot express natively goes inField(metadata={"dynamicio": {"validations": {...}, "metrics": [...]}}). Note Pandera's default isnullable=False; dynamicio's YAML allowed nulls unlesshas_no_null_valueswas declared - fix: when the same validation fails on several columns,
SchemaValidationError.messagenow holds every column's message (joined with;under the one validation-name key). Previously only the last column's message survived - feat:
schema=constructor argument andSCHEMA_FROM_ARGUMENTplaceholder, so one IO class can take a different Pandera model per instance (the Pandera counterpart ofSCHEMA_FROM_FILE). The argument takes precedence over the class attribute - feat:
dynamicio migrate <schemas dir> -o schemas.pyconverts YAML schemas to Pandera models (dataset names, validations and metrics preserved, verified by a parity check) and writesMIGRATION_PROMPT.mdlisting the call sites to update by hand or with an LLM. The existing-b/-sschema generator is unchanged - docs: README Step 4.1 (Pandera models) and migration guide, ADR 0005; the
demo/project now declares its schemas as Pandera models and keeps its resource definitions, models and IO classes together underdemo/src/resources(withenvironment.pyandconstants.pyunderdemo/src/config), withdemo/testsmirroringdemo/src, documented in the README as the recommended pipeline project structure
v9.2.3
- fix:
WithPostgresbuilds one SQLAlchemy model per table definition and reuses it, instead of a new same-named class on every read or write. The classes piled up in the declarative registry, and SQLAlchemy's cleanup of them could raise aRecursionErrorduring garbage collection (seen as a failing test on the lowest-dependency CI job). Each definition has its own declarative base, so two definitions of one table cannot alter each other's columns, and the cache is thread-safe forasync_read/async_write. Queries and tables are unchanged.
v9.2.2
- fix: an IO object can be read or written repeatedly. A read or write removed values from
self.options, so a second call on the same object failed or read the wrong data. The second call of a path with template fields (for example{runner_id}) raisedValueError: Expected values for all fields. Three cases gave no error. The S3 path-prefix read removedno_disk_space. The Athena read removedquery, and the Postgres read removedsql_query, so the second read ran the default query and returned wrong data. Each call now works on a copy of the options, andself.optionsdoes not change. The two private methodsWithS3._read_parquet_objectandWithS3._concat_parquet_objectsare removed. The module-level functions with the same names are unchanged
v9.2.1
- fix: transient S3 connection errors no longer fail a read or write. The shared boto3 client now uses the
standardretry mode with 10 attempts,connect_timeout=10andread_timeout=60. The in-memory and file transfers (put_object,upload_fileobj,get_object,download_fileobj) are repeated, up to 5 attempts with backoff, onConnectionClosedError,ConnectTimeoutError,EndpointConnectionError,ReadTimeoutError,IncompleteReadError,ResponseStreamingErrorand theRetriesExceededErrorthat the transfer manager raises (s3transfer's and boto3's) when a download above 16 MiB gave up on a broken part. botocore does not retry these once the body is being sent or read. The buffer is rewound before each attempt (bytes written are identical), each retry is logged at WARNING with the bucket and key, and the shared client is rebuilt so the next attempt opens fresh connections. Other errors (for exampleClientError) are raised as before, and nothing changes when no error occurs
v9.2.0
- feat: a single-file S3 parquet read with
filtersreads only the row groups whose footer statistics can match, over rangedGetObjectrequests throughpyarrow.fs.S3FileSystem(one for each process and bucket). The rows, index and dtypes are the same as before. Reads withoutfiltersor withfilters=[], fastparquet reads, reads with a non-pyarrowfilesystem, prefix reads and reads with wrangler-only kwargs do not change. On the ranged read, a missing key or a permission problem raises pyarrowFileNotFoundErrororOSErrorin place of botocoreClientError, and a custom S3 endpoint of boto3 (AWS_ENDPOINT_URL) does not apply. See ADR 0004 - feat: new postgres write option
create_table(defaultFalse). WhenTrue, the writer creates a missing table from the schema columns, with no primary key and with unbounded string columns, in the same transaction as the write. The postgres writer now lists its options (truncate_and_append,create_table), so it drops other options with a warning, as the other writers do. Acreate_tablevalue that is not a bool (for example the YAML string"false") raisesTypeError - fix: the cache of pyarrow S3 file systems is cleared at exit, before the pyarrow S3 finalizer runs. With pyarrow 24 and later, a process that read with
filterson a pod with web identity credentials did not exit (https://github.com/apache/arrow/issues/50188) - chore: dev dependency
moto[server]>=5for the tests of the ranged read
v9.1.0
- chore(deps): dependency audit, ranges widened to what the code needs and CI proves. Removed packages the code never imports (
fsspec,libcst,logzero,no_implicit_optional); dropped exact pins (confluent-kafka>=2.4.0,<3,psycopg2-binary>=2.9.3,<3,python-json-logger>=2.0.1,simplejson>=3.17.2); lifted thenumpy<2ceiling (numpy 2.x passes the suite) and thepython<3.13ceiling (3.13 and 3.14 pass; added to CI and classifiers). Raised floors so every allowed combination works with numpy 2 as well as numpy 1:pandas>=2.2.2(JSON writes withindex=Falseforrecords/values, relied on since 8.1.1, need pandas 2.1, and pandas before 2.2.2 breaks with numpy 2),pyarrow>=16.0.0,fastparquet>=2024.5.0,awswrangler>=3.9.0(older ones cannot import against numpy 2 or pyarrow 16),PyYAML>=6.0(5.4.1 does not build with current toolchains). New CI jobdependency-rangestests the minimums (Python 3.10), the minimums with numpy 2 (Python 3.11) and the newest releases (Python 3.14); publishing waits for it - fix:
python-json-logger3.1+ import path (pythonjsonlogger.json) with fallback to the old module, avoiding aDeprecationWarningon import
v9.0.0
Breaking: Athena resources must rename s3_staging_dir to s3_output (see the first Athena entry below).
- docs: add Backstage
catalog-info.yamlto enable TechDocs - docs: README documents Athena I/O (resource options,
ctas_approach, dtypes,s3_staging_dirmigration); ADR 0003 - chore: migrate project and dependency management from Poetry to uv (PEP 621
pyproject.toml,uv.lock,hatchling+hatch-vcsbuild backend). The version is still derived from the git tag and the published wheel is unchanged apart from standard metadata. Contributors now usemake dev-env-setup/uv sync; seeCONTRIBUTING.mdandAGENTS.md - feat(RND-13624)!:
WithAthenareads through AWS Data Wrangler (wr.athena.read_sql_query) instead of PyAthena, and thepyathenadependency is dropped. Breaking (resource definitions): thes3_staging_dirkey of anathena:section is removed. Loading such a definition now fails with an error saying to rename it tos3_output(same value); nothing else changes in the YAML.region_nameandquerystay optional, anddatabase(defaultdefault, Athena's own default) andworkgroupare new optional keys. Reads use the shared per-process boto3 session unlessregion_nameis set. Extraoptionsare filtered to whatread_sql_queryaccepts (e.g.ctas_approach,unload_approach,athena_cache_settings);chunksizeis rejected with an error, because a chunked read returns an iterator rather than a DataFrame ctas_approachdefaults toFalse(as PyAthena, which read the query's result file): column names keep the case used in the query (CTAS lower-cases them) and no temporary Glue table is created. Passctas_approach: trueinoptionsto opt in for large results- note: results now come back with pandas nullable dtypes (
Int64,string,boolean) anddateas Python dates, as returned by awswrangler; declare the dtypes in the resource schema to cast them on read, as for any other source
v8.1.1
- fix: S3 JSON reads and writes use pandas' defaults again:
lines=False(a JSON array fororient="records", as written by 7.x and earlier) unless the caller passeslines. 8.1.0 defaulted to JSON Lines fororient="records", which could not read existing files and wrote files that 7.x readers cannot parse. Setlines: truein the resourceoptionsto opt in to JSON Lines - fix: S3 JSON writes default
indexper orient as pandas does (Trueforindex/columns/split/table,Falseforrecords/values) instead of alwaysFalse, which madeorient="index"andorient="columns"writes fail or drop the index
Metadata
Release files for dynamicio 12.0.0rc2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dynamicio-12.0.0rc2.tar.gz | 3.3 MB | Without provenance | Details for dynamicio-12.0.0rc2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dynamicio-12.0.0rc2-py3-none-any.whl | Python 3 | none | any | Details for dynamicio-12.0.0rc2-py3-none-any.whl |
Total release size: 3.5 MB
Release files / dynamicio-12.0.0rc2.tar.gz
| Download URL | dynamicio-12.0.0rc2.tar.gz |
|---|---|
| Size | 3.3 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a2157e20fb7888f8d462a15991361b5e02d5a42e9e0d9f51858ca5e44c9ae716
|
|
BLAKE2b-256 checksum How to use checksums |
0408eb99c09ee466d1cb6d5c41161db57d31ce2a85c2a2b11608b30b9d96b6bc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / dynamicio-12.0.0rc2-py3-none-any.whl
| Download URL | dynamicio-12.0.0rc2-py3-none-any.whl |
|---|---|
| Size | 195.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
91f2493412531f66eeab455bc838322f1ca4d3b80e01ec3b8c9919a065d52c56
|
|
BLAKE2b-256 checksum How to use checksums |
1845d33040e971ef45cf1a14a2df222860e46dc55c9fd6d666c41fcf23e0f85d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.9 {"installer":{"name":"uv","version":"0.13.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for dynamicio 12.0.0rc2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.