bynat

bynat is a Python utility package for tabular-data cleaning and small SQLite-backed caches. Its first-class cleaning API is class-based through BaseData, JsonFile, and CsvFile; the optional df.bynat accessor provides a complementary interface for ordinary pandas.DataFrame objects. The caching API stores per-function dictionaries in a SQLite database.

Status

Version 0.6.0 is the project version declared in pyproject.toml. See CHANGELOG.md for behavior changes since 0.1.1 and ROADMAP.md for planned work.

Installation

python -m pip install bynat

For a checkout, install it in editable mode:

python -m pip install -e .

Requirements

• Python >=3.11.
• Mandatory runtime distributions: numpy and pandas.
• Optional integrations are grouped in pyproject.toml:
  • translation: deep-translator, langdetect
  • geocoding: geopy
  • currency: Babel, pycountry, yfinance
  • categorization: pyahocorasick for the ahocorasick import
  • nlp: spacy
  • sql: SQLAlchemy

Install an integration group when needed, for example:

python -m pip install "bynat[translation]"

These integrations are imported only when their methods are used. Basic imports and core dataframe operations do not require them. Calling a method that needs a missing integration raises an ImportError naming the required optional dependency. Install the extra corresponding to the feature you use.

extract_entities_nlp also needs a compatible spaCy model, defaulting to en_core_web_sm. save_to_sql needs the DBAPI driver required by the SQLAlchemy URL. Neither model files nor database drivers are bundled.

Public API

import bynat                          # registers the df.bynat accessor
from bynat import read_csv, read_json
from bynat.caching import CacheManager, shared_cache_manager
from bynat.cleaning import BaseData, CsvFile, JsonFile
from bynat.cleaning.enrich import Geocoder, RateSource, Translator

• bynat: read_json, read_csv, BynatAccessor. Importing bynat (or any submodule) registers the bynat accessor on pandas.DataFrame.
• bynat.caching: CacheManager, shared_cache_manager, CACHE_DIR_ENV_VAR, DEFAULT_CACHE_DIR
• bynat.cleaning: BynatAccessor, read_json, read_csv, and the supported BaseData, JsonFile, CsvFile classes
• bynat.cleaning.enrich: the provider protocols Translator, Geocoder, RateSource and their defaults GoogleTranslatorProvider, NominatimGeocoder, YFinanceRateSource

Module layout

Each function in columns, text, stats, enrich, and io takes the DataFrame as its first argument and returns a new one by default; the supported classes and optional accessor are thin layers over them. The legacy module paths bynat.caching.Caching and bynat.cleaning.Cleaning still import and resolve to bynat.caching.manager and bynat.cleaning.base.

Caching

CacheManager

CacheManager stores each function's cache as rows in a SQLite database named cache.sqlite3 under cache_dir. When cache_dir is omitted (here and in BaseData, JsonFile, and CsvFile), the BYNAT_CACHE_DIR environment variable is used if set, otherwise Settings/Cache Data/Automated. Relative cache paths are resolved from the current working directory; ~ and environment variables are expanded. The cache directory and database are created lazily: constructing a manager creates neither, and the SQLite backend is created on first cache access or on the first save of in-memory changes. Legacy <func_name>_cache.json files, if present, are read only for a one-time migration into SQLite.

from pathlib import Path
from bynat.caching import CacheManager

cache = CacheManager(Path(".cache") / "bynat")
translations = cache.get_column_cache("translate_categorical_column", "status")
translations["pending"] = "Pending"
cache.save_function_cache("translate_categorical_column")

# Load the same function cache later.
loaded = cache.load_function_cache("translate_categorical_column")
assert loaded["status"]["pending"] == "Pending"

Cache keys must be non-empty strings and cannot contain path separators. Values are JSON-serialized inside SQLite rows, so supported values stay JSON-compatible: strings, numbers, None, lists, and dictionaries. save_function_cache applies the added, changed, and removed entries in one SQLite transaction; serialization, locking, or commit failures are logged rather than raised, committed rows stay intact, and the pending in-memory changes are kept so the next save retries them. A legacy <func_name>_cache.json file is migrated into SQLite at most once: existing SQLite values take precedence over stale JSON, malformed or non-object files are quarantined to a .corrupt name for manual recovery, and unsupported nested structures are left in place, unmarked, for recovery.

CacheManager methods and parameters

_validate_cache_key and _get_file_path are implementation helpers and are not part of the public API.

Cleaning

The class-based API remains first-class: instantiate BaseData(data), JsonFile, or CsvFile and call the cleaning methods directly. After import bynat, every pandas.DataFrame also has the optional bynat accessor for the same cleaning operations. Accessor methods return new DataFrames and do not accept inplace; class methods return a new object by default, while supported inplace=True calls mutate the original object and return None (or the documented statistics result). Methods that perform translation, geocoding, NLP extraction, or exchange-rate lookup use one shared CacheManager per cache directory: the BYNAT_CACHE_DIR directory or the default, unless df.bynat.with_cache(...) selects another directory or an explicit CacheManager.

Practical examples

Basic column cleaning and reshaping:

import bynat
import pandas as pd

records = pd.DataFrame({
    "user.display_name": ["Ada", "Grace"],
    "user_id": [1, 2],
    "description": ["Blue shoes", "Red shirt"],
})

cleaned = records.bynat.rename_all_DOTS().bynat.drop_by_substring("user")

labeled = records.bynat.extract_keywords(
    "description",
    colors=["blue", "red"],
)

Load files with the loader helpers, which return plain DataFrames:

import bynat

from_csv = bynat.read_csv("input.csv", sep=";")
from_json = bynat.read_json("input.json", key="items")

from_csv.bynat.save_to_json("output.json", orient="records")

Hierarchical statistics compute one statistic per group at every level of a hierarchy, from the full hierarchy down to its first column. Pass one strategy or a list of median, mean, and mode (mod is accepted as an alias):

import bynat
import pandas as pd

salaries = pd.DataFrame({
    "country": ["EG", "EG", "EG", "US", "US"],
    "title": ["Analyst", "Analyst", "Engineer", "Analyst", "Analyst"],
    "salary": [10.0, 14.0, 30.0, 70.0, 90.0],
})

stats = salaries.bynat.get_hierarchical_stats(
    "salary",
    strategy=["median", "mode"],
    hierarchy_cols=["country", "title"],
)

stats is returned as the caller's class (here pandas.DataFrame) with the columns hierarchy_level, hierarchy_columns, group_key, target_column, strategy, and statistic. The median rows are:

group_key is always a tuple with one value per hierarchy column, including single-column levels such as ("EG",). Null group keys are kept. mean and median need numeric targets and raise pandas' TypeError otherwise; mode works for any hashable values and breaks ties by first occurrence. Choose another container with output_type:

as_frame = salaries.bynat.get_hierarchical_stats("salary", hierarchy_cols="country", output_type=pd.DataFrame)
as_dict = salaries.bynat.get_hierarchical_stats("salary", hierarchy_cols="country", output_type=dict)
# as_dict["group_key"] == [("EG",), ("US",)]; as_dict["statistic"] == [14.0, 80.0]

output_type accepts None (default, the caller's class), pandas.DataFrame, dict (column name to list of values), or a BaseData subclass. Target columns cannot also be hierarchy columns.

Network-backed enrichment is explicit in the method behavior and should be used with an appropriate cache directory and service credentials/configuration where applicable:

translated = records.bynat.with_cache(".cache/bynat").translate_categorical_column(
    "category",
    target_lang="en",
    save_every=100,
)

External-service providers

Translation, geocoding, and currency conversion call external services through small provider protocols in bynat.cleaning.enrich. The defaults are used unless a method receives provider=:

A provider is any object with the right method; no subclassing or monkeypatching is needed:

class UpperTranslator:
    def translate(self, text, target_lang):
        return text.upper()

df.bynat.translate_categorical_column("status", provider=UpperTranslator())

Translators with translate_batch receive the values in chunks of save_every. Otherwise translate runs in a thread pool sized by the provider's optional max_workers attribute (default 1), with request starts spaced by its optional min_interval attribute in seconds (default 0). GoogleTranslatorProvider uses four workers and at most five requests per second. A falsy translation or a raised exception leaves that value untranslated and uncached. convert_to_usd makes one yearly_rates call per currency, covering every year its look-back can need.

All four cached methods, and extract_entities_nlp, run through one engine: unique non-null values are looked up in the cache, the misses are fetched, the cache is checkpointed every save_every values, and failures are logged and left uncached so the next run retries them.

Class-based API

BaseData, JsonFile, and CsvFile are supported public classes. BaseData is a supported pandas.DataFrame subclass, and all three use the same cleaning implementations as the optional accessor.

Equivalent class-based and accessor styles:

# Class-based style
from bynat.cleaning import JsonFile
jobs = JsonFile(json_file_path="jobs.json", key="items", cache_dir="cache")
jobs = jobs.rename_all_DOTS()
jobs.translate_categorical_column("status", inplace=True)

# Optional accessor style
import bynat
jobs = bynat.read_json("jobs.json", key="items")
jobs = jobs.bynat.rename_all_DOTS()
jobs = jobs.bynat.with_cache("cache").translate_categorical_column("status")

Every supported BaseData method below also accepts inplace=False where in-place operation is meaningful. With inplace=True, the method mutates the original object and returns None (or the documented statistics result); it is supported and is not scheduled for removal. Without it, every method returns a new object, including early exits that previously returned the original object itself. BaseData creates its CacheManager on first use and shares it with every frame pandas derives from it.

Methods and parameters

The optional accessor and the supported classes expose the same cleaning methods. Accessor methods take no inplace argument; class methods accept it where in-place operation is meaningful. The statistics-only get_hierarchical_stats method returns statistics and does not expose an in-place mode.

Most methods skip missing source/target columns with a logged warning and return an unchanged copy. Always use the method's return value. Invalid arguments that would otherwise produce silently wrong results raise instead: an unknown strategy, an empty file_path, invalid save_to_sql connection settings, or an unusable extract_entities_nlp configuration.

For handle_outliers, drop_outliers, and fill_missing, passing get_hierarchical_stats=True reuses get_hierarchical_stats() and computes the statistics from the original data before the cleaning operation changes values or rows. The return value is (result, stats); with inplace=True, the operation mutates the original object and returns stats directly. The outlier methods use the default hierarchical median statistic; fill_missing uses its requested strategy.

Limitations and operational notes

• Optional integrations are lazy-loaded, so basic bynat.cleaning imports and core dataframe methods work with only numpy and pandas. Methods that need an absent integration raise a clear ImportError when called.

• Translation uses deep-translator by default and may make external requests. Geocoding uses Nominatim through geopy with a one-second rate limit. Currency conversion uses Yahoo Finance history through yfinance. These services can fail, rate-limit, or change results; cached values persist locally in SQLite.

• NLP extraction requires a compatible spaCy model installed separately. extract_entities_nlp(disable=...) accepts None, a component name, or an iterable, defaults to disabling parser and ner, rejects components required for lemmatization, and warns before continuing when a requested component is not in the loaded pipeline. A missing or unloadable model raises spaCy's OSError.

• save_to_sql lets database exceptions propagate to the caller and always disposes its engine. It receives database credentials as ordinary arguments; do not hard-code secrets in application code.

• Cache entries live in a single SQLite database rather than one JSON file per function. Saves run inside SQLite transactions and WAL checkpoints, so an interrupted or failed save is logged and rolled back without raising; rows from earlier successful saves stay intact and the pending in-memory changes are retried on the next save. Concurrent managers sharing one database merge disjoint key updates, and for a key written by more than one manager the last committed write wins. Values must be JSON serializable.

• Messages go through the standard logging module under the bynat logger. Skipped columns and failed lookups are WARNINGs and appear on stderr by default; progress and checkpoints are INFO. To see progress, or to silence warnings:

  import logging
  
  logging.basicConfig(level=logging.INFO)              # show progress
  logging.getLogger("bynat").setLevel(logging.ERROR)   # or hide warnings
  

• The top-level bynat namespace exposes only read_json, read_csv, and BynatAccessor; import the supported class API from bynat.cleaning (for example, from bynat.cleaning import BaseData).

Development and verification

The repository test suite uses pytest and avoids network access, secrets, user files, and external services. Network-backed integrations (translation, geocoding, exchange rates, spaCy) are exercised with fake modules:

python -m pip install -e ".[dev]"
python -m pytest -q
python -m compileall -q bynat
ruff check bynat tests benchmarks
python -m mypy

Install ".[categorization,sql]" to also run the tests that use the real pyahocorasick and SQLAlchemy/SQLite; they are skipped otherwise. CI runs the suite on Ubuntu and Windows with Python 3.11 and 3.13, plus Ruff, mypy, and an extras job.

Type checking is gradual. python -m mypy reads its scope from [tool.mypy] in pyproject.toml: the cache manager and the cleaning helper modules (_common, accessor, columns, text, stats, io, enrich), checked against pandas-stubs. The supported BaseData classes in bynat/cleaning/base.py remain a separate gradual-typing boundary. To bring a module into scope, add it to files, fix what mypy reports with precise annotations rather than Any or # type: ignore, and keep the runtime behavior unchanged.

benchmarks/ holds timing scripts that compare the current implementations with the 0.3.0 algorithms; see benchmarks/README.md.

Building a wheel or source archive requires the configured setuptools build backend and wheel tooling.

License

MIT. See LICENSE.