bynat

bynat is a Python utility package for tabular-data cleaning and small JSON-backed caches. Its cleaning API is exposed through subclasses of pandas.DataFrame; the caching API stores per-function dictionaries as JSON files.

Status

Version 0.2.0 is the project version declared in pyproject.toml. See CHANGELOG.md for behavior changes since 0.1.1 and ROADMAP.md for planned work.

Installation

python -m pip install bynat

For a checkout, install it in editable mode:

python -m pip install -e .

Requirements

• Python >=3.11.
• Mandatory runtime distributions: numpy and pandas.
• Optional integrations are grouped in pyproject.toml:
  • translation: deep-translator, langdetect
  • geocoding: geopy
  • currency: Babel, pycountry, yfinance
  • categorization: pyahocorasick for the ahocorasick import
  • nlp: spacy
  • sql: SQLAlchemy

Install an integration group when needed, for example:

python -m pip install "bynat[translation]"

These integrations are imported only when their methods are used. Basic imports and core dataframe operations do not require them. Calling a method that needs a missing integration raises an ImportError naming the required optional dependency. Install the extra corresponding to the feature you use.

extract_entities_nlp also needs a compatible spaCy model, defaulting to en_core_web_sm. save_to_sql needs the DBAPI driver required by the SQLAlchemy URL. Neither model files nor database drivers are bundled.

Public API

The top-level bynat/__init__.py is empty. Import public objects from their subpackages:

from bynat.caching import CacheManager
from bynat.cleaning import BaseData, CsvFile, JsonFile

The subpackage __init__.py files re-export the names listed by each implementation module's __all__:

• bynat.caching: CacheManager
• bynat.cleaning: BaseData, JsonFile, CsvFile

Caching

CacheManager

CacheManager manages JSON files named <func_name>_cache.json under cache_dir. When cache_dir is omitted (here and in BaseData, JsonFile, and CsvFile), the BYNAT_CACHE_DIR environment variable is used if set, otherwise Settings/Cache Data/Automated. Relative cache paths are resolved from the current working directory; ~ and environment variables are expanded. Cache directories are created when a cache is saved, not when the manager is constructed.

from pathlib import Path
from bynat.caching import CacheManager

cache = CacheManager(Path(".cache") / "bynat")
translations = cache.get_column_cache("translate_categorical_column", "status")
translations["pending"] = "Pending"
cache.save_function_cache("translate_categorical_column")

# Load the same function cache later.
loaded = cache.load_function_cache("translate_categorical_column")
assert loaded["status"]["pending"] == "Pending"

Cache keys must be non-empty strings and cannot contain path separators. A cache file that is not a valid JSON object is renamed to <name>.corrupt.<timestamp> (so its data can be recovered by hand), logged as a warning, and treated as an empty dictionary. Saves are atomic: the JSON is written to a temporary file that replaces the old one. JSON serialization errors and file-write errors are logged rather than raised by save_function_cache, and the previous file is kept.

CacheManager methods and parameters

_validate_cache_key and _get_file_path are implementation helpers and are not part of the public API.

Cleaning

BaseData subclasses pandas.DataFrame. Most cleaning methods return a new BaseData-derived object by default; pass inplace=True to mutate the current object and receive None. Methods that perform translation, geocoding, NLP extraction, or exchange-rate lookup use the instance's CacheManager.

Practical examples

Basic column cleaning and reshaping:

from bynat.cleaning import BaseData

records = BaseData({
    "user.display_name": ["Ada", "Grace"],
    "user_id": [1, 2],
    "description": ["Blue shoes", "Red shirt"],
})

cleaned = (
    records
    .rename_all_DOTS()
    .drop_by_substring("user")
)

labeled = records.extract_keywords(
    "description",
    colors=["blue", "red"],
)

Load files with the convenience subclasses:

from bynat.cleaning import CsvFile, JsonFile

from_csv = CsvFile(csv_file_path="input.csv", read_kwargs={"sep": ";"})
from_json = JsonFile(json_file_path="input.json", key="items")

from_csv.save_to_json("output.json", orient="records")

Hierarchical statistics compute one statistic per group at every level of a hierarchy, from the full hierarchy down to its first column. Pass one strategy or a list of median, mean, and mode (mod is accepted as an alias):

import pandas as pd
from bynat.cleaning import BaseData

salaries = BaseData({
    "country": ["EG", "EG", "EG", "US", "US"],
    "title": ["Analyst", "Analyst", "Engineer", "Analyst", "Analyst"],
    "salary": [10.0, 14.0, 30.0, 70.0, 90.0],
})

stats = salaries.get_hierarchical_stats(
    "salary",
    strategy=["median", "mode"],
    hierarchy_cols=["country", "title"],
)

stats is returned as the caller's class (here BaseData) with the columns hierarchy_level, hierarchy_columns, group_key, target_column, strategy, and statistic. The median rows are:

group_key is always a tuple with one value per hierarchy column, including single-column levels such as ("EG",). Null group keys are kept. mean and median need numeric targets and raise pandas' TypeError otherwise; mode works for any hashable values and breaks ties by first occurrence. Choose another container with output_type:

as_frame = salaries.get_hierarchical_stats("salary", hierarchy_cols="country", output_type=pd.DataFrame)
as_dict = salaries.get_hierarchical_stats("salary", hierarchy_cols="country", output_type=dict)
# as_dict["group_key"] == [("EG",), ("US",)]; as_dict["statistic"] == [14.0, 80.0]

output_type accepts None (default, the caller's class), pandas.DataFrame, dict (column name to list of values), or a BaseData subclass. Target columns cannot also be hierarchy columns.

Network-backed enrichment is explicit in the method behavior and should be used with an appropriate cache directory and service credentials/configuration where applicable:

translated = records.translate_categorical_column(
    "category",
    target_lang="en",
    save_every=100,
)

Constructors

BaseData methods and parameters

Most methods skip missing source/target columns with a logged warning and return the original or copied frame, depending on inplace. Check the method's return value when chaining operations. Invalid arguments that would otherwise produce silently wrong results raise instead: an unknown strategy, an empty file_path, invalid save_to_sql connection settings, or an unusable extract_entities_nlp configuration.

Limitations and operational notes

• Optional integrations are lazy-loaded, so basic bynat.cleaning imports and core dataframe methods work with only numpy and pandas. Methods that need an absent integration raise a clear ImportError when called.

• Translation uses deep-translator and may make external requests. Geocoding uses Nominatim through geopy with a one-second rate limit. Currency conversion uses Yahoo Finance history through yfinance. These services can fail, rate-limit, or change results; cached values persist locally as JSON.

• NLP extraction requires a compatible spaCy model installed separately. A missing or unloadable model raises spaCy's OSError.

• save_to_sql lets database exceptions propagate to the caller and always disposes its engine. It receives database credentials as ordinary arguments; do not hard-code secrets in application code.

• Cache files are ordinary JSON. Each save writes a temporary file and atomically replaces the old one, so an interrupted save never leaves a truncated cache. Files are not locked: concurrent writers to the same cache can still overwrite each other's updates. Values must be JSON serializable.

• Messages go through the standard logging module under the bynat logger. Skipped columns and failed lookups are WARNINGs and appear on stderr by default; progress and checkpoints are INFO. To see progress, or to silence warnings:

  import logging
  
  logging.basicConfig(level=logging.INFO)              # show progress
  logging.getLogger("bynat").setLevel(logging.ERROR)   # or hide warnings
  

• The package's top-level namespace is intentionally empty; from bynat import BaseData is not supported by the current __init__.py.

Development and verification

The repository test suite uses pytest and avoids network access, secrets, user files, and external services. Network-backed integrations (translation, geocoding, exchange rates, spaCy) are exercised with fake modules:

python -m pytest -q
python -m compileall -q bynat
ruff check bynat tests

Install ".[categorization,sql]" to also run the tests that use the real pyahocorasick and SQLAlchemy/SQLite; they are skipped otherwise. CI runs the suite on Ubuntu and Windows with Python 3.11 and 3.13, plus Ruff and an extras job.

Building a wheel or source archive requires the configured setuptools build backend and wheel tooling.

License

MIT. See LICENSE.