bynat

bynat is a Python utility package for tabular-data cleaning and small SQLite-backed caches. Its cleaning API is exposed through subclasses of pandas.DataFrame; the caching API stores per-function dictionaries in a SQLite database.

Status

Version 0.3.0 is the project version declared in pyproject.toml. See CHANGELOG.md for behavior changes since 0.1.1 and ROADMAP.md for planned work.

Installation

python -m pip install bynat

For a checkout, install it in editable mode:

python -m pip install -e .

Requirements

• Python >=3.11.
• Mandatory runtime distributions: numpy and pandas.
• Optional integrations are grouped in pyproject.toml:
  • translation: deep-translator, langdetect
  • geocoding: geopy
  • currency: Babel, pycountry, yfinance
  • categorization: pyahocorasick for the ahocorasick import
  • nlp: spacy
  • sql: SQLAlchemy

Install an integration group when needed, for example:

python -m pip install "bynat[translation]"

These integrations are imported only when their methods are used. Basic imports and core dataframe operations do not require them. Calling a method that needs a missing integration raises an ImportError naming the required optional dependency. Install the extra corresponding to the feature you use.

extract_entities_nlp also needs a compatible spaCy model, defaulting to en_core_web_sm. save_to_sql needs the DBAPI driver required by the SQLAlchemy URL. Neither model files nor database drivers are bundled.

Public API

The top-level bynat/__init__.py is empty. Import public objects from their subpackages:

from bynat.caching import CacheManager
from bynat.cleaning import BaseData, CsvFile, JsonFile

The subpackage __init__.py files re-export the names listed by each implementation module's __all__:

• bynat.caching: CacheManager
• bynat.cleaning: BaseData, JsonFile, CsvFile

Caching

CacheManager

CacheManager stores each function's cache as rows in a SQLite database named cache.sqlite3 under cache_dir. When cache_dir is omitted (here and in BaseData, JsonFile, and CsvFile), the BYNAT_CACHE_DIR environment variable is used if set, otherwise Settings/Cache Data/Automated. Relative cache paths are resolved from the current working directory; ~ and environment variables are expanded. The cache directory and database are created lazily: constructing a manager creates neither, and the SQLite backend is created on first cache access or on the first save of in-memory changes. Legacy <func_name>_cache.json files, if present, are read only for a one-time migration into SQLite.

from pathlib import Path
from bynat.caching import CacheManager

cache = CacheManager(Path(".cache") / "bynat")
translations = cache.get_column_cache("translate_categorical_column", "status")
translations["pending"] = "Pending"
cache.save_function_cache("translate_categorical_column")

# Load the same function cache later.
loaded = cache.load_function_cache("translate_categorical_column")
assert loaded["status"]["pending"] == "Pending"

Cache keys must be non-empty strings and cannot contain path separators. Values are JSON-serialized inside SQLite rows, so supported values stay JSON-compatible: strings, numbers, None, lists, and dictionaries. save_function_cache applies the added, changed, and removed entries in one SQLite transaction; serialization, locking, or commit failures are logged rather than raised, committed rows stay intact, and the pending in-memory changes are kept so the next save retries them. A legacy <func_name>_cache.json file is migrated into SQLite at most once: existing SQLite values take precedence over stale JSON, malformed or non-object files are quarantined to a .corrupt name for manual recovery, and unsupported nested structures are left in place, unmarked, for recovery.

CacheManager methods and parameters

_validate_cache_key and _get_file_path are implementation helpers and are not part of the public API.

Cleaning

BaseData subclasses pandas.DataFrame. Most cleaning methods return a new BaseData-derived object by default; pass inplace=True to mutate the current object and receive None. Methods that perform translation, geocoding, NLP extraction, or exchange-rate lookup use the instance's CacheManager.

Practical examples

Basic column cleaning and reshaping:

from bynat.cleaning import BaseData

records = BaseData({
    "user.display_name": ["Ada", "Grace"],
    "user_id": [1, 2],
    "description": ["Blue shoes", "Red shirt"],
})

cleaned = (
    records
    .rename_all_DOTS()
    .drop_by_substring("user")
)

labeled = records.extract_keywords(
    "description",
    colors=["blue", "red"],
)

Load files with the convenience subclasses:

from bynat.cleaning import CsvFile, JsonFile

from_csv = CsvFile(csv_file_path="input.csv", read_kwargs={"sep": ";"})
from_json = JsonFile(json_file_path="input.json", key="items")

from_csv.save_to_json("output.json", orient="records")

Hierarchical statistics compute one statistic per group at every level of a hierarchy, from the full hierarchy down to its first column. Pass one strategy or a list of median, mean, and mode (mod is accepted as an alias):

import pandas as pd
from bynat.cleaning import BaseData

salaries = BaseData({
    "country": ["EG", "EG", "EG", "US", "US"],
    "title": ["Analyst", "Analyst", "Engineer", "Analyst", "Analyst"],
    "salary": [10.0, 14.0, 30.0, 70.0, 90.0],
})

stats = salaries.get_hierarchical_stats(
    "salary",
    strategy=["median", "mode"],
    hierarchy_cols=["country", "title"],
)

stats is returned as the caller's class (here BaseData) with the columns hierarchy_level, hierarchy_columns, group_key, target_column, strategy, and statistic. The median rows are:

group_key is always a tuple with one value per hierarchy column, including single-column levels such as ("EG",). Null group keys are kept. mean and median need numeric targets and raise pandas' TypeError otherwise; mode works for any hashable values and breaks ties by first occurrence. Choose another container with output_type:

as_frame = salaries.get_hierarchical_stats("salary", hierarchy_cols="country", output_type=pd.DataFrame)
as_dict = salaries.get_hierarchical_stats("salary", hierarchy_cols="country", output_type=dict)
# as_dict["group_key"] == [("EG",), ("US",)]; as_dict["statistic"] == [14.0, 80.0]

output_type accepts None (default, the caller's class), pandas.DataFrame, dict (column name to list of values), or a BaseData subclass. Target columns cannot also be hierarchy columns.

Network-backed enrichment is explicit in the method behavior and should be used with an appropriate cache directory and service credentials/configuration where applicable:

translated = records.translate_categorical_column(
    "category",
    target_lang="en",
    save_every=100,
)

Constructors

BaseData methods and parameters

Most methods skip missing source/target columns with a logged warning and return the original or copied frame, depending on inplace. Check the method's return value when chaining operations. Invalid arguments that would otherwise produce silently wrong results raise instead: an unknown strategy, an empty file_path, invalid save_to_sql connection settings, or an unusable extract_entities_nlp configuration.

For handle_outliers, drop_outliers, and fill_missing, passing get_hierarchical_stats=True reuses get_hierarchical_stats() and computes the statistics from the original data before the cleaning operation changes values or rows. With inplace=False, the return value is (result, stats); with inplace=True, the operation still mutates the original object and returns stats directly. The outlier methods use the default hierarchical median statistic; fill_missing uses its requested strategy.

Limitations and operational notes

• Optional integrations are lazy-loaded, so basic bynat.cleaning imports and core dataframe methods work with only numpy and pandas. Methods that need an absent integration raise a clear ImportError when called.

• Translation uses deep-translator and may make external requests. Geocoding uses Nominatim through geopy with a one-second rate limit. Currency conversion uses Yahoo Finance history through yfinance. These services can fail, rate-limit, or change results; cached values persist locally in SQLite.

• NLP extraction requires a compatible spaCy model installed separately. extract_entities_nlp(disable=...) accepts None, a component name, or an iterable, defaults to disabling parser and ner, rejects components required for lemmatization, and warns before continuing when a requested component is not in the loaded pipeline. A missing or unloadable model raises spaCy's OSError.

• save_to_sql lets database exceptions propagate to the caller and always disposes its engine. It receives database credentials as ordinary arguments; do not hard-code secrets in application code.

• Cache entries live in a single SQLite database rather than one JSON file per function. Saves run inside SQLite transactions and WAL checkpoints, so an interrupted or failed save is logged and rolled back without raising; rows from earlier successful saves stay intact and the pending in-memory changes are retried on the next save. Concurrent managers sharing one database merge disjoint key updates, and for a key written by more than one manager the last committed write wins. Values must be JSON serializable.

• Messages go through the standard logging module under the bynat logger. Skipped columns and failed lookups are WARNINGs and appear on stderr by default; progress and checkpoints are INFO. To see progress, or to silence warnings:

  import logging
  
  logging.basicConfig(level=logging.INFO)              # show progress
  logging.getLogger("bynat").setLevel(logging.ERROR)   # or hide warnings
  

• The package's top-level namespace is intentionally empty; from bynat import BaseData is not supported by the current __init__.py.

Development and verification

The repository test suite uses pytest and avoids network access, secrets, user files, and external services. Network-backed integrations (translation, geocoding, exchange rates, spaCy) are exercised with fake modules:

python -m pytest -q
python -m compileall -q bynat
ruff check bynat tests

Install ".[categorization,sql]" to also run the tests that use the real pyahocorasick and SQLAlchemy/SQLite; they are skipped otherwise. CI runs the suite on Ubuntu and Windows with Python 3.11 and 3.13, plus Ruff and an extras job.

Building a wheel or source archive requires the configured setuptools build backend and wheel tooling.

License

MIT. See LICENSE.