Skip to main content

deepcsv

PyPI Downloads Version Stars

"You think you saved a list. You open it tomorrow, and it's a string."

deepcsv exists to fix that.


See it in action

deepcsv demo: data_details() finds the problems in an Excel file, auto_fix(), process_file() and auto_outliers() fix them, and the second report is clean

A messy Excel file with numbers saved as text, lists saved as strings and 3 outliers. data_details() finds all three, one line fixes each, and the second report comes back clean. The output in the GIF is real, not a mock-up.


The Problem

Your CSV files are lying to you.

  • You save a list, open the file tomorrow, and it's a string
  • You have lists and dicts stored as strings, nested inside other lists and dicts
  • Your column looks like numbers but secretly holds 3 different data types
  • You have 200 CSV files across 40 folders and process them one by one
  • You load a file and spend 20 minutes just picking the right reader
  • You have nulls scattered everywhere with no clean way to handle them

Small problems like these quietly break pipelines.


The Solution

deepcsv handles all of it with one import.

  • Walks through every folder and subfolder automatically
  • Detects columns storing lists as strings and converts them to real NumPy arrays
  • Catches mixed-type columns and fixes them automatically
  • Saves in any format you choose, not just Parquet
  • Reads any file format with one function, so there is no more picking the right reader
  • Cleans nulls by column, row, index, value or type

Installation

pip install deepcsv

# with the ML tools (auto_fs, auto_outliers)
pip install deepcsv[ml]

Functions Overview

Function What it does
process_file() Converts string lists to NumPy arrays, fixes mixed types
process_all_files() Batch processes entire folder trees
read_any() Reads any file format automatically
clean_values() Cleans nulls, values and types
auto_fix() Finds and fixes mixed data types and numbers stored as text
auto_fs() Feature selection: keeps the columns that matter for your target
auto_outliers() Detects outliers and removes their rows (IQR, Z-score or Isolation Forest)
data_details() Report about your data (types, missing values, outliers), exportable as CSV or TXT
merge_data() Merges two datasets (any format) by a specific column

Functions

process_file(data_input, file_format=None, auto_fix=False, to_list=False, deep_check=False, col_name="all")

Reads a file or DataFrame, converts array-like strings to NumPy arrays, fixes mixed-type columns, and optionally saves the result.

import deepcsv

# Process only
df = deepcsv.process_file('path/to/file.csv')

# Process and save as parquet
df = deepcsv.process_file('path/to/file.csv', file_format='parquet')

# Process and convert to real Python lists
df = deepcsv.process_file('path/to/file.csv', to_list=True)

# Deep parse nested lists and dicts inside arrays
df = deepcsv.process_file('path/to/file.csv', deep_check=True)

# Only some columns
df = deepcsv.process_file('path/to/file.csv', col_name=["col1","col2"])
Parameter Type Default Description
deep_check bool False True: also parses lists and dicts nested inside the arrays. Slower on large datasets
col_name str | list "all" Column name or list of names to process. The default "all" processes every column. Pass "genres" or ["genres", "tags"] to pick columns

Supported save formats: .csv .tsv .txt .xlsx .json .parquet .pkl .feather .html .xml


process_all_files(directory_path, output_dir="All CSV Files is Converted Here", file_format="parquet", auto_fix=False, to_list=False, deep_check=False)

Walks through all folders and subfolders, applies process_file on every supported file, and saves results.

import deepcsv

# Default: saves as parquet
deepcsv.process_all_files('path/to/folder')

# Custom output folder
deepcsv.process_all_files('path/to/folder', output_dir='Converted Files')

# Save as CSV
deepcsv.process_all_files('path/to/folder', file_format='csv')

Supported input formats: .csv .txt .tsv .xls .xlsx .json .parquet .pkl .feather .db .sqlite


read_any(file_path)

Reads any supported file format and returns a pandas DataFrame. One function for everything.

from deepcsv import read_any

df = read_any('data/users.csv')
df = read_any('reports/sales.xlsx')
df = read_any('warehouse/orders.parquet')
df = read_any('local.db')

Supported formats: .csv .txt .tsv .xls .xlsx .json .parquet .pkl .feather .db .sqlite


clean_values(data_input, ...)

Cleans a DataFrame by removing nulls, specific values, specific types, or rows by index.

from deepcsv import clean_values

# Drop fully-null columns
df = clean_values('data.csv', cols=['age', 'salary'])

# Drop rows that have nulls in specific cols
df = clean_values('data.csv', cols=['age', 'salary'], ax_0=True)

# Drop rows by index
df = clean_values(df, index=[0, 5, 12])

# Remove rows where a specific value exists
df = clean_values(df, cols=['status'], finding_value='N/A')

# Remove rows where value meets a condition
df = clean_values(df, cols=['score'], finding_value='N/A', condition=['>=', 500])

# Remove rows by Python type
df = clean_values(df, cols=['age'], finding_type=str)

# Apply on all columns except some
df = clean_values('data.csv', all_cols_except=['id', 'name'])
Parameter Type Default Description
data_input str | DataFrame required File path or DataFrame
cols list None Columns to apply on
ax_0 bool False True: drop rows with nulls. False: drop fully-null columns
index list None Row indexes to drop
condition list None [operator, value], for example ['>=', 500]
all_cols_except list None Apply on all columns except these
finding_value any None Find and remove rows containing this value
finding_type type None Find and remove rows matching this Python type

Supported operators: >= <= > < == !=


auto_fix(data_input)

Automatically detects columns with mixed data types and fixes them by converting all values to the most dominant type. Logs every change made.

from deepcsv import auto_fix

df = auto_fix('data.csv')
df = auto_fix(my_dataframe)
from deepcsv import auto_fix

df = auto_fix('data.csv')
df = auto_fix(my_dataframe)

# Fix only specific columns
df = auto_fix('data.csv', col_name='age')
df = auto_fix('data.csv', col_name=['age', 'score', 'price'])
Parameter Type Default Description
data_input str | DataFrame required File path or DataFrame
col_name str | list "all" Column name or list of names to fix. Default "all" applies to every column

auto_fs(df, target, model=None, mode="balanced", corr_threshold=0.3)

Automatically selects the most important features for a given target column. Three modes, so you can pick speed or accuracy.

from deepcsv.ml import auto_fs

# Balanced mode (default): Ridge + cross-validation drops weak features
result = auto_fs(df, target='price')

# Accurate mode: GradientBoosting + cross-validation, slowest but most precise
result = auto_fs(df, target='price', mode='accurate')

# Fast mode: correlation threshold only, much faster on large datasets
result = auto_fs(df, target='price', mode='fast')

# Custom correlation threshold in fast mode
result = auto_fs(df, target='price', mode='fast', corr_threshold=0.4)
Parameter Type Default Description
df DataFrame required Input DataFrame
target str required Target column name
model model None Custom sklearn model. Defaults to auto-tuned Ridge
mode str "balanced" "fast": correlation filter only. "balanced": Ridge + cross-validation loop. "accurate": GradientBoosting + cross-validation, slowest but most precise
corr_threshold float 0.3 Minimum correlation to keep a feature (used in fast mode)

auto_fs lives in the deepcsv.ml subpackage. Install the extras with pip install deepcsv[ml] (scikit-learn and tqdm) to use it.


auto_outliers(df, cols=None, method="iqr", threshold=None)

Detects outliers in numeric columns and removes their rows. A row is removed if it is an outlier in any checked column. Missing values are never counted as outliers.

from deepcsv.ml import auto_outliers

# IQR rule on every numeric column (default)
clean = auto_outliers(df)

# Check only some columns (keep your target column untouched)
clean = auto_outliers(df, cols=['age', 'salary'])

# Z-score with a stricter threshold
clean = auto_outliers(df, method='zscore', threshold=2.5)

# Look at all columns together, which catches rows that are only strange as a combination
clean = auto_outliers(df, method='isolation_forest', threshold=0.1)
Parameter Type Default Description
df str | DataFrame required File path or DataFrame
cols list None Numeric columns to check. Default: all numeric columns
method str "iqr" "iqr": outside Q1 - t*IQR / Q3 + t*IQR. "zscore": abs(z) > t. "isolation_forest": looks at all columns together (needs scikit-learn)
threshold float None t of the method. Defaults: 1.5 (iqr), 3.0 (zscore), 0.05 = share of outliers (isolation_forest)

The original index is kept, so you can see which rows were removed.


data_details(data_input, export=None, output_path="data_details", show=True)

Prints a report about your data: size, memory, missing values, duplicates, types, unique values, numeric stats and outliers. Each warning names the deepcsv function that fixes it.

from deepcsv import data_details

report = data_details('data/users.csv')            # print the report
report = data_details(df, export='csv')            # also save the columns table
report = data_details(df, export='txt')            # also save the full report
report = data_details(df, export=['csv', 'txt'], output_path='reports/users')

report['overview']   # dict with rows, columns, memory, duplicates, missing cells
report['columns']    # DataFrame, one row per column
report['warnings']   # list of problems found
============================================================
 DATA DETAILS
============================================================
Rows            : 6
Columns         : 4  (numeric: 1, other: 3)
Memory          : 0.001 MB
Duplicate rows  : 0
Missing cells   : 1 (4.17%)

------------------------------------------------------------
 COLUMNS
------------------------------------------------------------
column  dtype  non_null  nulls  null_%  unique
   age  int64         6      0    0.00       6
 price object         6      0    0.00       6
  tags object         6      0    0.00       1
  city object         5      1   16.67       2

------------------------------------------------------------
 NUMERIC STATS
------------------------------------------------------------
column   mean     std  min  25%  50%  75%   max  outliers
   age 74.167 110.671 25.0 27.5 30.0 32.5 300.0         1

------------------------------------------------------------
 WARNINGS
------------------------------------------------------------
- (age) has 1 outliers - remove them with ml.auto_outliers().
- (price) stores numbers as text - fix it with auto_fix().
- (tags) has a single value - it carries no information.
- (tags) stores lists/dicts as strings - convert it with process_file().
============================================================
Parameter Type Default Description
data_input str | DataFrame required File path (any supported format) or DataFrame
export str | list None "csv": columns table. "txt": full report. ["csv", "txt"]: both
output_path str "data_details" File path without extension
show bool True False: do not print, only return the report

merge_data(left, right, on, how="inner", suffixes=("_x", "_y"))

Merges two datasets on a column (or a list of columns). Each dataset can be a file in any supported format or a DataFrame, so you can merge a CSV with an Excel file directly.

from deepcsv import merge_data

df = merge_data('users.csv', 'orders.xlsx', on='user_id')
df = merge_data(users_df, orders_df, on='user_id', how='left')
df = merge_data('a.csv', 'b.csv', on=['country', 'year'], how='outer')
Parameter Type Default Description
left, right str | DataFrame required File path or DataFrame
on str | list required Column name(s) that exist in both datasets
how str "inner" "inner", "left", "right" or "outer"
suffixes tuple ("_x", "_y") Added to columns with the same name in both datasets

A warning is printed when repeated keys can multiply rows.


Function Signatures

process_file(data_input: Union[str, pd.DataFrame], file_format: str = None, auto_fix: bool = False, to_list: bool = False, deep_check: bool = False, col_name:  Union[str, list] = "all") -> pd.DataFrame
process_all_files(directory_path: str, output_dir="All CSV Files is Converted Here",file_format= "parquet",auto_fix=False, to_list=False, deep_check=False) -> None
read_any(file_path: str) -> pd.DataFrame
clean_values(data_input, cols=None, ax_0=False, index=None, condition=None, all_cols_except=None, finding_value=None, finding_type=None) -> pd.DataFrame
auto_fix(data_input: Union[str, pd.DataFrame], col_name: Union[str, list] = "all") -> pd.DataFrame
auto_fs(df: pd.DataFrame, target: str, model=None, mode: str = "balanced", corr_threshold: float = 0.3) -> pd.DataFrame
auto_outliers(df: Union[str, pd.DataFrame], cols: list = None, method: str = "iqr", threshold: float = None) -> pd.DataFrame
data_details(data_input: Union[str, pd.DataFrame], export: Union[str, list] = None, output_path: str = "data_details", show: bool = True) -> dict
merge_data(left: Union[str, pd.DataFrame], right: Union[str, pd.DataFrame], on: Union[str, list], how: str = "inner", suffixes: tuple = ("_x", "_y")) -> pd.DataFrame

What you get

  • Lists stored as strings become real NumPy arrays (or Python lists). deep_check also handles lists and dicts nested inside them.
  • The cleaning functions work on one column, a list of columns, or all of them.
  • Mixed-type columns and numbers stored as text are found and fixed, and every change is printed.
  • Cleaning by null, value, type or index, with six comparison operators for conditions.
  • One reader for 10+ formats, and saving to CSV, Excel, JSON, Parquet, Feather and more.
  • Folder trees are processed recursively.
  • auto_fs selects features in three modes, and auto_outliers removes outliers with three methods.
  • data_details reports on your data and exports it as CSV or TXT, and merge_data joins files of different formats.

Notes

  • Requires pyarrow for Parquet and Feather support
  • process_all_files only saves files that contain converted list columns

Requirements

  • Python >= 3.7
  • pandas
  • pyarrow
  • scikit-learn and tqdm (optional, only needed for deepcsv.ml: pip install deepcsv[ml])

PyPI · GitHub · Kaggle

By: Abdullah Bakr

Metadata

Release files for deepcsv 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for deepcsv 0.8.0
File Size Uploaded Provenance Actions
deepcsv-0.8.0.tar.gz 32.3 kB Without provenance Details for deepcsv-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for deepcsv 0.8.0
File Interpreter ABI Platform Actions
deepcsv-0.8.0-py3-none-any.whl Python 3 none any Details for deepcsv-0.8.0-py3-none-any.whl

Total release size: 57.1 kB

Release files / deepcsv-0.8.0.tar.gz

Download URL deepcsv-0.8.0.tar.gz
Size 32.3 kB
Tags Source
SHA-256 checksum
How to use checksums
708671a3d9db4934a630d3a43ae228603355e0d7d63dc1a79c2f2fbc02f654e5
BLAKE2b-256 checksum
How to use checksums
c24c465844373f5c0e1fc75708010125b96653d4935a96d99d7f2c8707ae8b1f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.12

Release files / deepcsv-0.8.0-py3-none-any.whl

Download URL deepcsv-0.8.0-py3-none-any.whl
Size 24.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2f56d28dba1da96c32d05bf051c8a60410d211711a35f425691ca625fbde22ac
BLAKE2b-256 checksum
How to use checksums
b2cdb8899898f8d77c0da85c869a111f877c2816d7a93210a3494b596a88a7f4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.12

Security for deepcsv 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page