filoma 1.7.6

Modular Python tool for profiling files, analyzing directory structures, and inspecting image data

Fast, multi-backend file/directory profiling and data preparation for machine learning workflows.

Installation • Quickstart • Cookbook • Source Code

---

filoma helps you analyze file directory trees, inspect file metadata, and prepare your data for exploration and modelling. It can achieve this blazingly fast using the best available backend (Rust, fd, or pure Python) ⚡🍃

---

Key Features

• 🚀 High-Performance Backends: Automatic selection of Rust, fd, or Python for the best performance.
• 📊 Rich Directory Analysis: Get detailed statistics on file counts, extensions, sizes, and more.
• 🔍 Smart File Search: Use regex and glob patterns to find files with FdFinder.
• 📈 DataFrame Integration: Convert scan results to Polars (or pandas) DataFrames for powerful analysis.
• 🖼️ File/Image Profiling: Extract metadata and statistics from various file formats.
• 🔀 ML-Ready Splits: Create deterministic train/validation/test datasets with ease.

Scope of filoma

Feature Highlights

Quick, copyable examples showing filoma's standout capabilities and where to learn more.

• Automatic multi-backend scanning: filoma picks the fastest available backend (Rust → fd → pure Python). You can also force a backend for reproducibility. See the backends docs: docs/backends.md.

import filoma as flm

# filoma will pick Rust > fd > Python depending on availability
analysis = flm.probe('.')
analysis.print_summary()

• Polars-first DataFrame wrapper & enrichment: Returns a filoma.DataFrame (Polars) with helpers to add path components, depth, and file stats for immediate analysis. Docs: docs/dataframe.md.

df = flm.probe_to_df('.', enrich=True)  # returns a filoma.DataFrame
print(df.head())

• Ultra-fast discovery with fd: When fd is available filoma uses it for very fast file discovery. Advanced usage and patterns: docs/advanced-usage.md.

if flm.fd.is_available():
    files = flm.fd.find(pattern=r"\\.py$", path='src', max_depth=3)
    print(len(files), 'python files found')

• ML-ready, deterministic splits: Group-aware, reproducible train/validation/test splitting to avoid leakage. See docs/ml.md for grouping options and examples.

df = flm.probe_to_df('.', enrich=False)
train, val, test = flm.ml.split_data(df, train_val_test=(70,15,15), seed=42)

• Lightweight, lazy top-level API: Importing filoma is cheap; heavy dependencies load only when used. Quickstart and one-line helpers: docs/quickstart.md.

info = flm.probe_file('README.md')
df = flm.probe_to_df('.')

Installation

Install filoma using uv or pip:

uv pip install filoma

---

Workflow Demo

This guide follows a typical filoma workflow, from basic file profiling to creating machine learning datasets.

1. Profile a Single File

Start by inspecting a single file. filoma provides a detailed dataclass with metadata.

import filoma as flm

# Profile a file
file_info = flm.probe_file("README.md")

print(f"Path: {file_info.path}")
print(f"Size: {file_info.size_str}")
print(f"Modified: {file_info.modified}")

For images, probe_image gives you additional details like shape and pixel statistics.

# Profile an image
img_info = flm.probe_image("images/logo.png")
print(f"Type: {img_info.file_type}")
print(f"Shape: {img_info.shape}")

2. Analyze a Directory

Scan an entire directory to get a high-level overview.

# Analyze the current directory
analysis = flm.probe('.')

# Print a summary report
analysis.print_summary()

Directory Analysis: /project (🦀 Rust (Parallel)) - 0.27s
Total Files: 17,330    Total Folders: 2,427    Analysis Time: 0.27 s

3. Convert to a DataFrame

For detailed analysis, convert the scan results into a Polars DataFrame.

# Scan a directory and get a DataFrame
df = flm.probe_to_df('.')

print(df.head())

4. Enrich Your Data

Add more context to your DataFrame, like file depth and path components, with the enrich() method.

# The DataFrame returned by flm.probe_to_df is a filoma.DataFrame
# with extra capabilities.
df_enriched = df.enrich()

print(df_enriched.head())

5. Create ML-Ready Splits

filoma makes it easy to split your files into training, validation, and test sets for machine learning. You can even group files by parts of their path to prevent data leakage.

# Split the data, grouping by parent directory
train, val, test = flm.ml.split_data(df, how='parts', parts=(-2,), seed=42)

print(f"Train: {len(train)}, Validation: {len(val)}, Test: {len(test)}")

---

License

Shield:

This work is licensed under a Creative Commons Attribution 4.0 International License.

Contributing

Contributions welcome! Please check the issues for planned features and bug reports.

---

filoma - Fast, multi-backend file/directory profiling and data preparation for Python.