filoma logo

Dataset CI for ML — folder → verified dataset → insights → agent, in one pipeline.

PyPI version Python versions License Ruff Actions status Documentation Status

Documentation: https://filoma.readthedocs.io

Source Code: https://github.com/kalfasyan/filoma

Filoma profiles file trees, builds DataFrames, finds duplicates, runs integrity checks, and lets you talk to your filesystem — all with a tiny API and automatic backend selection (Rust → fd → Python).

The key features are:

  • Fast: Scans 1M files in 7 seconds with the Rust backend. Auto-detected at runtime — no config needed.
  • DataFrame-native: Polars-powered wrapper with enrichment, filter, sort, and multi-format export.
  • Dedup: Exact duplicates, text near-duplicates, and image near-duplicate detection.
  • Integrity: Snapshots, manifests, and SHA-256 verification for dataset versioning.
  • CI-ready: YAML quality gates (filoma-gates.yml) with pass/fail for your pipelines.
  • Agentic: 28 filesystem tools via chat, flm.ask(), MCP server, and schema proposals.
  • Vector search: LanceDB-backed RAG — search your files by meaning.
  • Watch mode: Snapshot → drift detect → gates check → JSON export.
  • Extensible: Third-party plugins via filoma.tools entry points and bundled skill workflows.

Installation

$ pip install filoma

---> 100%

or with uv:

uv add filoma

Note: Filoma auto-detects the fastest available backend. For maximum speed, build the Rust extension:

maturin develop --release

Optional extras:

pip install "filoma[dedup]"   # near-duplicate detection
pip install "filoma[rag]"     # LanceDB vector search
pip install "filoma[stats]"   # statistical analysis extras

Example

Your first scan

import filoma as flm

# Scan a directory — the Rust backend kicks in automatically
analysis = flm.probe("./my-dataset")

# Get a Polars DataFrame, enriched with depth, path components, and file stats
df = flm.probe_to_df("./my-dataset")

# Filter and explore
images = df.filter("mime_type", "image/*")
large = df.filter_size(5, "MB", "gt")
print(df.summary())

Run it

python -c "import filoma as flm; flm.probe('.').print_summary()"

📊 Directory Analysis: /home/user/my-dataset
   Total files: 12,847
   Total size: 3.2 GB
   Types: .png (4,201), .json (3,100), .txt (2,546), .jpg (3,000)

Example Upgrade

Audit a dataset (scan → enrich → verify → report)

import filoma as flm

# One fluent pipeline — a single filesystem walk
flm.Pipeline("./data").scan().enrich().verify().report()

filoma audit ./data --export report.html

This produces an HTML audit report with file counts, type breakdowns, integrity status, and warnings.

Talk to your filesystem

import filoma as flm

result = flm.ask("how many corrupted images are in ./dataset?")
print(result.output)

filoma ask "find all python files modified in the last week"

Validate in CI

import filoma as flm

ds = flm.Dataset("./data")
ds.snap().verify().check_gates("gates.yml")

filoma watch ./data --snapshot baseline.json --gates gates.yml

Define quality gates in filoma-gates.yml:

gates:
  - name: "max file count"
    rule: "file_count <= 100000"
  - name: "no corrupt images"
    rule: "corrupt_images == 0"
  - name: "allowed types only"
    rule: 'mime_types in ["image/png", "application/json"]'

Backends

Filoma auto-detects the fastest available backend at runtime:

Backend 1M files (local SSD) 200K files (network)
Rust 7.3s — 136K files/sec 2.3s — 86K files/sec
fd / Async 11.5s — 87K files/sec 2.8s — 70K files/sec
Python 35.5s — 28K files/sec 15.1s — 13K files/sec

No configuration required — the fastest backend is selected for you.

Recap

In summary, you get:

  • One-line scans — flm.probe(path) gives you a full directory analysis.
  • Fluent pipelines — chain .scan().enrich().verify().report() with shared state.
  • Agentic queries — flm.ask("question") talks to your filesystem in natural language.
  • DataFrame power — Polars-native, with enrichment, lineage tracking, and export.
  • CI integration — Snapshot, verify, and gate-check in any pipeline.
  • Auto backend — Rust when available, fd as fallback, Python as last resort.

Dependencies

Filoma stands on the shoulders of:

Guides

Persona Start here
ML Engineer Audit a Dataset
Data Engineer Explore a Dataset
Researcher Talk to Your Data
DevOps / CI Watch for Drift
Package author Plugin Discovery

License

This project is licensed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0).

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.12.10)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for kalfasyan from gravatar.com kalfasyan

Requires

Python <3.15, >=3.11

Provides Extra

dedup dev docs pd rag stats

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

filoma-1.12.7.tar.gz (753.5 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

filoma-1.12.7-cp311-cp311-win_amd64.whl (529.9 kB view details)

Uploaded CPython 3.11Windows x86-64

filoma-1.12.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (697.0 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

filoma-1.12.7-cp311-cp311-macosx_11_0_arm64.whl (642.3 kB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file filoma-1.12.7.tar.gz.

File metadata

  • Download URL: filoma-1.12.7.tar.gz
  • Upload date:
  • Size: 753.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for filoma-1.12.7.tar.gz
Algorithm Hash digest
SHA256 9fa672315c9c54050ad7eae5a10afeec0d6600adcd04aee8d126ccfa3628285c
MD5 2d1914801a12fa4cc5dcbf2611b045ea
BLAKE2b-256 a12642677f68cf55ecf3d9a7c79839e756fb6cb3e39cef1414a8b41760acb16d

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.7.tar.gz:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.7-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: filoma-1.12.7-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 529.9 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for filoma-1.12.7-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 20e8e4bf1bde1f59692ae19a3470d2504afe422993352f00e0921ac39cd5006f
MD5 5a972c92bd8d2d31145cd58815d3448d
BLAKE2b-256 69946af35b7f6d5233938ac6d0319c0b434e5dedd0ef2b8fab99028aac69d82e

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.7-cp311-cp311-win_amd64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for filoma-1.12.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 f45385282b49450957b45cb4f37955f1296ba14c67686c2633918784cb1cd4d4
MD5 ffe8b593937b3054cf52c0fa55361dbd
BLAKE2b-256 a87c3c40160eaf59d4127204aa03925a16e825c4cb898d0afc33f326653c22d6

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.7-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for filoma-1.12.7-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 1d13d4c823c430d5741efb1746c353b557fa3ae9fae6f4abfd9c118150b2b471
MD5 ea1c1d50168f6ca81586f7ca0b6247eb
BLAKE2b-256 d97e1ccd2964c5f77b76058e80f8a2c25be4db0ace16d40127d6339ddb798b35

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.7-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.12.10

4 files

1.12.9

4 files

1.12.8

4 files

This release

1.12.7 This release

4 files

1.12.5

4 files

1.12.4

4 files

1.12.3

4 files

1.12.2

4 files

1.12.1

4 files

1.12.0

4 files

1.11.18

4 files

1.11.17

4 files

1.11.16

4 files

1.11.12

4 files

1.11.11

4 files

1.11.10

4 files

1.11.7

4 files

1.11.6

4 files

1.11.5

4 files

1.11.4

4 files

1.11.1

4 files

1.11.0

4 files

1.10.2

4 files

1.10.1

4 files

1.10.0

4 files

1.9.6

4 files

1.9.5

4 files

1.9.4

4 files

1.9.3

4 files

1.9.2

4 files

1.9.1

4 files

1.9.0

4 files

1.8.1

4 files

1.8.0

4 files

1.7.7

4 files

1.7.6

4 files

1.7.5

4 files

1.7.4

4 files

1.7.3

4 files

1.7.2

4 files

1.7.1

4 files

1.7.0

4 files

1.6.2

4 files

1.6.1

4 files

1.4.0

4 files

1.3.4

4 files

1.3.3

4 files

1.3.2

4 files

1.3.1

4 files

1.3.0

4 files

1.2.1

4 files

1.2.0

4 files

1.1.1

4 files

1.1.0

4 files

1.0.3

4 files

1.0.2

4 files

0.1.3

2 files

0.1.2

2 files

0.1.0

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page