filoma logo

Dataset CI for ML — folder → verified dataset → insights → agent, in one pipeline.

PyPI version Python versions License Ruff Actions status Documentation Status

Documentation: https://filoma.readthedocs.io

Source Code: https://github.com/kalfasyan/filoma

Filoma profiles file trees, builds DataFrames, finds duplicates, runs integrity checks, and lets you talk to your filesystem — all with a tiny API and automatic backend selection (Rust → fd → Python).

The key features are:

  • Fast: Scans 1M files in 7 seconds with the Rust backend.* Auto-detected at runtime — no config needed.
  • DataFrame-native: Polars-powered wrapper with enrichment, filter, sort, and multi-format export.
  • Dedup: Exact duplicates, text near-duplicates, and image near-duplicate detection.
  • Integrity: Snapshots, manifests, and SHA-256 verification for dataset versioning.
  • CI-ready: filoma-gates.yml quality gates with pass/fail exit codes, plus drift detection via filoma watch.
  • Agentic: Natural-language filesystem queries via flm.ask(), interactive chat, or an MCP server.
  • Vector search: LanceDB-backed RAG — search your files by meaning, or attach embeddings straight to the DataFrame with add_embedding_cols() / add_semantic_similarity_cols() to surface semantic relationships between files.
  • Extensible: Third-party plugins via filoma.tools entry points and bundled skill workflows.

* Benchmarked on a MacBook Air M4, local NVMe SSD. Your numbers will vary by hardware and filesystem — see Benchmarks.

Installation

pip install filoma

or with uv:

uv add filoma

Note: The Rust extension (fastest backend) is bundled automatically by pip/uv above — no separate build step needed. See the Installation guide if filoma falls back to the fd/Python backend on your platform.

Optional extras:

pip install "filoma[dedup]"   # near-duplicate detection
pip install "filoma[rag]"     # LanceDB vector search
pip install "filoma[stats]"   # statistical analysis extras

Example

Your first scan

import filoma as flm

# Scan a directory — the Rust backend kicks in automatically
analysis = flm.probe("./my-dataset")
analysis.print_summary()

# Get a Polars DataFrame, enriched with depth, path components, and file stats
df = flm.probe_to_df("./my-dataset")

# Filter and explore
images = df.filter_by_extension([".png", ".jpg"])
print(df.extension_counts())

Run it

python -c "import filoma as flm; flm.probe('.').print_summary()"

📊 Directory Analysis: /home/user/my-dataset
   Total files: 12,847
   Total size: 3.2 GB
   Types: .png (4,201), .json (3,100), .txt (2,546), .jpg (3,000)

Use with GitHub Copilot

The fastest way to try filoma without writing Python: plug it into GitHub Copilot (VS Code chat, Copilot CLI, or Copilot coding agent).

Agent Skill — teaches Copilot to drive filoma's CLI for you:

filoma skills install --scope vscode   # writes ./.github/skills/filoma-*/SKILL.md

MCP server — gives Copilot real, callable tools (probe_directory, audit_dataset, search_files, and more):

copilot mcp add filoma -- uvx -p 3.11 filoma mcp serve

Both work via uvx — no pip install filoma needed to try them out. See the Filaraki guide for VS Code chat setup, nanobot, and other MCP clients.

Example Upgrade

Audit a dataset (scan → enrich → verify → report)

import filoma as flm

# One fluent pipeline — a single filesystem walk
flm.Pipeline("./data").scan().enrich().verify().report()

filoma audit ./data --export report.html --format html

This produces an HTML audit report with file counts, type breakdowns, integrity status, and warnings.

Talk to your filesystem

import filoma as flm

result = flm.ask("how many corrupted images are in ./dataset?")
print(result.output)

filoma ask "find all python files modified in the last week"

Validate in CI

filoma audit ./data --gates filoma-gates.yml

Exits 0 on pass, 1 on any gate failure — wire it straight into a CI job. Track drift between runs with filoma watch ./data --snapshot baseline.json.

Define gates in filoma-gates.yml:

gates:
  corrupted_files: 0
  zero_byte_files: 0
  duplicate_ratio_pct: 5
  hygiene_score: 80

Backends

Filoma auto-detects the fastest available backend at runtime:

Backend 1M files (local SSD) 200K files (network)
Rust 7.3s — 136K files/sec 2.3s — 86K files/sec
fd / Async 11.5s — 87K files/sec 2.8s — 70K files/sec
Python 35.5s — 28K files/sec 15.1s — 13K files/sec

Measured on a MacBook Air M4 / local NVMe SSD (1M files) and a network NFS mount (200K files) — see full methodology and hardware. Your results will vary by hardware, filesystem, and directory shape.

No configuration required — the fastest backend is selected for you.

Recap

In summary, you get:

  • One-line scans — flm.probe(path) gives you a full directory analysis.
  • Fluent pipelines — chain .scan().enrich().verify().report() with shared state.
  • Agentic queries — flm.ask("question") talks to your filesystem in natural language.
  • DataFrame power — Polars-native, with enrichment, lineage tracking, and export.
  • CI integration — Snapshot, verify, and gate-check in any pipeline.
  • Auto backend — Rust when available, fd as fallback, Python as last resort.

Dependencies

Filoma stands on the shoulders of:

Guides

Persona Start here
ML Engineer Audit a Dataset
Data Engineer Explore a Dataset
Researcher Talk to Your Data
DevOps / CI Validate in CI
Package author Plugin Discovery

License

This project is licensed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0).

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.12.10)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for kalfasyan from gravatar.com kalfasyan

Requires

Python <3.15, >=3.11

Provides Extra

dedup dev docs pd rag stats

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

filoma-1.12.8.tar.gz (774.1 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

filoma-1.12.8-cp311-cp311-win_amd64.whl (543.5 kB view details)

Uploaded CPython 3.11Windows x86-64

filoma-1.12.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (710.5 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

filoma-1.12.8-cp311-cp311-macosx_11_0_arm64.whl (655.9 kB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file filoma-1.12.8.tar.gz.

File metadata

  • Download URL: filoma-1.12.8.tar.gz
  • Upload date:
  • Size: 774.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for filoma-1.12.8.tar.gz
Algorithm Hash digest
SHA256 6dcc3d5878221ad743c1a58aae13f7365c2d44303e2949640269505567fbf034
MD5 575b77b3ccb04f6e453a6390ab367a91
BLAKE2b-256 10b73e7ad0f3f991cf409899ac05bd64337bb7cb585ff651cc1d8dea9f3b4519

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.8.tar.gz:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.8-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: filoma-1.12.8-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 543.5 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for filoma-1.12.8-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 52df473cd59dfd4ca63795d908978daabc3e50c4b75c28091aef428b95ea6b3e
MD5 870498f9df35a38115e879130d7f96b6
BLAKE2b-256 1aa3e2e22a08fea9563e46010464fac9ae886361ef78806c42994a23c272c530

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.8-cp311-cp311-win_amd64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for filoma-1.12.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 6ad702b53a9680d4ef89ae9df45414d4f02eb319331bdd6dbe4eb3eee87df479
MD5 d5683b62271f346beb75e5016c078635
BLAKE2b-256 ee5c4bf3c144ba5286176ad540a16e3431bd8447df86bad501f9971af365ad55

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.8-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for filoma-1.12.8-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 425c5e5b48cd8ca71bb32c5ea43feac4bee32183b3660c9f0fd8e2d9e8f23f0c
MD5 a3cbe587b74fe3a08d8e0f3745769413
BLAKE2b-256 d11d4814cce7a06f3d79eaafda8ce83014ba34987a818b3b9c98eec648165bc7

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.8-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.12.10

4 files

1.12.9

4 files

This release

1.12.8 This release

4 files

1.12.7

4 files

1.12.5

4 files

1.12.4

4 files

1.12.3

4 files

1.12.2

4 files

1.12.1

4 files

1.12.0

4 files

1.11.18

4 files

1.11.17

4 files

1.11.16

4 files

1.11.12

4 files

1.11.11

4 files

1.11.10

4 files

1.11.7

4 files

1.11.6

4 files

1.11.5

4 files

1.11.4

4 files

1.11.1

4 files

1.11.0

4 files

1.10.2

4 files

1.10.1

4 files

1.10.0

4 files

1.9.6

4 files

1.9.5

4 files

1.9.4

4 files

1.9.3

4 files

1.9.2

4 files

1.9.1

4 files

1.9.0

4 files

1.8.1

4 files

1.8.0

4 files

1.7.7

4 files

1.7.6

4 files

1.7.5

4 files

1.7.4

4 files

1.7.3

4 files

1.7.2

4 files

1.7.1

4 files

1.7.0

4 files

1.6.2

4 files

1.6.1

4 files

1.4.0

4 files

1.3.4

4 files

1.3.3

4 files

1.3.2

4 files

1.3.1

4 files

1.3.0

4 files

1.2.1

4 files

1.2.0

4 files

1.1.1

4 files

1.1.0

4 files

1.0.3

4 files

1.0.2

4 files

0.1.3

2 files

0.1.2

2 files

0.1.0

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page