filoma logo

Dataset CI for ML — folder → verified dataset → insights → agent, in one pipeline.

PyPI version Python versions License Ruff Actions status Documentation Status

Documentation: https://filoma.readthedocs.io

Source Code: https://github.com/kalfasyan/filoma

Filoma profiles file trees, builds DataFrames, finds duplicates, runs integrity checks, and lets you talk to your filesystem — all with a tiny API and automatic backend selection (Rust → fd → Python).

The key features are:

  • Fast: Scans 1M files in 7 seconds with the Rust backend.* Auto-detected at runtime — no config needed.
  • DataFrame-native: Polars-powered wrapper with enrichment, filter, sort, and multi-format export.
  • Dedup: Exact duplicates, text near-duplicates, and image near-duplicate detection.
  • Integrity: Snapshots, manifests, and SHA-256 verification for dataset versioning.
  • CI-ready: filoma-gates.yml quality gates with pass/fail exit codes, plus drift detection via filoma watch.
  • Agentic: Natural-language filesystem queries via flm.ask(), interactive chat, or an MCP server.
  • Vector search: LanceDB-backed RAG — search your files by meaning, or attach embeddings straight to the DataFrame with add_embedding_cols() / add_semantic_similarity_cols() to surface semantic relationships between files.
  • Extensible: Third-party plugins via filoma.tools entry points and bundled skill workflows.

* Benchmarked on a MacBook Air M4, local NVMe SSD. Your numbers will vary by hardware and filesystem — see Benchmarks.

Installation

pip install filoma

or with uv:

uv add filoma

Note: The Rust extension (fastest backend) is bundled automatically by pip/uv above — no separate build step needed. See the Installation guide if filoma falls back to the fd/Python backend on your platform.

Optional extras:

pip install "filoma[dedup]"   # near-duplicate detection
pip install "filoma[rag]"     # LanceDB vector search
pip install "filoma[stats]"   # statistical analysis extras

Example

Your first scan

import filoma as flm

# Scan a directory — the Rust backend kicks in automatically
analysis = flm.probe("./my-dataset")
analysis.print_summary()

# Get a Polars DataFrame, enriched with depth, path components, and file stats
df = flm.probe_to_df("./my-dataset")

# Filter and explore
images = df.filter_by_extension([".png", ".jpg"])
print(df.extension_counts())

Run it

python -c "import filoma as flm; flm.probe('.').print_summary()"

📊 Directory Analysis: /home/user/my-dataset
   Total files: 12,847
   Total size: 3.2 GB
   Types: .png (4,201), .json (3,100), .txt (2,546), .jpg (3,000)

Use with GitHub Copilot

The fastest way to try filoma without writing Python: plug it into GitHub Copilot (VS Code chat, Copilot CLI, or Copilot coding agent).

Agent Skill — teaches Copilot to drive filoma's CLI for you:

filoma skills install --scope vscode   # writes ./.github/skills/filoma-*/SKILL.md

MCP server — gives Copilot real, callable tools (probe_directory, audit_dataset, search_files, and more):

copilot mcp add filoma -- uvx -p 3.11 filoma mcp serve

Both work via uvx — no pip install filoma needed to try them out. See the Filaraki guide for VS Code chat setup, nanobot, and other MCP clients.

Example Upgrade

Audit a dataset (scan → enrich → verify → report)

import filoma as flm

# One fluent pipeline — a single filesystem walk
flm.Pipeline("./data").scan().enrich().verify().report()

filoma audit ./data --export report.html --format html

This produces an HTML audit report with file counts, type breakdowns, integrity status, and warnings.

Talk to your filesystem

import filoma as flm

result = flm.ask("how many corrupted images are in ./dataset?")
print(result.output)

filoma ask "find all python files modified in the last week"

Validate in CI

filoma audit ./data --gates filoma-gates.yml

Exits 0 on pass, 1 on any gate failure — wire it straight into a CI job. Track drift between runs with filoma watch ./data --snapshot baseline.json.

Define gates in filoma-gates.yml:

gates:
  corrupted_files: 0
  zero_byte_files: 0
  duplicate_ratio_pct: 5
  hygiene_score: 80

Backends

Filoma auto-detects the fastest available backend at runtime:

Backend 1M files (local SSD) 200K files (network)
Rust 7.3s — 136K files/sec 2.3s — 86K files/sec
fd / Async 11.5s — 87K files/sec 2.8s — 70K files/sec
Python 35.5s — 28K files/sec 15.1s — 13K files/sec

Measured on a MacBook Air M4 / local NVMe SSD (1M files) and a network NFS mount (200K files) — see full methodology and hardware. Your results will vary by hardware, filesystem, and directory shape.

No configuration required — the fastest backend is selected for you.

Recap

In summary, you get:

  • One-line scans — flm.probe(path) gives you a full directory analysis.
  • Fluent pipelines — chain .scan().enrich().verify().report() with shared state.
  • Agentic queries — flm.ask("question") talks to your filesystem in natural language.
  • DataFrame power — Polars-native, with enrichment, lineage tracking, and export.
  • CI integration — Snapshot, verify, and gate-check in any pipeline.
  • Auto backend — Rust when available, fd as fallback, Python as last resort.

Dependencies

Filoma stands on the shoulders of:

Guides

Persona Start here
ML Engineer Audit a Dataset
Data Engineer Explore a Dataset
Researcher Talk to Your Data
DevOps / CI Validate in CI
Package author Plugin Discovery

License

This project is licensed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0).

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.12.10)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for kalfasyan from gravatar.com kalfasyan

Requires

Python <3.15, >=3.11

Provides Extra

dedup dev docs pd rag stats

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

filoma-1.12.9.tar.gz (816.0 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

filoma-1.12.9-cp311-cp311-win_amd64.whl (556.6 kB view details)

Uploaded CPython 3.11Windows x86-64

filoma-1.12.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (723.5 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

filoma-1.12.9-cp311-cp311-macosx_11_0_arm64.whl (668.8 kB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file filoma-1.12.9.tar.gz.

File metadata

  • Download URL: filoma-1.12.9.tar.gz
  • Upload date:
  • Size: 816.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for filoma-1.12.9.tar.gz
Algorithm Hash digest
SHA256 f0479c25561e18a065ebb5b249314e89864b10e7654791b5d6204b4fffee98f2
MD5 8d8b685741f57e12b5f978e36510389c
BLAKE2b-256 557d18a4c3e065185d67f29901c2d8317b2e8110740d0d77d799474fc0ef89e4

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.9.tar.gz:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.9-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: filoma-1.12.9-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 556.6 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for filoma-1.12.9-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 ec74f14d69ae2addb05c810a72940c90d939a75626da1bd8b376ad4b2a72bae6
MD5 5e9aca74b4e0ba89dbfae498664eb8f9
BLAKE2b-256 1ecdc6babb9ffb0f84bb769c5b90e1f6e4f189b0091834d04b69b67ac686710a

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.9-cp311-cp311-win_amd64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for filoma-1.12.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9a069a6e917ac550ccf6e2da5365ddd4610e7744961b7c0548a9558ee43e09fb
MD5 26cecd6046cd9099fd2e05368599018b
BLAKE2b-256 ac97c858fbf4f45d3740082f6843a38558aea1c71f430d25959f0cff0216e4ab

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.9-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for filoma-1.12.9-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 53c0b4e6f840132a701666e1ab6b8008bded9657d1a761036b76b5ef58b82c5e
MD5 1fe6cc6d984e393ae50466294a46d1b2
BLAKE2b-256 46d8246aabc87f6870f4ea733110ef3d99866f363765f0e3dbe432007e07d7bc

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.9-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.12.10

4 files

This release

1.12.9 This release

4 files

1.12.8

4 files

1.12.7

4 files

1.12.5

4 files

1.12.4

4 files

1.12.3

4 files

1.12.2

4 files

1.12.1

4 files

1.12.0

4 files

1.11.18

4 files

1.11.17

4 files

1.11.16

4 files

1.11.12

4 files

1.11.11

4 files

1.11.10

4 files

1.11.7

4 files

1.11.6

4 files

1.11.5

4 files

1.11.4

4 files

1.11.1

4 files

1.11.0

4 files

1.10.2

4 files

1.10.1

4 files

1.10.0

4 files

1.9.6

4 files

1.9.5

4 files

1.9.4

4 files

1.9.3

4 files

1.9.2

4 files

1.9.1

4 files

1.9.0

4 files

1.8.1

4 files

1.8.0

4 files

1.7.7

4 files

1.7.6

4 files

1.7.5

4 files

1.7.4

4 files

1.7.3

4 files

1.7.2

4 files

1.7.1

4 files

1.7.0

4 files

1.6.2

4 files

1.6.1

4 files

1.4.0

4 files

1.3.4

4 files

1.3.3

4 files

1.3.2

4 files

1.3.1

4 files

1.3.0

4 files

1.2.1

4 files

1.2.0

4 files

1.1.1

4 files

1.1.0

4 files

1.0.3

4 files

1.0.2

4 files

0.1.3

2 files

0.1.2

2 files

0.1.0

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page