filoma logo

Dataset CI for ML — folder → verified dataset → insights → agent, in one pipeline.

PyPI version Python versions License Ruff Actions status Documentation Status

Documentation: https://filoma.readthedocs.io

Source Code: https://github.com/kalfasyan/filoma

Filoma profiles file trees, builds DataFrames, finds duplicates, runs integrity checks, and lets you talk to your filesystem — all with a tiny API and automatic backend selection (Rust → fd → Python).

The key features are:

  • Fast: Scans 1M files in 7 seconds with the Rust backend.* Auto-detected at runtime — no config needed.
  • DataFrame-native: Polars-powered wrapper with enrichment, filter, sort, and multi-format export.
  • Dedup: Exact duplicates, text near-duplicates, and image near-duplicate detection.
  • Integrity: Snapshots, manifests, and SHA-256 verification for dataset versioning.
  • CI-ready: filoma-gates.yml quality gates with pass/fail exit codes, plus drift detection via filoma watch.
  • Agentic: Natural-language filesystem queries via flm.ask(), interactive chat, or an MCP server.
  • Vector search: LanceDB-backed RAG — search your files by meaning, or attach embeddings straight to the DataFrame with add_embedding_cols() / add_semantic_similarity_cols() to surface semantic relationships between files.
  • Extensible: Third-party plugins via filoma.tools entry points and bundled skill workflows.

* Benchmarked on a MacBook Air M4, local NVMe SSD. Your numbers will vary by hardware and filesystem — see Benchmarks.

Installation

pip install filoma

or with uv:

uv add filoma

Note: The Rust extension (fastest backend) is bundled automatically by pip/uv above — no separate build step needed. See the Installation guide if filoma falls back to the fd/Python backend on your platform.

Optional extras:

pip install "filoma[dedup]"   # near-duplicate detection
pip install "filoma[rag]"     # LanceDB vector search
pip install "filoma[stats]"   # statistical analysis extras

Example

Your first scan

import filoma as flm

# Scan a directory — the Rust backend kicks in automatically
analysis = flm.probe("./my-dataset")
analysis.print_summary()

# Get a Polars DataFrame, enriched with depth, path components, and file stats
df = flm.probe_to_df("./my-dataset")

# Filter and explore
images = df.filter_by_extension([".png", ".jpg"])
print(df.extension_counts())

Run it

python -c "import filoma as flm; flm.probe('.').print_summary()"

📊 Directory Analysis: /home/user/my-dataset
   Total files: 12,847
   Total size: 3.2 GB
   Types: .png (4,201), .json (3,100), .txt (2,546), .jpg (3,000)

Use with GitHub Copilot

The fastest way to try filoma without writing Python: plug it into GitHub Copilot (VS Code chat, Copilot CLI, or Copilot coding agent).

Agent Skill — teaches Copilot to drive filoma's CLI for you:

filoma skills install --scope vscode   # writes ./.github/skills/filoma-*/SKILL.md

MCP server — gives Copilot real, callable tools (probe_directory, audit_dataset, search_files, and more):

copilot mcp add filoma -- uvx -p 3.11 filoma mcp serve

Both work via uvx — no pip install filoma needed to try them out. See the Filaraki guide for VS Code chat setup, nanobot, and other MCP clients.

Example Upgrade

Audit a dataset (scan → enrich → verify → report)

import filoma as flm

# One fluent pipeline — a single filesystem walk
flm.Pipeline("./data").scan().enrich().verify().report()

filoma audit ./data --export report.html --format html

This produces an HTML audit report with file counts, type breakdowns, integrity status, and warnings.

Talk to your filesystem

import filoma as flm

result = flm.ask("how many corrupted images are in ./dataset?")
print(result.output)

filoma ask "find all python files modified in the last week"

Validate in CI

filoma audit ./data --gates filoma-gates.yml

Exits 0 on pass, 1 on any gate failure — wire it straight into a CI job. Track drift between runs with filoma watch ./data --snapshot baseline.json.

Define gates in filoma-gates.yml:

gates:
  corrupted_files: 0
  zero_byte_files: 0
  duplicate_ratio_pct: 5
  hygiene_score: 80

Backends

Filoma auto-detects the fastest available backend at runtime:

Backend 1M files (local SSD) 200K files (network)
Rust 7.3s — 136K files/sec 2.3s — 86K files/sec
fd / Async 11.5s — 87K files/sec 2.8s — 70K files/sec
Python 35.5s — 28K files/sec 15.1s — 13K files/sec

Measured on a MacBook Air M4 / local NVMe SSD (1M files) and a network NFS mount (200K files) — see full methodology and hardware. Your results will vary by hardware, filesystem, and directory shape.

No configuration required — the fastest backend is selected for you. Inside Rust, the default local engine is the dua-core parallel walker (parallel read_dir + metadata); opt out with walker="walkdir" in DirectoryProfilerConfig.

Recap

In summary, you get:

  • One-line scans — flm.probe(path) gives you a full directory analysis.
  • Fluent pipelines — chain .scan().enrich().verify().report() with shared state.
  • Agentic queries — flm.ask("question") talks to your filesystem in natural language.
  • DataFrame power — Polars-native, with enrichment, lineage tracking, and export.
  • CI integration — Snapshot, verify, and gate-check in any pipeline.
  • Auto backend — Rust when available, fd as fallback, Python as last resort.

Dependencies

Filoma stands on the shoulders of:

Guides

Persona Start here
ML Engineer Audit a Dataset
Data Engineer Explore a Dataset
Researcher Talk to Your Data
DevOps / CI Validate in CI
Package author Plugin Discovery

License

This project is licensed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0).

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for kalfasyan from gravatar.com kalfasyan

Requires

Python <3.15, >=3.11

Provides Extra

dedup dev docs pd rag stats

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

filoma-1.12.10.tar.gz (825.7 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

filoma-1.12.10-cp311-cp311-win_amd64.whl (630.5 kB view details)

Uploaded CPython 3.11Windows x86-64

filoma-1.12.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (803.6 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.17+ x86-64

filoma-1.12.10-cp311-cp311-macosx_11_0_arm64.whl (741.4 kB view details)

Uploaded CPython 3.11macOS 11.0+ ARM64

File details

Details for the file filoma-1.12.10.tar.gz.

File metadata

  • Download URL: filoma-1.12.10.tar.gz
  • Upload date:
  • Size: 825.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for filoma-1.12.10.tar.gz
Algorithm Hash digest
SHA256 d36b05506e5d3332545c270a660a6c27e015eda07a4b6dffebeea84a2f526034
MD5 dc010120ce6e56c1d18c304c027df3b5
BLAKE2b-256 2ff195a776a716733a136e3cb192c6b25f699ab457b5f9b8a1556c7501bd89ce

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.10.tar.gz:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.10-cp311-cp311-win_amd64.whl.

File metadata

  • Download URL: filoma-1.12.10-cp311-cp311-win_amd64.whl
  • Upload date:
  • Size: 630.5 kB
  • Tags: CPython 3.11, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for filoma-1.12.10-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 f899d98f8264189594f2777643d9d16d3c7caef0736b9614fbcbbd28e1834b13
MD5 91b990d4dbc39ef17ce62dbfda9f8d95
BLAKE2b-256 06898532c363958276c55a6d2e7ff26da767a9ed33c0d0fc1323f947d64472e5

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.10-cp311-cp311-win_amd64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for filoma-1.12.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e2f25c55b9982b4e13bb4a8f1302aa46884fb6a9e488e93b3fc7f8737b661b95
MD5 4d3555896ea297ff90d84f1f071ea8a3
BLAKE2b-256 f12eeaeba810b4351dab6c7ae42c5decf464612fe1a17a763e1418950709d67b

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file filoma-1.12.10-cp311-cp311-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for filoma-1.12.10-cp311-cp311-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 d1d5f07f468f2f5fbf9dd92b4ec34a6bb2183b147ee0d42bef15579e18aacfee
MD5 a6628d8f1e1e1ca1f02d97773cadba62
BLAKE2b-256 5bb9bc5045485295c9a07207bc8821528da6cdb3f47cbfb81856fb4ca5a564ee

See more details on using hashes here.

Provenance

The following attestation bundles were made for filoma-1.12.10-cp311-cp311-macosx_11_0_arm64.whl:

Publisher: publish.yml on kalfasyan/filoma

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

1.12.10 This release

4 files

1.12.9

4 files

1.12.8

4 files

1.12.7

4 files

1.12.5

4 files

1.12.4

4 files

1.12.3

4 files

1.12.2

4 files

1.12.1

4 files

1.12.0

4 files

1.11.18

4 files

1.11.17

4 files

1.11.16

4 files

1.11.12

4 files

1.11.11

4 files

1.11.10

4 files

1.11.7

4 files

1.11.6

4 files

1.11.5

4 files

1.11.4

4 files

1.11.1

4 files

1.11.0

4 files

1.10.2

4 files

1.10.1

4 files

1.10.0

4 files

1.9.6

4 files

1.9.5

4 files

1.9.4

4 files

1.9.3

4 files

1.9.2

4 files

1.9.1

4 files

1.9.0

4 files

1.8.1

4 files

1.8.0

4 files

1.7.7

4 files

1.7.6

4 files

1.7.5

4 files

1.7.4

4 files

1.7.3

4 files

1.7.2

4 files

1.7.1

4 files

1.7.0

4 files

1.6.2

4 files

1.6.1

4 files

1.4.0

4 files

1.3.4

4 files

1.3.3

4 files

1.3.2

4 files

1.3.1

4 files

1.3.0

4 files

1.2.1

4 files

1.2.0

4 files

1.1.1

4 files

1.1.0

4 files

1.0.3

4 files

1.0.2

4 files

0.1.3

2 files

0.1.2

2 files

0.1.0

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page