fylex 1.1.3

Fylex: A production-ready, high-performance file utility tailored for Linux — with smart copy/move, hashing, filters, undo/redo, and conflict resolution.

Fylex: The Linux File Ninja

Fylex is a production-ready, Linux-tailored file management tool that combines the best of features of rsync, cp, and Python’s shutil — but goes beyond with:

• Smart Copy & Move with hashing (xxhash, blake3, SHA, MD5)
• Advanced conflict resolution (rename, skip, replace, larger/smaller, newer/older, prompt)
• Filters: regex, glob, exact filename matches, inclusion/exclusion
• Safety nets: undo, redo, backup of deprecated files
• Data integrity: hash verification, SQLite-backed hash cache for deduplication
• Metadata preservation: permissions, timestamps, xattrs, ACLs (Linux-specific)
• CLI & Python API for flexible usage

Feature comparison

Feature / Tool	Fylex	cp (coreutils)	rsync	shutil (Python stdlib)
Primary purpose	Smart local copy/move with safety nets	Basic copy	Fast sync (local/remote)	Library-level file ops
Undo / Redo	Yes — built-in JSON journaling	No	No	No
Hash verification	Yes — xxhash, blake3, sha256, etc.	No	Partial — checksums optional	No
Hash cache (SQLite)	Yes — avoids rehashing unchanged files	No	No	No
Duplicate detection (dest)	Yes — size + hash	No	Partial — based on size/checksums	No
Conflict resolution	Extensive — rename, replace, skip, newer/older, larger/smaller, prompt	None — overwrite only	Limited — flags like --backup, --suffix	None
Metadata preservation	Yes — mtime, perms, xattrs, ACLs on Linux	Partial — -a preserves many	Partial — -a preserves many	Partial — copystat only
Atomic writes	Yes — via fylex.tmp	No	Partial — temp options exist	No
Logging / audit trail	Yes — JSON logs per process	No	Partial — verbose logs only	No
CLI + Python API	Yes — both	CLI only	CLI only (bindings exist)	Python API only
Delta transfer (network)	No — local only	No	Yes	No
Remote / cloud support	No — local-first	No	Yes — ssh/rsyncd	No
Cross-platform	Partial — Linux-first (xattrs/ACL best)	Yes	Yes	Yes
Performance (local)	Very good — uses copy_file_range / sendfile	Good	Very good — efficient I/O	Moderate
Learning curve	Moderate — many options	Very low	Moderate to high — many options	Low
Best fit	Local integrity-critical workflows, reversible ops	Quick one-off copies	Local/remote sync and bandwidth-efficient backups	Small Python scripts

---

Strengths

• Undo / Redo — Most competitors don’t offer a built-in reversible operation for arbitrary copy/move workflows. This is a major safety feature for power users.
• JSON audit trail — Every process writes machine-readable logs that enable reproducibility and automation (and fuel the undo/redo).
• Hash verification + hash cache — Optional verification plus an on-disk SQLite cache avoids repeated hashing of unchanged files and speeds repeated operations.
• Flexible conflict resolution — Many realistic conflict policies (rename with suffix, replace, choose larger/newer, prompt) are available out-of-the-box.
• Linux metadata handling — Attempts to preserve xattrs/ACLs when system tools are available—valuable for server or system admin workflows.
• Atomic writes & backups — Writes to a .tmp area and moves replaced files to fylex.deprecated/PROCESS_ID to avoid data loss during partial operations.
• Good local performance — Uses copy_file_range or sendfile and falls back sensibly for portability.

Limitations

• Not a network delta-sync tool — If you need efficient remote sync over low bandwidth, rsync or rclone is better (rsync implements delta transfer).
• Not a full backup system — For encrypted, deduplicated backups with retention/versioning, choose borg/restic.
• Linux-first — Windows/macOS will work for basic operations, but xattr/ACL preservation and some system calls are Linux-specific; Fylex is intentionally optimized for Linux environments.
• No built-in remote cloud backends — If you need native S3/GoogleDrive/OneDrive support, rclone is the tool of choice.
• Single-process / scale — Current design favors correctness and simplicity; large-scale parallel distributed copying may require additional engineering (e.g., a distributed worker model).
• Delta-copy library replacement — Fylex does not implement network delta transfers (it focuses on local efficiency and correctness).

Safety Nets: Reliability First

Fylex is built with data safety as priority:

• Undo: Rollback the last copy/move operation (removes created files, restores moved ones).
• Redo: Replay an operation exactly as before.
• Backups: Replaced/conflicting files are moved into fylex.deprecated/ (per process).
• JSON logs: Every operation is journaled (json/{process_id}.json + .jsonl).
• Verification: Optional hash verification ensures copy integrity.
• Retries: Up to 5 retries (MAX_RETRIES) on hash mismatch.
• Protections: Prevents overwriting itself, copying into subdirectories, or backup loops.

Installation

pip install fylex

Requires Python 3.8+. Linux recommended (for full xattr/ACL support).

Command Line Usage

Copy files

fylex copy ~/Downloads ~/Backup --resolve rename --algo xxhash --verify --verbose

• Smartly copies, resolves conflicts by renaming, verifies integrity via hash.
• Deprecated/replaced files stored under ~/Backup/fylex.deprecated/PROCESS_ID/.

Move files

fylex move ./data ./archive --resolve newer --match-glob "*.csv"

• Moves only .csv files, replacing only if source is newer.

Undo

fylex undo 1002

• Undoes process with ID 1002.

Redo

fylex redo 1002

• Replays process with ID 1002.

Python API Usage

from fylex import filecopy, filemove, undo, redo

# Copy with filters & conflict resolution
filecopy(
    src="~/Downloads",
    dest="~/Backup",
    resolve="rename",
    algo="xxhash",
    verify=True,
    match_glob="*.jpg",
    verbose=True
)

# Move and preserve metadata
filemove("project/", "archive/", resolve="newer", preserve_meta=True)

# Undo / Redo
undo("1002")
redo("1002")

Function Reference

filecopy(src, dest, ...)

Description: Smartly copies files from src to dest with conflict handling, filters, and safety nets.

Param	Type	Default	Description
src	str|Path	required	Source file or directory
dest	str|Path	required	Destination directory
resolve	str	"rename"	Conflict strategy: rename, replace, skip, larger, smaller, newer, older, prompt
algo	str	"xxhash"	Hash algorithm: xxhash, blake3, md5, sha256, sha512
chunk_size	int	16 * 1024 * 1024	Buffer size (bytes) for reading files
verbose	bool	True	Log operations to stdout
dry_run	bool	False	Simulate actions without making changes
summary	str|Path	None	Path to copy fylex.log summary
match_regex	str	None	Regex pattern to include files
match_names	list[str]	None	Exact filenames to include
match_glob	list[str]	None	Glob patterns to include
exclude_regex	str	None	Regex pattern to exclude files
exclude_names	list[str]	None	Exact filenames to exclude
exclude_glob	list[str]	None	Glob patterns to exclude
recursive_check	bool	False	Deduplication check recursively in dest
verify	bool	False	Verify file hashes after copying
has_extension	bool	False	Include file extension in deduplication check
no_create	bool	False	Do not create dest if it does not exist
preserve_meta	bool	True	Preserve timestamps, permissions, xattrs, ACLs
backup	str|Path	"fylex.deprecated"	Folder for deprecated or conflicting files
recurse	bool	False	Traverse subdirectories in src

Example:

filecopy("photos", "photos_backup", resolve="newer", match_glob="*.png", verify=True)

filemove(src, dest, ...)

Same params as filecopy, but moves files instead. If conflicts exist, originals are moved into deprecated folders within src or dest depending on the origin of the file being deprecated.

undo(p_id, verbose=True, force=False)

Rollback a process by ID.

Param	Type	Description
p_id	str	Process ID (JSON log ID)
verbose	bool	Enable logs
force	bool	Continue undo even if some entries fail
summary	str | Path	Path to copy fylex.log summary

redo(p_id, verbose=True, force=False)

Replay a process by ID. Same parameters as undo.

Example Use Cases

• Daily backup with safety nets

  fylex copy ~/work ~/backup --resolve newer --verify --summary=backup.log
  

• Selective move

  fylex move ./data ./archive --match-regex ".*2025.*\.csv"
  

• Quick rollback

  fylex undo 1021
  

• Replay last operation for reproducibility

  fylex redo 1021
  

Internals

• Hash cache: Stored in file_cache.db (SQLite). All file hashes computed with the chosen algorithm (xxhash, blake3, etc.) are cached here to avoid redundant computation.

• Logging: JSON logs are stored in the json/ folder in both .jsonl (line-delimited JSON) and .json formats. These logs track all file operations, conflicts, renames, and metadata preservation.

• Temporary files: During copy or move operations, temporary files are written to dest/fylex.tmp/ to ensure atomic writes. Only after successful completion are files renamed to their final names.

• Backups / Deprecated files: Conflicting or replaced files are moved to fylex.deprecated/{process_id}/.

  • If the source file is within src, the backup is created at src/fylex.deprecated/{process_id}/.
  • If the source file is within dest, the backup goes to dest/fylex.deprecated/{process_id}/.

• Move operations: Fylex implements move by first copying the file to the destination, verifying the integrity of the copy (via hash comparison if verify=True), and then deleting the source file. This ensures safety and prevents data loss.

• Conflict resolution: Handled according to the resolve strategy (rename, replace, skip, larger, smaller, newer, older, prompt). Conflicting files are either renamed, replaced, skipped, or backed up depending on the configuration.

• Metadata preservation: File timestamps, permissions, extended attributes (xattrs), and ACLs are preserved if preserve_meta=True.

• Recursive deduplication: When recursive_check=True, Fylex scans the destination folder to detect duplicates using cached hashes. Optional has_extension filtering allows deduplication by considering file extensions.

• Filters: Files can be included or excluded using regex patterns (match_regex, exclude_regex), exact names (match_names, exclude_names), or glob patterns (match_glob, exclude_glob).

• Atomicity and safety: Temporary files, hash verification, and backup mechanisms make all operations safe against crashes or interruptions.

License

MIT © 2025 Sivaprasad Murali

---

✨ With Fylex, file management on Linux is no longer just copying and moving — it’s safe, verifiable, reversible, and smart.

---