A flexible tool for traversing, filtering, and processing file trees
Project description
Treeverse
Treeverse is a Python tool for traversing, processing, and accumulating data from file trees using custom code.
Installation
pip install treeverse
Usage
Treeverse operates in three main steps:
- Build: Traverse the directory and build the file tree
- Process: Apply custom processing to each node
- Accumulate: Aggregate data from child nodes to parent nodes
1. Build the tree
treeverse build -p /path/to/your/project -e py -e yaml \
-c your_module:your_filter_function -o tree.yaml
Options:
-p, --path
: Specify the path to traverse (default is current directory)-e, --extensions
: Specify file extensions to include (can be used multiple times)-c, --callback
: Path to filter callback functions (can be used multiple times)-o, --output
: Output file (default: tree.yaml)--write-to-stream
: Write output to stdout instead of a file
2. Process the tree
treeverse process -i tree.yaml -c your_module:your_processing_function -o processed_tree.yaml
Options:
-i, --input
: Input file (default: tree.yaml)-c, --callback
: Path to the processing function-o, --output
: Output file (default: processed_tree.yaml)--read-from-stream
: Read input from stdin instead of a file--write-to-stream
: Write output to stdout instead of a file
3. Accumulate results
treeverse accumulate -i processed_tree.yaml -c your_module:your_accumulation_function -o final_result.yaml
Options:
-i, --input
: Input file (default: processed_tree.yaml)-c, --callback
: Path to the accumulation function-o, --output
: Output file (default: accumulated_tree.yaml)--read-from-stream
: Read input from stdin instead of a file--write-to-stream
: Write output to stdout instead of a file
Example Use Case
Process all Python and YAML files in a project, analyze them, and accumulate results:
treeverse build -p ~/path/to/your/project -e py -e yaml -o tree.yaml
treeverse process -i tree.yaml -c ~/path/to/your/tools.py:analyze_file -o processed_tree.yaml
treeverse accumulate -i processed_tree.yaml -c ~/path/to/your/tools.py:accumulate_results -o final_analysis.yaml
To use streaming for the entire pipeline:
treeverse build -p ~/path/to/your/project -e py -e yaml --write-to-stream | \
treeverse process -c ~/path/to/your/tools.py:analyze_file --read-from-stream --write-to-stream | \
treeverse accumulate -c ~/path/to/your/tools.py:accumulate_results --read-from-stream > final_analysis.yaml
In this example, tools.py
would contain custom analyze_file
and accumulate_results
functions.
Reference Implementations
For examples of callable functions that can be used with Treeverse, check the simple_callbacks.py
file in the project repository. This file contains reference implementations for filtering, processing, and accumulation functions.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for treeverse-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00271fa13953c99ae49240dfe0196238c5c42edd359eaa77c36e519519fd5b9f |
|
MD5 | 2853e79d8d82994565110b3b8879dc87 |
|
BLAKE2b-256 | 52e0b7bd55f3e48274177cc15e85331815c327de87c9416c1ca3123d5a9c7ec7 |