Walk file systems and collect stats
Walk file systems and collect stats.
Statwalker is a command-line program that scan files recursively (normally called as “walk”) and collects stats, basically file names and metadata (inode information in Linux systems). It runs in parallel in a single machine, and the output is a comma-separated file (csv), one line per file. These results can be analysed using other tools (see below).
The csv file will look like this:
- INODE: device identifier and inode (Linux)
- ATIME: last access time in unix format (seconds since epoc)
- MTIME: last modified time in unix fromat
- UID: user ID
- GID: group ID
- MODE: mode, which is file type and permissions
- SIZE: real size in bytes, same value reported with command du -b
- DISK: disk usage, which is number of blocks times 512
- PATH: full path
Collecting stats is as simple as this one-liner in bash:
There are many tools doing the same thing, the problem is performance. After trying some tools in a file system with many terabytes of data and millions of files, the problem became untractable. I run statwalker in a storage with 100+ millions of files, with a reading rate over NFS folders of 3000 files/second on average, and much faster if disks are local.
$ pip install statwalker
# run it from the command line to see available parameters: $ statwalker -h # run it with options $ statwalker -o output.csv /home
Clone the github repository:
$ git clone https://github.com/sganis/statwalker.git
- Add documentation with analysis tools, resolution, agregation, benchmark with c++ and mpi versions.
- Add notes for windows users