Walk file systems and collect stats
Project description
Walk file systems and collect stats.
Summary
Statwalker is a command-line program that scan files recursively (normally called as “walk”) and collects stats, basically file names and metadata (inode information in Linux systems). It runs in parallel in a single machine, and the output is a comma-separated file (csv), one line per file. These results can be analysed using other tools (see below).
Output
The csv file will look like this:
INODE,ATIME,MTIME,UID,GID,MODE,SIZE,DISK,PATH
Colum description:
INODE: device identifier and inode (Linux)
ATIME: last access time in unix format (seconds since epoc)
MTIME: last modified time in unix fromat
UID: user ID
GID: group ID
MODE: mode, which is file type and permissions
SIZE: real size in bytes, same value reported with command du -b
DISK: disk usage, which is number of blocks times 512
PATH: full path
How it works
Collecting stats is as simple as this one-liner in bash:
$ TODO
There are many tools doing the same thing, the problem is performance. After trying some tools in a file system with many terabytes of data and millions of files, the problem became untractable. I run statwalker in a storage with 100+ millions of files, with a reading rate over NFS folders of 3000 files/second on average, and much faster if disks are local.
Installation
Use pip:
$ pip install statwalker
Usage
# run it from the command line to see available parameters:
$ statwalker -h
# run it with options
$ statwalker -o output.csv /home
Contribute
Clone the github repository:
$ git clone https://github.com/sganis/statwalker.git
TODO
Add documentation with analysis tools, resolution, agregation, benchmark with c++ and mpi versions.
Add notes for windows users
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.