iterfilesystem 0.2.0

Threaded directory iteration via os.scandir() with progress indicator and resume function.

IterFilesystem

Threaded directory iteration via os.scandir() with progress indicator and resume function.

among other things these packages are used:

• UniqueAckQ from persist-queue

• progress bar tqdm

Requirement:

• Python 3.6 or newer.

• Pipenv. Packages and virtual environment manager.

Please: try, fork and contribute! ;)

	travis-ci.org/jedie/IterFilesystem
	ci.appveyor.com/project/jedie/IterFilesystem
	codecov.io/gh/jedie/IterFilesystem
	coveralls.io/r/jedie/IterFilesystem
	requires.io/github/jedie/IterFilesystem/requirements/

Example

Use example CLI, e.g.:

~$ git clone https://github.com/jedie/IterFilesystem.git
~$ cd IterFilesystem
~/IterFilesystem$ pipenv install
~/IterFilesystem$ pipenv shell
(IterFilesystem) ~/IterFilesystem$ print_fs_stats --help
usage: print_fs_stats.py [-h] [-v] [--path PATH]
                         [--skip_dirs [SKIP_DIRS [SKIP_DIRS ...]]]
                         [--skip_filenames [SKIP_FILENAMES [SKIP_FILENAMES ...]]]
                         [--force_restart] [--complete_cleanup]

Scan filesystem and print some information

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  --path PATH           The file path that should be scanned e.g.: "~/foobar/"
                        default is "~"
  --skip_dirs [SKIP_DIRS [SKIP_DIRS ...]]
                        Directory names to exclude from scan.
  --skip_filenames [SKIP_FILENAMES [SKIP_FILENAMES ...]]
                        File names to ignore.
  --force_restart       Don't use resume data: Delete persist queue data
                        before start.
  --complete_cleanup    Delete persist queue after complete scan.

example output looks like this:

(IterFilesystem) ~/IterFilesystem$ $ print_fs_stats --path ~/IterFilesystem --skip_dirs .tox .pytest_cache
Read/process: '/home/foobar/IterFilesystem'...
Skip directories:
    * .tox
    * .pytest_cache

[INFO:  61 2019-09-15 12:37:17,973] Save persist queue to: /home/foobar/IterFilesystem/persisted_queue_4Aubn
[INFO:  37 2019-09-15 12:37:18,252] Skip dir: '.idea'
[INFO:  37 2019-09-15 12:37:18,252] Skip dir: '.tox'
[INFO:  37 2019-09-15 12:37:18,270] Skip dir: '.pytest_cache'
Read filesystem with 83 items in 0.3sec.
*** all items processed ***: 100%|████████████████████████████████████| 83/83 [00:00<00:00, 85.25it/s]

total filesystem items: 83
Finish in.............: 1.0sec.
File count.....: 73 with 10991860 Bytes
Directory count: 10
Other count....: 0

History

• dev - compare v0.2.0…master

  • TBC

• 15.09.2019 - compare v0.1.0…v0.2.0

  • store persist queue in temp directory

  • Don’t catch process_path_item errors, this should be made in child class

• 15.09.2019 - compare v0.0.1…v0.1.0

  • add some project meta files and tests

  • setup CI

  • fix tests

• 15.09.2019 - v0.0.1

  • first Release on PyPi

Links

• https://pypi.python.org/pypi/IterFilesystem/

• https://github.com/jedie/IterFilesystem/

Donating

• paypal.me/JensDiemer

• Flattr This!

• Send Bitcoins to 1823RZ5Md1Q2X5aSXRC5LRPcYdveCiVX6F