Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Faster recursive directory walk for Python.

Project description

<meta name='keywords' content='walk, os.walk, python, faster, speed, benchmark, rapido, velocidad, optimizacion, cython, pypy, cpython, docker, json, ndjson, rapido, veloz, performance, critical, compiled, module, modulo, loc, minimalismo, minimalism, simple, small, tiny, argentina, spanish, compare, mejora'>

Faster-than-Walk

screenshot

Library Speed
os.walk (std lib) 225.98
faster_than_walk 70.0
  • Benchmarks run on Docker from Dockerfile on this repo.
  • Speed is IRL time to complete 10000 full recursive directory walks to /usr/share/.
  • Stats as of year 2019.
  • x86_64 64Bit AMD, SSD, Arch Linux.

Use

import faster_than_walk as ftw

print(ftw.walks("/path/to/some/folder"))                   # From path string
print(ftw.walks("/path/to/some/folder", [".py", ".txt"]))  # Filter by Extension
print(ftw.walks("/path/to/some/folder", [".py"], followlinks=True))  # Follow SymLinks
print(ftw.walks("/path/to/some/folder", [".py"], followlinks=True, yieldfiles=False)) # Folders only
print(ftw.walks_glob("*.txt"))           # Standard POSIX Globs
print(ftw.walks_glob("**/*.txt"))        # Recursive standard POSIX Globs

walks()

Description: Takes a path string, makes a full recursive walk of folders and returns a list with the result. Recursive directory walk from path to list of strings.

Arguments:

  • folderpath A local path, str type, required, must not be empty string.
  • extensions list of file Extensions, list type, optional, list must contain str, must not be empty list.
  • followlinks Set to True to follow SymLinks, bool type, optional.
  • yieldfiles Set to False to return folders only, bool type, optional.
  • debugs Set to True for Debug mode, bool type, optional, not needed for normal usage.

Returns: Result, list type, values of the list are str type.

walks_glob()

Description: Takes a path string, makes a full recursive walk of folders and returns a list with the result. Recursive directory walk from standard POSIX like Glob Pattern string to list of strings, allows Recursive and Non-Recursive Globs.

Arguments:

  • globpattern A standard POSIX like Glob Pattern string, str type, required, must not be empty string.

Returns: Result, list type, values of the list are str type.

For more Examples check the Examples.

Instead of having a pair of functions with a lot of arguments that you should provide to make it work, we have tiny functions with very few arguments that do one thing and do it as fast as possible.

Install

  • pip install faster_than_walk

Dependencies

  • None

Docker

  • Make a quick test drive on Docker!.
$ ./build-docker.sh
$ ./run-docker.sh
$ ./run-benchmark.sh  # Inside Docker.

Platforms

  • ✅ Linux
  • ✅ Windows
  • ✅ Mac
  • ✅ Android
  • ✅ Raspberry Pi
  • ✅ BSD

Extras

More Faster Libraries...

Requisites

  • Python 3.
  • GCC.
  • 64 Bit.

FAQ

  • Whats the idea, inspiration, reason, etc ?.

Feel free to Fork, Clone, Download, Improve, Reimplement, Play with this Open Source. Make it 10 times faster, 10 times smaller.

  • This requires Cython ?.

No.

  • This runs on PyPy ?.

No.

  • This runs on Python2 ?.

I dunno. (Not supported)

  • This runs on 32Bit ?.

No.

  • This runs with Clang ?.

No.

  • How can I Install it ?.

pip install faster_than_walk

  • How can be faster than os.walk ?.

I dunno.

  • How to Filter by File Extensions ?.

extensions = [".py", ".txt", ".tar.gz"]

(Glob Pattern may be faster for several file extensions than the extensions argument)

  • How to force Follow SymLinks ?.

followlinks = True

  • How to force return Folders only and Not Files ?.

yieldfiles = False

  • How to enable Debug ?.

debugs = True

  • How to use a Glob Pattern ?.

globpattern = "*.txt"

  • How to use a Recursive Glob Pattern ?.

globpattern = "**/*.txt"

  • I need to import glob on my code to use a Glob Pattern ?.

No.

  • Whats a Glob Pattern ?.

https://en.wikipedia.org/wiki/Glob_%28programming%29

  • Why needs 64Bit ?.

Maybe it works on 32Bit, but is not supported, integer sizes are too small, and performance can be worse.

  • Why needs Python 3 ?.

Maybe it works on Python 2, but is not supported, and performance can be worse, we suggest to migrate to Python3.

  • Can I wrap the functions on a try: except: block ?.

Functions do not have internal try: except: blocks, so you can wrap them inside try: except: blocks if you need very resilient code.

  • PIP fails to install or fails build the wheel ?.

Add at the end of the PIP install command:

--isolated --disable-pip-version-check --no-cache-dir --no-binary :all:

Not my Bug.

  • How to Build the project ?.

build.sh

  • How to Package the project ?.

package.sh

  • This requires Nimble ?.

No.

  • Whats the unit of measurement for speed ?.

Unmmodified raw output of Python timeit module.

Please send Pull Request to Python to improve the output of timeit.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for faster-than-walk, version 0.5
Filename, size File type Python version Upload date Hashes
Filename, size faster_than_walk-0.5.zip (115.7 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page