Recursively Walk Into Directories and Archives

This module primarily provides the function unzipwalk(), which recursively walks into directories and compressed files and returns all files, directories, etc. found, together with binary file handles (file objects) for reading the files. Currently supported are zip, gz, bz2, xz, and the various tar compressed formats tar.gz/tgz, tar.xz/txz, and tar.bz2/tbz/tbz2, plus 7z files if the Python package py7zr is installed (to get the latter, you can install this package with pip install unzipwalk[7z]). File types are detected based on the aforementioned extensions.

>>> from unzipwalk import unzipwalk
>>> results :list[tuple[str, ...]] = []
>>> for result in unzipwalk('.'):
...     names = tuple( name.as_posix() for name in result.names )
...     if result.hnd:  # result is a file opened for reading (binary)
...         # could use result.hnd.read() here, or for line-by-line:
...         for line in result.hnd:
...             pass  # do something interesting with the data here
...     results.append(names + (result.typ.name,))
>>> print(sorted(results))
[('bar.zip', 'ARCHIVE'),
 ('bar.zip', 'bar.txt', 'FILE'),
 ('bar.zip', 'test.tar.gz', 'ARCHIVE'),
 ('bar.zip', 'test.tar.gz', 'hello.csv', 'FILE'),
 ('bar.zip', 'test.tar.gz', 'test', 'DIR'),
 ('bar.zip', 'test.tar.gz', 'test/cool.txt.gz', 'ARCHIVE'),
 ('bar.zip', 'test.tar.gz', 'test/cool.txt.gz', 'test/cool.txt', 'FILE'),
 ('foo.txt', 'FILE')]

Note that unzipwalk() automatically closes files as it goes from file to file. This means that you must use the handles as soon as you get them from the generator - something as seemingly simple as sorted(unzipwalk('.')) would cause the code above to fail, because all files will have been opened and closed during the call to sorted() and the handles to read the data would no longer be available in the body of the loop. This is why the above example first processes all the files before sorting the results. You can also use recursive_open() with result.raw_names to open the files later, though using that function is less efficient that unzipwalk() if you are opening multiple files inside of Zip or tar archives.

The yielded file handles can be wrapped in io.TextIOWrapper to read them as text files. For example, to read all CSV files in the current directory and below, including within compressed files:

>>> from unzipwalk import unzipwalk, FileType
>>> from io import TextIOWrapper
>>> import csv
>>> for result in unzipwalk('.'):
...     if result.typ==FileType.FILE and result.names[-1].suffix.lower()=='.csv':
...         assert result.hnd is not None
...         print([ name.as_posix() for name in result.names ])
...         with TextIOWrapper(result.hnd, encoding='UTF-8', newline='') as handle:
...             csv_rd = csv.reader(handle, strict=True)
...             for row in csv_rd:
...                 print(repr(row))
['bar.zip', 'test.tar.gz', 'hello.csv']
['Id', 'Name', 'Address']
['42', 'Hello', 'World']

NOTE

The original names of files compressed with gzip, bzip2, and lzma are derived by simply removing the respective .gz, .bz2, or .xz extensions. If the filename consists only of the extension (for example .gz), the derived basename is noname, preserving any directory prefix.

Using the original filename from the gzip file’s header is currently not possible due to limitations in the underlying library.

SEE ALSO

• zipfile Issues
• tarfile Issues
• Compression issues (gzip, bzip2, lzma)
• py7zr Issues

API

unzipwalk.unzipwalk(paths: str | PathLike[str] | bytes | Iterable[str | PathLike[str] | bytes], *, matcher: Callable[[Sequence[str]], bool] | None = None, raise_errors: bool = True) → Generator[UnzipWalkResult, None, None]

This generator recursively walks into directories and compressed files and yields named tuples of type UnzipWalkResult.

• Parameters:

  • paths – A filename or iterable of filenames. Symbolic links are reported as FileType.SYMLINK and are not followed, including when supplied directly as input paths.

  • matcher –

    When you provide this optional argument, it must be a callable that accepts a sequence of filename strings corresponding to raw_names as its only argument, and returns a boolean value whether this filename should be further processed or not. If a file is skipped, a UnzipWalkResult of type FileType.SKIP is yielded. Matching takes place before filesystem metadata is accessed, so excluded input paths can also be missing or inaccessible.

    Be aware that within Zip and tar archives, all files are basically a flat list, so if your matcher excludes a directory inside an archive, it must also exclude all files within that directory as well. This behavior is different for physical directories in the file system: if you exclude a directory there, it will not be descended into, so you won’t have to exclude the files inside (though it’s good practice to write your matcher to exclude them anyway; for example, for physical paths, you can construct a Path and use is_relative_to()).

  • raise_errors – When this is turned on (the default), any errors are raised immediately, aborting the iteration. If this is turned off, when file access, directory traversal, or decompression errors occur, a UnzipWalkResult of type FileType.ERROR is yielded for those files instead.

NOTE

If py7zr is not installed, those archives will not be descended into.

NOTE

Do not rely on the order of results! But see also the discussion in the main documentation about why e.g. sorted(unzipwalk(...)) automatically closes files and so may not be what you want.

NOTE

Archive entries with identical names are yielded separately with their own file handles. recursive_open() cannot distinguish these occurrences after iteration, even when given raw_names.

• Raises: Exception – Because of the various underlying libraries, both this function and recursive_open() can raise a variety of exceptions: zipfile.BadZipFile, tarfile.TarError, py7zr.exceptions.ArchiveError and its subclasses like py7zr.Bad7zFile, gzip.BadGzipFile, zlib.error, lzma.LZMAError, EOFError, various OSErrors, and other exceptions may be possible. Therefore, you may need to catch all Exceptions to play it safe.

IMPORTANT

Errors from gzip, bz2, and lzma (.gz, .bz2, and .xz files, respectively) may not be raised until the file is actually read, so you’ll probably also want to add an exception handler around your read() call!

class unzipwalk.UnzipWalkResult(names: tuple[PurePath, ...], raw_names: tuple[str, ...], typ: FileType, hnd: IO[bytes] | None = None, size: int | None = None)

Return type for unzipwalk().

names : tuple[PurePath, ...]

A tuple of the filename(s) as pathlib objects. The first element is always the physical file in the file system. If the tuple has more than one element, then the yielded file is contained in a compressed file, possibly nested in other compressed file(s), and the last element of the tuple will contain the file’s normalized name. Note that Path objects normalize filenames, for example by removing ./ prefixes and repeated separators, and you can use raw_names to access the exact archive member names, for example for use in recursive_open().

raw_names : tuple[str, ...]

The filename sequence as strings, preserving archive member names exactly as reported by the archive library (though for gzip, bzip2, and lzma files, the extension is removed and an extension-only basename becomes noname). Pass this sequence to recursive_open() to avoid path normalization. This field must have the same number of elements as names.

typ : FileType

A FileType value representing the type of the current file.

hnd : IO[bytes] | None

When typ is FileType.FILE, this is a file handle (file object) for reading the file contents in binary mode, validated at runtime against ReadOnlyBinary. Otherwise, this is None. If this object was produced by from_checksum_line(), this handle will read the checksum of the data, not the data itself!

size : int | None

When typ is FileType.FILE or FileType.ARCHIVE, this field may hold the size of the file, if the compression format and library support knowing the compressed file’s size in advance. Otherwise, this is None.

validate() → UnzipWalkResult

Validate whether the object’s fields are set properly and throw errors if not.

Intended for internal use, mainly when type checkers are not being used. unzipwalk() validates all the results it returns.

• Returns: The object itself, for method chaining.
• Raises: ValueError, TypeError – If the object is invalid.

checksum_line(hash_algo: str, *, raise_errors: bool = True) → str

Encodes this object into a line of text suitable for use as a checksum line.

Intended mostly for internal use by the --checksum CLI option. See from_checksum_line() for the inverse operation. Uses raw_names to preserve the exact spelling of archive member names.

WARNING

Requires that the file handle be open (for files), and will read from it to generate the checksum!

• Parameters: hash_algo – The hashing algorithm to use, as recognized by hashlib.new().
• Returns: The checksum line, without trailing newline.

classmethod from_checksum_line(line: str, *, windows: bool = False) → UnzipWalkResult | None

Decodes a checksum line as produced by checksum_line().

Intended as a utility function for use when reading files produced by the --checksum CLI option. The filename strings are preserved in raw_names, alongside the normalized path objects in names.

WARNING

The hnd of the returned object will not be a handle to the data from the file, instead it will be a handle to read the checksum of the file! (You could use recursive_open() to open the files themselves.)

• Parameters:
  • line – The line to parse, optionally ending with LF or CRLF on any platform.
  • windows – Whether the physical pathname (and gzip, bzip2, or lzma paths derived from it) are in Windows format. Defaults to the current platform. Archive member names always use POSIX path objects, including any files nested inside archives.
• Returns: The UnzipWalkResult object, or None for empty or comment lines.
• Raises: ValueError – If the line could not be parsed.

class unzipwalk.FileType(value)

Used in UnzipWalkResult to indicate the type of the file.

WARNING

Don’t rely on the numeric value of the enum elements, they are automatically generated and may change!

FILE = 1

A regular file.

ARCHIVE = 2

An archive file, will be descended into.

DIR = 3

A directory.

SYMLINK = 4

A symbolic link.

OTHER = 5

Some other file type (e.g. FIFO).

SKIP = 6

A file was skipped due to the matcher filter.

ERROR = 7

An error was encountered with this file, when the raise_errors option is off.

unzipwalk.recursive_open(fns: Sequence[str | PathLike[str]], encoding: str | None = None, errors: str | None = None, newline: str | None = None) → Generator[IO[bytes] | TextIOWrapper, None, None]

This context manager allows opening files nested inside archives directly.

unzipwalk() automatically closes files as it iterates through directories and archives; this function exists to allow you to open the returned files after the iteration (using their raw_names). However, this function will be less efficient that unzipwalk() if you’re opening multiple files inside of Zip or tar archives.

NOTE

If any of encoding, errors, or newline is specified, the returned file is wrapped in io.TextIOWrapper!

NOTE

If the last file in the list of files is an archive file, then it won’t be decompressed, instead you’ll be able to read the archive’s raw compressed data from the handle.

NOTE

Strings used as archive member names are matched literally, including ./ prefixes and repeated separators. pathlib objects introduce filename normalization and may therefore refer to a different member. Windows path objects used as archive member names are converted to POSIX format. Other path-like objects preserve the exact spelling returned by os.fspath().

WARNING

If an archive contains multiple entries with exactly the same name, this function cannot select a particular occurrence. The underlying archive library determines which entry is returned, or may raise an error. Read each entry’s handle during unzipwalk() iteration to access its individual contents.

In this example, we open a gzip-compressed file, stored inside a tgz archive, which in turn is stored in a Zip file:

>>> from unzipwalk import recursive_open
>>> with recursive_open(('bar.zip', 'test.tar.gz', 'test/cool.txt.gz', 'test/cool.txt'), encoding='UTF-8') as fh:
...     print(fh.read())
Hi, I'm a compressed file!

• Raises:
  • ImportError – If you try to open a 7z file but py7zr is not installed.
  • Exception – See description in unzipwalk().

class unzipwalk.ReadOnlyBinary(*args, **kwargs)

Common readable interface for the file handles used in UnzipWalkResult.

This protocol is used for runtime validation. Returned handles are annotated as typing.IO with binary contents for compatibility with standard I/O utilities. The concrete stream depends on the compression format; attributes such as name are not available on every backend. Use UnzipWalkResult.raw_names for filenames.

close() → None

Close the file.

NOTE

unzipwalk() automatically closes files.

property closed : bool

readable() → bool

read(n: int = -1, /) → bytes

readline(limit: int = -1, /) → bytes

seekable() → bool

See io.IOBase.seekable().

WARNING

Some underlying classes may return True even in cases where seek() will fail! (e.g. GH python/cpython#77354)

seek(offset: int, whence: int = 0, /) → int

unzipwalk.ARCHIVE_RE = re.compile('\\.(?:tar(?:\\.gz|\\.bz2|\\.xz)?|tgz|txz|tbz2?|zip|7z|bz2|xz|gz)\\Z', re.IGNORECASE)

A regular expression that can be used to search against filenames which this module supports.

Command-Line Interface

usage: unzipwalk [-h] [-a] [-d | -c ALGO] [-e EXCLUDE] [-r] [-o OUTFILE]
                 [PATH ...]

Recursively walk into directories and archives

positional arguments:
  PATH                  paths to process (default is current directory)

options:
  -h, --help            show this help message and exit
  -a, --all-files       also list dirs, symlinks, etc.
  -d, --dump            also dump file contents
  -c ALGO, --checksum ALGO
                        generate a checksum for each file**
  -e EXCLUDE, --exclude EXCLUDE
                        filename globs to exclude*
  -r, --raise-errors    raise errors instead of reporting them in output
  -o OUTFILE, --outfile OUTFILE
                        output filename (must not already exist)

* Note --exclude matches the full final name in the sequence, including
directory components. For archive members, this is the path within the
innermost archive. For more control, use the library instead of this command-
line tool. ** Possible values for ALGO: blake2b, blake2s, md5, md5-sha1,
ripemd160, sha1, sha224, sha256, sha384, sha3_224, sha3_256, sha3_384,
sha3_512, sha512, sha512_224, sha512_256, sm3

The available checksum algorithms may vary depending on your system and Python version. Run the command with --help to see the list of algorithms available on your system.

Author, Copyright, and License

Copyright (c) 2022-2026 Hauke Dämpfling (haukex@zero-g.net) at the Leibniz Institute of Freshwater Ecology and Inland Fisheries (IGB), Berlin, Germany, https://www.igb-berlin.de/

This library is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see https://www.gnu.org/licenses/