This release is a pre-release and may not be stable for production use.
AFF4 -The Advanced Forensics File Format
The Advanced Forensics File Format 4 (AFF4) is an open source format used for the storage of digital evidence and data.
It was originally designed and published in [1] and has since been standardised as the AFF4 Standard v1.0, which is available at https://github.com/aff4/Standard.
The focus of this implementation is:
- Standards compliance with AFF4 Standard v1.0 (for physical images)
- Ongoing development of the AFF4-L Standard v1.0 (for logical images)
Canonical images for the v1.0 physical image specification are provided in the AFF4 Reference Images github project at https://github.com/aff4/ReferenceImages
What is currently supported.
- Reading, writing & appending to ZipFile style volumes.
- Reading striped ZipFile volumes.
- Reading & writing AFF4 ImageStreams using the deflate or snappy compressor.
- Reading RDF metadata using Turtle and HDT (see below).
- Verification of linear and block hashed images.
- Reading & writing logical images.
- Reading & writing deduplicated logical images.
- Encrypted AFF4 logical volumes.
RDF metadata and HDT
AFF4 metadata is RDF. Containers always store it as Turtle and optionally as
Header Dictionary Triples (information.hdt), a compressed, queryable binary form which supports fast
opening of large containers.
- Nothing is unpacked: the dictionary, bitmaps and packed sequences of the HDT strucure are read in
place. An optional
.opidxsidecar (as the Java tools write, and as a container may hold asinformation.hdt.opidx) saves building the object index. - Wheels are
cp312-abi3for Linux (manylinux2014, x86_64 and aarch64), macOS 11+ (x86_64 and arm64) and Windows (x86_64 and arm64). Other platforms get thepy312-none-anywheel, which runs the HDT kernels interpreted and is about 100× slower on HDT queries.
pyaff4 bundles two packages for HDT:
rdflib_hdt_ng: a read-onlyrdflib.Storeover HDT bytes, andgraph_to_hdtto write an rdflib graph as HDT.pyhdt2_cython: the HDT reader and writer behind it, in Python and NumPy with the hot loops compiled by Cython.
UTF-8 extended HDT format
HDT v1 relies on NULL terminators for determining the end of strings (and hence length), which fails for valid UTF-8 strings containing the Unicode NUL code point.
This implementation reads and writes HDT serializations which include the full range of UTF-8, by defining a new DictionarySection type. pyaff4 writes every dictionary section in this type, and reads it alongside the standard type 2. hdt-cpp 1.3.3 and hdt-java 2.1.2 cannot read type 8, so they cannot read the HDT files pyaff4 writes. With reference section 4 of "HDT Binary Format" (2015) [1], the newly defined PlainFrontCodingUtf8 = 8 dictionary section reuses the design of the existing PlainFrontCoding DictionarySection, instead employing a length-prepended string encoding, per the following structure:
<Buffer of Blocks>
<VByte size of following string in bytes>
<First String of Block>
<VByte count of Common prefix characters> <VByte size of following string in bytes> <Suffix String>
… as many times as necessary to fit "BlockSize" strings.
<CRC32 of Buffer of Blocks>
The included test file "hdtv1-failing-utf8.hdt" contains two triples constructed from such strings which demonstrate the limitation, while the test file "hdt-ng-utf8.hdt" demonstrates the proposed new UTF-8 capable variant of format.
Literal text is stored raw, as hdt-java and hdt-cpp store it: the characters between a literal's quotes are stored as they are, without N-Triples escapes, and a reader finds the end of the text at the last ".
Porting from v0.x
pyaff4 2.x needs Python 3.12 or later. While 2.x is in pre-release, install it
with pip install --pre pyaff4; without --pre, pip installs 0.x. The
command-line tool is the aff4 entry point. Code written against 0.x needs
these changes:
| 0.x | 2.x |
|---|---|
rdfvalue.XSDString, XSDInteger, XSDDateTime, … |
rdflib.Literal(value, datatype=XSD.…) |
rdfvalue.URN("aff4://…") |
rdfvalue.ARN("aff4://…"), an rdflib.URIRef. URN() takes no arguments; use URN.FromFileName(path) for files |
Lexicon constants are strings, wrapped in rdfvalue.URN(...) |
Lexicon constants are URIRefs; pass them directly |
hashes.newImmutableHash(hexdigest, hash_type) |
rdflib.Literal(hexdigest, datatype=hash_type) |
pyaff4.utils (SmartUnicode, SmartStr, …) |
Removed; use str() and bytes |
Container.createURN returns a hash-based container unless zip_based=True |
It returns a logical container; pass hashbased=True for a hash-based one |
createURN(compression_method=…) and writeLogicalStream(…, allow_large_zipsegments=True) |
createURN(zipstream_compression_method=…, always_write_zipstreams=True), set once per container |
Metadata in information.turtle only |
Containers also hold information.hdt and its .opidx index, which readers prefer; encrypted containers keep Turtle only |
| The local header of a zip member has a Zip64 field only when needed | It always has one, so each member is 20 bytes larger than 0.x wrote it |
Google Cloud Storage backend (aff4_cloud), aes_keywrap module |
Removed; key wrapping uses cryptography's aes_key_wrap |
fastchunking for hash-based containers; snappy optional |
fastcdc; python-snappy is a required dependency |
Development
uv sync installs pyaff4 editable and compiles the HDT kernels. The tests also
read the AFF4 reference images from the reference_images submodule: clone with
--recurse-submodules, or run git submodule update --init.
just test # all tests, then the HDT engine's tests with interpreted kernels
just build # sdist and a wheel for this host
just release-build # cross-build, audit and install-test every release wheel (Linux)
python cicd/lint_kernels.py # no CPython API calls in the compiled hot loops
References
[1] "Extending the advanced forensic format to accommodate multiple data sources, logical evidence, arbitrary information and forensic workflow" M.I. Cohen, Simson Garfinkel and Bradley Schatz, digital investigation 6 (2009) S57-S68.
Metadata
Release files for pyaff4 2.0.0a20
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pyaff4-2.0.0a20.tar.gz | 392.1 kB | With provenance | Details for pyaff4-2.0.0a20.tar.gz |
Built distributions (wheels)
Total release size: 3.0 MB
Release files / pyaff4-2.0.0a20.tar.gz
| Download URL | pyaff4-2.0.0a20.tar.gz |
|---|---|
| Size | 392.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ae550c8eb0c9d141fafbaaf74c28b1508ef289f5bb5ee98f8699d0eaf10d0e61
|
|
BLAKE2b-256 checksum How to use checksums |
4b712d887338155872f296cea3ba519ce39b4c519a313fee303f4f288c26b816
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / pyaff4-2.0.0a20-py312-none-any.whl
| Download URL | pyaff4-2.0.0a20-py312-none-any.whl |
|---|---|
| Size | 142.5 kB |
| Tags | Python 3.12 |
|
SHA-256 checksum How to use checksums |
d8c85b00169b136e456655d00f7c6c69c067f22775d5e59548885e04ed2826c9
|
|
BLAKE2b-256 checksum How to use checksums |
1a614d850d02397392cdab91d30281883a4e6d62d6e04120184b665fd0949096
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / pyaff4-2.0.0a20-cp312-abi3-win_arm64.whl
| Download URL | pyaff4-2.0.0a20-cp312-abi3-win_arm64.whl |
|---|---|
| Size | 325.1 kB |
| Tags | CPython 3.12 Windows ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
0543b8c662c5c2c781785e8a890059bb3c13677e354a30e1ca6df70acb186a90
|
|
BLAKE2b-256 checksum How to use checksums |
11265bfa8e31badcb76e17c2ac3f4be1906c7e2648c74307df242219dadc083f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / pyaff4-2.0.0a20-cp312-abi3-win_amd64.whl
| Download URL | pyaff4-2.0.0a20-cp312-abi3-win_amd64.whl |
|---|---|
| Size | 341.4 kB |
| Tags | CPython 3.12 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
e1242357c6028857b5cd32ac03e3e6e9b1b29a6d08d32c9e1ad3570f96eaf37f
|
|
BLAKE2b-256 checksum How to use checksums |
718ff9fc547d07f5eae747516a9b11fd4484678163584a3cbf1b988030049135
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 624.8 kB |
| Tags | CPython 3.12 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
47e281cf8f70310be2a2df44ee501f1763200683c00b920f23f5cc17d5346e80
|
|
BLAKE2b-256 checksum How to use checksums |
b3e9186f108f0d2ecc1dd0f82f3c45a0f1ea0feaad0a2906d160e73dd670a083
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 607.3 kB |
| Tags | CPython 3.12 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
8749330ae26daa671f86f1e0960ab6a6dc19b06d3d8556f977ad3ea2be0fd0bb
|
|
BLAKE2b-256 checksum How to use checksums |
7cdc2499f3954c2f4dd9944a4472b5a7b1fe83c93ad8ee0353ac2b3a2aca877f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_x86_64.whl
| Download URL | pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_x86_64.whl |
|---|---|
| Size | 265.5 kB |
| Tags | CPython 3.12 abi3 macOS 11.0+ x86-64 |
|
SHA-256 checksum How to use checksums |
aab8f11466b59dabc48bc48ffc518ea06e74ae77b53ef33b7105d2a09d91b6df
|
|
BLAKE2b-256 checksum How to use checksums |
27ab1072f5e004fcd2dd11bde8b839ae12098a8ec4904788460f22f0438c417c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_arm64.whl
| Download URL | pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 264.1 kB |
| Tags | CPython 3.12 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
b98541a8db4736e9b74a506b20d5d2f896d5e2263092bde44efa0759c781a94d
|
|
BLAKE2b-256 checksum How to use checksums |
fcb9c37081b3d7bd4dac9ae943743f292a20b3637d2bb0702edc1dd18eb68345
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logSecurity for pyaff4 2.0.0a20
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 8 files in this release have been uploaded with provenance metadata.
Attestation for pyaff4-2.0.0a20.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.