Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

AFF4 -The Advanced Forensics File Format

The Advanced Forensics File Format 4 (AFF4) is an open source format used for the storage of digital evidence and data.

It was originally designed and published in [1] and has since been standardised as the AFF4 Standard v1.0, which is available at https://github.com/aff4/Standard.

The focus of this implementation is:

  1. Standards compliance with AFF4 Standard v1.0 (for physical images)
  2. Ongoing development of the AFF4-L Standard v1.0 (for logical images)

Canonical images for the v1.0 physical image specification are provided in the AFF4 Reference Images github project at https://github.com/aff4/ReferenceImages

What is currently supported.

  1. Reading, writing & appending to ZipFile style volumes.
  2. Reading striped ZipFile volumes.
  3. Reading & writing AFF4 ImageStreams using the deflate or snappy compressor.
  4. Reading RDF metadata using Turtle and HDT (see below).
  5. Verification of linear and block hashed images.
  6. Reading & writing logical images.
  7. Reading & writing deduplicated logical images.
  8. Encrypted AFF4 logical volumes.

RDF metadata and HDT

AFF4 metadata is RDF. Containers always store it as Turtle and optionally as Header Dictionary Triples (information.hdt), a compressed, queryable binary form which supports fast opening of large containers.

  • Nothing is unpacked: the dictionary, bitmaps and packed sequences of the HDT strucure are read in place. An optional .opidx sidecar (as the Java tools write, and as a container may hold as information.hdt.opidx) saves building the object index.
  • Wheels are cp312-abi3 for Linux (manylinux2014, x86_64 and aarch64), macOS 11+ (x86_64 and arm64) and Windows (x86_64 and arm64). Other platforms get the py312-none-any wheel, which runs the HDT kernels interpreted and is about 100× slower on HDT queries.

pyaff4 bundles two packages for HDT:

  • rdflib_hdt_ng: a read-only rdflib.Store over HDT bytes, and graph_to_hdt to write an rdflib graph as HDT.
  • pyhdt2_cython: the HDT reader and writer behind it, in Python and NumPy with the hot loops compiled by Cython.

UTF-8 extended HDT format

HDT v1 relies on NULL terminators for determining the end of strings (and hence length), which fails for valid UTF-8 strings containing the Unicode NUL code point.

This implementation reads and writes HDT serializations which include the full range of UTF-8, by defining a new DictionarySection type. pyaff4 writes every dictionary section in this type, and reads it alongside the standard type 2. hdt-cpp 1.3.3 and hdt-java 2.1.2 cannot read type 8, so they cannot read the HDT files pyaff4 writes. With reference section 4 of "HDT Binary Format" (2015) [1], the newly defined PlainFrontCodingUtf8 = 8 dictionary section reuses the design of the existing PlainFrontCoding DictionarySection, instead employing a length-prepended string encoding, per the following structure:

<Buffer of Blocks>
 <VByte size of following string in bytes>
 <First String of Block>
 <VByte count of Common prefix characters> <VByte size of following string in bytes> <Suffix String>
 … as many times as necessary to fit "BlockSize" strings.
<CRC32 of Buffer of Blocks>

The included test file "hdtv1-failing-utf8.hdt" contains two triples constructed from such strings which demonstrate the limitation, while the test file "hdt-ng-utf8.hdt" demonstrates the proposed new UTF-8 capable variant of format.

Literal text is stored raw, as hdt-java and hdt-cpp store it: the characters between a literal's quotes are stored as they are, without N-Triples escapes, and a reader finds the end of the text at the last ".

Porting from v0.x

pyaff4 2.x needs Python 3.12 or later. While 2.x is in pre-release, install it with pip install --pre pyaff4; without --pre, pip installs 0.x. The command-line tool is the aff4 entry point. Code written against 0.x needs these changes:

0.x 2.x
rdfvalue.XSDString, XSDInteger, XSDDateTime, … rdflib.Literal(value, datatype=XSD.…)
rdfvalue.URN("aff4://…") rdfvalue.ARN("aff4://…"), an rdflib.URIRef. URN() takes no arguments; use URN.FromFileName(path) for files
Lexicon constants are strings, wrapped in rdfvalue.URN(...) Lexicon constants are URIRefs; pass them directly
hashes.newImmutableHash(hexdigest, hash_type) rdflib.Literal(hexdigest, datatype=hash_type)
pyaff4.utils (SmartUnicode, SmartStr, …) Removed; use str() and bytes
Container.createURN returns a hash-based container unless zip_based=True It returns a logical container; pass hashbased=True for a hash-based one
createURN(compression_method=…) and writeLogicalStream(…, allow_large_zipsegments=True) createURN(zipstream_compression_method=…, always_write_zipstreams=True), set once per container
Metadata in information.turtle only Containers also hold information.hdt and its .opidx index, which readers prefer; encrypted containers keep Turtle only
The local header of a zip member has a Zip64 field only when needed It always has one, so each member is 20 bytes larger than 0.x wrote it
Google Cloud Storage backend (aff4_cloud), aes_keywrap module Removed; key wrapping uses cryptography's aes_key_wrap
fastchunking for hash-based containers; snappy optional fastcdc; python-snappy is a required dependency

Development

uv sync installs pyaff4 editable and compiles the HDT kernels. The tests also read the AFF4 reference images from the reference_images submodule: clone with --recurse-submodules, or run git submodule update --init.

just test             # all tests, then the HDT engine's tests with interpreted kernels
just build            # sdist and a wheel for this host
just release-build    # cross-build, audit and install-test every release wheel (Linux)
python cicd/lint_kernels.py   # no CPython API calls in the compiled hot loops

References

[1] "Extending the advanced forensic format to accommodate multiple data sources, logical evidence, arbitrary information and forensic workflow" M.I. Cohen, Simson Garfinkel and Bradley Schatz, digital investigation 6 (2009) S57-S68.

Metadata

Release files for pyaff4 2.0.0a20

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyaff4 2.0.0a20
File Size Uploaded Provenance Actions
pyaff4-2.0.0a20.tar.gz 392.1 kB With provenance Details for pyaff4-2.0.0a20.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for pyaff4 2.0.0a20
File Actions
pyaff4-2.0.0a20-py312-none-any.whl Python 3.12 none any Details for pyaff4-2.0.0a20-py312-none-any.whl
pyaff4-2.0.0a20-cp312-abi3-win_arm64.whl CPython 3.12 abi3 Windows ARM64 Details for pyaff4-2.0.0a20-cp312-abi3-win_arm64.whl
pyaff4-2.0.0a20-cp312-abi3-win_amd64.whl CPython 3.12 abi3 Windows x86-64 Details for pyaff4-2.0.0a20-cp312-abi3-win_amd64.whl
pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.12 abi3 Linux glibc 2.17+ x86-64 Details for pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.12 abi3 Linux glibc 2.17+ ARM64 Details for pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_x86_64.whl CPython 3.12 abi3 macOS 11.0+ x86-64 Details for pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_x86_64.whl
pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_arm64.whl CPython 3.12 abi3 macOS 11.0+ ARM64 Details for pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_arm64.whl

Total release size: 3.0 MB

Release files / pyaff4-2.0.0a20.tar.gz

Download URL pyaff4-2.0.0a20.tar.gz
Size 392.1 kB
Tags Source
SHA-256 checksum
How to use checksums
ae550c8eb0c9d141fafbaaf74c28b1508ef289f5bb5ee98f8699d0eaf10d0e61
BLAKE2b-256 checksum
How to use checksums
4b712d887338155872f296cea3ba519ce39b4c519a313fee303f4f288c26b816
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a20-py312-none-any.whl

Download URL pyaff4-2.0.0a20-py312-none-any.whl
Size 142.5 kB
Tags Python 3.12
SHA-256 checksum
How to use checksums
d8c85b00169b136e456655d00f7c6c69c067f22775d5e59548885e04ed2826c9
BLAKE2b-256 checksum
How to use checksums
1a614d850d02397392cdab91d30281883a4e6d62d6e04120184b665fd0949096
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a20-cp312-abi3-win_arm64.whl

Download URL pyaff4-2.0.0a20-cp312-abi3-win_arm64.whl
Size 325.1 kB
Tags CPython 3.12 Windows ARM64 abi3
SHA-256 checksum
How to use checksums
0543b8c662c5c2c781785e8a890059bb3c13677e354a30e1ca6df70acb186a90
BLAKE2b-256 checksum
How to use checksums
11265bfa8e31badcb76e17c2ac3f4be1906c7e2648c74307df242219dadc083f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a20-cp312-abi3-win_amd64.whl

Download URL pyaff4-2.0.0a20-cp312-abi3-win_amd64.whl
Size 341.4 kB
Tags CPython 3.12 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
e1242357c6028857b5cd32ac03e3e6e9b1b29a6d08d32c9e1ad3570f96eaf37f
BLAKE2b-256 checksum
How to use checksums
718ff9fc547d07f5eae747516a9b11fd4484678163584a3cbf1b988030049135
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 624.8 kB
Tags CPython 3.12 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
47e281cf8f70310be2a2df44ee501f1763200683c00b920f23f5cc17d5346e80
BLAKE2b-256 checksum
How to use checksums
b3e9186f108f0d2ecc1dd0f82f3c45a0f1ea0feaad0a2906d160e73dd670a083
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL pyaff4-2.0.0a20-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 607.3 kB
Tags CPython 3.12 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
8749330ae26daa671f86f1e0960ab6a6dc19b06d3d8556f977ad3ea2be0fd0bb
BLAKE2b-256 checksum
How to use checksums
7cdc2499f3954c2f4dd9944a4472b5a7b1fe83c93ad8ee0353ac2b3a2aca877f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_x86_64.whl

Download URL pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_x86_64.whl
Size 265.5 kB
Tags CPython 3.12 abi3 macOS 11.0+ x86-64
SHA-256 checksum
How to use checksums
aab8f11466b59dabc48bc48ffc518ea06e74ae77b53ef33b7105d2a09d91b6df
BLAKE2b-256 checksum
How to use checksums
27ab1072f5e004fcd2dd11bde8b839ae12098a8ec4904788460f22f0438c417c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_arm64.whl

Download URL pyaff4-2.0.0a20-cp312-abi3-macosx_11_0_arm64.whl
Size 264.1 kB
Tags CPython 3.12 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
b98541a8db4736e9b74a506b20d5d2f896d5e2263092bde44efa0759c781a94d
BLAKE2b-256 checksum
How to use checksums
fcb9c37081b3d7bd4dac9ae943743f292a20b3637d2bb0702edc1dd18eb68345
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for pyaff4 2.0.0a20

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 8 files in this release have been uploaded with provenance metadata.

Select from 8 files

Attestation for pyaff4-2.0.0a20.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page