Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

AFF4 -The Advanced Forensics File Format

The Advanced Forensics File Format 4 (AFF4) is an open source format used for the storage of digital evidence and data.

It was originally designed and published in [1] and has since been standardised as the AFF4 Standard v1.0, which is available at https://github.com/aff4/Standard.

The focus of this implementation is:

  1. Standards compliance with AFF4 Standard v1.0 (for physical images)
  2. Ongoing development of the AFF4-L Standard v1.0 (for logical images)

Canonical images for the v1.0 physical image specification are provided in the AFF4 Reference Images github project at https://github.com/aff4/ReferenceImages

What is currently supported.

  1. Reading, writing & appending to ZipFile style volumes.
  2. Reading striped ZipFile volumes.
  3. Reading & writing AFF4 ImageStreams using the deflate or snappy compressor.
  4. Reading RDF metadata using Turtle and HDT (see below).
  5. Verification of linear and block hashed images.
  6. Reading & writing logical images.
  7. Reading & writing deduplicated logical images.
  8. Encrypted AFF4 logical volumes.

RDF metadata and HDT

AFF4 metadata is RDF. Containers always store it as Turtle and optionally as Header Dictionary Triples (information.hdt), a compressed, queryable binary form which supports fast opening of large containers.

  • Nothing is unpacked: the dictionary, bitmaps and packed sequences of the HDT strucure are read in place. An optional .opidx sidecar (as the Java tools write, and as a container may hold as information.hdt.opidx) saves building the object index.
  • Wheels are cp312-abi3 for Linux (manylinux2014, x86_64 and aarch64), macOS 11+ (x86_64 and arm64) and Windows (x86_64 and arm64). Other platforms get the py312-none-any wheel, which runs the HDT kernels interpreted and is about 100× slower on HDT queries.

pyaff4 bundles two packages for HDT:

  • rdflib_hdt_ng: a read-only rdflib.Store over HDT bytes, and graph_to_hdt to write an rdflib graph as HDT.
  • pyhdt2_cython: the HDT reader and writer behind it, in Python and NumPy with the hot loops compiled by Cython.

UTF-8 extended HDT format

HDT v1 relies on NULL terminators for determining the end of strings (and hence length), which fails for valid UTF-8 strings containing the Unicode NUL code point.

This implementation reads and writes HDT serializations which include the full range of UTF-8, by defining a new DictionarySection type. pyaff4 writes every dictionary section in this type, and reads it alongside the standard type 2. hdt-cpp 1.3.3 and hdt-java 2.1.2 cannot read type 8, so they cannot read the HDT files pyaff4 writes. With reference section 4 of "HDT Binary Format" (2015) [1], the newly defined PlainFrontCodingUtf8 = 8 dictionary section reuses the design of the existing PlainFrontCoding DictionarySection, instead employing a length-prepended string encoding, per the following structure:

<Buffer of Blocks>
 <VByte size of following string in bytes>
 <First String of Block>
 <VByte count of Common prefix characters> <VByte size of following string in bytes> <Suffix String>
 … as many times as necessary to fit "BlockSize" strings.
<CRC32 of Buffer of Blocks>

The included test file "hdtv1-failing-utf8.hdt" contains two triples constructed from such strings which demonstrate the limitation, while the test file "hdt-ng-utf8.hdt" demonstrates the proposed new UTF-8 capable variant of format.

Literal text is stored raw, as hdt-java and hdt-cpp store it: the characters between a literal's quotes are stored as they are, without N-Triples escapes, and a reader finds the end of the text at the last ".

Porting from v0.x

pyaff4 2.x needs Python 3.12 or later. While 2.x is in pre-release, install it with pip install --pre pyaff4; without --pre, pip installs 0.x. The command-line tool is the aff4 entry point. Code written against 0.x needs these changes:

0.x 2.x
rdfvalue.XSDString, XSDInteger, XSDDateTime, … rdflib.Literal(value, datatype=XSD.…)
rdfvalue.URN("aff4://…") rdfvalue.ARN("aff4://…"), an rdflib.URIRef. URN() takes no arguments; use URN.FromFileName(path) for files
Lexicon constants are strings, wrapped in rdfvalue.URN(...) Lexicon constants are URIRefs; pass them directly
hashes.newImmutableHash(hexdigest, hash_type) rdflib.Literal(hexdigest, datatype=hash_type)
pyaff4.utils (SmartUnicode, SmartStr, …) Removed; use str() and bytes
Container.createURN returns a hash-based container unless zip_based=True It returns a logical container; pass hashbased=True for a hash-based one
createURN(compression_method=…) and writeLogicalStream(…, allow_large_zipsegments=True) createURN(zipstream_compression_method=…, always_write_zipstreams=True), set once per container
Metadata in information.turtle only Containers also hold information.hdt and its .opidx index, which readers prefer; encrypted containers keep Turtle only
The local header of a zip member has a Zip64 field only when needed It always has one, so each member is 20 bytes larger than 0.x wrote it
Google Cloud Storage backend (aff4_cloud), aes_keywrap module Removed; key wrapping uses cryptography's aes_key_wrap
fastchunking for hash-based containers; snappy optional fastcdc; python-snappy is a required dependency

Development

uv sync installs pyaff4 editable and compiles the HDT kernels. The tests also read the AFF4 reference images from the reference_images submodule: clone with --recurse-submodules, or run git submodule update --init.

just test             # all tests, then the HDT engine's tests with interpreted kernels
just build            # sdist and a wheel for this host
just release-build    # cross-build, audit and install-test every release wheel (Linux)
python cicd/lint_kernels.py   # no CPython API calls in the compiled hot loops

References

[1] "Extending the advanced forensic format to accommodate multiple data sources, logical evidence, arbitrary information and forensic workflow" M.I. Cohen, Simson Garfinkel and Bradley Schatz, digital investigation 6 (2009) S57-S68.

Metadata

Release files for pyaff4 2.0.0a21

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pyaff4 2.0.0a21
File Size Uploaded Provenance Actions
pyaff4-2.0.0a21.tar.gz 392.2 kB With provenance Details for pyaff4-2.0.0a21.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for pyaff4 2.0.0a21
File Actions
pyaff4-2.0.0a21-py312-none-any.whl Python 3.12 none any Details for pyaff4-2.0.0a21-py312-none-any.whl
pyaff4-2.0.0a21-cp312-abi3-win_arm64.whl CPython 3.12 abi3 Windows ARM64 Details for pyaff4-2.0.0a21-cp312-abi3-win_arm64.whl
pyaff4-2.0.0a21-cp312-abi3-win_amd64.whl CPython 3.12 abi3 Windows x86-64 Details for pyaff4-2.0.0a21-cp312-abi3-win_amd64.whl
pyaff4-2.0.0a21-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.12 abi3 Linux glibc 2.17+ x86-64 Details for pyaff4-2.0.0a21-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pyaff4-2.0.0a21-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.12 abi3 Linux glibc 2.17+ ARM64 Details for pyaff4-2.0.0a21-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
pyaff4-2.0.0a21-cp312-abi3-macosx_11_0_x86_64.whl CPython 3.12 abi3 macOS 11.0+ x86-64 Details for pyaff4-2.0.0a21-cp312-abi3-macosx_11_0_x86_64.whl
pyaff4-2.0.0a21-cp312-abi3-macosx_11_0_arm64.whl CPython 3.12 abi3 macOS 11.0+ ARM64 Details for pyaff4-2.0.0a21-cp312-abi3-macosx_11_0_arm64.whl

Total release size: 3.0 MB

Release files / pyaff4-2.0.0a21.tar.gz

Download URL pyaff4-2.0.0a21.tar.gz
Size 392.2 kB
Tags Source
SHA-256 checksum
How to use checksums
01533e0d9fb84fd0bc9785d6e73d3b190771809e675465acabf0390cacfccf10
BLAKE2b-256 checksum
How to use checksums
fd2ecc1e17e73733e26f0276e77fff7134787071eb91f66385b9b45eacdd680e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a21-py312-none-any.whl

Download URL pyaff4-2.0.0a21-py312-none-any.whl
Size 142.5 kB
Tags Python 3.12
SHA-256 checksum
How to use checksums
fc36125f7c9cca348a7b0e41a1c53b57c927ca0b39b9495f1facf33d40e5d08c
BLAKE2b-256 checksum
How to use checksums
4de149191f8a3e89d18db2e162f93c49889cf746e4951f1cdc792eb26994baf9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a21-cp312-abi3-win_arm64.whl

Download URL pyaff4-2.0.0a21-cp312-abi3-win_arm64.whl
Size 325.0 kB
Tags CPython 3.12 Windows ARM64 abi3
SHA-256 checksum
How to use checksums
8a8ac9824ab89a4d58123644e2299924ee4216b8b92c5f3c2f62dbdc9003019a
BLAKE2b-256 checksum
How to use checksums
47a1969302f4fbc52cce3d5893741e4ac9619bc5b30fbe21422a106dfb3e6d32
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a21-cp312-abi3-win_amd64.whl

Download URL pyaff4-2.0.0a21-cp312-abi3-win_amd64.whl
Size 341.4 kB
Tags CPython 3.12 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
e58f99ea38dc9c6de6b523a4909fd45ecbda7d36040d81d3b3686be1ffa0c6a0
BLAKE2b-256 checksum
How to use checksums
b676abe097728ed33946c8fd9d0d03002870aadee23a2412a109930a4598282d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a21-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL pyaff4-2.0.0a21-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 624.7 kB
Tags CPython 3.12 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
eeccee8dbdb17f43bfe35e517737e3f22669383993d8c17dfd28f9c3c8982343
BLAKE2b-256 checksum
How to use checksums
22a0f944557f4f48c10c2468a0e319ae5a1ae9da642c4eda1168717f53eb58bb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a21-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL pyaff4-2.0.0a21-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 607.5 kB
Tags CPython 3.12 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
16f6ac265af842cc81dc5e9d0cab48a8418579e8f9f4c6dd90b3f0975bb28053
BLAKE2b-256 checksum
How to use checksums
d1212e3e8ee6eca1ead2835b3485af613f4d668cba7acc3448de29a2748d443a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a21-cp312-abi3-macosx_11_0_x86_64.whl

Download URL pyaff4-2.0.0a21-cp312-abi3-macosx_11_0_x86_64.whl
Size 265.5 kB
Tags CPython 3.12 abi3 macOS 11.0+ x86-64
SHA-256 checksum
How to use checksums
f6c84c4523e170afb2122311ce90d7f002a894ec010b072a5514dc52e880af4f
BLAKE2b-256 checksum
How to use checksums
6db7fb1c16d7a3377f061a4d1791fc1561478dbd1c10b94a76b649b89fdac5b5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / pyaff4-2.0.0a21-cp312-abi3-macosx_11_0_arm64.whl

Download URL pyaff4-2.0.0a21-cp312-abi3-macosx_11_0_arm64.whl
Size 264.0 kB
Tags CPython 3.12 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
368230dfa0e4fe281c658a8958ed522d2d24d7040b85ec7f03e49d30218b4725
BLAKE2b-256 checksum
How to use checksums
b9c5e5bdd0033b4b8429acfcbfc47a887cac6b0011484a54bd87d697c2391971
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for pyaff4 2.0.0a21

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 8 files in this release have been uploaded with provenance metadata.

Select from 8 files

Attestation for pyaff4-2.0.0a21.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page