AI Model Dynamic Offloader

This project is a pytorch VRAM allocator that implements on-demand offloading of model weights when the primary pytorch VRAM allocator comes under pressure.

Support:

  • Nvidia GPUs (CUDA) and AMD GPUs (ROCm/HIP)
  • PyTorch 2.8+
  • CUDA 12.8+ (Nvidia) / ROCm 7+ (AMD)
  • Windows 11+ / Linux as per python ManyLinux support

How it works:

  • The pytorch application creates a Virtual Base Address Register (VBAR) for a model. Creating a VBAR doesn't cost any VRAM, only GPU virtual address space (which is pretty much free).
  • The pytorch application allocates tensors for model weights within the VBAR. These tensors are initially un-allocated and will segfault if touched.
  • The pytorch application faults in the tensors using the fault() API at the time the tensor is needed. This is where VRAM actually gets allocated.
If the fault() is successful (sufficient VRAM for this tensor):
  1. If the fault() resultant signature is changed or unknown:
    • The application uses tensor::_copy() to populate the weight data on the GPU.
    • The application saves the returned signature against this weight for future comparison
  2. The layer uses the weight tensor.
  3. The application calls unpin() on the tensor to allow it to be freed under pressure later if needed.
If the fault() is unsuccessful (offloaded weight):
  1. The application allocates a temporary regular GPU tensor.
  2. Uses _copy to populate weight data on the GPU.
  3. The layer uses the temporary as the weight.
  4. Pytorch garbage collects the temp when the layer is finished.

see examples/example.py

Priorities:

  • The most recent VBARs are the highest priority and lower addresses in the VBAR take priority over higher addresses.
  • Applications should order their tensor allocations in the VBAR in load-priority order with the lowest addresses for the highest priority weights.
  • Calling fault() on a weight that is higher priority than other weights will cause those lower priority weights to get freed to make space.
  • Having a weight evicted sets that VBAR's watermark to that weight's level. Any weights in the same VBAR above the watermark automatically fail the fault() API. This avoids constantly faulting in all weights each model iteration while allowing the application to just blindly call fault() every layer and check the results. There is no need for the application to manage any VRAM quotas or watermarks.
  • Existing VBARs can be pushed to top priority with the prioritize() API. This allows use of an already loaded or partially model (e.g. using the same model twice in a complex workflow). Using prioritize resets the offload watermark of that model to no offloading, giving its weights priority over any other currently loaded models.

Backend:

  • VBAR allocation is done with cuMemAddressReserve(), faulting with cuMemCreate() and cuMemMap() and all frees done with appropriate converse APIs.
  • For consistency with VBAR memory management, main pytorch allocator plugin is also implemented with cuMemAddressReserve -> cuMemCreate -> cuMemMap. This also behaves a lot better on Windows systems with System Memory fallback.

On AMD, the equivalent HIP APIs (hipMemAddressReserve -> hipMemCreate -> hipMemMap, and their converse calls) are used throughout via the same flow.

Caveats:

  • There is no real way for this allocator to tell the difference between high usage and bad fragmentation in the pytorch caching allocator. As we always return success to the pytorch caching allocator it experiences no pressure while weights are being offloaded which means it can run in an extremely fragmented mode. The assumption is model weight access patterns are reasonably regular over blocks or iterations and it finds a good set of sizes to cache. What you should generally do though, is completely flush the pytorch caching allocator before each new model run, which avoids completely un-used reservations from taking priority over the next models weights.

Project links

Data verified by PyPI on Aug 23, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

Owner

PyPI data
Data sourced directly from PyPI's database.

Credits

Author: rattus

GitHub Statistics

Data verified by PyPI on Aug 23, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

Requires

Python >=3.9

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

comfy_aimdo-0.4.14-py3-none-any.whl (22.9 kB view details)

Uploaded Python 3

comfy_aimdo-0.4.14-cp39-abi3-win_arm64.whl (227.6 kB view details)

Uploaded CPython 3.9+Windows ARM64

comfy_aimdo-0.4.14-cp39-abi3-win_amd64.whl (253.8 kB view details)

Uploaded CPython 3.9+Windows x86-64

comfy_aimdo-0.4.14-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (567.1 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64

comfy_aimdo-0.4.14-cp39-abi3-manylinux2010_x86_64.manylinux2014_x86_64.manylinux_2_12_x86_64.manylinux_2_17_x86_64.whl (342.3 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.12+ x86-64manylinux: glibc 2.17+ x86-64

File details

Details for the file comfy_aimdo-0.4.14-py3-none-any.whl.

File metadata

  • Download URL: comfy_aimdo-0.4.14-py3-none-any.whl
  • Upload date:
  • Size: 22.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for comfy_aimdo-0.4.14-py3-none-any.whl
Algorithm Hash digest
SHA256 4397b34107b75f2438b321755a6b2aaadf44a648a16313e08b487515b9719ace
MD5 b130bd0e8e647590df5306bbc0b841f0
BLAKE2b-256 413bf9304fad30b394aca05216b4a695c01119703e37bd5a81bec2ee0f034ed4

See more details on using hashes here.

Provenance

The following attestation bundles were made for comfy_aimdo-0.4.14-py3-none-any.whl:

Publisher: build-wheels.yml on Comfy-Org/comfy-aimdo

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file comfy_aimdo-0.4.14-cp39-abi3-win_arm64.whl.

File metadata

  • Download URL: comfy_aimdo-0.4.14-cp39-abi3-win_arm64.whl
  • Upload date:
  • Size: 227.6 kB
  • Tags: CPython 3.9+, Windows ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for comfy_aimdo-0.4.14-cp39-abi3-win_arm64.whl
Algorithm Hash digest
SHA256 77787cf91c680b4764141a9ff252889640e2780eccfad799cf323439769d7b71
MD5 d36ea9b53cd8264f77bfab246f6684ee
BLAKE2b-256 8e9330c1a34f2d552adf0a5264fd058a89b12161afd31f8fe6851ca87fc8052c

See more details on using hashes here.

Provenance

The following attestation bundles were made for comfy_aimdo-0.4.14-cp39-abi3-win_arm64.whl:

Publisher: build-wheels.yml on Comfy-Org/comfy-aimdo

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file comfy_aimdo-0.4.14-cp39-abi3-win_amd64.whl.

File metadata

  • Download URL: comfy_aimdo-0.4.14-cp39-abi3-win_amd64.whl
  • Upload date:
  • Size: 253.8 kB
  • Tags: CPython 3.9+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for comfy_aimdo-0.4.14-cp39-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 cae0d6d2e7d8510a475c94e89cfcd413f881b01887f74a94c1e2411d9e87b92e
MD5 6ab83c8523d511b51878162b73195be9
BLAKE2b-256 a6ec8175d9296bcf66cb715bb7ff64ea4a31fca6e16a854cd44db89b4ae481b1

See more details on using hashes here.

Provenance

The following attestation bundles were made for comfy_aimdo-0.4.14-cp39-abi3-win_amd64.whl:

Publisher: build-wheels.yml on Comfy-Org/comfy-aimdo

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file comfy_aimdo-0.4.14-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for comfy_aimdo-0.4.14-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 b30a083cf7e7222bb083969bcb3d735a296db5ad2eb321e6c8f009aaf24686c3
MD5 a62b8a1e3fe38681668e1dfc71d9a130
BLAKE2b-256 ef88c0520d371c51c5b2b8df1a0864505bc45b87a67558c70b3651111458b882

See more details on using hashes here.

Provenance

The following attestation bundles were made for comfy_aimdo-0.4.14-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl:

Publisher: build-wheels.yml on Comfy-Org/comfy-aimdo

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file comfy_aimdo-0.4.14-cp39-abi3-manylinux2010_x86_64.manylinux2014_x86_64.manylinux_2_12_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for comfy_aimdo-0.4.14-cp39-abi3-manylinux2010_x86_64.manylinux2014_x86_64.manylinux_2_12_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 1c5b3a9653526fadc6492247621a6d8c0d3e9bccc626d4dba17a496c74f77cd8
MD5 b144ea18ea83aa0ef29da6b2ab35be01
BLAKE2b-256 0ffc6c3c1ade49d53a4d1945e42f03e57b38583a55be5a13c32dc83e82e7134f

See more details on using hashes here.

Provenance

The following attestation bundles were made for comfy_aimdo-0.4.14-cp39-abi3-manylinux2010_x86_64.manylinux2014_x86_64.manylinux_2_12_x86_64.manylinux_2_17_x86_64.whl:

Publisher: build-wheels.yml on Comfy-Org/comfy-aimdo

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.4.14 This release

5 files

0.4.13

5 files

0.4.12

5 files

0.4.11

5 files

0.4.10

5 files

0.4.9

5 files

0.4.8

5 files

0.4.7

5 files

0.4.6

5 files

0.4.5

5 files

0.4.4

5 files

0.4.3

5 files

0.4.2

5 files

0.4.1

5 files

0.4.0

5 files

0.3.0

5 files

0.2.14

3 files

0.2.12

3 files

0.2.11

3 files

0.2.10

3 files

0.2.9

3 files

0.2.8

3 files

0.2.7

3 files

0.2.6

3 files

0.2.5

3 files

0.2.4

3 files

0.2.3

3 files

0.2.2

3 files

0.2.1

3 files

0.2.0

3 files

0.1.8

3 files

0.1.7

3 files

0.1.6

3 files

0.1.5

3 files

0.1.4

3 files

0.1.3

3 files

0.1.2

11 files

0.1.1

11 files

0.1.0

11 files

0.0.214

3 files

0.0.213

3 files

0.0.4

11 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page