Aftab paper


🇪🇸🇲🇽🇨🇺 Español | 🇮🇷🇦🇫🇹🇯 فارسی | 🇮🇹🇨🇭 Italiano | 🇫🇷🇧🇪🇨🇭 Français | 🇩🇪🇦🇹🇨🇭 Deutsch | 🇳🇱🇧🇪🇸🇷 Nederlands | 🇵🇹🇧🇷🇦🇴 Português | 🇸🇦🇱🇧🇮🇶 العربية | 🇷🇺🇧🇾🇰🇿 Русский | 🇨🇳🇸🇬🇹🇼 中文 | 🇯🇵 日本語 | 🇰🇷 한국어 | 🇮🇳 हिन्दी | 🇮🇩 Bahasa Indonesia | 🇧🇩🇮🇳 বাংলা | 🇻🇳 Tiếng Việt | 🇹🇷 Türkçe

Overview

Aftab (Persian: آفتاب, meaning "sun" or "sun rays") is a benchmarking framework for evaluating CNN-based encoders in PQN across Atari games. It provides standardized training, evaluation, and reproducibility tools for deep reinforcement learning research.

We have compiled a few videos comparing PQN and Aftab agents. Watch them here.

Encoder Experiments

IQM HNS
IQM HNS (Last 50M Frames)

Hadamax Experiments

IQM HNS
IQM HNS (Last 50M Frames)

References:

Q-Values Experiments

IQM HNS
IQM HNS (Last 50M Frames)

References:

Procgen (Overfitting Prevention) Experiments

IQM PNS
IQM PNS (Last 50M Frames)

Installation

Install via pip:

pip install aftab

Alternatively, you can clone the repository and install in editable mode.

git clone https://github.com/tahashieenavaz/aftab.git aftab_source
pip install -e aftab_source

We highly recommend using Micromamba for creating virtual environments with instructions detailed here.

Training Agents

Currently JAX API is under development and is planned to be finished by the end of 2026. Contributions are highly encouraged.

from aftab import Aftab
from aftab import aftab_environments

seeds = [1, 2, 3, 4]

for environment in aftab_environments:
    agent = Aftab(encoder="gamma", frames="pilot")
    for seed in seeds:
        agent.train(environment=environment, seed=seed)
        agent.log()

Custom Encoder Injection

You can define your own encoder as a PyTorch module and pass it to the agent:

import torch
from aftab import Aftab

class CustomImageEncoder(torch.nn.Module):
    pass

agent = Aftab(encoder=CustomImageEncoder)

Results

All experimental results are organized by experiment category. Each section contains:

  • Tables: numerical results (HNS/PHS and raw scores)
  • Charts: IQM normalized scores and training curves

Encoder Experiments

Tables

Charts

Hadamax Experiments

Tables

Charts

Q-Value Experiments

Tables

Charts

Procgen Experiments

Tables

Model Complexity

Base Variants

Variant Encoder Parameters Regression Head Parameters Total Parameters Encoder FLOPs Regression Head FLOPs Total FLOPs
PQN 78,304 1,686,500 1,764,804 7.734 1.610 9.347
Alpha 174,752 1,782,948 1,957,700 27.541 1.610 29.151
Beta 89,008 1,782,948 1,871,956 61.515 1.610 63.126
Gamma 117,168 1,725,364 1,842,532 22.901 1.610 24.512
Delta 78,552 1,850,588 1,929,140 6.143 1.774 7.917
Epsilon 80,112 2,179,828 2,259,940 13.252 2.101 15.354
Zeta 77,232 2,537,396 2,614,628 25.362 2.462 27.824
Eta 78,400 23,739,460 23,817,860 28.422 23.663 52.085
Theta 76,288 1,127,428 1,203,716 9.065 1.053 10.118

Note: The Eta variant has significantly more parameters than other variants, primarily due to the encoder producing a large number of features.

Hadamax Variants

Variant Encoder Parameters Regression Head Parameters Total Parameters Encoder FLOPs Regression Head FLOPs Total FLOPs
Hadamax 156,608 3,968,516 4,125,124 159.014 3.969 162.984
Gamma-Hadamax-Valid 234,336 1,609,220 1,843,556 122.001 1.610 123.611
Gamma-Hadamax-Same 234,336 3,280,388 3,514,724 129.300 3.281 132.581

Hyperparameters

Hyperparameter Value
Learning rate $2.5 \times 10^{-4}$
Training environments 128
Test environments 8
Optimizer Rectified Adam
Weight decay 0
$\epsilon$ $1 \times 10^{-5}$
$\beta_{1}$ 0.9
$\beta_{2}$ 0.999
Total Frames 200,000,000
Loss function Mean Squared Error
Scheduler Linear Annealing
$\epsilon$-greedy exploration 10% of total frames
Discount factor ($\gamma$) 0.99
GAE ($\lambda$) 0.65
Epochs 2
Batch size 4096

Used in encoder and Hadamax experiments.

Statistical Significance

Encoder Experiments

Wilcoxon Signed Rank Test Wilcoxon Signed Rank Test (Corrected)
Probability of Improvement

Hadamax Experiments

Wilcoxon Signed Rank Test Wilcoxon Signed Rank Test (Corrected)
Probability of Improvement

Q-Value Experiments

Wilcoxon Signed Rank Test Wilcoxon Signed Rank Test (Corrected)
Probability of Improvement

Reproducibility

Due to the stochastic nature of deep reinforcement learning, exact reproducibility via fixed datasets is not feasible.
Instead, we provide a set of random seeds used in our experiments.

from aftab import aftab_seeds

print(aftab_seeds)

Full experiment replication:

from aftab import Aftab
from aftab import aftab_environments
from aftab import aftab_seeds

for environment in aftab_environments:
    agent = Aftab()
    for seed in aftab_seeds:
        agent.train(environment=environment, seed=seed)
        agent.log()

A comprehensive set of Atari environments is available via EnvPool:
https://envpool.readthedocs.io/en/latest/env/atari.html#available-tasks

Procgen environments use their native RGB observations with shape (3, 64, 64). Aftab reads each task's EnvPool configuration and only applies supported options. Atari-only options such as noop, frame_skip, frame_stack, train_episodic_life, and EnvPool reward clipping are therefore not passed to Procgen.

A comprehensive set of Procgen environments is available via EnvPool:

https://envpool.readthedocs.io/en/latest/env/procgen.html#available-tasks

Hardware

Nvidia A40 GPUs were used to run all the experiments in this experiment.

Specification Details
GPU Memory 48 GB GDDR6 with error-correcting code (ECC)
GPU Memory Bandwidth 696 GB/s
Interconnect NVIDIA NVLink 112.5 GB/s (bidirectional); PCIe Gen4: 64 GB/s
NVLink 2-way low profile (2-slot)
Display Ports 3x DisplayPort 1.4*
Max Power Consumption 300 W
Form Factor 4.4" (H) x 10.5" (L), Dual Slot
Thermal Passive
vGPU Software Support NVIDIA Virtual PC, NVIDIA Virtual Applications, NVIDIA RTX Virtual Workstation, NVIDIA Virtual Compute Server, NVIDIA AI Enterprise
vGPU Profiles Supported See the Virtual GPU Licensing Guide
NVENC / NVDEC 1x / 2x (includes AV1 decode)
Secure Boot Secure and Measured Boot with Hardware Root of Trust (optional)
NEBS Ready Level 3
Power Connector 8-pin CPU

Citation

@article{aftab2026drl,
  title={Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks},
  author={Shieenavaz, Taha and Zareshahraki, Shabnam and Nanni, Loris},
  journal={arXiv preprint arXiv:YYMM.NNNNN},
  year={2026}
}

Related Works

@misc{2407.04811,
  Title = {Simplifying Deep Temporal Difference Learning},
  Author = {Matteo Gallici and Mattie Fellows and Benjamin Ellis and Bartomeu Pou and Ivan Masmitja and Jakob Nicolaus Foerster and Mario Martin},
  Year = {2024},
  Eprint = {arXiv:2407.04811},
}

@misc{2403.03950,
  Title = {Stop Regressing: Training Value Functions via Classification for Scalable Deep RL},
  Author = {Jesse Farebrother and Jordi Orbay and Quan Vuong and Adrien Ali Taïga and Yevgen Chebotar and Ted Xiao and Alex Irpan and Sergey Levine and Pablo Samuel Castro and Aleksandra Faust and Aviral Kumar and Rishabh Agarwal},
  Year = {2024},
  Eprint = {arXiv:2403.03950},
}

@misc{1511.06581,
  Title = {Dueling Network Architectures for Deep Reinforcement Learning},
  Author = {Ziyu Wang and Tom Schaul and Matteo Hessel and Hado van Hasselt and Marc Lanctot and Nando de Freitas},
  Year = {2015},
  Eprint = {arXiv:1511.06581},
}

@misc{1806.04613,
  Title = {Improving Regression Performance with Distributional Losses},
  Author = {Ehsan Imani and Martha White},
  Year = {2018},
  Eprint = {arXiv:1806.04613},
}

@misc{1602.04621,
  Title = {Deep Exploration via Bootstrapped DQN},
  Author = {Ian Osband and Charles Blundell and Alexander Pritzel and Benjamin Van Roy},
  Year = {2016},
  Eprint = {arXiv:1602.04621},
}

Useful Links

License

© 2025 Taha Shieenavaz.
Licensed under CC BY-NC 4.0: https://creativecommons.org/licenses/by-nc/4.0/

Project links

Data verified by PyPI on Aug 10, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.0.5)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for underdash from gravatar.com underdash

Credits

Author: Taha Shieenavaz

GitHub Statistics

Data verified by PyPI on Aug 10, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (Creative Commons Attribution-NonCommercial 4.0 International)

Requires

Python >=3.9

Tags

reinforcement learning dqn aftab drl pqn deep reinforcement learning cnn

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

aftab-1.0.0.tar.gz (59.5 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

aftab-1.0.0-py3-none-any.whl (67.2 kB view details)

Uploaded Python 3

File details

Details for the file aftab-1.0.0.tar.gz.

File metadata

  • Download URL: aftab-1.0.0.tar.gz
  • Upload date:
  • Size: 59.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for aftab-1.0.0.tar.gz
Algorithm Hash digest
SHA256 416b5a09ae2383cf4c77ef23efc7cc53542a589175cb39fa94e717b94c76b9be
MD5 c76081acaa4045641cd0333351c156b8
BLAKE2b-256 27f70cb8d69c32f88712a2316e1d5ce7325b55046b0db629b1a38ab63342660a

See more details on using hashes here.

Provenance

The following attestation bundles were made for aftab-1.0.0.tar.gz:

Publisher: publish.yaml on tahashieenavaz/aftab

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file aftab-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: aftab-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 67.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for aftab-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 5aa139d76d99b8e52b54e7e400b565aba415c6cc59263e3f6317280d4b89e54a
MD5 6f6c17183fcd043d5d180f3c54c60f1b
BLAKE2b-256 81249325dcb9ab3284649d5862e48a5a61854b9fea20253acd5d7fd6844613f8

See more details on using hashes here.

Provenance

The following attestation bundles were made for aftab-1.0.0-py3-none-any.whl:

Publisher: publish.yaml on tahashieenavaz/aftab

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

This release

1.0.0 This release

2 files

0.1.61

2 files

0.1.60

2 files

0.1.59

2 files

0.1.58

2 files

0.1.57

2 files

0.1.56

2 files

0.1.55

2 files

0.1.54

2 files

0.1.53

2 files

0.1.52

2 files

0.1.51

2 files

0.1.50

2 files

0.1.49

2 files

0.1.48

2 files

0.1.47

2 files

0.1.46

2 files

0.1.45

2 files

0.1.44

2 files

0.1.43

2 files

0.1.42

2 files

0.1.41

2 files

0.1.40

2 files

0.1.39

2 files

0.1.38

2 files

0.1.37

2 files

0.1.36

2 files

0.1.35

2 files

0.1.34

2 files

0.1.33

2 files

0.1.31

2 files

0.1.30

2 files

0.1.29

2 files

0.1.28

2 files

0.1.27

2 files

0.1.26

2 files

0.1.25

2 files

0.1.24

2 files

0.1.23

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

0.1.19

2 files

0.1.18

2 files

0.1.16

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

0.1.1

2 files

0.1.0

2 files

0.0.99

2 files

0.0.97

2 files

0.0.96

2 files

0.0.95

2 files

0.0.94

2 files

0.0.93

2 files

0.0.92

2 files

0.0.91

2 files

0.0.90

2 files

0.0.89

2 files

0.0.88

2 files

0.0.87

2 files

0.0.86

2 files

0.0.85

2 files

0.0.84

2 files

0.0.83

2 files

0.0.82

2 files

0.0.81

2 files

0.0.80

2 files

0.0.79

2 files

0.0.78

2 files

0.0.77

2 files

0.0.76

2 files

0.0.75

2 files

0.0.74

2 files

0.0.73

2 files

0.0.72

2 files

0.0.71

2 files

0.0.70

2 files

0.0.69

2 files

0.0.68

2 files

0.0.67

2 files

0.0.66

2 files

0.0.65

2 files

0.0.64

2 files

0.0.63

2 files

0.0.62

2 files

0.0.61

2 files

0.0.60

2 files

0.0.59

2 files

0.0.58

2 files

0.0.57

2 files

0.0.56

2 files

0.0.55

2 files

0.0.54

2 files

0.0.53

2 files

0.0.52

2 files

0.0.51

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page