Aftab Header


🇪🇸🇲🇽🇨🇺 Español | 🇮🇷🇦🇫🇹🇯 فارسی | 🇮🇹🇨🇭 Italiano | 🇫🇷🇧🇪🇨🇭 Français | 🇩🇪🇦🇹🇨🇭 Deutsch | 🇳🇱🇧🇪🇸🇷 Nederlands | 🇵🇹🇧🇷🇦🇴 Português | 🇸🇦🇱🇧🇮🇶 العربية | 🇷🇺🇧🇾🇰🇿 Русский | 🇨🇳🇸🇬🇹🇼 中文 | 🇯🇵 日本語 | 🇰🇷 한국어 | 🇮🇳 हिन्दी | 🇮🇩 Bahasa Indonesia | 🇧🇩🇮🇳 বাংলা | 🇻🇳 Tiếng Việt | 🇹🇷 Türkçe

Overview

Aftab (Persian: آفتاب, meaning "sun" or "sun rays") is a benchmarking framework for evaluating CNN-based encoders in PQN across Atari games. It provides standardized training, evaluation, and reproducibility tools for deep reinforcement learning research.

We have compiled a few videos comparing PQN and Aftab agents. Watch them here.

Encoder Experiments

IQM HNS
IQM HNS (Last 50M Frames)

Hadamax Experiments

IQM HNS
IQM HNS (Last 50M Frames)

References:

Q-Values Experiments

IQM HNS
IQM HNS (Last 50M Frames)

References:

Procgen (Overfitting Prevention) Experiments

Since there are no public benchmarks comparing human-normalized-scores of Procgen environments, we created PNS (Procgen Normalized Score) that is a minimal min-max normalization of scores across seeds.

IQM PNS
IQM PNS (Last 50M Frames)

Installation

Install via pip:

pip install aftab

Alternatively, you can clone the repository and install in editable mode.

git clone https://github.com/tahashieenavaz/aftab.git aftab_source
pip install -e aftab_source

We highly recommend using Micromamba for creating virtual environments with instructions detailed here.

Training Agents

Currently JAX API is under development and is planned to be finished by the end of 2026. Contributions are highly encouraged.

from aftab import Aftab
from aftab import aftab_environments

seeds = [1, 2, 3, 4]

for environment in aftab_environments:
    agent = Aftab(encoder="gamma", frames="pilot")
    for seed in seeds:
        agent.train(environment=environment, seed=seed)
        agent.log()

Custom Encoder Injection

You can define your own encoder as a PyTorch module and pass it to the agent:

import torch
from aftab import Aftab

class CustomImageEncoder(torch.nn.Module):
    pass

agent = Aftab(encoder=CustomImageEncoder)

Results

All experimental results are organized by experiment category. Each section contains:

  • Tables: numerical results (HNS/PHS and raw scores)
  • Charts: IQM normalized scores and training curves

Encoder Experiments

Tables

Charts

Hadamax Experiments

Tables

Charts

Q-Value Experiments

Tables

Charts

Procgen Experiments

Tables

AUC

Model Complexity

Base Variants

Variant Encoder Parameters Regression Head Parameters Total Parameters Encoder FLOPs Regression Head FLOPs Total FLOPs
PQN 78,304 1,686,500 1,764,804 7.734 1.610 9.347
Alpha 174,752 1,782,948 1,957,700 27.541 1.610 29.151
Beta 89,008 1,782,948 1,871,956 61.515 1.610 63.126
Gamma 117,168 1,725,364 1,842,532 22.901 1.610 24.512
Delta 78,552 1,850,588 1,929,140 6.143 1.774 7.917
Epsilon 80,112 2,179,828 2,259,940 13.252 2.101 15.354
Zeta 77,232 2,537,396 2,614,628 25.362 2.462 27.824
Eta 78,400 23,739,460 23,817,860 28.422 23.663 52.085
Theta 76,288 1,127,428 1,203,716 9.065 1.053 10.118

Note: The Eta variant has significantly more parameters than other variants, primarily due to the encoder producing a large number of features.

Hadamax Variants

Variant Encoder Parameters Regression Head Parameters Total Parameters Encoder FLOPs Regression Head FLOPs Total FLOPs
Hadamax 156,608 3,968,516 4,125,124 159.014 3.969 162.984
Gamma-Hadamax-Valid 234,336 1,609,220 1,843,556 122.001 1.610 123.611
Gamma-Hadamax-Same 234,336 3,280,388 3,514,724 129.300 3.281 132.581

Hyperparameters

The following tables reflect the defaults defined by Aftab. The experiment_name argument is required and has no default.

Training and Environment

Hyperparameter (Aftab argument) Default
Encoder (encoder) Gamma-Hadamax-Valid
Network (network) Distributional Bootstrapped (Ensemble) Duelling
Total frames (frames) 200,000,000
Frame skip (frame_skip) 4
Frame stack (frame_stack) 4
No-op maximum (noop) 30
Learning rate (lr) $2.5 \times 10^{-4}$
Training environments (train_environments) 128
Test environments (test_environments) 8
Steps per update (steps_per_update) 32
Batch size (derived) 4,096
Mini-batches (mini_batches) 32
Mini-batch size (derived) 128
Discount factor ($\gamma$) 0.99
Return $\lambda$ (return_lambda) 0.65
Epochs (epochs) 2
Gradient norm (gradient_norm) 10.0
Embedding dimension (embedding_dimension) 512
Training episodic life (train_episodic_life) True
Test episodic life (test_episodic_life) False
Training reward clipping (train_reward_clip) True
Test reward clipping (test_reward_clip) True
Epsilon Schedule Linear
Epsilon Annealing Ratio 10%

Optimizer

Hyperparameter (Aftab argument) Default
Optimizer (optimizer) Rectified Adam ("radam")
Epsilon (optimizer_epsilon) $1 \times 10^{-5}$
Weight decay (optimizer_weight_decay) 0.0
$\beta_1$ (optimizer_first_beta) 0.9
$\beta_2$ (optimizer_second_beta) 0.999

Distributional and Bootstrapped (Ensemble) Q-Values

Hyperparameter (Aftab argument) Default
Distributional bins (distributional_bins) 51
Distributional minimum (distributional_min_value) -10.0
Distributional maximum (distributional_max_value) 10.0
Distributional sigma (distributional_sigma) None (derived from the sigma ratio)
Distributional sigma ratio (distributional_sigma_ratio) 0.75
Distributional value clip (distributional_value_clip) 0.0
Bootstrap heads (bootstrap_heads) 10
Bootstrap probability (bootstrap_probability) 1.0

Procgen Overrides

Hyperparameter Default Procgen
Training environments 128 64 (procgen_train_environments)
Steps per update 32 256 (procgen_steps_per_update)
Batch size 4,096 16,384
Mini-batch size 128 512

For Procgen environments, Aftab automatically applies the two overrides above; other defaults remain unchanged.

Statistical Significance

Encoder Experiments

Wilcoxon Signed Rank Test Wilcoxon Signed Rank Test (Corrected)
Probability of Improvement

Hadamax Experiments

Wilcoxon Signed Rank Test Wilcoxon Signed Rank Test (Corrected)
Probability of Improvement

Q-Value Experiments

Wilcoxon Signed Rank Test Wilcoxon Signed Rank Test (Corrected)
Probability of Improvement

Reproducibility

Due to the stochastic nature of deep reinforcement learning, exact reproducibility via fixed datasets is not feasible.
Instead, we provide a set of random seeds used in our experiments.

from aftab import aftab_seeds

print(aftab_seeds)

Full experiment replication:

from aftab import Aftab
from aftab import aftab_environments
from aftab import aftab_seeds

for environment in aftab_environments:
    agent = Aftab()
    for seed in aftab_seeds:
        agent.train(environment=environment, seed=seed)
        agent.log()

A comprehensive set of Atari environments is available via EnvPool:
https://envpool.readthedocs.io/en/latest/env/atari.html#available-tasks

Procgen environments use their native RGB observations with shape (3, 64, 64). Aftab reads each task's EnvPool configuration and only applies supported options. Atari-only options such as noop, frame_skip, frame_stack, train_episodic_life, and EnvPool reward clipping are therefore not passed to Procgen.

A comprehensive set of Procgen environments is available via EnvPool:

https://envpool.readthedocs.io/en/latest/env/procgen.html#available-tasks

Hardware

Nvidia A40 GPUs were used to run all the experiments in this experiment.

Specification Details
GPU Memory 48 GB GDDR6 with error-correcting code (ECC)
GPU Memory Bandwidth 696 GB/s
Interconnect NVIDIA NVLink 112.5 GB/s (bidirectional); PCIe Gen4: 64 GB/s
NVLink 2-way low profile (2-slot)
Display Ports 3x DisplayPort 1.4*
Max Power Consumption 300 W
Form Factor 4.4" (H) x 10.5" (L), Dual Slot
Thermal Passive
vGPU Software Support NVIDIA Virtual PC, NVIDIA Virtual Applications, NVIDIA RTX Virtual Workstation, NVIDIA Virtual Compute Server, NVIDIA AI Enterprise
vGPU Profiles Supported See the Virtual GPU Licensing Guide
NVENC / NVDEC 1x / 2x (includes AV1 decode)
Secure Boot Secure and Measured Boot with Hardware Root of Trust (optional)
NEBS Ready Level 3
Power Connector 8-pin CPU

Citation

@article{aftab2026drl,
  title={Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks},
  author={Shieenavaz, Taha and Zareshahraki, Shabnam and Nanni, Loris},
  journal={arXiv preprint arXiv:YYMM.NNNNN},
  year={2026}
}

Related Works

@misc{2407.04811,
  Title = {Simplifying Deep Temporal Difference Learning},
  Author = {Matteo Gallici and Mattie Fellows and Benjamin Ellis and Bartomeu Pou and Ivan Masmitja and Jakob Nicolaus Foerster and Mario Martin},
  Year = {2024},
  Eprint = {arXiv:2407.04811},
}

@misc{2403.03950,
  Title = {Stop Regressing: Training Value Functions via Classification for Scalable Deep RL},
  Author = {Jesse Farebrother and Jordi Orbay and Quan Vuong and Adrien Ali Taïga and Yevgen Chebotar and Ted Xiao and Alex Irpan and Sergey Levine and Pablo Samuel Castro and Aleksandra Faust and Aviral Kumar and Rishabh Agarwal},
  Year = {2024},
  Eprint = {arXiv:2403.03950},
}

@misc{1511.06581,
  Title = {Dueling Network Architectures for Deep Reinforcement Learning},
  Author = {Ziyu Wang and Tom Schaul and Matteo Hessel and Hado van Hasselt and Marc Lanctot and Nando de Freitas},
  Year = {2015},
  Eprint = {arXiv:1511.06581},
}

@misc{1806.04613,
  Title = {Improving Regression Performance with Distributional Losses},
  Author = {Ehsan Imani and Martha White},
  Year = {2018},
  Eprint = {arXiv:1806.04613},
}

@misc{1602.04621,
  Title = {Deep Exploration via Bootstrapped DQN},
  Author = {Ian Osband and Charles Blundell and Alexander Pritzel and Benjamin Van Roy},
  Year = {2016},
  Eprint = {arXiv:1602.04621},
}

Useful Links

License

© 2025 Taha Shieenavaz.
Licensed under CC BY-NC 4.0: https://creativecommons.org/licenses/by-nc/4.0/

Project links

Data verified by PyPI on Aug 12, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.0.5)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for underdash from gravatar.com underdash

Credits

Author: Taha Shieenavaz

GitHub Statistics

Data verified by PyPI on Aug 12, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (Creative Commons Attribution-NonCommercial 4.0 International)

Requires

Python >=3.9

Tags

reinforcement learning dqn aftab drl pqn deep reinforcement learning cnn

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

aftab-1.0.4.tar.gz (59.5 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

aftab-1.0.4-py3-none-any.whl (68.0 kB view details)

Uploaded Python 3

File details

Details for the file aftab-1.0.4.tar.gz.

File metadata

  • Download URL: aftab-1.0.4.tar.gz
  • Upload date:
  • Size: 59.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for aftab-1.0.4.tar.gz
Algorithm Hash digest
SHA256 cf4886d328bbf1d69a03470d904a84ac19a88ff0ae3d820c34e07c3643302e96
MD5 db98cf64ef16f1c8a8f5ead7ec7b0cc3
BLAKE2b-256 ad64f7093625224704e1f0d9fb7b438f470344ce900088e7bd55c6cf14c8e6c9

See more details on using hashes here.

Provenance

The following attestation bundles were made for aftab-1.0.4.tar.gz:

Publisher: publish.yaml on tahashieenavaz/aftab

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file aftab-1.0.4-py3-none-any.whl.

File metadata

  • Download URL: aftab-1.0.4-py3-none-any.whl
  • Upload date:
  • Size: 68.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for aftab-1.0.4-py3-none-any.whl
Algorithm Hash digest
SHA256 c750d773a67b53c8394b3562da01ec557b59871ea2b1fc71dbabf56b7fdad926
MD5 fa7683e62f2f44e206fab9d3cabffeb4
BLAKE2b-256 733b30da4792a68b298cb9d6f6c338890093190288d94afbd1d5046d2d77062b

See more details on using hashes here.

Provenance

The following attestation bundles were made for aftab-1.0.4-py3-none-any.whl:

Publisher: publish.yaml on tahashieenavaz/aftab

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.0.5

2 files

This release

1.0.4 This release

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

0.1.61

2 files

0.1.60

2 files

0.1.59

2 files

0.1.58

2 files

0.1.57

2 files

0.1.56

2 files

0.1.55

2 files

0.1.54

2 files

0.1.53

2 files

0.1.52

2 files

0.1.51

2 files

0.1.50

2 files

0.1.49

2 files

0.1.48

2 files

0.1.47

2 files

0.1.46

2 files

0.1.45

2 files

0.1.44

2 files

0.1.43

2 files

0.1.42

2 files

0.1.41

2 files

0.1.40

2 files

0.1.39

2 files

0.1.38

2 files

0.1.37

2 files

0.1.36

2 files

0.1.35

2 files

0.1.34

2 files

0.1.33

2 files

0.1.31

2 files

0.1.30

2 files

0.1.29

2 files

0.1.28

2 files

0.1.27

2 files

0.1.26

2 files

0.1.25

2 files

0.1.24

2 files

0.1.23

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

0.1.19

2 files

0.1.18

2 files

0.1.16

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

0.1.1

2 files

0.1.0

2 files

0.0.99

2 files

0.0.97

2 files

0.0.96

2 files

0.0.95

2 files

0.0.94

2 files

0.0.93

2 files

0.0.92

2 files

0.0.91

2 files

0.0.90

2 files

0.0.89

2 files

0.0.88

2 files

0.0.87

2 files

0.0.86

2 files

0.0.85

2 files

0.0.84

2 files

0.0.83

2 files

0.0.82

2 files

0.0.81

2 files

0.0.80

2 files

0.0.79

2 files

0.0.78

2 files

0.0.77

2 files

0.0.76

2 files

0.0.75

2 files

0.0.74

2 files

0.0.73

2 files

0.0.72

2 files

0.0.71

2 files

0.0.70

2 files

0.0.69

2 files

0.0.68

2 files

0.0.67

2 files

0.0.66

2 files

0.0.65

2 files

0.0.64

2 files

0.0.63

2 files

0.0.62

2 files

0.0.61

2 files

0.0.60

2 files

0.0.59

2 files

0.0.58

2 files

0.0.57

2 files

0.0.56

2 files

0.0.55

2 files

0.0.54

2 files

0.0.53

2 files

0.0.52

2 files

0.0.51

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page