Aftab Header


🇪🇸🇲🇽🇨🇺 Español | 🇮🇷🇦🇫🇹🇯 فارسی | 🇮🇹🇨🇭 Italiano | 🇫🇷🇧🇪🇨🇭 Français | 🇩🇪🇦🇹🇨🇭 Deutsch | 🇳🇱🇧🇪🇸🇷 Nederlands | 🇵🇹🇧🇷🇦🇴 Português | 🇸🇦🇱🇧🇮🇶 العربية | 🇷🇺🇧🇾🇰🇿 Русский | 🇨🇳🇸🇬🇹🇼 中文 | 🇯🇵 日本語 | 🇰🇷 한국어 | 🇮🇳 हिन्दी | 🇮🇩 Bahasa Indonesia | 🇧🇩🇮🇳 বাংলা | 🇻🇳 Tiếng Việt | 🇹🇷 Türkçe

Overview

Aftab (Persian: آفتاب, meaning "sun" or "sun rays") is a benchmarking framework for evaluating CNN-based encoders in PQN across Atari games. It provides standardized training, evaluation, and reproducibility tools for deep reinforcement learning research.

See how the Aftab architecture compares to standard PQN baselines in these video demonstrations.

This research was done without receiving any funds; therefore, if you found our work useful, please consider sponsoring on GitHub 💛.

Encoder Experiments

IQM HNS
IQM HNS (Last 50M Frames)

Hadamax Experiments

IQM HNS
IQM HNS (Last 50M Frames)

References:

Q-Values Experiments

IQM HNS
IQM HNS (Last 50M Frames)

References:

Procgen (Overfitting Prevention) Experiments

Since there are no public benchmarks comparing human-normalized-scores of Procgen environments, we created PNS (Procgen Normalized Score) that is a minimal min-max normalization of scores across seeds.

IQM PNS
IQM PNS (Last 50M Frames)

Installation

Install via pip:

pip install aftab

Alternatively, you can clone the repository and install in editable mode.

git clone https://github.com/tahashieenavaz/aftab.git aftab_source
pip install -e aftab_source

We highly recommend using Micromamba for creating virtual environments with instructions detailed here.

Training Agents

Currently JAX API is under development and is planned to be finished by the end of 2026. Contributions are highly encouraged.

from aftab import Aftab
from aftab import aftab_environments

seeds = [1, 2, 3, 4]

for environment in aftab_environments:
    agent = Aftab(encoder="gamma", frames="pilot")
    for seed in seeds:
        agent.train(environment=environment, seed=seed)
        agent.log()

Custom Encoder Injection

You can define your own encoder as a PyTorch module and pass it to the agent:

import torch
from aftab import Aftab

class CustomImageEncoder(torch.nn.Module):
    pass

agent = Aftab(encoder=CustomImageEncoder)

Results

All experimental results are organized by experiment category. Each section contains:

  • Tables: numerical results (HNS/PHS and raw scores)
  • Charts: IQM normalized scores and training curves

Encoder Experiments

Tables

Charts

Hadamax Experiments

Tables

Charts

Q-Value Experiments

Tables

Charts

Procgen Experiments

Tables

Charts

Model Complexity

Base Variants

Variant Encoder Parameters Regression Head Parameters Total Parameters Encoder FLOPs Regression Head FLOPs Total FLOPs
PQN 78,304 1,686,500 1,764,804 7.734 1.610 9.347
Alpha 174,752 1,782,948 1,957,700 27.541 1.610 29.151
Beta 89,008 1,782,948 1,871,956 61.515 1.610 63.126
Gamma 117,168 1,725,364 1,842,532 22.901 1.610 24.512
Delta 78,552 1,850,588 1,929,140 6.143 1.774 7.917
Epsilon 80,112 2,179,828 2,259,940 13.252 2.101 15.354
Zeta 77,232 2,537,396 2,614,628 25.362 2.462 27.824
Eta 78,400 23,739,460 23,817,860 28.422 23.663 52.085
Theta 76,288 1,127,428 1,203,716 9.065 1.053 10.118

Note: The Eta variant has significantly more parameters than other variants, primarily due to the encoder producing a large number of features.

Hadamax Variants

Variant Encoder Parameters Regression Head Parameters Total Parameters Encoder FLOPs Regression Head FLOPs Total FLOPs
Hadamax 156,608 3,968,516 4,125,124 159.014 3.969 162.984
Gamma-Hadamax-Valid 234,336 1,609,220 1,843,556 122.001 1.610 123.611
Gamma-Hadamax-Same 234,336 3,280,388 3,514,724 129.300 3.281 132.581

Hyperparameters

The following tables reflect the defaults defined by Aftab. The experiment_name argument is required and has no default.

Training and Environment

Hyperparameter (Aftab argument) Default
Encoder (encoder) Gamma-Hadamax-Valid
Network (network) Distributional Bootstrapped (Ensemble) Dueling
Total frames (frames) 200,000,000
Frame skip (frame_skip) 4
Frame stack (frame_stack) 4
No-op maximum (noop) 30
Learning rate (lr) $2.5 \times 10^{-4}$
Training environments (train_environments) 128
Test environments (test_environments) 8
Steps per update (steps_per_update) 32
Batch size (derived) 4,096
Mini-batches (mini_batches) 32
Mini-batch size (derived) 128
Discount factor ($\gamma$) 0.99
Return $\lambda$ (return_lambda) 0.65
Epochs (epochs) 2
Gradient norm (gradient_norm) 10.0
Embedding dimension (embedding_dimension) 512
Training episodic life (train_episodic_life) True
Test episodic life (test_episodic_life) False
Training reward clipping (train_reward_clip) True
Test reward clipping (test_reward_clip) True
Epsilon Schedule Linear
Epsilon Annealing Ratio 10%

Optimizer

Hyperparameter (Aftab argument) Default
Optimizer (optimizer) Rectified Adam ("radam")
Epsilon (optimizer_epsilon) $1 \times 10^{-5}$
Weight decay (optimizer_weight_decay) 0.0
$\beta_1$ (optimizer_first_beta) 0.9
$\beta_2$ (optimizer_second_beta) 0.999

Distributional and Bootstrapped (Ensemble) Q-Values

Hyperparameter (Aftab argument) Default
Distributional bins (distributional_bins) 51
Distributional minimum (distributional_min_value) -10.0
Distributional maximum (distributional_max_value) 10.0
Distributional sigma (distributional_sigma) None (derived from the sigma ratio)
Distributional sigma ratio (distributional_sigma_ratio) 0.75
Distributional value clip (distributional_value_clip) 0.0
Bootstrap heads (bootstrap_heads) 10
Bootstrap probability (bootstrap_probability) 1.0

Procgen Overrides

Hyperparameter Default Procgen
Training environments 128 64 (procgen_train_environments)
Steps per update 32 256 (procgen_steps_per_update)
Batch size 4,096 16,384
Mini-batch size 128 512

For Procgen environments, Aftab automatically applies the two overrides above; other defaults remain unchanged.

Statistical Significance

Encoder Experiments

Wilcoxon Signed Rank Test Wilcoxon Signed Rank Test (Corrected)
Probability of Improvement

Hadamax Experiments

Wilcoxon Signed Rank Test Wilcoxon Signed Rank Test (Corrected)
Probability of Improvement

Q-Value Experiments

Wilcoxon Signed Rank Test Wilcoxon Signed Rank Test (Corrected)
Probability of Improvement

Reproducibility

Due to the stochastic nature of deep reinforcement learning, exact reproducibility via fixed datasets is not feasible.
Instead, we provide a set of random seeds used in our experiments.

from aftab import aftab_seeds

print(aftab_seeds)

Full experiment replication:

from aftab import Aftab
from aftab import aftab_environments
from aftab import aftab_seeds

for environment in aftab_environments:
    agent = Aftab()
    for seed in aftab_seeds:
        agent.train(environment=environment, seed=seed)
        agent.log()

A comprehensive set of Atari environments is available via EnvPool:
https://envpool.readthedocs.io/en/latest/env/atari.html#available-tasks

Procgen environments use their native RGB observations with shape (3, 64, 64). Aftab reads each task's EnvPool configuration and only applies supported options. Atari-only options such as noop, frame_skip, frame_stack, train_episodic_life, and EnvPool reward clipping are therefore not passed to Procgen.

A comprehensive set of Procgen environments is available via EnvPool:

https://envpool.readthedocs.io/en/latest/env/procgen.html#available-tasks

Hardware

Nvidia A40 GPUs were used to run all the experiments in this experiment.

Specification Details
GPU Memory 48 GB GDDR6 with error-correcting code (ECC)
GPU Memory Bandwidth 696 GB/s
Interconnect NVIDIA NVLink 112.5 GB/s (bidirectional); PCIe Gen4: 64 GB/s
NVLink 2-way low profile (2-slot)
Display Ports 3x DisplayPort 1.4*
Max Power Consumption 300 W
Form Factor 4.4" (H) x 10.5" (L), Dual Slot
Thermal Passive
vGPU Software Support NVIDIA Virtual PC, NVIDIA Virtual Applications, NVIDIA RTX Virtual Workstation, NVIDIA Virtual Compute Server, NVIDIA AI Enterprise
vGPU Profiles Supported See the Virtual GPU Licensing Guide
NVENC / NVDEC 1x / 2x (includes AV1 decode)
Secure Boot Secure and Measured Boot with Hardware Root of Trust (optional)
NEBS Ready Level 3
Power Connector 8-pin CPU

Citation

@article{aftab2026drl,
  title={Aftab: A Comprehensive Benchmark of CNN Encoders and Advanced Value Functions in Parallelized Q-Networks},
  author={Shieenavaz, Taha and Zareshahraki, Shabnam and Nanni, Loris},
  journal={arXiv preprint arXiv:YYMM.NNNNN},
  year={2026}
}

Related Works

@misc{2407.04811,
  Title = {Simplifying Deep Temporal Difference Learning},
  Author = {Matteo Gallici and Mattie Fellows and Benjamin Ellis and Bartomeu Pou and Ivan Masmitja and Jakob Nicolaus Foerster and Mario Martin},
  Year = {2024},
  Eprint = {arXiv:2407.04811},
}

@misc{2403.03950,
  Title = {Stop Regressing: Training Value Functions via Classification for Scalable Deep RL},
  Author = {Jesse Farebrother and Jordi Orbay and Quan Vuong and Adrien Ali Taïga and Yevgen Chebotar and Ted Xiao and Alex Irpan and Sergey Levine and Pablo Samuel Castro and Aleksandra Faust and Aviral Kumar and Rishabh Agarwal},
  Year = {2024},
  Eprint = {arXiv:2403.03950},
}

@misc{1511.06581,
  Title = {Dueling Network Architectures for Deep Reinforcement Learning},
  Author = {Ziyu Wang and Tom Schaul and Matteo Hessel and Hado van Hasselt and Marc Lanctot and Nando de Freitas},
  Year = {2015},
  Eprint = {arXiv:1511.06581},
}

@misc{1806.04613,
  Title = {Improving Regression Performance with Distributional Losses},
  Author = {Ehsan Imani and Martha White},
  Year = {2018},
  Eprint = {arXiv:1806.04613},
}

@misc{1602.04621,
  Title = {Deep Exploration via Bootstrapped DQN},
  Author = {Ian Osband and Charles Blundell and Alexander Pritzel and Benjamin Van Roy},
  Year = {2016},
  Eprint = {arXiv:1602.04621},
}

Useful Links

Font

The Vazirmatn font is used for both Persian and English text throughout the GitHub repository header and the project's landing page.

GitHub | Google Fonts

License

© 2025 Taha Shieenavaz.
Licensed under CC BY-NC 4.0: https://creativecommons.org/licenses/by-nc/4.0/

Project links

Data verified by PyPI on Aug 15, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for underdash from gravatar.com underdash

Credits

Author: Taha Shieenavaz

GitHub Statistics

Data verified by PyPI on Aug 15, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (Creative Commons Attribution-NonCommercial 4.0 International)

Requires

Python >=3.9

Tags

reinforcement learning dqn aftab drl pqn deep reinforcement learning cnn

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

aftab-1.0.5.tar.gz (61.4 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

aftab-1.0.5-py3-none-any.whl (68.4 kB view details)

Uploaded Python 3

File details

Details for the file aftab-1.0.5.tar.gz.

File metadata

  • Download URL: aftab-1.0.5.tar.gz
  • Upload date:
  • Size: 61.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for aftab-1.0.5.tar.gz
Algorithm Hash digest
SHA256 656d21c5cb47cacda79b86b7c49887e149e160799205712380fd0c30ddb2a4c6
MD5 d3a1910ca5e33c6299680a8f6d99b32b
BLAKE2b-256 59ff139f95e82af51f25205e50eb0961f5da387aff762d54a806b99ffd2b9a55

See more details on using hashes here.

Provenance

The following attestation bundles were made for aftab-1.0.5.tar.gz:

Publisher: publish.yaml on tahashieenavaz/aftab

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file aftab-1.0.5-py3-none-any.whl.

File metadata

  • Download URL: aftab-1.0.5-py3-none-any.whl
  • Upload date:
  • Size: 68.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for aftab-1.0.5-py3-none-any.whl
Algorithm Hash digest
SHA256 eb7557524966ad18ac0b9d696b6a740f960a85c5e078e94f3911eaf284db46e0
MD5 f5b1c4beb7420d25109b13bb8bf05f72
BLAKE2b-256 eb8922aa3b7423b7bda7bc0ff3cf6148d7e1189f95b3b67d9fcb21a79ba53495

See more details on using hashes here.

Provenance

The following attestation bundles were made for aftab-1.0.5-py3-none-any.whl:

Publisher: publish.yaml on tahashieenavaz/aftab

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

1.0.5 This release

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

0.1.61

2 files

0.1.60

2 files

0.1.59

2 files

0.1.58

2 files

0.1.57

2 files

0.1.56

2 files

0.1.55

2 files

0.1.54

2 files

0.1.53

2 files

0.1.52

2 files

0.1.51

2 files

0.1.50

2 files

0.1.49

2 files

0.1.48

2 files

0.1.47

2 files

0.1.46

2 files

0.1.45

2 files

0.1.44

2 files

0.1.43

2 files

0.1.42

2 files

0.1.41

2 files

0.1.40

2 files

0.1.39

2 files

0.1.38

2 files

0.1.37

2 files

0.1.36

2 files

0.1.35

2 files

0.1.34

2 files

0.1.33

2 files

0.1.31

2 files

0.1.30

2 files

0.1.29

2 files

0.1.28

2 files

0.1.27

2 files

0.1.26

2 files

0.1.25

2 files

0.1.24

2 files

0.1.23

2 files

0.1.22

2 files

0.1.21

2 files

0.1.20

2 files

0.1.19

2 files

0.1.18

2 files

0.1.16

2 files

0.1.15

2 files

0.1.14

2 files

0.1.13

2 files

0.1.12

2 files

0.1.11

2 files

0.1.1

2 files

0.1.0

2 files

0.0.99

2 files

0.0.97

2 files

0.0.96

2 files

0.0.95

2 files

0.0.94

2 files

0.0.93

2 files

0.0.92

2 files

0.0.91

2 files

0.0.90

2 files

0.0.89

2 files

0.0.88

2 files

0.0.87

2 files

0.0.86

2 files

0.0.85

2 files

0.0.84

2 files

0.0.83

2 files

0.0.82

2 files

0.0.81

2 files

0.0.80

2 files

0.0.79

2 files

0.0.78

2 files

0.0.77

2 files

0.0.76

2 files

0.0.75

2 files

0.0.74

2 files

0.0.73

2 files

0.0.72

2 files

0.0.71

2 files

0.0.70

2 files

0.0.69

2 files

0.0.68

2 files

0.0.67

2 files

0.0.66

2 files

0.0.65

2 files

0.0.64

2 files

0.0.63

2 files

0.0.62

2 files

0.0.61

2 files

0.0.60

2 files

0.0.59

2 files

0.0.58

2 files

0.0.57

2 files

0.0.56

2 files

0.0.55

2 files

0.0.54

2 files

0.0.53

2 files

0.0.52

2 files

0.0.51

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page