self-play-pretrain-zero-data

Self-Play Pretraining with Zero Data (wip)

Implementation of Self-Play Pretraining with Zero Data

Usage

import torch
from self_play_pretrain_zero_data import Brainfuck, SelfPlay, Transformer

executor = Brainfuck()

generator = Transformer(num_tokens = executor.num_tokens, dim = 512, depth = 6)
learner = Transformer(num_tokens = 256 + 1, dim = 512, depth = 6)

self_play = SelfPlay(generator, learner, executor)
self_play(epochs = 10)

torch.save(learner.state_dict(), './learner.pt')

Citations

@misc{cowsik2026selfplaypretrainingzerodata,
    title    = {Self-Play Pretraining with Zero Data},
    author   = {Aditya Cowsik and Kfir Dolev and Michael Y. Li and G. Bruno De Luca and Nourya Cohen and Noah D. Goodman and Yoav Levine},
    year     = {2026},
    eprint   = {2609.30063},
    archivePrefix = {arXiv},
    primaryClass = {cs.AI},
    url      = {https://arxiv.org/abs/2609.30063},
}

@software{Morehead_JVP_Flash_Attention_2025,
    author  = {Morehead, Alex},
    doi     = {10.5281/zenodo.17050188},
    license = {MIT},
    month   = {sep},
    title   = {JVP Flash Attention},
    url     = {https://github.com/amorehead/jvp_flash_attention},
    version = {0.14.0},
    year    = {2025}
}

@misc{bloem2025universalpretrainingiteratedrandom,
    title   = {Universal pre-training by iterated random computation},
    author  = {Peter Bloem},
    year    = {2025},
    eprint  = {2506.20057},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url     = {https://arxiv.org/abs/2506.20057},
}

@misc{lee2026traininglanguagemodelsneural,
    title    = {Training Language Models via Neural Cellular Automata},
    author   = {Dan Lee and Seungwook Han and Akarsh Kumar and Pulkit Agrawal},
    year     = {2026},
    eprint   = {2603.10055},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url      = {https://arxiv.org/abs/2603.10055},
}

Metadata

Release files for self-play-pretrain-zero-data 0.0.21

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for self-play-pretrain-zero-data 0.0.21
File	Size	Uploaded
self_play_pretrain_zero_data-0.0.21.tar.gz	19.9 kB	Oct 5, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for self-play-pretrain-zero-data 0.0.21
File	Interpreter	ABI	Platform	Reset
self_play_pretrain_zero_data-0.0.21-py3-none-any.whl 22.8 kB Oct 5, 2026	Python 3	none	any	Details

Total release size: 42.7 kB

Release files / self_play_pretrain_zero_data-0.0.21.tar.gz

Download URL	self_play_pretrain_zero_data-0.0.21.tar.gz
Size	19.9 kB
Tags	Source
SHA-256 checksum How to use checksums	`84a3a581faa731b7ff95506e83536af67efc5e3cc2ab72427348c3e1c53cc0c3`
BLAKE2b-256 checksum How to use checksums	`da831dc46a1a6a051637fe567a70ce7811ca233aff8cad378c89fe019ae4e0e9`
Upload date	Oct 5, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.8.17`

Release files / self_play_pretrain_zero_data-0.0.21-py3-none-any.whl

Download URL	self_play_pretrain_zero_data-0.0.21-py3-none-any.whl
Size	22.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`bde68139b3cf973a7e9d440e5c3f65cbd56d51564310eab13cbdcb35d925785c`
BLAKE2b-256 checksum How to use checksums	`a171a46e9e8991ef4433593aa25cf48ba78c32efc63d590f1c8d597cfd8fbd9f`
Upload date	Oct 5, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.8.17`