Project description

Pytorch-Dataset

A PyTorch Code Dataset for Cutting-Edge Fine-tuning

Installation

You can install the package using pip

pip install pytorch-dataset

Usage

Downloader that downloads and unzips each repository in an account

from pytorch import GitHubRepoDownloader

downloader = GitHubRepoDownloader(username="lucidrains", download_dir="lucidrains_repositories")
downloader.download_repositories()

Processor that cleans, formats, and submits the cleaned dataset to huggingface

from pytorch import CodeDatasetBuilder

code_builder = CodeDatasetBuilder("lucidrains_repositories")
code_builder.save_dataset("lucidrains_python_code_dataset")
code_builder.push_to_hub("lucidrains_python_code_dataset", organization="kye")

License

MIT

Project details

These details have not been verified by PyPI

Project links

Release history Release notifications | RSS feed

This version

0.0.4

Sep 15, 2023

0.0.3

Sep 15, 2023

0.0.1

Sep 6, 2023

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pytorch_dataset-0.0.4.tar.gz (4.6 kB view details)

Uploaded Sep 15, 2023 Source

Built Distribution

pytorch_dataset-0.0.4-py3-none-any.whl (5.9 kB view details)

Uploaded Sep 15, 2023 Python 3

File details

Details for the file pytorch_dataset-0.0.4.tar.gz.

File metadata

Download URL: pytorch_dataset-0.0.4.tar.gz
Upload date: Sep 15, 2023
Size: 4.6 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: poetry/1.3.2 CPython/3.11.0 Darwin/22.4.0

File hashes

Hashes for pytorch_dataset-0.0.4.tar.gz
Algorithm	Hash digest
SHA256	`376f68737a66b4cb67b4446ab8c415206e22596f6023e8bac8537669ff2b1b65`
MD5	`5e42fb06c6c7970ab5a6d0c00063a081`
BLAKE2b-256	`128c0ccc030c799d5e8f93cd4cc4116061649575938a14237ef0765ec0cb895a`

See more details on using hashes here.

File details

Details for the file pytorch_dataset-0.0.4-py3-none-any.whl.

File metadata

Download URL: pytorch_dataset-0.0.4-py3-none-any.whl
Upload date: Sep 15, 2023
Size: 5.9 kB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: poetry/1.3.2 CPython/3.11.0 Darwin/22.4.0

File hashes

Hashes for pytorch_dataset-0.0.4-py3-none-any.whl
Algorithm	Hash digest
SHA256	`41b98a61c71ca72d772c9d783ff851b8de14f4773fe1579ec9952cfa7daeed6a`
MD5	`78f3644d7a2f24c23a2882367d27a306`
BLAKE2b-256	`743058bc378d038083c0b0499b5f23285faf17ffbd2aa367d76dd5310244238c`