magi-dataset 1.0.4

Convenient access to massive corpus of GitHub repositories

MAGI Dataset

Install

pip install magi_dataset

If you plan on using magi_dataset to periodically crawl data, set the following variables in your environment:

export GH_TOKEN="Your token"

Read Creating a personal access token for more information on creating GitHub personal access token. If using the default data without crawling new data, you may safely ignore this token.

Usage

The recommended way to use magi_dataset is to run the collection process in chunked mode. First create an empty dataset and initiate index from GitHub:

</code></pre>
<p>Initialize an empty instance and collect data:</p>
<pre><code class="language-python">>>> from magi_dataset import GitHubDataset

>>> github_dataset = GitHubDataset(
...     empty = True
... )

github_dataset.init_repos(fully_initialize=True)

Download default data (not guranteed to be the newest):

>>> from magi_dataset import GitHubDataset

>>> github_dataset3 = GitHubDataset(
...	    empty = False
... )

The default data may be found at Enoch2090/github_semantic_search on HuggingFace. We will update the data periodically.

After the dataset is created, access the data with either number index:

>>> github_dataset[5]
GitHubRepo(name='ytdl-org/youtube-dl', stars=114798, description='Command-line program to download videos from YouTube.com and other video sites', _fully_initialized=True)

Or the full name:

>>> github_dataset['ytdl-org/youtube-dl']
GitHubRepo(name='ytdl-org/youtube-dl', stars=114798, description='Command-line program to download videos from YouTube.com and other video sites', _fully_initialized=True)

And you can access the corpus by accessing the readme and hn_comments attributes of GitHubRepo objects.

>>> github_dataset[5].readme[0:100]
'[![Build Status](https://github.com/ytdl-org/youtube-dl/workflows/CI/badge.svg)](https'

Future Works

• The current idle handler design is primordial, will switch to async pipelines to relieve CPU sleep time.
• Elasticsearch database builder
• Pinecone database builder (wrapper only)
• Hash verification of persistence files