GH-INFERENCE

Text in, text out. Quantized models run on GitHub servers.

Your Python client only connects, submits prompts, and retrieves answers. Model downloads, model loading, inference, and optional prompt refinement all happen inside GitHub Actions. No model runtime or GPU is required on your machine.

Start

Install the client in your environment (Python 3.9+):

pip install gh-inference
export GH_AI_TOKEN="your-github-token"

Use a classic GitHub token with repo and workflow scopes. Do not commit it.

from gh_inference import Agent

agent = Agent()
print(agent.run("Explain recursion briefly."))

That is the whole flow. The first call automatically creates or reuses the ai-inference-runner repository, installs the workflow, starts a GitHub worker, and returns its answer as a plain string. agent("prompt") is equivalent. You can also pass Agent(github_token=...) instead of setting the environment.

agent.connect() optionally prepares the repository ahead of the first prompt; it is never required. Connecting does not start or reserve a model worker.

Models and every published GGUF variant

The 16 pinned aliases below remain convenient defaults. Model support is open: use any public, ungated Hugging Face GGUF repository supported by the installed llama.cpp runtime. Search discovers Qwen, Llama, Gemma, Phi, Mistral, DeepSeek, SmolLM and other families, with no project parameter-count ceiling. A model being discoverable does not mean it fits your runner or its architecture is supported.

from gh_inference import Agent, search_models, list_quantizations

print(search_models("Qwen3"))
print(list_quantizations("Qwen/Qwen2.5-0.5B-Instruct-GGUF"))
agent = Agent(model="Qwen/Qwen2.5-0.5B-Instruct-GGUF", quantization="Q8_0")
print(agent("Explain quantization in one sentence."))

quantization defaults to Q4_K_M for repository selections. All published GGUF labels can be selected, including Q2/Q3/Q4/Q5/Q6/Q8, K-quants, IQ, TQ and floating point variants when provided and supported by your backend. We do not manufacture missing variants or convert weights. AWQ/GPTQ/Safetensors require other backends and are not accepted by this GGUF CPU runner. Model licenses still apply.

For ambiguous repositories use Agent(model="owner/repo", filename="exact.gguf"). Split GGUF files are grouped, all shards are checked and downloaded, and the first shard is loaded. Incomplete shard groups are excluded. list_quantizations lists exact filenames and aggregate bytes; projector files are excluded (text only).

Selections resolve metadata once per client process and pin a Hub commit plus each file's SHA-256 and size. Workers verify downloads before loading. Subsequent jobs reuse the Actions weights cache when eligible; run_many() keeps one model resident in RAM throughout that job. An Actions job ending releases its RAM. Use revision="commit-or-tag" for reproducibility; clear_model_cache() refreshes cached branch metadata. Hub metadata calls download no model weights locally.

Context defaults to about 8K (or a known smaller ceiling). Existing aliases retain their documented limits; arbitrary repositories use Hub GGUF context_length when supplied. If absent, the advertised ceiling is unknown, not invented. RAM estimates for arbitrary models are approximate; architecture and KV cache shape affect actual usage. Limits and Agent.experimental() below remain available, but cannot create RAM, extend GitHub quotas, or add backend support.

gh-inference models Qwen3
gh-inference variants Qwen/Qwen2.5-0.5B-Instruct-GGUF
gh-inference run "What is recursion?" --model qwen0.5b --track-time

For source development, pip install -e '.[dev]' installs this checkout. The legacy gh_ai_runner import remains supported; new code can use gh_inference.

Publishing

The publish.yml workflow builds and tests the wheel before publishing to PyPI using GitHub OIDC, with no stored PyPI token. Configure the PyPI trusted publisher for owner TanishC4444, repository GH-INFERENCE, workflow publish.yml, and environment pypi, then run it with publish=true or publish a v* release. You can also publish by updating .github/release.json on main to the package version; the workflow checks that they match before building and uploading. Each version is immutable on PyPI; bump gh_ai_runner.__version__ for a release.

New repositories are private by default. Existing visibility is never changed. Actions must be enabled. Private Actions usage consumes included minutes and can incur charges. Use a dedicated repository: the package manages and can overwrite run_inference.py and .github/workflows/inference.yml there.

Experiments: remove project ceilings

from gh_ai_runner import Agent

agent = Agent.experimental(
    model="qwen0.5b",
    n_ctx=65536,          # Deliberately above this model's advertised 32K.
    max_tokens=None,     # Ask the runtime to use remaining context for the answer.
    track_time=True,
)
print(agent.run("Explain context windows."))
print(agent.last_result.usage)

Agent.experimental() uses Limits.unrestricted(), trim_history=False, and max_history_turns=None. It removes our advertised model/output checks, estimated RAM/disk rejection, temperature bounds, prompt-character cap, batch-count cap, formatting reserves, cache-size threshold, and project workflow-input budget. Default context remains 8K (2K for TinyLlama) unless you set n_ctx; default answer length remains 512 unless you change max_tokens. All computation stays on GitHub.

This permits an attempt; it does not guarantee that the backend can execute it. A bigger n_ctx allocates a bigger window. Sending more messages alone does not. Going beyond a model's trained/advertised length may require positional scaling and can produce poor answers. Disabling memory checks cannot create memory.

Edit individual rules

from gh_ai_runner import Agent, Limits

limits = Limits.unrestricted(
    check_resources=True,
    max_prompts=100,
    max_cached_model_gb=6,
    job_timeout_minutes=90,
)
agent = Agent.experimental(limits=limits, n_ctx=16384, max_tokens=2048)

Limits() preserves standard checks. Limits.unrestricted(**overrides) disables project ceilings and applies your overrides. Both accept every field below. Pass the configuration to Agent, GitHubAIRunner, ai_call, or an individual InferenceRequest/submit(); request settings override runner defaults. Configurations are immutable: make a new one or use dataclasses.replace().

The unrestricted preset sets the three enforcement flags to False; optional prompt, workflow-budget, batch, temperature, and cache-size limits to None; formatting reserves to 0; and remote timeout to 360. Other defaults remain. The experimental agent factory additionally disables trimming and turn-count caps. Explicit arguments to the factory override those choices.

Model loading and sampling

agent = Agent.experimental(
    n_ctx=65536,
    max_tokens=4096,
    model_options={
        "n_threads": 2,
        "n_batch": 256,
        "rope_scaling_type": "yarn",
        "rope_freq_scale": 0.5,
        "yarn_orig_ctx": 32768,
        "yarn_ext_factor": 1.0,
    },
    generation_options={"top_k": 40, "top_p": 0.9, "repeat_penalty": 1.1, "seed": 42},
)

model_options forwards JSON-compatible keywords to llama_cpp.Llama, including CPU threads, batch sizes, KV settings, and positional scaling. rope_scaling_type accepts a backend integer enum or a name such as "yarn" or "linear". These are experimental settings, not verified 64K quality claims. Increasing context does not retrain the weights. Qwen 7B/14B still apply their published factor-4 YaRN recipe above 32K unless you override it. Beyond advertised lengths, choose and evaluate scaling yourself. Other entries keep GGUF metadata.

generation_options forwards sampling keywords to create_chat_completion, including top_k, top_p, min_p, penalties, seed, and stop. It applies to main answers; refinement uses separate deterministic sampling. Unknown keywords are rejected by the installed backend. Use named n_ctx, max_tokens, and temperature arguments for those values. Extra-option dictionaries cannot replace the managed model path, CPU/RAM placement, messages, or nonstreaming text contract.

max_tokens=None requests remaining context. Standard model-limit enforcement still applies a recorded publisher output ceiling; unrestricted mode does not. The runtime can shorten an explicit answer allowance to remaining space and stop early at EOS. Input exceeding allocated n_ctx is rejected by llama-cpp-python. Custom policies/options are sent to the worker and covered by request checksums.

Connection and waiting

from gh_ai_runner import Agent, GitHubAIRunner, Limits

runner = GitHubAIRunner(
    limits=Limits.unrestricted(job_timeout_minutes=90),
    completion_timeout=5500,
    run_discovery_timeout=300,
    poll_interval=10,
    http_timeout=60,
    artifact_timeout=120,
    workflow_registration_timeout=120,
)
agent = Agent.experimental(runner=runner, limits=runner.limits)

Runner defaults, all in seconds: completion_timeout=1800, run_discovery_timeout=180, poll_interval=8, http_timeout=30, artifact_timeout=60, workflow_registration_timeout=60. Values must be finite and positive; there is no project upper bound. HTTP timeout applies per API request. Discovery/completion/artifact deadlines apply to their own phases. agent.run(prompt, timeout=...) bounds waiting/result retrieval after dispatch, not repository preparation or model execution. A client timeout does not cancel the remote job. Increase client waiting alongside longer remote job timeouts.

External limits remain

• GitHub workflow inputs have a 65,535-character service limit including history, configuration, escaping, and JSON overhead. Characters are not tokens. This version has no alternate large-prompt transport, even when model context is 128K or larger.
• GitHub Actions limits include the six-hour hosted-job maximum, account minutes, concurrency, storage, and API limits. A larger configured timeout cannot extend the service maximum.
• Real RAM/disk, CPU time, backend numeric ranges, supported architectures and scaling, and model quality remain constraints. Allocation, disk-full, timeout, and worker-termination failures are possible when resource checks are disabled.

The curated 16B catalog is not a mathematical parameter-count cap; larger entries need verified GGUF download metadata and appropriate workers. Valid input types, checksums, and job identity checks remain required in both presets.

Choose a model or role

agent = Agent(model="qwen", instructions="You are a concise coding assistant.")
answer = agent.run("Write a function that merges overlapping intervals.")

The default qwen0.5b is a roughly 491 MB Q4_K_M starter model. Larger models can provide better answers but take more download time, RAM, and CPU time.

All catalog entries are prequantized GGUF files. Their sizes and SHA-256 hashes are recorded in models.py; new downloads must match both before loading.

Reuse one loaded model

answers = agent.run_many([
    "Explain a Python list.",
    "Compare it with a tuple.",
    "Give a small example using both.",
])
for answer in answers:
    print(answer)

By default, up to 32 prompts run sequentially in one GitHub job. Change Limits.max_prompts to raise/remove that cap. The full quantized model loads into CPU RAM once and remains loaded for all those prompts and refinement steps. Memory mapping and GPU offloading are disabled; memory locking is requested, subject to the worker OS's permissions.

Separate run() calls create separate workers. Hosted GitHub workers cannot keep RAM between jobs. Group prompts with run_many() to avoid repeated model loading. GitHub caches downloaded weights and the Python environment across jobs, although caches can be evicted. Model cache keys include the expected weight checksum. Setup prefers official prebuilt CPU packages so a new worker normally avoids compiling the inference runtime; source compilation is a fallback.

By default, files at least 6 GB skip weight caching because uploading their archive can need another model-sized copy on the worker's limited disk. They still load once per run_many() job, but a new job downloads them again. Set max_cached_model_gb=None to permit all sizes, or cache=False on Agent to skip weight caching. Unrestricted mode permits all sizes; cache archives can still fail if the worker's disk or GitHub cache storage is too small.

Timing

agent = Agent(track_time=True)
print(agent.run("What is a B-tree?"))
print(agent.last_result.timings)

The return value stays plain text. Optional details are in agent.last_result and agent.last_results: token usage, download/load/generation timing, weight-cache hits, and within-job model reuse. agent.last_job exposes the Actions URL and low-level status/cancellation handle.

total_seconds measures worker time for that prompt, including shared setup on the first prompt. remote_seconds measures the full client operation, including repository preparation, queuing, waiting, and result retrieval. For run_many(), the whole-job remote duration appears on every result; do not sum it. Override tracking per call with agent.run(prompt, track_time=True).

Optional prompt refinement

agent = Agent(refine_prompt=True, track_time=True)
print(agent.run("Your detailed task and constraints ..."))
print(agent.last_result.refinement.tokens_saved)

Refinement runs on the same loaded model on GitHub, not a separate service or second model. Short prompts are skipped. A rewrite is accepted only when it uses fewer main-model tokens, finishes normally, and preserves protected numbers, URLs, double-quoted literals, and code. Otherwise the original prompt is retained.

It can reduce the main generation's input processing, but adds its own tokens and time. Total usage includes that work; last_result.refinement.usage reports it separately. Shorter input does not shrink model weights or automatically reduce the allocated context cache. Literal checks do not guarantee unchanged meaning.

Conversation and limits

Conversation is remembered on the agent object and passed to subsequent GitHub jobs. run_many() manages it within the same worker. No prompt/output history is automatically written to disk by Agent. Use remember=False for independent prompts and agent.reset() to forget previous turns.

Older conversation turns are dropped as needed to leave room for the new prompt and answer. Current prompt text is never silently truncated. The main options are:

Context includes instructions, history, formatting, and the answer. Worker tokenization supplies actual usage; context planning includes a conservative formatting allowance. The model can stop before reaching max_tokens.

Context ceilings

The old blanket 8,192-context and 4,096-output caps are removed. Standard starting windows are 8,192 total tokens, except TinyLlama, whose supported maximum is only 2,048. These advertised ceilings are enforced by standard policy and bypassable with enforce_model_limits=False. They describe support, not a promise that a worker has enough RAM or CPU time to use them. The catalog includes each limit's publisher source in MODELS[key]["context_source"].

— means no separate output ceiling is recorded; the available total context still bounds generation. It does not mean unlimited or tested long-form output. Sources include the Qwen 0.5B model card, Qwen 7B long-context recipe, Gemma 2 model card, and the individual publisher links in the catalog.

agent = Agent(model="qwen0.5b", n_ctx=32768, max_tokens=8192)
print(agent.run("Explain the design in detail."))

The answer allowance shares the window with instructions, history, and your prompt. Increasing context increases RAM usage even if the current prompt is short. The worker automatically enables Qwen's factor-4 YaRN settings above 32,768 tokens for the 7B/14B Qwen and Qwen Coder entries. Other models retain the scaling encoded in their GGUF files. Long-context quality and peak RAM are not verified at every ceiling.

When a conversation grows past its window

A 32K model can participate in a conversation totaling 64K tokens across many calls. It can only receive one window's worth on any given call. This agent keeps the instructions and current prompt, reserves the requested answer allowance, and removes the oldest complete user/assistant pairs until the request fits. It does not automatically summarize or retrieve the removed messages. Details that appeared only in those messages are no longer available to the model.

The default history cap is also six previous exchanges, even when context has space. Increase max_history_turns or set it to None to retain more exchanges; the token budget and workflow transport can still trim them earlier. n_ctx=32768 must be set explicitly to use a 32K window instead of the 8K default.

agent = Agent(model="qwen0.5b", n_ctx=32768, max_history_turns=32)
print(agent.run("Continue our discussion."))
print(agent.last_result.dropped_history_turns)  # history removed to fit context/transport

With the default trimming policy, one oversized current prompt is rejected, rather than cut silently. A 64K-token prompt cannot fit an allocated 32K window. Standard policy rejects allocations above the advertised maximum; experimental mode allows those attempts to reach the runtime.

Experiment without automatic history removal

agent = Agent(
    model="qwen0.5b",
    n_ctx=8192,             # Try 16384 or 32768 in a separate experiment.
    trim_history=False,
    track_time=True,
)
print(agent.run("Remember this fact: my project is called Cedar."))
print(agent.run("What is my project called?"))
print(agent.last_result.usage)

trim_history=False disables all automatic history removal: the exchange-count cap, token-budget trimming on GitHub, and client-side transport trimming. max_history_turns is ignored in this mode. Completed exchanges stay on the agent until you call reset() or discard the object; remember=False still disables conversation memory entirely. This does not restore messages removed earlier.

The full retained history is passed to llama-cpp-python without our pre-generation fit check. It does not enlarge n_ctx, bypass model ceilings or RAM checks, or promise that the runtime accepts overflow. The runtime may reject an oversized input or shorten generation to the remaining context. Use Agent.experimental() or custom Limits to bypass the separate advertised-ceiling/resource policies. Check the Actions job/logs via agent.last_job.actions_url if a call fails. A failed call does not replace the client's previously retained conversation or append its failed prompt.

GitHub's workflow-input transport limit still applies. When it is exceeded, the call fails instead of dropping history, before starting a new worker. Setting a 32K context and sending 64K of messages therefore does not create 64K memory. For controlled comparisons, start fresh agents with the same prompts, disable refinement (the default), and vary n_ctx while keeping answer length fixed.

Standard policy limits prompt plus instructions to 60,000 characters, and the complete workflow-input payload must fit GitHub's limit. Oversized jobs are rejected before provisioning. These are separate transport limits, not token limits: a model can allow 128K context while this version cannot transmit a full 128K-token prompt or history through workflow inputs. With trim_history=True, history may be trimmed to satisfy transport before it reaches the model. Workflow timeout and client completion wait default to 30 minutes; both are editable as described above. A client timeout does not automatically cancel a job; use agent.last_job when you need to inspect or cancel it.

Tools, file access, Python callbacks, and shell execution are deliberately out of scope. This version is text in, text out.

Larger workers

The default client budget is 8 GB RAM and 14 GB workspace disk. GitHub lists 8 GB RAM for standard private Linux workers and 16 GB for public ones. See GitHub runner specifications.

from gh_ai_runner import Agent, GitHubAIRunner

runner = GitHubAIRunner(
    runner_label="your-linux-32gb-runner",
    runner_ram_gb=32,
    runner_disk_gb=40,
)
agent = Agent(model="deepseek-coder16b", runner=runner)
print(agent.run("Explain this algorithm."))

The runner label must already be available to your repository; changing budgets does not purchase hardware. To deliberately create a public standard runner, configure a new repo_name, private=False, and runner_ram_gb=16. Existing visibility is never changed automatically.

RAM/disk estimates are checked before dispatch and available resources are checked again on the worker. Large CPU models can be slow even when they fit.

Quantization shrinks weight storage, not the parameter count. Idealized 16B weights take 32 GB at 16 bits or 8 GB at 4 bits; actual Q4_K_M mixes precisions and includes scales/metadata. The catalog's 16B file is 10.36 GB. RAM also needs context/KV cache and computation buffers. See llama.cpp quantization.

Existing APIs

ai_call(token, prompt, ...) remains a blocking text-returning helper. Explicit track_time=True returns an InferenceResult with .output and .timings. GitHubAIRunner() also reads GH_AI_TOKEN automatically and retains submission, batch/comparison, polling, cancellation, retry, resume, and hosted-provider APIs.

submit_batch() launches independent workflows; Agent.run_many() reuses one worker/model. Only failed workflow conclusions are automatically retried when requested by RetryPolicy, avoiding duplicate work after uncertain dispatches.

Privacy and verification

Prompts, history, and answers are sent to GitHub. Results are Actions artifacts, retained for three days. Public repository permissions and Actions logs matter. Checksums verify job identity, requests, output, and structured agent results; they are not signatures protecting against a compromised worker/repository.

Test/build CI runs on GitHub. The manual GitHub model smoke test workflow also downloads a real small model and checks that two prompts reuse it in RAM. Client orchestration tests use simulated API/model responses. No model runtime is installed by this package on the client.

MIT license. See LICENSE.