pr-arena 0.2.0

PR-Arena: Arena Setting for GitHub Issue Resolver

⚔️ OpenHands PR Arena ⚔️

👐 We welcome your feedback. Feel free to fill out the google form, send an email, or open an issue on this repository. 👐

OpenHands PR Arena is a platform for evaluating and benchmarking agentic coding assistants through paired pull request (PR) generations. PR Arena enables developers to compare multiple LLMs in real-world issue resolution by presenting side-by-side pull requests and allowing users to select the better fix.

This project is built upon OpenHands GitHub Backlog Resolver and inspired by Copilot Arena, an open source AI coding assistant that provides paired autocomplete completions from different LLMs.

Follow the instruction below to setup the Arena setting for the OpenHands resolver.

Maintainer

How to Get Started with the OpenHands PR Arena GitHub App

⭐️ You can use PR Arena without an API Key for a limited time!

How to use

⭐️ Please watch the guideline video that explains how to use the OpenHands PR Arena GitHub App!

1. Install OpenHands PR Arena to your GitHub Repository
   • Go to the installation page
   • Under Repository access, select the repositories you want to install the app on
2. Navigate to one of the selected repositories
3. Label an issue with pr-arena to trigger the automated fix:
   • Open or create an issue, click Labels in the sidebar, and select pr-arena
4. Wait for the agent to resolve the issue and open the Arena (this may take 20–40 minutes)
5. Click the link in the comment to enter the Arena and choose your preferred model
6. The selected fix will be automatically submitted as a Pull Request

⭐️ Progress is updated via comments on the issue—keep an eye on them!

Privacy Notification

1. The only code we collect is the git_diff generated during issue resolution. We never access or store the entire codebase, access GitHub secrets, or release any user data.
2. Important: Installing this App will automatically add a workflow file named pr-arena-workflow.yml to your repository. This file redirects to the actual resolver workflow located at: https://github.com/neulab/pr-arena/blob/main/.github/workflows/pr-arena-resolver.yml. If you are concerned about repository workflows, we encourage you to review the resolver workflow to understand the operations it performs.
3. Do not modify the injected workflow. Any modifications will prevent it from being triggered.
4. Please install and use this app only on repositories where you consent to having code snippets (i.e., git_diff) processed by the LLM provider.
5. The following metadata is collected for research purpose:
   • User info: owner, repo, repo URL
   • Model info: user preference on model, duration of an attempt
   • Code info: agent code (git_diffs), commit hash, repository language composition

Q&A

Q. How can I track the progress?

A. The agent will automatically comment on the issue at each stage of the process:

• PR-Arena workflow triggered successfully! 🎉 ...
  • Step 0. Tips and instructions to guide you through the PR-Arena workflow.
• OpenHands started fixing the issue! You can monitor the progress [here]
  • Step 1. OpenHands begins resolving the issue. Please wait 10 ~ 20 minutes for the next comment.
• ⚔️PR-Arena is now open⚔️! You can view the proposed fixes and make a decision at [this link]
  • Step 2. The Arena is open. Click the link to review both fixes and choose your preferred one.
• PR has been created based on the fix you've selected. Please review the changes.
  • Step 3. A pull request has been created. You can now review and merge it.

Q. What happens if an error occurs?

A. If an error occurs, the agent will comment on the issue with an appropriate message. You can retry by removing the pr-arena label, waiting 5 seconds, and adding it again.

There are three types of errors:

• Agent side error: ❌ PR-Arena has failed due to the agent error. Please remove the 'pr-arena' label and add it back to retry.
• Workflow side error: ❌ PR-Arena encountered an error while ___. Please remove the 'pr-arena' label and add it back to retry.
• Timeout error: ⏱️ PR-Arena workflow has been cancelled due to exceeding the 30-minute timeout limit. This may be due to a complex task or an agent error. Please remove the 'pr-arena' label and add it back to retry.

Q. How long does the process take?

A. The time depends on the complexity of the issue. Some reasoning models may take longer to process. Typically, it should take less than 30 minutes, so please be patient!

Q. How does this affect my GitHub Actions build minutes?

A. The workflow makes API calls to our backend infrastructure where OpenHands agents run remotely. Your GitHub Actions runner only handles lightweight tasks like triggering the workflow and creating pull requests. The actual AI processing and code generation happens on our servers, so it consumes minimal GitHub Actions minutes (typically just a few minutes per issue).

Security & Permission (🔒)

This GitHub App requires the following permissions:

• Read & Write access to Issues and Pull Requests — to analyze issues and generate PRs
• Workflow execution — to trigger automated fixes via GitHub Actions
• Access to repository contents — to apply code changes and submit pull requests

No user secrets or sensitive information are stored in your repository. All sensitive operations are securely handled through our backend infrastructure.

Support

This project is an extension of OpenHands GitHub Backlog Resolver. If you have any issues, please open an issue on this github repo, we're happy to help! Alternatively, you can email us or join the OpenHands Slack workspace and ask there.