Allows `Aider` to use CEDARScript as an edit format
Project description
CEDARScript Integration: Aider
CEDARScript Integration: Aider
enables Aider
to use
CEDARScript
as an edit format.
Table of Contents
- What is CEDARScript?
- Installation
- Usage
- Why Use CEDARScript?
- Performance Comparison
- Contributing
- License
What is CEDARScript?
CEDARScript (Concise Examination, Development, And Refactoring Script) is a domain-specific language designed to improve how AI coding assistants interact with codebases and communicate their code modification intentions. It provides a standardized way to express complex code modification and analysis operations, making it easier for AI-assisted development tools to understand and execute these tasks.
How to use it
- Install Aider, if you haven't.
- Now, simply use the
--edit-format
switch and selectcedarscript
:
aider --model gemini/gemini-1.5-flash-latest --edit-format cedarscript
Why use CEDARScript?
TL;DR
: You can get higher success rates when compared to other edit formats.
- Higher Success Rates: Significantly improves the performance of AI models in code refactoring tasks.
- Cost-Effective Performance: Enables more affordable models to compete with top-tier options.
- Standardized Communication: Provides a consistent format for AI-code interaction in coding tasks.
- Enhanced Accuracy: Reduces errors and improves the quality of AI-generated code modifications.
Performance Comparison
CEDARScript has shown remarkable improvements in AI model performance for code refactoring:
Model | Format | Pass Rate | Well-Formed Cases | Syntax Errors | Indentation Errors | Cost | Avg. Time per case |
---|---|---|---|---|---|---|---|
Gemini 1.5 PRO | CEDARScript | 77.5% | 86.5% | 4 | 3 | 26.2 | 29 |
Gemini 1.5 Flash | CEDARScript | 76.4% | 94.4% | 3 | 5 | 0.68 | 14.7 |
Claude 3.5 Sonnet | diff | 64.0% | 76.4% | n/a | n/a | n/a | n/a |
Gemini 1.5 PRO | diff-fenced | 49.4% | 7.9% | 21 | 93 | 28.3 | 110.1 |
Notable Achievements:
- Gemini 1.5 PRO with CEDARScript outperformed both its diff-fenced format and Claude 3.5 Sonnet.
- Most remarkably, the more cost-effective Gemini 1.5 Flash model, using CEDARScript, outperformed Claude 3.5 Sonnet.
- It goes to show that even a more affordable model can surpass top-tier competitors when equipped with the right tools.
This suggests that CEDARScript can level the playing field, enabling more accessible AI models to compete with and even exceed the capabilities of more expensive options in certain coding tasks.
Raw Metrics
**Sonnet 3.5 + `diff`**
- dirname: refac-claude-3.5-sonnet-diff-not-lazy
model: claude-3.5-sonnet (diff)
edit_format: diff
pass_rate_1: 64.0
percent_cases_well_formed: 76.4
Gemini 1.5 PRO + `diff-fenced` (leaderboard site)
- dirname: refac-gemini
model: gemini/gemini-1.5-pro-latest
edit_format: diff-fenced
pass_rate_1: 49.4
percent_cases_well_formed: 7.9
Gemini 1.5 PRO + `diff-fenced` (own tests)
- dirname: 2024-10-05-00-43-21--diff-fenced-Gemini-Refactoring
test_cases: 89
model: gemini/gemini-1.5-pro-latest
edit_format: diff-fenced
commit_hash: 772710b-dirty
pass_rate_1: 18.0
pass_rate_2: 21.3
pass_rate_3: 24.7
percent_cases_well_formed: 34.8
error_outputs: 180
num_malformed_responses: 180
num_with_malformed_responses: 58
user_asks: 128
lazy_comments: 2
syntax_errors: 21
indentation_errors: 93
exhausted_context_windows: 0
test_timeouts: 0
command: aider --model gemini/gemini-1.5-pro-latest
date: 2024-10-05
versions: 0.57.2.dev
seconds_per_case: 110.1
total_cost: 28.2515
Gemini 1.5 PRO + `CEDARScript`
- dirname: 2024-10-19-22-48-07--cedarscript-0.3.1-refactoring-gemini1.5pro
test_cases: 89
model: gemini/gemini-1.5-pro-latest
edit_format: cedarscript-g
commit_hash: 4da1e9b-dirty
pass_rate_1: 77.5
percent_cases_well_formed: 86.5
error_outputs: 337
num_malformed_responses: 19
num_with_malformed_responses: 12
user_asks: 12
lazy_comments: 0
syntax_errors: 4
indentation_errors: 3
exhausted_context_windows: 0
test_timeouts: 0
command: aider --model gemini/gemini-1.5-pro-latest
date: 2024-10-19
versions: 0.59.2.dev
seconds_per_case: 29.0
total_cost: 26.2374
Gemini 1.5 _Flash_ + `CEDARScript`
- dirname: 2024-10-20-00-33-27--cedarscript-0.3.1-refactoring-gemini1.5flash
test_cases: 89
model: gemini/gemini-1.5-flash-latest
edit_format: cedarscript-g
commit_hash: 4da1e9b-dirty
pass_rate_1: 76.4
percent_cases_well_formed: 94.4
error_outputs: 403
num_malformed_responses: 13
num_with_malformed_responses: 5
user_asks: 21
lazy_comments: 0
syntax_errors: 3
indentation_errors: 5
exhausted_context_windows: 0
test_timeouts: 0
command: aider --model gemini/gemini-1.5-flash-latest
date: 2024-10-20
versions: 0.59.2.dev
seconds_per_case: 14.7
total_cost: 0.6757
functional_Functional__conform_to_reference_input
diff-fenced
"cost": 0.33188854999999995,
"duration": 27.793912172317505,
"test_timeouts": 0,
"commit_hash": "772710b-dirty",
"num_error_outputs": 2,
"num_user_asks": 3,
"num_exhausted_context_windows": 0,
"num_malformed_responses": 2,
"syntax_errors": 0,
"indentation_errors": 3,
"lazy_comments": 0,
cedarscript
"cost": 0.18178265,
"duration": 11.176445960998535,
"test_timeouts": 0,
"commit_hash": "772710b-dirty",
"num_error_outputs": 0,
"num_user_asks": 1,
"num_exhausted_context_windows": 0,
"num_malformed_responses": 0,
"syntax_errors": 0,
"indentation_errors": 0,
"lazy_comments": 0,
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
License
This project is licensed under the MIT License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cedarscript_integration_aider-0.0.8.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 88d7958c3231283b80381f3cb191a53fc9024960909b72715db2686af72ede93 |
|
MD5 | 1fa2088bf734fbe367a7120c7a3bd4d3 |
|
BLAKE2b-256 | f2c0c9a07e98c694093f039a4e444535e31b5c63a4b13a6186b4125c3de0e984 |
Hashes for cedarscript_integration_aider-0.0.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9332c987d6d2d68c68fd70765812cb651e4f2aed4cb98268b4458b8051183f5b |
|
MD5 | 82349441f11730f672e91887874d33b6 |
|
BLAKE2b-256 | e2b6220d33a65824d2b918566247b70bb561cc755798b69f3fcf21a8313f2d14 |