SuperCoder 2.0
Basic Information
Website: https://superagi.com/supercoder/
Short description: Open source autonomous software development system [source]
Intended uses: What does the developer state that the system is intended for?: General-purpose coding and software development
Date(s) deployed: Earliest GitHub commits from July 3, 2024 [source]
Developer
Website: https://superagi.com/
Legal name: SuperAGI, Inc [source]
Entity type: Corporation [source]
Country (location of developer or first author's first affiliation): USA [source]
Safety policies: What safety and/or responsibility policies are in place?: Unknown
System Components
Backend model(s): What model(s) are used to power the system?: Variable, including Claude 3.5 Sonnet and GPT-4o
Public model specification: Is there formal documentation on the system’s intend...: None
Description of reasoning, planning, and memory implementation: How does the syst...: It uses its "PlannerAgent, a sophisticated tool designed to navigate through the code files, identify buggy locations, and determine the necessary changes required to rectify the issues [...] In some instances, the PlannerAgent may need to insert an entirely new method to resolve the problem or at least propose a viable solution". SuperCoder 2.0 uses retrieval augmented generation and its 'AssimilatorAgent' to retrieve relevant files [source].
Observation space: What is the system able to observe while 'thinking'?: SuperCoder 2.0 looks at an entire codebase, using retrieval augmented generation and an agent to select specific files [source].
Action space/tools: What direct actions can the system take?: SuperCoder 2.0 looks can retrieve, generate, and replace code in a codebase [source].
User interface: How do users interact with the system?: They have a visual application user interface where a user can specify tasks for the agent [source].
Development cost and compute: What is known about the development costs?: Unknown
Guardrails & Oversight
Accessibility of components
Weights: Are model parameters available?: N/A; backends various models
Data: Is data available?: N/A; backends various models
Code: Is code available?: Available [source]
Documentation: Is documentation available?: Available [source]
Scaffolding: Is system scaffolding available?: Available [source]
Controls and guardrails: What notable methods are used to protect against harmfu...: None
Monitoring and shutdown procedures: Are there any notable methods or protocols t...: Unknown
Customer and usage restrictions: Are there know-your-customer measures or other ...: None
Evaluation
Notable benchmark evaluations (e.g., on SWE-Bench Verified): "SuperCoder 2.0 achieves 34% success rate in SWE-bench Lite, ranking #4 globally and #1 among all open-source coding systems" [source].
Bespoke testing (e.g., demos): None
Safety: Have safety evaluations been conducted by the developers? What were the ...: None
Publicly reported external red-teaming or comparable auditing
Personnel: Who were the red-teamers/auditors?: None
Scope, scale, access, and methods: What access did red-teamers/auditors have and...: None
Findings: What did the red-teamers/auditors conclude?: None