Code Droid
Basic information
Website: https://web.archive.org/web/20240905220438/https://www.factory.ai/droids
Short description: Agent for software development [source]
Intended uses: What does the developer say it’s for? General-purpose coding and software development
Date(s) deployed: Code Droid documentation was posted June 18, 2024
Developer
Website: https://web.archive.org/web/20241224152919/https://www.factory.ai/
Legal name: The San Francisco AI Factory Inc [source]
Entity type: Corporation [source]
Country (location of developer or first author’s first affiliation): USA [source]
Safety policies: What safety and/or responsibility policies are in place? Unknown
System components
Backend model: What model(s) are used to power the system? Variable, including models from Anthropic and OpenAI [source]
Publicly available model specification: Is there formal documentation on the system’s intended uses and how it is designed to behave in them? None
Reasoning, planning, and memory implementation: How does the system ‘think’? Only high level details are given. Droid can decompose tasks, “simulate decisions, perform self-criticism, and reflect on real and imagined decisions”. Droid uses “HyperCode to construct a multi-resolution representation of a given engineering system” and ByteRank for retrieving task-relevant information [source].
Observation space: What is the system able to observe while ‘thinking’? From the codebase, Droid can “autonomously construct explicit (graph) and implicit (latent space similarity) relationships within low-level data and extract insights about these relationships at different levels of abstraction” [source].
Action space/tools: What direct actions can the system take? Droid has access to developer tools and environments (intended to be similar to what is available to human developers) [source].
User interface: How do users interact with the system? Unknown
Development cost and compute: What is known about the development costs? Unknown
Guardrails and oversight
Accessibility of components:
- Weights: Are model parameters available? N/A; backends various models
- Data: Is data available? N/A; backends various models
- Code: Is code available? Closed source
- Scaffolding: Is system scaffolding available? Closed source
- Documentation: Is documentation available? Unavailable, but they have a technical report [source]
Controls and guardrails: What notable methods are used to protect against harmful actions? 1. “Code Droid operates within a strictly defined, sandboxed environment that isolates its operational scope from main development environments.” 2. “Droids log and report the reasoning behind all of their actions.” 3. “DroidShield performs real-time static code analysis to detect potential security vulnerabilities, bugs, or intellectual property breaches before they are committed to code” [source].
Customer and usage restrictions: Are there know-your-customer measures or other restrictions on customers? None
Monitoring and shutdown procedures: Are there any notable methods or protocols that allow for the system to be shut down if it is observed to behave harmfully? Unknown
Evaluation
Notable benchmark evaluations: 19.27% on SWE-bench Full and 31.67% on SWE-bench Lite [source]
Bespoke testing: None
Safety: Have safety evaluations been conducted by the developers? What were the results? None
Publicly reported external red-teaming or comparable auditing:
- Personnel: Who were the red-teamers/auditors? None
- Scope, scale, access, and methods: What access did red-teamers/auditors have and what actions did they take? None
- Findings: What did the red-teamers/auditors conclude? None
Ecosystem information
Interoperability with other systems: What tools or integrations are available? None
Usage statistics and patterns: Are there any notable observations about usage? Unknown
Additional notes
None