Claude 3.5 Sonnet (2024-10-22)
Basic information
Website: https://web.archive.org/web/20241229024749/https://www.anthropic.com/news/developing-computer-use
Short description: An LLM that can use specialized tools for using a computer just from screenshots.
Intended uses: What does the developer say it’s for? The latest version of Claude 3.5 Sonnet [source] can, when run through the appropriate software setup, follow a user’s commands to move a cursor around their computer’s screen, click on relevant locations, and input information via a virtual keyboard, emulating the way people interact with their own computer [source]
Date(s) deployed: October 22, 2024 [source]
Developer
Website: https://web.archive.org/web/20241231211939/http://anthropic.com/
Legal name: Anthropic, PBC [source]
Entity type: Benefit Corporation [source]
Country (location of developer or first author’s first affiliation): Incorporation: Delaware, USA (Anthropic, PBC (4860621)) [source]
Safety policies: What safety and/or responsibility policies are in place? Available [source]
System components
Backend model: What model(s) are used to power the system? Claude 3.5 Sonnet Upgraded
Publicly available model specification: Is there formal documentation on the system’s intended uses and how it is designed to behave in them? Available [source]
Reasoning, planning, and memory implementation: How does the system ‘think’? N/A, it’s an LLM and doesn’t include scaffolding.
Observation space: What is the system able to observe while ‘thinking’? Images (of computer screens) and text.
Action space/tools: What direct actions can the system take? A large collection of computer tools, including mouse clicking, typing into GUIs, and reading/writing files using a CLI [source]
User interface: How do users interact with the system? LLM used through the Anthropic API [source]
Development cost and compute: What is known about the development costs? Unknown
Guardrails and oversight
Accessibility of components:
- Weights: Are model parameters available? Closed source
- Data: Is data available? Closed source
- Code: Is code available? Closed source
- Scaffolding: Is system scaffolding available? Some demo code for computer use is shared [source]
- Documentation: Is documentation available? Available [source]
Controls and guardrails: What notable methods are used to protect against harmful actions? Unknown
Customer and usage restrictions: Are there know-your-customer measures or other restrictions on customers? None
Monitoring and shutdown procedures: Are there any notable methods or protocols that allow for the system to be shut down if it is observed to behave harmfully? Anthropic recommends users run the agents on VMs instead of their PCs [source]
Evaluation
Notable benchmark evaluations: 53% on SWE-bench Verified [source]
Bespoke testing: Demonstrations on Anthropic’s YouTube channel [source]
Safety: Have safety evaluations been conducted by the developers? What were the results? Addendum to Claude 3 Model Card, concluding Claude 3.5 Sonnet (2024-10-22) is an ASL-2 model [source].
Publicly reported external red-teaming or comparable auditing:
- Personnel: Who were the red-teamers/auditors? Pre-deployment testing by US AISI and UK AISI, alongside independent assessment by METR [source]
- Scope, scale, access, and methods: What access did red-teamers/auditors have and what actions did they take? Biological, cyber, software and AI development, and safeguard efficacy evaluations [source]
- Findings: What did the red-teamers/auditors conclude? Results reported but no statements about safety or appropriateness to release [source]
Ecosystem information
Interoperability with other systems: What tools or integrations are available? Anthropic’s Model Context Protocol allows for Claude to be integrated flexibly with many apps [source].
Usage statistics and patterns: Are there any notable observations about usage? 7.1k stars on Anthropic Quickstarts which contains the computer use demo [source]
Additional notes
None