Agent S
Basic Information
Website: https://arxiv.org/abs/2410.08164v1
Short description: "A novel framework for developing fully Autonomous Graphical User Interface (GUI) agents that can perform a wide range of user queries by directly controlling the keyboard and mouse." [source]
Intended uses: What does the developer state that the system is intended for?: Automating complex, multi-step tasks by getting an agent to use computers like a human [source]
Date(s) deployed: Earliest GitHub commits from October 11, 2024 [source]
Developer
Website: https://www.simular.ai/
Legal name: Simular, Inc [source.docx.pdf)]
Entity type: Corporation [source]
Country (location of developer or first author's first affiliation): Incorporation: Delaware, USA (SIMULAR, INC (7608865)) [source]
Safety policies: What safety and/or responsibility policies are in place?: Terms of Use [source.docx.pdf)]
System Components
Backend model(s): What model(s) are used to power the system?: Variable, defaulting to GPT-4o and Claude-3 Sonnet [source]
Public model specification: Is there formal documentation on the system’s intend...: None
Description of reasoning, planning, and memory implementation: How does the syst...: Agent S leverages "an experience-augmented hierarchical planning method that uses experience from external web knowledge and the agent’s internal memory to decompose complex tasks into executable subtasks." The authors illustrate their implementation through Figures 3 and 4 in [source].
Observation space: What is the system able to observe while 'thinking'?: Agent S operates in an environment where it can observe screenshots of webpages and annotated GUI elements, along with its task memory [source]
Action space/tools: What direct actions can the system take?: The agent's design "incorporates a bounded action space. This space includes primitive actions like click, type, and hotkey." [source]
User interface: How do users interact with the system?: While the company's demos sometimes include a user interface (UI), there is no publicly available UI [source]
Development cost and compute: What is known about the development costs?: Unknown
Guardrails & Oversight
Accessibility of components
Weights: Are model parameters available?: N/A; backends various models
Data: Is data available?: N/A; backends various models
Code: Is code available?: Available [source]
Scaffolding: Is system scaffolding available?: Available [source]
Controls and guardrails: What notable methods are used to protect against harmfu...: The authors bound the agent's action space, in part, to improve its safety [source]
Monitoring and shutdown procedures: Are there any notable methods or protocols t...: Depends on what is implemented in a specific configuration [source].
Customer and usage restrictions: Are there know-your-customer measures or other ...: None
Evaluation
Notable benchmark evaluations (e.g., on SWE-Bench Verified): 20.58% success rate on the OSWorld full test set when running on GPT-4o [source]
Bespoke testing (e.g., demos): Demo [source]
Safety: Have safety evaluations been conducted by the developers? What were the ...: None
Publicly reported external red-teaming or comparable auditing
Personnel: Who were the red-teamers/auditors?: None
Scope, scale, access, and methods: What access did red-teamers/auditors have and...: None
Findings: What did the red-teamers/auditors conclude?: None
Ecosystem
Interoperability with other systems: What tools or integrations are available?: Agent S was designed for the Ubuntu operating system generalizes to the Windows operating system [source]
Usage statistics and patterns: Are there any notable observations about usage?: The GitHub repository has 99 forks and 734 stars [source]
Other notes (if any): --