Agent S
Basic information
Website: https://arxiv.org/abs/2410.08164v1
Short description: “A novel framework for developing fully Autonomous Graphical User Interface (GUI) agents that can perform a wide range of user queries by directly controlling the keyboard and mouse.” [source]
Intended uses: What does the developer say it’s for? Automating complex, multi-step tasks by getting an agent to use computers like a human [source]
Date(s) deployed: Earliest GitHub commits from October 11, 2024 [source]
Developer
Website: https://www.simular.ai/
Legal name: Simular, Inc [source]
Entity type: Corporation [source]
Country (location of developer or first author’s first affiliation): Incorporation: Delaware, USA (SIMULAR, INC (7608865)) [source]
Safety policies: What safety and/or responsibility policies are in place? Terms of Use [source]
System components
Backend model: What model(s) are used to power the system? Variable, defaulting to GPT-4o and Claude-3 Sonnet [source]
Publicly available model specification: Is there formal documentation on the system’s intended uses and how it is designed to behave in them? None
Reasoning, planning, and memory implementation: How does the system ‘think’? Agent S leverages “an experience-augmented hierarchical planning method that uses experience from external web knowledge and the agent’s internal memory to decompose complex tasks into executable subtasks.” The authors illustrate their implementation through Figures 3 and 4 in [source].
Observation space: What is the system able to observe while ‘thinking’? Agent S operates in an environment where it can observe screenshots of webpages and annotated GUI elements, along with its task memory [source]
Action space/tools: What direct actions can the system take? The agent’s design “incorporates a bounded action space. This space includes primitive actions like click, type, and hotkey.” [source]
User interface: How do users interact with the system? While the company’s demos sometimes include a user interface (UI), there is no publicly available UI [source]
Development cost and compute: What is known about the development costs? Unknown
Guardrails and oversight
Accessibility of components:
- Weights: Are model parameters available? N/A; backends various models
- Data: Is data available? N/A; backends various models
- Code: Is code available? Available [source]
- Scaffolding: Is system scaffolding available? Available [source]
- Documentation: Is documentation available? Documentation on GitHub [source] and pre-print [source]
Controls and guardrails: What notable methods are used to protect against harmful actions? The authors bound the agent’s action space, in part, to improve its safety [source]
Customer and usage restrictions: Are there know-your-customer measures or other restrictions on customers? None
Monitoring and shutdown procedures: Are there any notable methods or protocols that allow for the system to be shut down if it is observed to behave harmfully? Depends on what is implemented in a specific configuration [source].
Evaluation
Notable benchmark evaluations: 20.58% success rate on the OSWorld full test set when running on GPT-4o [source]
Bespoke testing: Demo [source]
Safety: Have safety evaluations been conducted by the developers? What were the results? None
Publicly reported external red-teaming or comparable auditing:
- Personnel: Who were the red-teamers/auditors? None
- Scope, scale, access, and methods: What access did red-teamers/auditors have and what actions did they take? None
- Findings: What did the red-teamers/auditors conclude? None
Ecosystem information
Interoperability with other systems: What tools or integrations are available? Agent S was designed for the Ubuntu operating system generalizes to the Windows operating system [source]
Usage statistics and patterns: Are there any notable observations about usage? The GitHub repository has 99 forks and 734 stars [source]
Additional notes
None