ScribeAgent
Basic information
Website: https://web.archive.org/web/20250115055003/https://scribehow.com/library/scribe-agent
Short description: ScribeAgent is a web agent trained to automate web workflows. Unlike most web agents, ScribeAgent is trained on Scribe’s vast dataset of real step-by-step business processes, which Scribe collects as a part of its main product [source]
Intended uses: What does the developer say it’s for? To complete complex web-based tasks, mostly in the business domain (e.g. working with CRM platforms, productivity tools like Notion, social platforms like Facebook, shopping sites like Amazon, and many others) [source]
Date(s) deployed: Not yet deployed, paper released November 22, 2024 [source]
Developer
Website: https://web.archive.org/web/20250109221924/https://scribehow.com/library/scribe-agent
Legal name: Colony Labs, Inc [source]
Entity type: Corporation
Country (location of developer or first author’s first affiliation): Incorporation: Delaware, USA (COLONY LABS, INC. 7498102) [source]
Safety policies: What safety and/or responsibility policies are in place? Unknown
System components
Backend model: What model(s) are used to power the system? Custom finetunes of Qwen2 [source]
Publicly available model specification: Is there formal documentation on the system’s intended uses and how it is designed to behave in them? None
Reasoning, planning, and memory implementation: How does the system ‘think’? ScribeAgent is a single agent system without specialized planning or memory modules. ScribeAgent is trained to perform next-action prediction and rollouts are autoregressive [source]
Observation space: What is the system able to observe while ‘thinking’? The HTML DOM of the page in text format, with some elements removed through a simple hardcoded filter [source]
Action space/tools: What direct actions can the system take? Take three types of web browser actions: 1) mouse click at element 2) type characters into an element 3) press hotkey combination (e.g. ctrl+c) [source]
User interface: How do users interact with the system? There is no publicly available UI, but ScribeAgent takes a single prompt describing the desired task [source]
Development cost and compute: What is known about the development costs? ScribeAgent- Small and Large can be finetuned using 8 H100 GPUs in four and ten days respectively [source]
Guardrails and oversight
Accessibility of components:
- Weights: Are model parameters available? Closed source
- Data: Is data available? Closed source
- Code: Is code available? Available [source]
- Scaffolding: Is system scaffolding available? Available [source]
- Documentation: Is documentation available? Basic documentation on Github [source] and pre-print [source]
Controls and guardrails: What notable methods are used to protect against harmful actions? None
Customer and usage restrictions: Are there know-your-customer measures or other restrictions on customers? None
Monitoring and shutdown procedures: Are there any notable methods or protocols that allow for the system to be shut down if it is observed to behave harmfully? None
Evaluation
Notable benchmark evaluations: 53% task success rate on WebArena [source]
Bespoke testing: None
Safety: Have safety evaluations been conducted by the developers? What were the results? None
Publicly reported external red-teaming or comparable auditing:
- Personnel: Who were the red-teamers/auditors? None
- Scope, scale, access, and methods: What access did red-teamers/auditors have and what actions did they take? None
- Findings: What did the red-teamers/auditors conclude? None
Ecosystem information
Interoperability with other systems: What tools or integrations are available? The WebArena benchmark code in the repository contains a minimal browser integration [source]
Usage statistics and patterns: Are there any notable observations about usage? None
Additional notes
None