OpenWebVoyager
Basic Information
Website: https://arxiv.org/abs/2410.19609
Short description: OpenWebVoyager is an agent for accomplishing open-ended tasks on the internet. This agent supports multimodal observations, and requires minimal human guidance.
Intended uses: What does the developer state that the system is intended for?: Designed to handle more complicated web scenarios, including dealing with multimodal input and sparse supervision. Standard internet interaction benchmarks are text-only: this work aims to extend this.
Date(s) deployed: Paper arXived Oct 25, 2024, and GitHub code initial commit Oct 21, 2024 [source]
Developer
System Components
Backend model(s): What model(s) are used to power the system?: Uses WebVoyager [source] (which is based on GPT-4o) to obtain imitation data for training multimodal web navigation, and uses GPT-4o directly to evaluate correctness of web trajectories in subsequent self-improvement loops. Agent itself build using Idefics2 [source]
Public model specification: Is there formal documentation on the system’s intend...: None
Description of reasoning, planning, and memory implementation: How does the syst...: System initially bootstraps basic web browsing capability from imitating a SOTA web browsing agent WebVoyager [source], and then explores real-world web environments in an open-ended way, and the successes are used to augment the initial IL dataset to further improve training.
Observation space: What is the system able to observe while 'thinking'?: The 3 past screenshots, and the accessibility tree of the current webpage.
Action space/tools: What direct actions can the system take?: Can navigate web pages autonomously, and interact with the user (i.e. the agent can click, input, scroll, go back, restart, wait, and provide an answer to the user).
User interface: How do users interact with the system?: The user can provide the system a query in natural language, but since the reward signal grounded in GPT-4o, unclear how the model grounds it (and, they self-report the system often hallucinates).
Development cost and compute: What is known about the development costs?: Unknown
Guardrails & Oversight
Accessibility of components
Weights: Are model parameters available?: Available [source]
Data: Is data available?: Available [source]
Code: Is code available?: Available [source]
Documentation: Is documentation available?: Available [source]
Scaffolding: Is system scaffolding available?: Available [source]
Controls and guardrails: What notable methods are used to protect against harmfu...: Unknown
Monitoring and shutdown procedures: Are there any notable methods or protocols t...: Unknown
Customer and usage restrictions: Are there know-your-customer measures or other ...: None
Evaluation
Notable benchmark evaluations (e.g., on SWE-Bench Verified): Mind2Web (20%) [source]
Bespoke testing (e.g., demos): See Figure 3 of paper [source]
Safety: Have safety evaluations been conducted by the developers? What were the ...: None
Publicly reported external red-teaming or comparable auditing
Personnel: Who were the red-teamers/auditors?: None
Scope, scale, access, and methods: What access did red-teamers/auditors have and...: None
Findings: What did the red-teamers/auditors conclude?: None
Ecosystem
Interoperability with other systems: What tools or integrations are available?: None
Usage statistics and patterns: Are there any notable observations about usage?: 7 forks and 65 starts on GitHub [source]
Other notes (if any): --