AutoWebGLM
Basic Information
Website: https://arxiv.org/abs/2404.03648
Short description: AutoWebGLM is an automated web navigation agent [source]
Intended uses: What does the developer state that the system is intended for?: Web browsing tasks [source]
Developer
Website: https://github.com/THUDM/AutoWebGLM
Legal name: Tsinghua University (et al.) [source]
Entity type: Academic Institution, Corporation
Country (location of developer or first author's first affiliation): China [source]
Safety policies: What safety and/or responsibility policies are in place?: None
System Components
Backend model(s): What model(s) are used to power the system?: ChatGLM3-6B model [source]
Public model specification: Is there formal documentation on the system’s intend...: None
Description of reasoning, planning, and memory implementation: How does the syst...: Paper arXived on April 4, 2024, and Github code initial commit April 3, 2024 [source] [source]
Observation space: What is the system able to observe while 'thinking'?: The observation space consists of states that include simplified HTML information, current location within webpage, and past operation records [source].
Action space/tools: What direct actions can the system take?: The action space includes the following: Click at an element, Hover on an element, Select option in an element, Type to an element, Scroll up or down of the page, Go forward or backward of the page, Jump to URL, Switch to i-th tab, Notify user to interact, Stop with answer [source]
User interface: How do users interact with the system?: A Chrome extension where users write prompts to perform operations on websites [source]
Development cost and compute: What is known about the development costs?: Unknown
Guardrails & Oversight
Accessibility of components
Weights: Are model parameters available?: Available [source]
Data: Is data available?: Available [source]
Code: Is code available?: Available [source]
Documentation: Is documentation available?: Some documentation in Github [source], and arXiv [source]
Scaffolding: Is system scaffolding available?: Available [source]
Controls and guardrails: What notable methods are used to protect against harmfu...: Unknown
Monitoring and shutdown procedures: Are there any notable methods or protocols t...: Unknown
Customer and usage restrictions: Are there know-your-customer measures or other ...: None
Evaluation
Notable benchmark evaluations (e.g., on SWE-Bench Verified): Mind2Web (59.5), MiniWoB++ (89.3%), WebArena (18.2%) [source]
Bespoke testing (e.g., demos): Reported results on their own benchmark called AutoWebBench [source]
Safety: Have safety evaluations been conducted by the developers? What were the ...: The authors "identify errors that occasionally occur during task execution, which can be broadly categorized into four types: hallucinations, poor graphical recognition, misinterpretation of task context, and pop-up interruptions". In Table 7, they report how often these types of errors occur [source]
Publicly reported external red-teaming or comparable auditing
Personnel: Who were the red-teamers/auditors?: None
Scope, scale, access, and methods: What access did red-teamers/auditors have and...: None
Findings: What did the red-teamers/auditors conclude?: None