WebRL
Basic information
Website: https://arxiv.org/abs/2411.02337v1
Short description: WebRL is a “reinforcement learning framework designed to train high-performance web agents using open LLMs.” [source]
Intended uses: What does the developer say it’s for? Intended to be used to train agents that can accomplish tasks (described in natural language) on the internet through a web browser [source]
Date(s) deployed: Paper first put on arXiv on November 4th 2024 [source]
Developer
Website: https://web.archive.org/web/20241107151506/https://github.com/THUDM/WebRL
Legal name: Tsinghua University (et al.) [source]
Entity type: Academic Institution and corporation.
Country (location of developer or first author’s first affiliation): China, [source]
Safety policies: What safety and/or responsibility policies are in place? None
System components
Backend model: What model(s) are used to power the system? The authors release three agents trained using WebRL based on Llama-3-8b, Llama-3-70b, and GLM-4-9b [source]
Publicly available model specification: Is there formal documentation on the system’s intended uses and how it is designed to behave in them? None
Reasoning, planning, and memory implementation: How does the system ‘think’? Reinforcement learning is used to train the model to plan internally. That is there are no separate specialized planning modules [source]
Observation space: What is the system able to observe while ‘thinking’? HTML content of the current web page along with the history of previous actions [source]
Action space/tools: What direct actions can the system take? In principle, WebRL can be used to train agents using any natural languages based action space. The authors use WebArena to train their released models, an environment with an “action space that emulates the keyboard and mouse operations available on web page” [source].
User interface: How do users interact with the system? There is no publicly available UI. Users can download the models released by the authors and personally host them. The agents are designed to be interacted with using natural language descriptions of tasks, just like a regular chatbot.
Development cost and compute: What is known about the development costs? Unknown
Guardrails and oversight
Accessibility of components:
- Weights: Are model parameters available? Authors release three open source agents created using the WebRL framework [source].
- Data: Is data available? Open source [source].
- Code: Is code available? Open source [source]
- Scaffolding: Is system scaffolding available? Public (see open source code).
- Documentation: Is documentation available? Basic documentation on Github [source] and paper [source]
Controls and guardrails: What notable methods are used to protect against harmful actions? None
Customer and usage restrictions: Are there know-your-customer measures or other restrictions on customers? None
Monitoring and shutdown procedures: Are there any notable methods or protocols that allow for the system to be shut down if it is observed to behave harmfully? None
Evaluation
Notable benchmark evaluations: Agent based on Llama-3-70b achieves 49.1% success rate on WebArena Lite [source].
Bespoke testing: Paper contains various ablation studies [source].
Safety: Have safety evaluations been conducted by the developers? What were the results? None
Publicly reported external red-teaming or comparable auditing:
- Personnel: Who were the red-teamers/auditors? None
- Scope, scale, access, and methods: What access did red-teamers/auditors have and what actions did they take? None
- Findings: What did the red-teamers/auditors conclude? None
Additional notes
None