ShowUI
Basic Information
Website: https://github.com/showlab/ShowUI
Short description: ShowUI is a multimodal GUI-based computer use agent that can complete tasks in the web browser, with desktop GUI applications, and on mobile devices [source]
Intended uses: What does the developer state that the system is intended for?: To complete GUI-based computer use tasks across a range of platforms (browser, desktop applications, and mobile) [source]
Date(s) deployed: Github repository created October 31, 2024 [source]
Developer
Website: https://github.com/showlab/ShowUI
Legal name: National University of Singapore (et al.) [source]
Entity type: Academic Institution, Industry Organization
Country (location of developer or first author's first affiliation): Singapore [source]
Safety policies: What safety and/or responsibility policies are in place?: Unknown
System Components
Backend model(s): What model(s) are used to power the system?: Qwen2-VL-2B with an efficient visual token selection design [source]
Public model specification: Is there formal documentation on the system’s intend...: None
Description of reasoning, planning, and memory implementation: How does the syst...: ShowUI is a single agent system without specialized planning modules, although it does keep a history of previous states and actions in its context. ShowUI is trained to perform next-action predictions and rollouts are autoregressive [source]
Observation space: What is the system able to observe while 'thinking'?: A screenshot of the current state as well as the full history of screenshots and actions in the trajectory [source]
Action space/tools: What direct actions can the system take?: For each deployment platform, a readme is provided with the available actions. These typically consist of positional click and type actions as well as device-specific buttons [source]
User interface: How do users interact with the system?: Users must launch a local server, but then they can issue commands through a webpage and ShowUI will control their MacOS and Windows computers directly [source]
Development cost and compute: What is known about the development costs?: "We utilize 32 V100 GPUs...for two days." [source]
Guardrails & Oversight
Accessibility of components
Weights: Are model parameters available?: Open source [source]
Data: Is data available?: Open source [source].
Code: Is code available?: Available [source]
Documentation: Is documentation available?: Basic documentation on Github [source] and pre-print [source]
Scaffolding: Is system scaffolding available?: Available [source]
Controls and guardrails: What notable methods are used to protect against harmfu...: None
Monitoring and shutdown procedures: Are there any notable methods or protocols t...: None
Customer and usage restrictions: Are there know-your-customer measures or other ...: None
Evaluation
Notable benchmark evaluations (e.g., on SWE-Bench Verified): 75% ground accuracy on ScreenSpot, 70% success rate on Android in the wild, and 35-37% step success rate on Mind2Web [source]
Bespoke testing (e.g., demos): There is one video demo [source], a HuggingFace space to play with the model [source], and an interface for user-friendly use [source]
Safety: Have safety evaluations been conducted by the developers? What were the ...: None
Publicly reported external red-teaming or comparable auditing
Personnel: Who were the red-teamers/auditors?: None
Scope, scale, access, and methods: What access did red-teamers/auditors have and...: None
Findings: What did the red-teamers/auditors conclude?: None
Ecosystem
Interoperability with other systems: What tools or integrations are available?: Can directly control MacOS and Windows computers [source]
Usage statistics and patterns: Are there any notable observations about usage?: The Github repository has 39 forks and 684 stars [source]
Other notes (if any): --