Runner H
Basic information
Website: https://web.archive.org/web/20241224155629/https://www.hcompany.ai/blog/introducing-h
Short description: Runner H is the flagship model for ‘H Company’s’ Ai Agent development platform, “Studio.”
Intended uses: What does the developer say it’s for? Runner H is a general-purpose agent designed to automate tasks on a filesystem and browser.
Date(s) deployed: Announced November 19, 2023. Currently in private beta [source]
Developer
Website: https://www.hcompany.ai/blog/introducing-h
Legal name: H.AI SAS [source]
Entity type: Société par actions simplifiée (~US LLC) [source]
Country (location of developer or first author’s first affiliation): Incorporation: France (927528471) [source]
Safety policies: What safety and/or responsibility policies are in place? Unknown
System components
Backend model: What model(s) are used to power the system? Their agents are based on in-house models: H-VLM and H-LLM. Their H-VLM is a 3B parameter vision language model trained to extract information from and interact with GUIs and images. Meanwhile, they also introduce that H-LLM family of language models, one of which is 2B parameters [source].
Publicly available model specification: Is there formal documentation on the system’s intended uses and how it is designed to behave in them? None
Reasoning, planning, and memory implementation: How does the system ‘think’? Unknown
Observation space: What is the system able to observe while ‘thinking’? H-VLM is able to observe images, diagrams, and GUIs [source]
Action space/tools: What direct actions can the system take? Runner H can interact with a browser. It is not clear what it can do beyond that.
User interface: How do users interact with the system? Chat [source]
Development cost and compute: What is known about the development costs? Unknown
Guardrails and oversight
Accessibility of components:
- Weights: Are model parameters available? Closed source
- Data: Is data available? Closed source
- Code: Is code available? Closed source
- Scaffolding: Is system scaffolding available? Closed source
- Documentation: Is documentation available? Unavailable
Controls and guardrails: What notable methods are used to protect against harmful actions? Unknown
Customer and usage restrictions: Are there know-your-customer measures or other restrictions on customers? Currently in a private beta.
Monitoring and shutdown procedures: Are there any notable methods or protocols that allow for the system to be shut down if it is observed to behave harmfully? Unknown
Evaluation
Notable benchmark evaluations: 67% on WebVoyage, 80% on Screenspot, and 65% average across several code and function calling benchmarks [source]
Bespoke testing: Demos [source]
Safety: Have safety evaluations been conducted by the developers? What were the results? None
Publicly reported external red-teaming or comparable auditing:
- Personnel: Who were the red-teamers/auditors? None
- Scope, scale, access, and methods: What access did red-teamers/auditors have and what actions did they take? None
- Findings: What did the red-teamers/auditors conclude? None
Ecosystem information
Interoperability with other systems: What tools or integrations are available? H-VLM is designed to integrate dynamically with web user interfaces.
Usage statistics and patterns: Are there any notable observations about usage? Unknown
Additional notes
None