The AI Agent Index

Documenting the technical and safety features of deployed agentic AI systems

OS-Copilot

Basic information

Website: https://os-copilot.github.io/

Short description: OS-Copilot is “a framework to build generalist agents capable of interfacing with comprehensive elements in an operating system (OS), including the web, code terminals, files, multimedia, and various third-party applications”. FRIDAY is “a self-improving embodied agent for automating general computer tasks”, built using the OS-Copilot framework [source]

Intended uses: What does the developer say it’s for? Potential use cases include operations requiring interactions with BASH, a Python interpreter, the keyboard/mouse, and an API. More details can be found in Appendix C [source]

Date(s) deployed: Paper arXived February 12, 2024 [source]

Developer

Website: https://os-copilot.github.io/

Legal name: Shanghai AI Laboratory (et al.) [source]

Entity type: Unknown

Country (location of developer or first author’s first affiliation): China [source]

Safety policies: What safety and/or responsibility policies are in place? Unknown

System components

Backend model: What model(s) are used to power the system? Variable for OS-Copilot. For FRIDAY, some backend models they have used include GPT-4 and GPT-4-Turbo plugins.

Publicly available model specification: Is there formal documentation on the system’s intended uses and how it is designed to behave in them? None

Reasoning, planning, and memory implementation: How does the system ‘think’? Implementation of OS-Copilot consists of three components: 1) A planner that “reasons over user requests and decomposes complex ones into simpler subtasks”; 2) A configurator, with working memory, that “takes a subtask from the planner and configures it to help the actor complete the subtask”; and 3) An actor that consists of an executor, proposing and executing appropriate actions, and a critic, assessing the outcomes of the executor’s actions [source]

Observation space: What is the system able to observe while ‘thinking’? Common observation space is user input and outcome of agent actions.

Action space/tools: What direct actions can the system take? The OS-Copilot framework “consolidates common practices for OS manipulation, including Python code interpreter (Significant-Gravitas, 2023), bash terminal, mouse/keyboard control (Cheng et al., 2024), and API calls (Qin et al., 2023)” [source]

User interface: How do users interact with the system? There is a user interface to interact with OS-Copilot [source].

Development cost and compute: What is known about the development costs? Unknown

Guardrails and oversight

Accessibility of components:

  • Weights: Are model parameters available? N/A; backends various models
  • Data: Is data available? N/A; backends various models
  • Code: Is code available? Available [source]
  • Scaffolding: Is system scaffolding available? Available [source]
  • Documentation: Is documentation available? Available [source]

Controls and guardrails: What notable methods are used to protect against harmful actions? Unknown

Customer and usage restrictions: Are there know-your-customer measures or other restrictions on customers? None

Monitoring and shutdown procedures: Are there any notable methods or protocols that allow for the system to be shut down if it is observed to behave harmfully? Unknown

Evaluation

Notable benchmark evaluations: On “GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods by 35%” (40.86% on level 1, 20.13% on level 2, 6.12% on level 3 tasks). Additionally, on the SheetCopilot-20 dataset, FRIDAY passes 60% of the tasks with each task being performed only once [source]

Bespoke testing: FRIDAY is essentially a demo for OS-Copilot [source].

Safety: Have safety evaluations been conducted by the developers? What were the results? None

Publicly reported external red-teaming or comparable auditing:

  • Personnel: Who were the red-teamers/auditors? None
  • Scope, scale, access, and methods: What access did red-teamers/auditors have and what actions did they take? None
  • Findings: What did the red-teamers/auditors conclude? None

Ecosystem information

Interoperability with other systems: What tools or integrations are available? FRIDAY can work with other systems since it is equipped with tools for OS manipulation that include interacting with a Python code interpreter, bash terminal, mouse/keyboard control, and API calls.

Usage statistics and patterns: Are there any notable observations about usage? The OS-Copilot github repository has 177 forks and 1.6k stars [source].

Additional notes

Some of the same developers have released a foundation model known as OS-ATLAS to serve as a backend for agents [source]