Product overview

What is Self-Operating Computer?

Self-Operating Computer is an open-source framework that enables multimodal models to operate a desktop using the same visual inputs and mouse or keyboard outputs as a human. The model views screenshots, decides on a sequence of actions, and repeats that loop until it reaches an objective. The repository describes the November 2023 release as an early example of full computer use.

How to Use Self-Operating Computer

Install the Python package or clone the repository.
Configure the API credential for the selected multimodal model.
On macOS, grant Terminal screen-recording and accessibility permissions.
Run operate and provide the objective.
Select an alternate model, voice input, OCR, or Set-of-Mark mode when needed.

Core Features

Screen understanding: Sends the current screen to a multimodal model.
Mouse and keyboard actions: Executes model-selected desktop interactions.
Multiple models: Lists GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL, and LLaVA.
Voice objectives: Accepts spoken task instructions with optional audio dependencies.
OCR mode: Maps visible text elements to clickable coordinates.
Set-of-Mark mode: Adds detected visual markers to improve grounding.

Use Cases

Computer-use research: Compare how vision models plan and execute desktop tasks.
Desktop automation: Prototype workflows that interact with existing graphical applications.
Model evaluation: Test different multimodal models against the same operating loop.
Local experimentation: Run LLaVA through Ollama, with the repository warning that its error rate is high.

Pricing

The framework is open source. Users supply any required model API credits.

Frequently Asked Questions

Which operating systems are supported?

The repository lists macOS, Windows, and Linux with an X server.

Does it support local models?

Yes. The repository documents LLaVA hosted through Ollama as an experimental option.

Back to product directory

Self-Operating Computer