Project Astra

What is Project Astra?

Project Astra is a Google DeepMind research prototype exploring capabilities for a universal AI assistant. It combines live audio, camera input, screen sharing, tools, personalization, and multimodal memory so an assistant can understand the user's current environment, hold a natural conversation, and take actions. Some capabilities first explored in Project Astra have been integrated into Gemini Live, while Google is also researching Search experiences and device formats such as glasses.

The prototype can respond proactively, ignore irrelevant background speech, highlight important objects on screen, and use Google services such as Search, Gmail, Calendar, Maps, Photos, and Lens. Its memory carries preferences and earlier interaction details into later responses and can persist when a user switches between an Android phone and prototype glasses.

How to Use Project Astra

1. Join the trusted-tester waitlist if eligible for the limited research program.
2. On a supported Android phone, point the camera at an object or environment or share the screen.
3. Begin a spoken conversation about what the assistant can see or what the user needs to do.
4. Let the prototype retrieve shared content, use approved Google tools, or react to changes in the visual scene.
5. Continue the conversation on another supported device while cross-device memory preserves context.

Core Features

• Natural voice interaction: Uses improved audio input and output for conversations across languages.
• Live visual understanding: Interprets camera views and screen sharing as the visual context changes.
• Proactive assistance: Can begin relevant responses and take actions without waiting for rigid commands.
• Google tool use: Uses services such as Search, Gmail, Calendar, Maps, Photos, and Lens.
• Personalized reasoning: Retains user preferences and explains the reasoning behind tailored answers.
• Multimodal memory: Combines different data types and remembers key details across interactions and devices.

Use Cases

• Visual assistance: Describe surroundings or identify objects through a phone camera.
• Low-vision support: Help blind and low-vision users understand unfamiliar spaces with professional human oversight in the tester program.
• Screen assistance: Discuss visible content while the user shares a phone screen.
• Personal tasks: Retrieve a shared manual or recipe and apply it to the current situation.
• Cross-device help: Move between a phone and prototype glasses without restarting the conversation.

Frequently Asked Questions

Is Project Astra publicly available?

No. Google DeepMind describes it as a research prototype used by a limited number of trusted testers and provides a waitlist.

Which devices does Project Astra use?

The page lists Android phones and prototype glasses.

Are any Project Astra capabilities available in a Google product?

Yes. Google says some capabilities were first explored in Project Astra and later brought to Gemini Live, including screen sharing and video understanding.