ShrewVoice: AI Voice Driven Browser Automation System

ShrewVoice: AI Voice Driven Browser Automation System

ShrewVoice is a voice driven browser automation system that combines a real time conversational AI agent with deterministic-yet-unbrittle execution flows generated by Shrew, our AI RPA platform. It lets users control browser workflows through natural speech while keeping execution strict and reliable using Playwright. At a high level, ShrewVoice connects voice interaction, AI reasoning, orchestration, and Shrew generated automation flows into a single working system.

System Overview

ShrewVoice is built on four main layers. The voice UI layer is a React application integrated with ElevenLabs conversational APIs, handling speech input and output in real time using WebRTC so the user can interact naturally. The agent layer handles reasoning by understanding user intent and deciding which tool to call, without directly controlling the browser. The MCP orchestration layer is written in Python using FastMCP and manages tool calls, session state, and flow execution between the agent and the runtime. The execution layer runs on Node.js with Playwright and executes flows generated by Shrew, without relying on any handwritten workflow logic.

Dependency on Shrew

ShrewVoice depends on Shrew for flow generation. Shrew is the AI RPA platform that builds the actual automation flows, which define steps such as navigation, clicking, form filling, scraping, and branching logic. ShrewVoice only executes these flows, and without Shrew generated flows it cannot run any automation.

Runtime Architecture

The frontend is a React voice interface using ElevenLabs useConversation for live voice sessions. The MCP server exposes tools such as flow fetch, step execution, and session reset, and these are the only interfaces through which the AI agent interacts with the system. A local step service connects MCP with the Node Playwright runtime and ensures execution of each step inside a live browser session. The execution engine runs Shrew generated flows inside a persistent browser session so state is preserved across steps.

Core Execution Model

ShrewVoice runs structured flows created by Shrew, where each flow contains a sequence of actions like navigate, click, fill, scrape, wait, and conditional checks. The system maintains a persistent session so browser state is preserved across tool calls, and variables are resolved at runtime so values can be dynamically injected during execution. Flows support branching based on page state so different paths can be taken at runtime, and if required input is missing the system requests it from the user instead of guessing. Scraped results are returned in structured form so the voice agent can respond naturally.

End to End Flow

The user starts a voice session from the React interface, after which the agent connects to MCP tools and requests a Shrew generated flow. The MCP layer loads the flow and begins executing it step by step, with each step run by Playwright through the Shrew runtime engine. Execution results are sent back to MCP and then to the agent, which decides the next step or requests missing input from the user. The frontend displays live step progress so the user can see execution happening in real time.

Design Principles

ShrewVoice is designed around a strict separation between intelligence and execution. The AI handles conversation and gathers missing information, while the Shrew execution engine handles all browser actions. The system enforces controlled, tool based access to the browser so the AI cannot directly manipulate it. All automation logic comes from Shrew generated flows to ensure consistency and predictability, and state is preserved across sessions so long running workflows remain stable. Human input is required for sensitive or missing data to prevent incorrect or unsafe automation.