Web Surfing Agent Core

Overview

This agent framework encapsulates sophisticated prompting logic designed to guide large language models (LLMs) through complex web browsing tasks. It moves beyond simple search queries by providing the model with detailed contextual information, including visible elements, page structure hints, and a decision matrix for action selection.

The core function is to help an AI decide how to answer a query—whether by clicking something immediately visible, scrolling to find content elsewhere on the current page, or initiating an entirely new search.

Capabilities

• Context-Aware Prompting: Provides structured prompts that include descriptions of the current viewport and surrounding page context.
• Interactive Element Identification: Processes and incorporates details about interactive elements (buttons, links, input fields) with bounding box information.
• Decision Framework Guidance: Explicitly guides the LLM to weigh three primary action scopes: Current Viewport actions, Full Page content analysis, or External Search necessity.
• Tool Selection Orchestration: Structures the final output requirement, forcing the model to select from a defined set of available tools or provide a direct answer.

Example Use Cases

1. Complex Form Filling: Guiding an agent to navigate through multi-step online applications by correctly identifying and interacting with sequential form fields.
2. Comparative Research: Instructing an agent to visit multiple product pages on different domains, summarize key specs from each, and synthesize a comparison table for the user.
3. Troubleshooting Guides: Having the agent follow instructions within a technical support website—scrolling past FAQs, clicking specific links, and extracting the final solution steps.