Rag Knowledge Graph Builder — Web Pages | Lagic

Configure Agent

Sample Data Preview

Overview

This tool crawls specified websites, intelligently chunks their content, and generates hypothetical questions for each section, creating a structured knowledge graph optimized for Retrieval Augmented Generation (RAG) and AI-driven insights.

### Build AI-Ready Knowledge Bases from Any Website To build effective AI applications, especially those using Retrieval Augmented Generation (RAG), you need structured, relevant data. This tool solves the challenge of turning raw website content into a finely tuned knowledge base. It acts as a specialized web crawler and content processor, designed to prepare data specifically for AI consumption. #### How it Works: From Crawl to Knowledge Graph Start by providing a list of URLs, and the tool will systematically crawl those pages and follow links within the same domain. You have granular control over the crawl: define the maximum number of pages and the depth of links to follow. This ensures you gather exactly the scope of information you need without over-collecting. Once content is extracted, the tool intelligently breaks it down into 'chunks' – manageable segments optimized for RAG performance. You set the target token size for these chunks and define how much overlap there should be between consecutive chunks to maintain context across segments. This careful chunking is crucial for ensuring your AI can retrieve precise and relevant information later. #### Intelligent Question Generation for Enhanced Retrieval A key feature is the generation of 'hypothetical questions' for each content chunk. These questions act as metadata, helping your AI system understand the core topics and potential queries a user might have about that specific piece of information. You can choose to generate these questions using a free, rule-based 'native' method, or opt for higher-quality, AI-generated questions by integrating with OpenAI or Anthropic (requiring your API key). #### Customization and Control The tool offers extensive customization to tailor your knowledge graph: * **URL Filtering:** Include or exclude specific URL patterns using glob patterns to focus your crawl. * **Content Exclusion:** Use CSS selectors to ignore unwanted elements like navigation, footers, or advertisements, ensuring only relevant content is processed. * **Metadata Inclusion:** Optionally include page titles, descriptions, and other metadata in your output for richer context. * **Proxy Support:** Configure proxy settings for complex crawling scenarios. #### Output for AI Applications The output is a structured dataset containing the original content chunks, the generated hypothetical questions, and comprehensive metadata for each chunk and page. This data is immediately ready to be ingested into vector databases, powering chatbots, AI assistants, search engines, and other RAG-based applications with accurate, context-rich information sourced directly from your chosen websites.