LAGIC
Lead Audience Growth Intelligence Computing
U

URLs List - Extract ALL website urls — Website | Lagic

Built For

Get a complete list of every URL on any website.

Curated by Lagic·Verified working

Configure Agent

List of website URLs to extract all pages from. You can add multiple websites for bulk processing. The scraper will automatically discover and extract all available URLs from each website.

If enabled, extracts all available URLs regardless of the maxResults setting. When disabled, the maxResults limit will be applied.

Maximum number of URLs to extract. This limit is ignored if 'Return All URLs' is enabled. Set to 0 for unlimited (same as enabling 'Return All URLs').

Filter URLs to only include those containing all specified keywords. For example, if you add 'blog', only URLs containing 'blog' will be returned (e.g., ). You can add multiple keywords - URLs must contain ALL of them.

Results to deliver

100 credits

This agent actively searches live listings — results may vary. You are only charged for what is delivered, up to this number.

Lagic Proxy

Country auto-rotated. Need a specific region? Contact support.

Pricing

1 credit per result
✓ 30 free credits on signup✓ Refund if 0 results✓ No card required

Sample Data Preview

The complete URL of each discovered page.The source sitemap URL where the page was listed, if applicable.The last modification date of the page, as reported by the server or sitemap.The suggested change frequency for search engines (e.g., daily, weekly, monthly).The priority of the page for crawling, on a scale from 0.0 to 1.0.The original website URL you provided for the crawl.
https://...https://...2026-04-05Value...Value...https://...
https://...https://...2026-04-05Value...Value...https://...
..................
Exports as:CSVXLSXJSON

Overview

Input a list of website homepages to discover and extract every single URL. Use the data for SEO audits, content inventories, or full website migrations.

This tool systematically crawls one or more websites to map out their complete structure, page by page. It starts from the homepage you provide and follows every internal link it can find, building a comprehensive list of all discoverable URLs. It's designed for SEO specialists, digital marketers, and web developers who need a complete inventory of a website's pages without clicking through every link manually. The output is a clean, structured list that can be used for site audits, planning a content migration, or analyzing a competitor's sitemap. ### How it Works You provide a list of starting URLs (like `https://example.com`). The tool then visits these pages and recursively follows all links that point to the same domain. You can optionally filter the results to only include URLs that contain specific keywords, such as `/blog/` or `/products/`, which is useful for targeting specific sections of large sites. The final output includes not just the discovered URLs, but also data often found in sitemap files, such as the page's last modification date (`lastmod`), its suggested update frequency (`changefreq`), and its relative priority (`priority`). This gives you a deeper understanding of how the site is structured for search engines.

Key Capabilities

  • ✓ The complete URL of each discovered page.
  • ✓ The source sitemap URL where the page was listed, if applicable.
  • ✓ The last modification date of the page, as reported by the server or sitemap.
  • ✓ The suggested change frequency for search engines (e.g., daily, weekly, monthly).
  • ✓ The priority of the page for crawling, on a scale from 0.0 to 1.0.
  • ✓ The original website URL you provided for the crawl.
  • ✓ Any error messages encountered while trying to access a page.
  • ✓ Conduct a comprehensive SEO audit: Generate a complete list of a website's URLs to identify orphaned pages, audit redirects, or check for inconsistent URL structures.
  • ✓ Create a full content inventory: Before a website redesign or content migration, get a full list of every page to ensure nothing is missed during the transition.
  • ✓ Analyze competitor website structures: Input a competitor's homepage to map out their entire site, understanding how they structure their product pages, blog content, and marketing funnels.
  • ✓ Verify a website migration: Crawl a site before and after a migration to a new domain or CMS, then compare the URL lists to ensure all pages were moved correctly.
  • ✓ Find specific pages in bulk across many domains: Use the keyword filter to find pages like 'contact', 'careers', or 'about us' across a list of hundreds of company websites for outreach or research.
  • ✓ Build targeted outreach lists: Filter a target website's URLs for '/blog/' to get a complete list of every article, creating a resource for targeted link-building or content partnership campaigns.

Field Dictionary

How To Run This Extractor

1

Paste one or more website homepages into the 'Website URLs' field.

2

Optionally, add keywords to the 'Keywords Filter' to only extract URLs containing those terms (e.g., 'blog').

3

Choose whether to extract all URLs or limit the number using the 'Max Results' field.

4

Run the tool to start the crawl.

5

Download the complete list of discovered URLs as a CSV or JSON file.

Frequently Asked Questions

Do I need technical skills to use this?
No, you only need to provide a list of website URLs. There is no coding required.
What formats can I export the data in?
Is it legal to crawl websites?
How many websites can I crawl at once?
Can I use this for client projects?
What's the difference between this and a sitemap generator?
How reliable is the crawl?
Can I schedule this to run automatically?
How does the 'Keywords Filter' work if I add multiple words?
What does 'lastmod' or 'priority' mean in the output?
What happens if a website doesn't have a sitemap.xml file?