Skip to main content

​
Overview

The Get URLs from Sitemap block extracts URLs from a website’s XML sitemap. Use it to discover all pages on a website, filter for specific sections, and feed URLs into other blocks for bulk scraping, analysis, or content processing.

​
Configuration

​
Sitemap URL

Enter the URL of the sitemap to parse. This is typically located at /sitemap.xml on most websites. Common sitemap locations:
  • https://example.com/sitemap.xml
  • https://example.com/sitemap_index.xml
  • https://example.com/post-sitemap.xml
This field supports placeholders for dynamic sitemap URLs:
Most websites list their sitemap location in the robots.txt file at https://example.com/robots.txt. Check there if you can’t find the sitemap.
Limit how many URLs are returned from the sitemap.
  • Default: 100 URLs
  • Minimum: 1 URL
Set a lower limit when:
  • Testing your workflow before running at scale
  • Processing only a sample of pages
  • Staying within credit budgets for subsequent scraping
Filter URLs to include only those matching specific patterns. Enter one or more text patterns separated by commas. How it works:
  • URLs must contain at least one of the specified patterns (OR logic)
  • Matching is case-sensitive
  • Partial matches work (e.g., /blog/ matches /blog/post-title)
Examples: This field supports placeholders:
Filter out URLs containing specific patterns. Enter one or more text patterns separated by commas. How it works:
  • URLs matching any pattern are removed (OR logic)
  • Applied after include filter
  • Useful for removing unwanted page types
Examples:

​
Output

The block returns an array of URL strings extracted from the sitemap.

​
Output Example

​
Accessing URLs

Get all URLs:
Get first URL:
Get URL count:
Loop through URLs: Use with the Loop block to process each URL individually.

​
Combining Filters

Include and exclude filters work together:
  1. First, include filter is applied (if set)
  2. Then, exclude filter removes unwanted URLs
  3. Finally, the limit is applied
Example:
  • Sitemap: https://example.com/sitemap.xml
  • Include: /blog/
  • Exclude: /tag/, /author/
  • Limit: 50
Result: Up to 50 blog post URLs, excluding tag and author pages.

​
Best Practices

  • Start with a small limit when testing workflows
  • Use include filters to target specific content types
  • Exclude pagination, tags, and archives for cleaner results
  • Check the sitemap structure first to understand URL patterns
  • Combine with Loop and Web Scrape blocks for bulk content extraction
  • Some sites have multiple sitemaps; check the sitemap index

​
Common Use Cases

​
Example Workflow: Bulk Content Analysis

Analyze all blog posts from a competitor:
  1. Get URLs from Sitemap Block:
    • Sitemap URL: https://competitor.com/sitemap.xml
    • Include: /blog/
    • Exclude: /tag/, /category/, /author/
    • Limit: 100
  2. Loop Block: Iterate through each URL
  3. Web Scrape Block:
    • URL: {{current}}
    • Format: Markdown
    • Only Main Content: On
  4. LLM Block: Analyze content themes and structure
  5. Google Sheets Block: Store analysis results

​
Example Workflow: Site Inventory

Create a complete inventory of a website’s pages:
  1. Get URLs from Sitemap Block:
    • Sitemap URL: https://yoursite.com/sitemap.xml
    • Limit: 500
  2. Loop Block: Process each URL
  3. Web Scrape Block:
    • URL: {{current}}
    • Format: Markdown
    • Include Metadata: On
  4. Google Sheets Block: Append URL, title, and description

​
Troubleshooting

​
What’s Next

Now that you understand the Get URLs from Sitemap block: