Configuring Crawl Operations

Configure the Get SiteMap operation.

Configure the Get SiteMap Operation

The Get SiteMap operation returns a sitemap in standard XML format following the sitemap protocol. The operation first tries to discover URLs from the site’s own sitemap and robots.txt, then falls back to a breadth-first crawl up to the specified depth. The sitemap results can be used to customize subsequent crawling of specific pages.

Priority is calculated based on depth: depth 0 = priority 1.0, depth 1 = priority 0.9, and so on. Depth greater than 10 always yields priority 0.0.

To configure the Get SiteMap operation:

Select the operation on the Anypoint Code Builder or Studio canvas.
In the General properties tab for the operation, enter these values:
- Path
  
  Relative path, resolved against the connection’s base URL, where the sitemap crawl starts.
- Maximum depth
  
  The maximum depth to reach while crawling a website. Defaults to 0.
- Encoding
  
  Encoding for emitted page bodies. Overrides the configuration’s encoding when set.
- Restrict crawl under URL
  
  If true, only internal pages under the base URL are crawled. Defaults to false.
- Regex URLs filter logic
  
  Type of filter logic to apply to regex URLs while crawling. Select INCLUDE to crawl only matching pages, or EXCLUDE to skip matching pages.
- Regex URLs List
  
  List of regex patterns for URLs to include or exclude while crawling, based on the filter logic selected above.

Crawl Options

Delay between pages

The delay introduced between each request, in the time unit selected below. Defaults to 0.
Delay time unit

Time unit for the delay between pages.
Enforce robots.txt

If true, enforce checks against the robots.txt file. Defaults to false.

Page Load Options

Wait on page load

The time to wait on page load, in the time unit selected below. On the HTTP connection, values greater than 0 fail with INVALID_PARAMETERS.
Wait on page load unit

Time unit for the page-load wait.

WebDriver Options (Remote WebDriver only)

These options apply only when using the Remote WebDriver connection type:

Wait for XPath

The XPath to wait for before proceeding with content extraction.
Extract Shadow DOM

If true, extracts the Shadow DOM content.
Shadow Host(s) XPath

Shadow host(s) to extract by XPath. If not set, the whole Shadow DOM is extracted.

Timeout Options

Response Timeout

Response timeout value. Overrides the configuration’s timeout when set.
Response Timeout Unit

Time unit for the response timeout.

XML Configuration

This is the XML for this operation:

<ms-webcrawler:get-sitemap
    config-ref="MuleSoft_WebCrawler_Config"
    doc:name="Get SiteMap"
    doc:id="6410aae9-21d4-4005-8d86-ac9a136657b4"
    path="#[payload.path]"
    maxDepth="#[payload.maxDepth]"
    restrictToPath="true"/>

Output Configuration

This operation responds with an XML payload that uses the sitemap protocol specification. This is an example response:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2024-01-15T10:30:00Z</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/docs</loc>
    <priority>0.9</priority>
  </url>
  <url>
    <loc>https://example.com/docs/getting-started</loc>
    <priority>0.8</priority>
  </url>
</urlset>

The <lastmod> and <changefreq> elements appear only for URLs where the crawled site’s own source sitemap supplies them, so some <url> entries include them while others contain only <loc> and <priority>.

The response attributes include:

count

The total number of URLs in the sitemap.
depth

The configured maximum crawl depth.
url

The root URL of the crawled site.