Contact Us 1-800-596-4880
  1. Homepage
  2. MuleSoft WebCrawler Connector (1.0)
  3. MuleSoft WebCrawler Connector Additional Configuration Information

  4. Configuring Page Operations

All Connectors

Configuring Page Operations

Configure the Get Page Content and Get Page Elements operations.

Configure the Get Page Content Operation

The Get Page Content operation retrieves the content of a specified web page in TEXT, HTML, or MARKDOWN format. Optionally target specific HTML tags via the Tag list parameter.

To configure the Get Page Content operation:

  1. Select the operation on the Anypoint Code Builder or Studio canvas.

  2. In the General properties tab for the operation, enter these values:

    • Path

      Relative path, resolved against the connection’s base URL, of the page to retrieve content from.

    • Output format

      Format of the extracted content. Select TEXT, HTML, or MARKDOWN. Default is TEXT.

    • Encoding

      Encoding for emitted page bodies. Overrides the configuration’s encoding when set.

    • Tag list

      List of HTML tags for which content must be retrieved. If specified, only content within the listed tags is returned.

    • Javascript

      Browser-context JavaScript executed after page load (not available for the HTTP connection). Trusted input only — never bind to user-controlled data; runs with full DOM access.

Crawl Options

  • Enforce robots.txt

    If true, enforce checks against the robots.txt file. Defaults to false.

Page Load Options

  • Wait on page load

    The time to wait on page load, in the time unit selected below. On the HTTP connection, values greater than 0 fail with INVALID_PARAMETERS.

  • Wait on page load unit

    Time unit for the page-load wait.

WebDriver Options (Remote WebDriver only)

These options apply only when using the Remote WebDriver connection type:

  • Wait for XPath

    The XPath to wait for before proceeding with content extraction.

  • Extract Shadow DOM

    If true, extracts the Shadow DOM content.

  • Shadow Host(s) XPath

    Shadow host(s) to extract by XPath. If not set, the whole Shadow DOM is extracted.

Timeout Options

  • Response Timeout

    Response timeout value. Overrides the configuration’s timeout when set.

  • Response Timeout Unit

    Time unit for the response timeout.

XML Configuration

This is the XML for this operation:

<ms-webcrawler:page-content
    config-ref="MuleSoft_WebCrawler_Config"
    doc:name="Get Page Content"
    doc:id="8f436a64-1795-42a0-9fe7-69be66db235b"
    path="#[payload.path]"
    outputFormat="MARKDOWN"
    waitOnPageLoad="5"
    waitForXPath="//results"
    extractShadowDom="true"
    shadowHostXPath="//results"/>

Output Configuration

This operation responds with a JSON payload. This is an example response:

{
    "title": "Example Page Title",
    "url": "https://example.com/page",
    "content": "The full text content of the page..."
}

  • title

    The page title.

  • url

    The page URL.

  • content

    The extracted content in the specified output format.

Configure the Get Page Elements Operation

The Get Page Elements operation fetches analytical data about a web page including word count, element or tag counts (H1–H5, DIV, P), link structures (internal, external, reference), and image links. Use these metrics to customize subsequent crawling operations.

To configure the Get Page Elements operation:

  1. Select the operation on the Anypoint Code Builder or Studio canvas.

  2. In the General properties tab for the operation, enter these values:

    • Path

      Relative path, resolved against the connection’s base URL, of the page to analyze.

    • Encoding

      Encoding for emitted page bodies. Overrides the configuration’s encoding when set.

    • Tag list

      List of HTML tags for which content must be retrieved.

    • Javascript

      Browser-context JavaScript executed after page load (not available for the HTTP connection). Trusted input only — never bind to user-controlled data; runs with full DOM access.

Crawl Options

  • Enforce robots.txt

    If true, enforce checks against the robots.txt file. Defaults to false.

Page Load Options

  • Wait on page load

    The time to wait on page load, in the time unit selected below. On the HTTP connection, values greater than 0 fail with INVALID_PARAMETERS.

  • Wait on page load unit

    Time unit for the page-load wait.

WebDriver Options (Remote WebDriver only)

These options apply only when using the Remote WebDriver connection type:

  • Wait for XPath

    The XPath to wait for before proceeding with content extraction.

  • Extract Shadow DOM

    If true, extracts the Shadow DOM content.

  • Shadow Host(s) XPath

    Shadow host(s) to extract by XPath. If not set, the whole Shadow DOM is extracted.

Timeout Options

  • Response Timeout

    Response timeout value. Overrides the configuration’s timeout when set.

  • Response Timeout Unit

    Time unit for the response timeout.

XML Configuration

This is the XML for this operation:

<ms-webcrawler:page-elements
    config-ref="MuleSoft_WebCrawler_Config"
    doc:name="Get Page Elements"
    doc:id="08dd9627-5220-41df-a7cf-869bff4eee91"
    path="#[payload.path]"/>

Output Configuration

This operation outputs a Binary payload containing page statistics and link details in JSON format. This is an example response:

{
    "pageStats": {
        "div": 36,
        "p": 6,
        "reference": 0,
        "internal": 7,
        "external": 2,
        "images": 4,
        "wordCount": 147,
        "h1": 1,
        "h2": 0,
        "h3": 4,
        "h4": 0,
        "h5": 0
    },
    "links": {
        "reference": [],
        "internal": ["https://example.com/docs", "https://example.com/about"],
        "external": ["https://github.com/example"],
        "images": ["https://example.com/images/logo.png"],
        "documents": [],
        "iframe": []
    },
    "title": "Example Page",
    "url": "https://example.com/"
}

  • pageStats - Counts of HTML elements and page metrics.

    • div, p, h1 through h5 - Counts of the respective HTML elements.

    • reference, internal, external - Counts of reference, internal, and external links.

    • images - Count of images on the page.

    • wordCount - Total word count of the page content.

  • links - Categorized lists of URLs found on the page.

    • reference - Reference links (anchors).

    • internal - Internal links to the same domain.

    • external - External links to other domains.

    • images - Image URLs.

    • documents - Document URLs.

    • iframe - Iframe URLs.

  • title - The page title.

  • url - The page URL.

See Also

Back to top