Contact Us 1-800-596-4880
  1. Homepage
  2. MuleSoft WebCrawler Connector (1.0)

  3. MuleSoft WebCrawler Connector Reference

All Connectors

MuleSoft WebCrawler Connector Reference

Anypoint Connector for MuleSoft WebCrawler (MuleSoft WebCrawler Connector) enhances data ingestion capabilities by allowing you to extract and process data from web pages.

Configurations

Config

Default configuration

Parameters

Name Type Description Default Value Required

Name

String

The name for this configuration. Connectors reference the configuration with this name.

x

Connection

The connection types to provide to this configuration.

x

Encoding

String

Encoding for emitted page bodies. Can be overridden per-operation. Falls back to the Mule context default encoding when unset.

Response Timeout

Number

Response timeout value.

60

Response Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Response Timeout field.

SECONDS

Name

String

The identifier of this element used to reference it in other components.

x

Expiration Policy

Expiration Policy

Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration. This doesn’t mean that the platform expires the instance at the exact moment that it becomes eligible. Mule purges the instances as appropriate.

Connection Types

HTTP

Use for static websites that return their content directly in the HTML response and can be crawled without a remote browser.

Parameters
Name Type Description Default Value Required

Base URL

String

Base URL against which relative paths are resolved.

x

Browser User Agent

String

Browser user-agent advertised on outbound requests.

Proxy Configuration

Default HTTP Proxy Config

Proxy configuration for outbound HTTP requests.

Response timeout

Number

Connect and read timeout for the connection’s HTTP client, in the time unit selected below.

30

Response timeout unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Response timeout field.

SECONDS

TLS Configuration

TLS

TLS configuration for secure HTTPS connections.

Reconnection

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.
Remote WebDriver

Use for websites that render content dynamically using JavaScript instead of returning it directly in the HTML response.

Parameters
Name Type Description Default Value Required

Base URL

String

Base URL against which relative paths are resolved.

x

Browser User Agent

String

Browser user-agent advertised on outbound requests.

Proxy Configuration

Default HTTP Proxy Config

Proxy configuration for outbound HTTP requests.

Response timeout

Number

Connect and read timeout for the connection’s HTTP client, in the time unit selected below.

30

Response timeout unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Response timeout field.

SECONDS

TLS Configuration

TLS

TLS configuration for secure HTTPS connections.

Remote Browser URL

String

URL of your own remote Selenium WebDriver endpoint (for example, Selenium Grid, standalone ChromeDriver, or BrowserStack). MuleSoft doesn’t provide or host a remote browser. Chromium-only (Chrome, Chromium, Edge).

x

Browser Authentication

Authentication for the remote browser hub connection. Applied when a session is created. Changed credentials take effect on the next fresh session.

Reconnection

Reconnection

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.

Pooling Profile

Pooling Profile

Characteristics of the connection pool.

Supported Operations

Associated Sources

Operations

Get SiteMap

<ms-webcrawler:get-sitemap>

Crawls a website to a specified depth and returns a sitemap in standard XML format following the sitemap protocol specification.

Parameters

Name Type Description Default Value Required

Configuration

String

The name of the configuration to use.

x

Path

String

Relative path, resolved against the connection’s base URL, where the sitemap crawl starts.

Maximum depth

Number

The maximum depth to reach while crawling a website.

0

Encoding

String

Encoding for emitted page bodies. Overrides the configuration’s encoding when set.

Output Mime Type

String

The mime type of the payload that this operation outputs.

Output Encoding

String

The encoding of the payload that this operation outputs.

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component.

x

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Restrict crawl under URL

Boolean

If true, only internal pages are crawled.

false

Regex URLs filter logic

Enumeration, one of:

  • INCLUDE

  • EXCLUDE

Type of filter logic to apply to regex URLs while crawling.

Regex URLs List

Array of String

List of regex patterns for URLs to include or exclude while crawling.

Delay between pages

Number

The delay introduced between each request.

0

Delay time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delay between pages field.

SECONDS

Enforce robots.txt

Boolean

If true, enforce checks against the robots.txt file.

false

Wait on page load

Number

The time to wait on page load. On the HTTP connection, values greater than 0 fail with INVALID_PARAMETERS.

0

Wait on page load unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Wait on page load field.

SECONDS

Wait for XPath

String

The XPath to wait for (not available for HTTP connection).

Extract Shadow DOM

Boolean

Extract the Shadow DOM content (not available for HTTP connection).

false

Shadow Host(s) XPath

String

Shadow host(s) to extract by XPath (not available for HTTP connection).

Response Timeout

Number

Response timeout value. Overrides the configuration’s timeout when set.

Response Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Response Timeout field.

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Error Mappings

Array of Error Mapping

Set of error mappings.

Reconnection Strategy

A retry strategy in case of connectivity errors.

Output

Type

urlset

Attributes Type

Sitemap Response Attributes

For Configurations

Throws

  • MS-WEBCRAWLER:CONNECTIVITY

  • MS-WEBCRAWLER:CRAWLER_ERROR

  • MS-WEBCRAWLER:INVALID_PARAMETERS

  • MS-WEBCRAWLER:PAGE_FETCH_FAILED

  • MS-WEBCRAWLER:PAGE_LOAD_TIMEOUT

  • MS-WEBCRAWLER:RETRY_EXHAUSTED

  • MS-WEBCRAWLER:ROBOTS_DISALLOWED

  • MS-WEBCRAWLER:TIMEOUT

Get Page Content

<ms-webcrawler:page-content>

Retrieves the content of a specified web page in TEXT, HTML, or MARKDOWN format.

Parameters

Name Type Description Default Value Required

Configuration

String

The name of the configuration to use.

x

Path

String

Relative path, resolved against the connection’s base URL, of the page to retrieve content from.

Javascript

String

Browser-context JavaScript executed after page load (not available for HTTP connection). Trusted input only — never bind to user-controlled data; runs with full DOM access.

Output format

Enumeration, one of:

  • TEXT

  • HTML

  • MARKDOWN

Format of the extracted content.

TEXT

Enforce robots.txt

Boolean

If true, enforce checks against the robots.txt file.

false

Encoding

String

Encoding for emitted page bodies. Overrides the configuration’s encoding when set.

Output Mime Type

String

The mime type of the payload that this operation outputs.

Output Encoding

String

The encoding of the payload that this operation outputs.

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component.

x

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Tag list

Array of String

List of HTML tags for which content must be retrieved.

Wait on page load

Number

The time to wait on page load. On the HTTP connection, values greater than 0 fail with INVALID_PARAMETERS.

0

Wait on page load unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Wait on page load field.

SECONDS

Wait for XPath

String

The XPath to wait for (not available for HTTP connection).

Extract Shadow DOM

Boolean

Extract the Shadow DOM content (not available for HTTP connection).

false

Shadow Host(s) XPath

String

Shadow host(s) to extract by XPath (not available for HTTP connection).

Response Timeout

Number

Response timeout value. Overrides the configuration’s timeout when set.

Response Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Response Timeout field.

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Error Mappings

Array of Error Mapping

Set of error mappings.

Reconnection Strategy

A retry strategy in case of connectivity errors.

Output

Type

Any

Attributes Type

Page Response Attributes

For Configurations

Throws

  • MS-WEBCRAWLER:CONNECTIVITY

  • MS-WEBCRAWLER:INVALID_PARAMETERS

  • MS-WEBCRAWLER:PAGE_FETCH_FAILED

  • MS-WEBCRAWLER:PAGE_LOAD_TIMEOUT

  • MS-WEBCRAWLER:RETRY_EXHAUSTED

  • MS-WEBCRAWLER:ROBOTS_DISALLOWED

  • MS-WEBCRAWLER:TIMEOUT

Get Page Elements

<ms-webcrawler:page-elements>

Fetches analytical data about a web page including word count, element or tag counts, link structures, and image links.

Parameters

Name Type Description Default Value Required

Configuration

String

The name of the configuration to use.

x

Path

String

Relative path, resolved against the connection’s base URL, of the page to analyze.

Javascript

String

Browser-context JavaScript executed after page load (not available for HTTP connection). Trusted input only — never bind to user-controlled data; runs with full DOM access.

Enforce robots.txt

Boolean

If true, enforce checks against the robots.txt file.

false

Encoding

String

Encoding for emitted page bodies. Overrides the configuration’s encoding when set.

Output Mime Type

String

The mime type of the payload that this operation outputs.

Output Encoding

String

The encoding of the payload that this operation outputs.

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component.

x

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Tag list

Array of String

List of HTML tags for which content must be retrieved.

Wait on page load

Number

The time to wait on page load. On the HTTP connection, values greater than 0 fail with INVALID_PARAMETERS.

0

Wait on page load unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Wait on page load field.

SECONDS

Wait for XPath

String

The XPath to wait for (not available for HTTP connection).

Extract Shadow DOM

Boolean

Extract the Shadow DOM content (not available for HTTP connection).

false

Shadow Host(s) XPath

String

Shadow host(s) to extract by XPath (not available for HTTP connection).

Response Timeout

Number

Response timeout value. Overrides the configuration’s timeout when set.

Response Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Response Timeout field.

Target Variable

String

Name of the variable that stores the operation’s output.

Target Value

String

An expression to evaluate against the operation’s output and store the expression outcome in the target variable.

#[payload]

Error Mappings

Array of Error Mapping

Set of error mappings.

Reconnection Strategy

A retry strategy in case of connectivity errors.

Output

Type

Binary

Attributes Type

Page Response Attributes

For Configurations

Throws

  • MS-WEBCRAWLER:CONNECTIVITY

  • MS-WEBCRAWLER:INVALID_PARAMETERS

  • MS-WEBCRAWLER:PAGE_FETCH_FAILED

  • MS-WEBCRAWLER:PAGE_LOAD_TIMEOUT

  • MS-WEBCRAWLER:RETRY_EXHAUSTED

  • MS-WEBCRAWLER:ROBOTS_DISALLOWED

  • MS-WEBCRAWLER:TIMEOUT

Sources

On Content Crawl

<ms-webcrawler:crawl-content-listener>

A source that crawls a website starting from a configured path, emitting each discovered page as a Mule event.

Parameters

Name Type Description Default Value Required

Configuration

String

The name of the configuration to use.

x

Path

String

Relative path, resolved against the connection’s base URL, where the crawl starts. When empty or /, the crawl starts at the base URL itself. Fully-qualified URLs are rejected with INVALID_PARAMETERS.

Additional seed URLs

String

Comma-separated list of additional fully-qualified seed URLs to enqueue alongside the path. Each entry must be a fully-qualified URL, and relative paths are dropped. Off-host and off-path entries are dropped and a warning is logged.

Maximum depth

Number

The maximum depth to reach while crawling a website.

0

Queue Name

String

VM queue used to schedule URLs for crawling. When omitted, defaults to webcrawler-{configName}-source-queue.

Object Store

ObjectStore

Object store used for crawl visit fingerprints (content hash, ETag, last-modified, and correlation ID) and the active crawl identity. Use a persistent store so that reconnection and app restart resume the same crawl.

x

Output format

Enumeration, one of:

  • TEXT

  • HTML

  • MARKDOWN

Format applied to each emitted page body.

TEXT

Number Of Consumers

Number

Number of concurrent queue consumers used by this source.

1

Hash route mode

Enumeration, one of:

  • IGNORE

  • PATH

  • PRESERVE

How to treat URLs containing #fragments. IGNORE strips fragments before crawling (default, fits classic server-rendered sites). PATH rewrites SPA-style hash routes (for example, #/page) into real paths (/page). PRESERVE keeps the URL verbatim, fragment included.

IGNORE

Crawl schedule

Periodic re-crawl mode. No Schedule crawls once at start (default). Interval re-seeds at a fixed frequency and time unit. Ticks that occur while a crawl is in progress are coalesced into one pending reseed (no consumer stacking). The schedule is registered on all replicas, but the actual reseed (new correlation ID and seed enqueue) runs on the primary node only.

Encoding

String

Encoding for emitted page bodies. Overrides the configuration’s encoding when set.

Primary Node Only

Boolean

Whether this source should only be executed on the primary node when running in a cluster.

Config Ref

ConfigurationProvider

The name of the configuration to use to execute this component.

x

Streaming Strategy

Configures how Mule processes streams. Repeatable streams are the default behavior.

Redelivery Policy

Redelivery Policy

Defines a policy for processing the redelivery of the same message.

Restrict crawl under URL

Boolean

If true, only internal pages are crawled.

false

Regex URLs filter logic

Enumeration, one of:

  • INCLUDE

  • EXCLUDE

Type of filter logic to apply to regex URLs while crawling.

Regex URLs List

Array of String

List of regex patterns for URLs to include or exclude while crawling.

Pages

Boolean

If true, page content of the crawled URL is emitted.

true

Images

Boolean

If true, images referenced on the crawled URL are emitted.

false

Max image number for each page

Number

Maximum number of images to fetch for each page. 0 means no limit.

0

Documents

Boolean

If true, documents referenced on the crawled URL are emitted.

false

Max document number for each page

Number

Maximum number of documents to fetch for each page. 0 means no limit.

0

Additional Allowed Base URLs

String

Optional comma-separated allowlist for image and document URLs. When set, assets must be under the base URL or one of these base URLs.

Retrieve meta tags

Boolean

If true, meta tags are retrieved from each crawled page.

false

Tag list

Array of String

List of HTML tags for which content must be retrieved.

Delay between pages

Number

The delay introduced between each request.

0

Delay time unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Delay between pages field.

SECONDS

Enforce robots.txt

Boolean

If true, enforce checks against the robots.txt file.

false

Wait on page load

Number

The time to wait on page load. On the HTTP connection, values greater than 0 fail with INVALID_PARAMETERS.

0

Wait on page load unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Wait on page load field.

SECONDS

Wait for XPath

String

The XPath to wait for (not available for HTTP connection).

Extract Shadow DOM

Boolean

Extract the Shadow DOM content (not available for HTTP connection).

false

Shadow Host(s) XPath

String

Shadow host(s) to extract by XPath (not available for HTTP connection).

Response Timeout

Number

Response timeout value. Overrides the configuration’s timeout when set.

Response Timeout Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Response Timeout field.

Reconnection Strategy

A retry strategy in case of connectivity errors.

Output

Type

Any

Attributes Type

Page Response Attributes

For Configurations

Types

Default HTTP Proxy Config

Field Type Description Default Value Required

Host

String

The proxy host.

x

Port

Number

The proxy port.

x

Username

String

Username for proxy authentication.

Password

String

Password for proxy authentication.

Non Proxy Hosts

String

Comma-separated list of hosts that shouldn’t go through the proxy.

TLS

Field Type Description Default Value Required

Enabled Protocols

String

A comma-separated list of protocols enabled for this context.

Enabled Cipher Suites

String

A comma-separated list of cipher suites enabled for this context.

Trust Store

Trust Store

Configures the trust store for TLS.

Key Store

Key Store

Configures the key store for TLS.

Revocation Check

Configures certificate revocation checking.

Trust Store

Field Type Description Default Value Required

Path

String

The location (resolved relative to the current classpath and file system, if possible) of the trust store.

Password

String

The password used to protect the trust store.

Type

String

The type of store used.

Algorithm

String

The algorithm used by the trust store.

Insecure

Boolean

If true, no certificate validations are performed, rendering connections vulnerable to attacks. Use at your own risk.

Key Store

Field Type Description Default Value Required

Path

String

The location (resolved relative to the current classpath and file system, if possible) of the key store.

Type

String

The type of store used.

Alias

String

When the key store contains many private keys, this attribute indicates the alias of the key to use. If not defined, the first key in the file is used.

Key Password

String

The password used to protect the private key.

Password

String

The password used to protect the key store.

Algorithm

String

The algorithm used by the key store.

Standard Revocation Check

Field Type Description Default Value Required

Only End Entities

Boolean

Only verify the last element of the certificate chain.

Prefer Crls

Boolean

Try CRL instead of OCSP first.

No Fallback

Boolean

Don’t use the secondary checking method (the one not selected before).

Soft Fail

Boolean

Avoid verification failure when the revocation server can’t be reached or is busy.

Custom OCSP Responder

Field Type Description Default Value Required

Url

String

The URL of the OCSP responder.

Cert Alias

String

Alias of the signing certificate for the OCSP response (must be in the trust store), if present.

CRL File

Field Type Description Default Value Required

Path

String

The path to the CRL file.

Reconnection

Field Type Description Default Value Required

Fails Deployment

Boolean

When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.

Reconnection Strategy

The reconnection strategy to use.

Reconnect

Field Type Description Default Value Required

Frequency

Number

How often in milliseconds to reconnect.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

Count

Number

How many reconnection attempts to make.

Reconnect Forever

Field Type Description Default Value Required

Frequency

Number

How often in milliseconds to reconnect.

Blocking

Boolean

If false, the reconnection strategy runs in a separate, non-blocking thread.

Pooling Profile

Field Type Description Default Value Required

Max Active

Number

Controls the maximum number of Mule components that can be borrowed from a session at one time. When set to a negative value, there’s no limit to the number of components that may be active at one time. When maxActive is exceeded, the pool is said to be exhausted.

Max Idle

Number

Controls the maximum number of Mule components that can sit idle in the pool at any time. When set to a negative value, there’s no limit to the number of Mule components that may be idle at one time.

Max Wait

Number

Specifies the number of milliseconds to wait for a pooled component to become available when the pool is exhausted and the exhaustedAction is set to WHEN_EXHAUSTED_WAIT.

Min Eviction Millis

Number

Determines the minimum amount of time an object may sit idle in the pool before it’s eligible for eviction. When non-positive, no objects are evicted from the pool due to idle time alone.

Eviction Check Interval Millis

Number

Specifies the number of milliseconds between runs of the object evictor. When non-positive, no object evictor is executed.

Exhausted Action

Enumeration, one of:

  • WHEN_EXHAUSTED_GROW

  • WHEN_EXHAUSTED_WAIT

  • WHEN_EXHAUSTED_FAIL

Specifies the behavior of the Mule component pool when the pool is exhausted. Possible values are: WHEN_EXHAUSTED_FAIL, which throws a NoSuchElementException; WHEN_EXHAUSTED_WAIT, which blocks by invoking Object.wait(long) until a new or idle object is available; or WHEN_EXHAUSTED_GROW, which creates a new Mule instance and returns it, essentially making maxActive meaningless. If a positive maxWait value is supplied, it blocks for at most that many milliseconds, after which a NoSuchElementException is thrown. If maxThreadWait is a negative value, it blocks indefinitely.

Initialisation Policy

Enumeration, one of:

  • INITIALISE_NONE

  • INITIALISE_ONE

  • INITIALISE_ALL

Determines how components in a pool should be initialized. Possible values are: INITIALISE_NONE (doesn’t load any components into the pool on startup), INITIALISE_ONE (loads one initial component into the pool on startup), or INITIALISE_ALL (loads all components in the pool on startup).

Disabled

Boolean

Whether pooling should be disabled.

Expiration Policy

Field Type Description Default Value Required

Max Idle Time

Number

The maximum amount of time a dynamic configuration instance can be idle before it’s considered eligible for expiration.

Time Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Max Idle Time field.

Basic

Field Type Description Default Value Required

Username

String

Username for basic authentication.

x

Password

String

Password for basic authentication.

x

Preemptive

Boolean

Whether to send credentials preemptively.

false

API Access Token

Field Type Description Default Value Required

Token

String

The API access token used to authenticate to the remote browser endpoint.

x

Interval

Field Type Description Default Value Required

Frequency

Number

How often the crawl re-seeds.

x

Unit

Enumeration, one of:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

Time unit for the Frequency field.

SECONDS

Page Response Attributes

Field Type Description Default Value Required

Cache Control

String

The cache-control header value from the page response.

Content Type

String

The content type of the page response.

Depth

Number

The crawl depth at which this page was discovered.

Etag

String

The ETag header value from the page response.

Event Type

Enumeration, one of:

  • PAGE

  • IMAGE

  • DOCUMENT

The type of emitted event.

File Name

String

The file name of the emitted image or document.

Last Modified

DateTime

The last-modified timestamp from the page response.

Meta Tags

String

The meta tags retrieved from the page.

Other Attributes

Object

Additional attributes returned by the operation.

Size

Number

The size of the emitted content in bytes.

Status Code

Number

The HTTP status code of the page response.

Title

String

The page title.

Url

String

The page URL.

Repeatable In Memory Stream

Field Type Description Default Value Required

Initial Buffer Size

Number

The amount of memory allocated to consume the stream and provide random access to it. If the stream contains more data than fits in this buffer, the buffer expands according to the Buffer Size Increment field, with an upper limit of Max Buffer Size.

Buffer Size Increment

Number

How much the buffer size expands if it exceeds its initial size. Setting a value of zero or lower means the buffer doesn’t expand, and a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised when the buffer gets full.

Max Buffer Size

Number

The maximum amount of memory to use. If more is used, a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value lower than or equal to zero means no limit.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the Initial Buffer Size, Buffer Size Increment, and Max Buffer Size fields.

Repeatable File Store Stream

Field Type Description Default Value Required

In Memory Size

Number

The maximum memory the stream uses to keep data in memory. If more is consumed, content is buffered on disk.

Buffer Unit

Enumeration, one of:

  • BYTE

  • KB

  • MB

  • GB

Unit for the In Memory Size field.

Redelivery Policy

Field Type Description Default Value Required

Max Redelivery Count

Number

The maximum number of times a message can be redelivered and processed unsuccessfully before triggering process-failed-message.

Message Digest Algorithm

String

The secure hashing algorithm to use. If not set, the default is SHA-256.

Message Identifier

Redelivery Policy Message Identifier

Defines which strategy is used to identify the messages.

Object Store

ObjectStore

The object store where the redelivery counter for each message is stored.

Redelivery Policy Message Identifier

Field Type Description Default Value Required

Use Secure Hash

Boolean

Whether to use a secure hash algorithm to identify a redelivered message.

Id Expression

String

Defines one or more expressions to use to determine when a message has been redelivered. This property may only be set if useSecureHash is false.

urlset

Field Type Description Default Value Required

Urlset

Anon Type_urlset

List of URL entries in the sitemap.

x

Anon Type_urlset

Field Type Description Default Value Required

Url

tURL

List of URL entries in the sitemap.

x

tURL

Field Type Description Default Value Required

Loc

String

The URL location.

x

Lastmod

String

The date of last modification.

Changefreq

Enumeration, one of:

  • always

  • hourly

  • daily

  • weekly

  • monthly

  • yearly

  • never

How frequently the page is likely to change.

Priority

Number

The priority of this URL relative to other URLs on the site (0.0 to 1.0).

Sitemap Response Attributes

Field Type Description Default Value Required

Count

Number

The total number of URLs in the sitemap.

Depth

Number

The configured maximum crawl depth.

Other Attributes

Object

Additional attributes.

Url

String

The root URL of the crawled site.

Error Mapping

Field Type Description Default Value Required

Source

Enumeration, one of:

  • ANY

  • REDELIVERY_EXHAUSTED

  • TRANSFORMATION

  • EXPRESSION

  • SECURITY

  • CLIENT_SECURITY

  • SERVER_SECURITY

  • ROUTING

  • CONNECTIVITY

  • RETRY_EXHAUSTED

  • TIMEOUT

The source error type to map from.

Target

String

The target error type to map to.

x
Back to top