MuleSoft WebCrawler Connector 1.0
Anypoint Connector for MuleSoft WebCrawler (MuleSoft WebCrawler Connector) is a highly customizable web-crawling connector that enables you to crawl websites, retrieve content, and automate web scraping processes within Mule flows. The connector is suited for gathering large-scale web data efficiently, with helper operations for fine-tuned crawling such as targeting specific content types and emitting referenced documents and images as Mule events. Use it to organize web data into structured formats for knowledge extraction, AI applications, and downstream processing.
For information about compatibility and fixed issues, see the MuleSoft WebCrawler Connector release notes.
Before You Begin
To use this connector, you must be familiar with:
-
Anypoint Connectors
-
Mule runtime engine (Mule)
-
Elements and global elements in a Mule flow
-
How to create a Mule app using Anypoint Code Builder or Anypoint Studio
-
Web crawling concepts and robots.txt compliance
Before creating an app, you must have:
-
Java 17 (required for compilation and runtime)
-
Apache Maven
-
Anypoint Platform
-
The latest versions of Anypoint Code Builder or Anypoint Studio
-
Your own Chromium-based remote WebDriver endpoint when content must be rendered in a real browser before scraping, such as JavaScript-heavy web applications. MuleSoft doesn’t provide or host a remote browser, so you must supply and maintain your own.
Connection Types
MuleSoft WebCrawler Connector supports these connection types for different crawling scenarios.
-
HTTP
Connects via standard HTTP requests with a configurable base URL, browser user agent, proxy, response timeout, and TLS settings. Use for static web content that doesn’t require JavaScript rendering.
-
Remote WebDriver
Connects through a remote Selenium WebDriver endpoint that you provide. Use for JavaScript-heavy or single-page application (SPA) pages that require browser rendering.
Key Features
MuleSoft WebCrawler Connector simplifies web data extraction into Mule applications with:
-
Comprehensive Web Crawling
Crawl entire websites or target specific content for large-scale data extraction with configurable depth and URL filtering.
-
Customizable Crawls
Control crawl depth, content filters, URL inclusion or exclusion patterns, and crawl delays tailored to specific business needs.
-
Content Extraction
Retrieve page content in TEXT, HTML, or MARKDOWN format. Target specific HTML tags to extract only relevant content.
-
Document and Image Extraction
Emit documents and images referenced on crawled pages as Mule events, with configurable limits and an optional allowlist of base URLs.
-
Page Elements
Get analytical data about web pages including word count, element or tag counts, and link structures to help customize crawls.
-
Dynamic Content Support
Crawl JavaScript-rendered pages using the Remote WebDriver connection, including Shadow DOM extraction and XPath-based wait conditions.
-
Robots.txt Compliance
Optionally enforce robots.txt rules to ensure respectful crawling practices.
-
Event-Driven Crawling
Use the On Content Crawl source to emit each discovered page, image, or document as a Mule event, with deduplication and URL queue management.
Use Cases
MuleSoft WebCrawler Connector supports a wide range of data ingestion and automation scenarios.
| Role | Goal |
|---|---|
Agentic Automation |
Scrape internal documents to ground agents and bots with up-to-date knowledge. |
Data Analyst |
Extract product information from e-commerce websites for competitive analysis. |
Marketing Team |
Gather customer reviews from various platforms for sentiment tracking. |
Business Development |
Collect contact information from industry directories for lead generation. |
Limitations
MuleSoft WebCrawler Connector has these limitations:
-
Analyzing or extracting the content within multimodal media, such as images and videos, isn’t supported. The connector can emit referenced image and document files as Mule events, but it doesn’t interpret what those files contain.
-
The recommended maximum size for a downloadable attachment is 25 MB.
Performance
The total crawl time depends on several factors, including:
-
Crawl depth
Higher crawl depths discover and process more pages.
-
Number of pages
Larger websites naturally take longer to crawl.
-
Website complexity
JavaScript-rendered websites require a Remote Browser and generally take longer than static HTML sites.
-
Network latency
Response times between the crawler, Remote Browser, and target website can affect overall performance.
-
Attachments
Downloading large files, such as PDFs or images, increases crawl time.
-
Remote Browser capacity
The CPU, memory, and concurrency of the Remote Browser deployment directly affect crawling throughput.
-
Number of consumers
Crawling throughput depends on the number of concurrent queue consumers. More consumers can increase throughput, but the Remote Browser must be able to handle that many concurrent sessions.
Best Practices
For the best crawling experience:
-
Deploy the Remote Browser with sufficient CPU and memory based on your expected crawl size.
-
Configure the Remote Browser reconnection strategy to Reconnect Forever to improve resilience during long-running crawls.
-
Use the HTTP connection for static websites whenever possible, and reserve the Remote Browser for JavaScript-rendered or interactive websites.
-
Limit crawl depth and exclude unnecessary pages or attachments to reduce crawl time.
-
For large websites, break the crawl into smaller jobs or use sitemap-based discovery to target only relevant pages.
-
Keep the crawl schedule in sync with the object store’s time-to-live (TTL) so that a full crawl runs on reseed. Otherwise, reseeding is skipped for that tick.
Next Step
After you complete the prerequisites, you are ready to create an app and configure the connector using Anypoint Studio or Anypoint Code Builder.