Nav
You are viewing an older version of this section. Click here to navigate to the latest version.

Auto-Paging in Anypoint Connectors

When an Anypoint Connector in your flow produces output that is significantly large, processing it may cause significant performance delays in your application. To prevent this from happening, you can split the connector’s output into several "pages" for more efficient processing. Within Mule this behavior in an Anypoint Connector is referred to as "auto-paging" because the connector automatically paginates its output to prevent memory issues. However, if memory use is not an issue, you can forgo any auto-paging configuration and simply treat the entire payload as a single unit.

Assumptions

This document assumes that you are familiar with the http://www.mulesoft.org/documentation/display/current/Mule+Studio+Essentials[Visual Editor in Anypoint Studio], http://www.mulesoft.org/documentation/display/current/Anypoint+Connectors[Anypoint Connectors] and http://www.mulesoft.org/documentation/display/current/Understand+Global+Mule+Elements[Global Elements] . Review the http://www.mulesoft.org/documentation/display/current/Getting+Started+with+Mule+Studio[Getting Started with Anypoint Studio] chapter to learn more about developing with Mule ESB’s graphical user interface.

Configuring Auto-Paging

The table below lists the http://www.mulesoft.org/documentation/display/current/Anypoint+Connectors[Anypoint Connectors] which support auto-paging functionality. (Additionally, you can use auto-paging with any custom-built connector which supports auto-paging functionality.)

Connectors with Auto-Paging Bundled with Studio

Box

 

Google Calendars

 

Google Drive

 

Google Contacts

 

Google Prediction

 

Google Spreadsheets

 

Google Tasks

 

Microsoft Dynamics CRM (on Demand)

 

Microsoft Dynamics CRM (on Premise)

 

NetSuite

 

QuickBooks

 

Salesforce

Learn how to http://www.mulesoft.org/documentation/display/current/Installing+Extensions[add connectors] to your instance of Anypoint Studio.

  1. To make the Paging section visible, you must first select an Operation which outputs a collection, for example Get groups. Otherwise, Studio does not display the Paging section in the properties editor.

  2. Enter an integer in the Fetch Size field to indicate the batch size of objects in a "page". For example, set the Fetch Size to 50 to return information in batches of 50 objects.

    googlecontacts1

    Attribute Default Value Description

    Fetch Size

    100

    Specify the number of objects that Mule returns in each page.

  1. To the connector in your flow, add a fetchSize attribute.

  2. Set the value of the attribute to an integer to indicate the batch size of objects in a "page". For example, set fetchSize to 50 to return information in batches of 50 objects.


    
            
         
1
2
3
<flow name="Flow1">
    <google-contacts:get-groups config-ref="google" doc:name="paging" fetchSize="50"/>
</flow>
Property Default Value Description

fetchSize

100

Specify the number of objects that Mule returns in each page.

Handling Auto-Paged Output

When you use a connector that employs paging, Mule creates an iterator object that splits the payload into manageable pages. Regardless of the page-size, the iterator pushes out objects one at a time, then fetches the next pages on demand. To process the paginated output, you must use elements in your flow that can handle collections, such as a Foreach scope, a collection splitter, a DataMapper transformer, etc. Because Mule processes the set of pages one page at a time, it prevents memory usage from exceeding its limits.

Example

  1. Drag an HTTP endpoint onto the canvas

  2. Set its Path to authenticate.

  3. Add a Google Contacts connector to the flow, then set its Operation to Authorize.

  4. Create a Google Contacts Global Element, then configure its Consumer Key and Consumer secret.

  5. Create a new flow with a new HTTP endpoint.

  6. Set its Path to get_contacts.

  7. Add a new Google Contacts connector in the new flow.

  8. Set its Operation to Get Contacts and its Fetch Size to 50.

    googlecontacts2

  9. Add a Foreach scope after the Google Contacts connector.

  10. Add a Logger inside the Foreach scope.

    autopaging+flow

    When a message reaches the Google Contacts connector, the Logger outputs a separate message for each object. If there are more than 50 objects, Mule paginates the output.

  1. Create a google-contacts Global Element, then define its Consumer Key and Consumer secret.

    
           
                    
                 
    1
    2
    3
    
    <google-contacts:config-with-oauth name="Google_Contacts" consumerKey="" consumerSecret="" doc:name="Google Contacts" applicationName="Mule-GoogleContactsConnector">
        <google-contacts:oauth-callback-config domain="localhost" path="auth" remotePort="8081"/>
    </google-contacts:config-with-oauth>
  2. Create an HTTP connector and set the value of its Path to authenticate.

    
           
                    
                 
    1
    
    <http:inbound-endpoint host="localhost" port="8080" path="authenticate" doc:name="HTTP"/>
  3. Add a Google Contacts connector setting its operation to authorize.

    
           
                    
                 
    1
    
    <google-contacts:authorize config-ref="Google_Contacts" doc:name="Google Contacts"/> 
  4. Create a new flow with a new HTTP endpoint. Set the value of its Path to get_contacts, and reference the same configuration element as the other connector.

    
           
                    
                 
    1
    
    <http:inbound-endpoint host="localhost" port="8080" path="get_contacts" doc:name="HTTP"/>
  5. Add a new Google Contacts connector in the new flow setting its operation to get-contacts and fetchSize to 50.

    
           
                    
                 
    1
    
    <google-contacts:get-contacts config-ref="Google_Contacts" doc:name="Google Contacts" fetchSize="50"/>
  6. After the Google Contacts connector, add a Foreach to the flow, then add a Logger as a child element inside Foreach element.


    
             
          
1
2
3
<foreach doc:name="For Each">
    <logger message="#[message.payload.getEmailAddresses()]" level="INFO" doc:name="Logger"/>
</foreach>

When a message reaches the Google Contacts connector, the Logger outputs a separate message for each object. If there are more than 50 objects, Mule paginates the output. See below for a [Complete Example].

Additional MEL Expressions

When working with paginated output in a flow, you can use MEL expressions to call two functions.

Function Syntax Description

size

#[payload.size()]

Returns the total amount of available objects.

close

#[payload.close()]

Abort iteration.
This function frees up the resources that auto-paging is using.

Example

You can call both the size() and the close() functions in any expression that supports MEL. The simple example below illustrates how to call size() in a logger so that it records the total amount of objects that the connector is outputting.


         
      
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns:google-contacts="http://www.mulesoft.org/schema/mule/google-contacts" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.5.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/google-contacts http://www.mulesoft.org/schema/mule/google-contacts/1.7.4/mule-google-contacts.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd">
 
 
    <google-contacts:config-with-oauth name="Google_Contacts" consumerKey="" consumerSecret="" doc:name="Google Contacts" applicationName="Mule-GoogleContactsConnector">
        <google-contacts:oauth-callback-config domain="localhost" path="auth" remotePort="8081"/>
    </google-contacts:config-with-oauth>
 
    <flow name="authorizationAndAuthenticationFlow">
        <http:inbound-endpoint host="localhost" port="8080" path="authenticate" doc:name="HTTP"/>
        <google-contacts:authorize config-ref="Google_Contacts" doc:name="Google Contacts"/>
    </flow>
    <flow name="googleContactsTest" doc:name="googleContactsTest">
        <http:inbound-endpoint host="localhost" port="8080" path="get_contacts" doc:name="HTTP"/>
        <google-contacts:get-contacts config-ref="Google_Contacts" doc:name="Google Contacts" fetchSize="50"/>
        <logger message="#[payload.size()]" level="INFO" doc:name="Log_Size"/>
        <foreach doc:name="For Each">
             <logger message="#[payload.getEmailAddresses()]" level="INFO" doc:name="Log_Size"/>
        </foreach>
    </flow>
</mule>

Complete Example


         
      
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns:google-contacts="http://www.mulesoft.org/schema/mule/google-contacts" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.5.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/google-contacts http://www.mulesoft.org/schema/mule/google-contacts/1.7.4/mule-google-contacts.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd">
 
 
    <google-contacts:config-with-oauth name="Google_Contacts" consumerKey="" consumerSecret="" doc:name="Google Contacts" applicationName="Mule-GoogleContactsConnector">
        <google-contacts:oauth-callback-config domain="localhost" path="auth" remotePort="8081"/>
    </google-contacts:config-with-oauth>
 
    <flow name="authorizationAndAuthenticationFlow">
        <http:inbound-endpoint host="localhost" port="8080" path="authenticate" doc:name="HTTP"/>
        <google-contacts:authorize config-ref="Google_Contacts" doc:name="Google Contacts"/>
    </flow>
    <flow name="googleContactsTest" doc:name="googleContactsTest">
        <http:inbound-endpoint host="localhost" port="8080" path="get_contacts" doc:name="HTTP"/>
        <google-contacts:get-contacts config-ref="Google_Contacts" doc:name="Google Contacts" fetchSize="50"/>
         <logger message="#[payload.size()]" level="INFO" doc:name="Log_Size"/>
         <foreach doc:name="For Each">
            <logger message="#[payload.getEmailAddresses()]"  level="INFO" doc:name="Logger"/>
        </foreach>
    </flow>
</mule>

See Also