Azure Data Lake Storage Connector Examples - Mule 4

The following Azure Data Lake Storage Connector examples show you how to:

Create an Azure Data Lake Storage file system.
Create a path inside an Azure Data Lake Storage file system.
Update a path inside an Azure Data Lake Storage file system.
Implement error handling.

These examples use variables for some field values. You can either:

Replace the variables with their values in the code.
Provide the values for each variable in a properties file and then refer to that file from the connector configuration.

For information about the benefits of using property placeholders and how to configure them, see Anypoint Connector Configuration.

Create an Azure Data Lake Storage file system

This example shows you how to create a file system in Azure Data Lake Storage, and includes these steps:

Set up the app flow.
Configure the File System operation.
Optionally, copy and paste the XML for the flow in to the Configuration XML tab.

Set Up the App Flow

Create a new Mule project in Studio.
In the Mule Palette view, search for http and select the Listener operation.
Drag the Listener operation onto the Studio canvas.
In the Listener configuration, click + next to the Connector configuration field to add a global element.
Accept the defaults and click OK.
In the General properties tab, set the Path field to create-fs:

HTTP Listener General properties tab with Path value set to create-fs

Configure the Create File System Operation

In the Mule Palette view, search for azure and select the Azure Data Lake Storage Connector > Create File System operation.
Drag the Create File System operation from the Mule Palette onto the Listener flow.
In the configuration screen for the operation, click the plus sign (+) next to the Connector configuration field to access the global element configuration fields.
On the General tab, in Connection, select the authentication method to configure and enter all the required values:
Click OK.
On the General properties tab, in the File System field, enter a name for the file system:
Save the project.
Open a browser and send a GET request to localhost:8081/create-fs to test your app.

XML for Creating a File System

Paste this code into the Configuration XML tab of your flow to quickly load the flow for this example into your Mule app:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns:azure-data-lake-storage="http://www.mulesoft.org/schema/mule/azure-data-lake-storage"
xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/azure-data-lake-storage http://www.mulesoft.org/schema/mule/azure-data-lake-storage/current/mule-azure-data-lake-storage.xsd">
<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config">
<http:listener-connection host="0.0.0.0" port="8081" />
</http:listener-config>
<azure-data-lake-storage:config name="Azure_Data_Lake_Storage_Connector_Config" doc:name="Azure Data Lake Storage Connector Config">
<azure-data-lake-storage:shared-access-signature-connection accountName="${account.name}" dnsSuffix="${dns.suffix}" sasToken="${sas.token}" />
</azure-data-lake-storage:config>
<flow name="create-file-system-flow">
<http:listener doc:name="Listener" config-ref="HTTP_Listener_config" path="create-fs"/>
<azure-data-lake-storage:create-file-system doc:name="Create File System" config-ref="Azure_Data_Lake_Storage_Connector_Config" filesystem="newfilesystem"/>
</flow>

</mule>

Create a Path

This example shows you how to create a path in Azure Data Lake Storage and includes the following steps:

Set up the app flow.
Configure the Create Path operation.
Optionally, copy and paste the XML for the flow in to the Configuration XML tab.

Set Up the App Flow

Follow these steps to create and configure an HTTP Listener source to initiate the flow each time it receives a request on the configured host and port:

Create a new Mule project in Studio.
In the Mule Palette view, search for http and select the Listener operation:
Drag the Listener operation onto the Studio canvas.
In the Listener configuration, click + next to the Connector configuration field to add a global element.
Accept the defaults and click OK.
In the General properties tab, set the Path field to create-path:

Configure the Create Path operation

Follow these steps to create a file or directory on the provided Azure Data Lake Storage file system:

In the Mule Palette view, search for azure and select the Create/Rename Path operation.
Drag the Create/Rename Path operation from the Studio canvas onto the Listener flow.
In the Create/Rename Path configuration, click + next to the Connector Configuration field to add a connection configuration.
On the General tab, in Connection, select the authentication method to configure and enter all the required values:
Click OK.
In the File System field, enter the value for your target Azure Data Lake Storage file system.
If the file system does not exist, an error is thrown.
In Path, enter your desired path name.
In Resource, enter either file or directory:
Save the project.
Test the app by sending a GET request to localhost:8081/create-path.

XML for Creating a Path

Paste this code into the Configuration XML tab of your flow to quickly load the flow for this example into your Mule app:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:http="http://www.mulesoft.org/schema/mule/http"
	xmlns:azure-data-lake-storage="http://www.mulesoft.org/schema/mule/azure-data-lake-storage"
	xmlns="http://www.mulesoft.org/schema/mule/core"
	xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="
		http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
		http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
		http://www.mulesoft.org/schema/mule/azure-data-lake-storage http://www.mulesoft.org/schema/mule/azure-data-lake-storage/current/mule-azure-data-lake-storage.xsd">
	<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config">
		<http:listener-connection host="0.0.0.0" port="8081" />
	</http:listener-config>
	<azure-data-lake-storage:config name="Azure_Data_Lake_Storage_Connector_Config" doc:name="Azure Data Lake Storage Connector Config">
		<azure-data-lake-storage:shared-access-signature-connection accountName="${account.name}" dnsSuffix="${dns.suffix}" sasToken="${sas.token}" />
	</azure-data-lake-storage:config>
	<flow name="create-path-flow">
		<http:listener doc:name="Listener" config-ref="HTTP_Listener_config" path="create-path"/>
		<azure-data-lake-storage:create-or-rename doc:name="Create/Rename Path" config-ref="Azure_Data_Lake_Storage_Connector_Config" fileSystem="newfilesystem" path="newpath" resource="directory"/>
	</flow>

</mule>

Configure Update Path Operations with Append and Flush

In this example, you configure two Update Path operations:

The first Update Path operation is configured with append to append contents to an existing file on the Azure Data Lake Storage file system.
To do this, you first set up the app flow, add the File Connector Read operation, configure the Set Variable component, then configure the Update Path operation with append.
The second Update Path operation is then configured with flush to flush the contents of an existing file on the Azure Data Lake Storage file system.

Set Up the App Flow

Follow these steps to create and configure an HTTP Listener source to initiate the flow each time it receives a request on the configured host and port:

Create a new Mule project in Studio.
In the Mule Palette view, search for http and select the Listener operation:
Drag the Listener operation onto the Studio canvas.
In the Listener configuration, click + next to the Connector configuration field to add a global element.
Accept the defaults and click OK.
In the General properties tab, set the Path field to update-path:

Listener General properties configuration

Add and Configure File Connector

Add File Connector to your project and configure the Read operation to read files on the Azure Data Lake Storage file system:

In the Mule Palette view, click (X) Search in Exchange.
In Add Modules to Project, type file connector in the search field.
Click File Connector in Available modules.
Click Add.
Click Finish.
In the Mule Palette view, select File > Read and drag the Read operation onto the flow.
In General properties tab for the Read operation, click + next to the Connector Configuration field to add a connection configuration.
Enter the values and click OK.
In the General properties for the Read operation, enter the absolute path to the file upload in File Path:

Configure the Set Variable Component

Configure a Set Variable component to store and use a variable later in the flow:

From the Mule Palette view, search for set variable and drag Set Variable next to the Read operation on the Studio canvas.
In the General properties tab, enter these values:
- Name
  fileSize
- Value
  #[attributes.size]+ This saves the length of the file variable for the next operations.

General properties configuration for the Set Variable component

Configure the Update Path Operation with Append

Configure the Update Path operation to add append so you can append contents to a file:

In the Mule Palette view, search for azure and select the Azure Data Lake Storage Connector Update Path operation.
Drag the Update Path operation from the Mule Palette view onto the Listener flow, next to the Set Variable component.
In the General configuration properties pane of Update Path, select the connection configuration for the operation.
In the General properties tab, enter the following values:
- File System
  Value for your target Azure Data Lake Storage file system
- Path
  Value for your target path
- Action
  append
- Position
  0
- Content Length
  #[vars.fileSize]
- Content
  payload
Save the project.

Configure the Update Path Operation with Flush

Configure the Update Path operation to add flush so you can flush the contents of a file:

In the Mule Palette view, search for azure and select the Azure Data Lake Storage Connector Update Path operation.
Drag the Update Path operation from the Studio canvas onto the Listener flow, next to the Update Path operation that you created in the step for append.
In the General configuration properties pane of Update Path, select the connection configuration for the operation.
In the General properties tab, enter the following values:
- File System
  Your target Azure Data Lake Storage file system
- Path
  Your target path
- Action
  flush
- Position
  #[vars.fileSize]
- Content Length
  0
Save the project.
Open a browser and send a GET request to localhost:8081/update-path to test the app.

XML for Updating a Path

Paste this code into the Configuration XML tab of your flow to quickly load the flow for this example into your Mule app:

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:file="http://www.mulesoft.org/schema/mule/file"
	xmlns:http="http://www.mulesoft.org/schema/mule/http"
	xmlns:azure-data-lake-storage="http://www.mulesoft.org/schema/mule/azure-data-lake-storage"
	xmlns="http://www.mulesoft.org/schema/mule/core"
	xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
		http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
		http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
		http://www.mulesoft.org/schema/mule/azure-data-lake-storage http://www.mulesoft.org/schema/mule/azure-data-lake-storage/current/mule-azure-data-lake-storage.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd">
	<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config">
		<http:listener-connection host="0.0.0.0" port="8081" />
	</http:listener-config>
	<azure-data-lake-storage:config name="Azure_Data_Lake_Storage_Connector_Config" doc:name="Azure Data Lake Storage Connector Config">
		<azure-data-lake-storage:shared-access-signature-connection accountName="${account.name}" dnsSuffix="${dns.suffix}" sasToken="${sas.token}" />
	</azure-data-lake-storage:config>
	<file:config name="File_Config" doc:name="File Config"/>
	<flow name="create-path-flow">
		<http:listener doc:name="Listener" config-ref="HTTP_Listener_config" path="update-path"/>
		<file:read doc:name="Read" config-ref="File_Config" path="${file.path}"/>
		<set-variable value="#[attributes.size]" doc:name="Set Variable" variableName="fileSize"/>
		<azure-data-lake-storage:update-path doc:name="Update Path" config-ref="Azure_Data_Lake_Storage_Connector_Config" fileSystem="${filesystem.name}" path="${path.name}" action="append" position="0" contentLength="#[vars.fileSize]"/>
		<azure-data-lake-storage:update-path doc:name="Update Path" config-ref="Azure_Data_Lake_Storage_Connector_Config" fileSystem="${filesystem.name}" path="${path.name}" action="flush" position="#[vars.fileSize]" contentLength="0"/>
	</flow>

</mule>

Implement Error Handling

This section shows you how to implement simple error handling on a flow that uses Azure Data Lake Storage Connector.

In any flow, from the Mule Palette view, select Core > On Error Continue and drag it to the Error Handling section of the flow:
From the Mule Palette view, drag Core > Transform Message to the flow:
In the Mule Palette view, search for logger and drag the Logger component to the right of Transform Message.
In the General properties tab for the Logger component, enter the following values:
```
 output application/json
 ---
 error.errorMessage.payload
```
To access the error code that comes from the Azure service, the expression is:
error.muleMessage.typedValue.error.code

To access the error message, the expression is:
error.muleMessage.typedValue.error.message