3.5.0 or later
Amazon S3 Connector - Mule 3
Where possible, we changed noninclusive terms to align with our company value of Equality. We maintained certain terms to avoid any effect on customer implementations. |
Anypoint Connector for Amazon S3 (Amazon S3 Connector) provides connectivity to the Amazon S3 API, enabling you to interface with Amazon S3 to store objects, download and use data with other AWS services, and build apps that call for internet storage.
Instant access to the Amazon S3 API enables seamless integrations between Amazon S3 and other databases, CMS apps such as Drupal, and CRM apps such as Salesforce. Amazon Simple Storage Service (Amazon S3) is storage for the internet. You can use Amazon S3 to store and retrieve any amount of data at any time, from anywhere on the web. You can accomplish these tasks using the simple and intuitive web interface of the AWS Management Console.
The AWS SDK for Java provides a Java API for AWS infrastructure services. The Amazon S3 connector is built using the SDK for Java.
Release Notes: Amazon S3 Connector Release Notes
Exchange: Amazon S3 Connector
Before You Begin
To use Amazon S3 Connector, you must have the following:
-
Access to Amazon Web Services
-
AWS Identity and Access Management (IAM) credentials
-
Anypoint Studio Enterprise edition
This document assumes that you are familiar with Amazon S3, Mule, Anypoint Connectors, Anypoint Studio, elements in a Mule flow, and global elements.
Install the Connector
-
In Anypoint Studio, click the Exchange icon in the Studio taskbar.
-
Sign in to Exchange with your Anypoint Platform credentials.
-
From Anypoint Exchange, click Provided by MuleSoft.
-
Search for the Mule 3 connector and click Install.
When Studio has an update, a message displays in the lower right corner, which you can click to install the update.
Configure a Global Element
To use Amazon S3 Connector in your Mule app, configure a global Amazon S3 element that can be used by all the Amazon S3 connectors in the app (read more about Global Elements).
-
Click the Global Elements tab at the base of the canvas, then click Create.
-
In the Choose Global Type window, expand Connector Configuration, and click Amazon S3: Configuration.
-
Click Ok
-
Enter the global element properties:
Parameter Description Name
Enter a name for the configuration to reference it.
Access Key
Alphanumeric text string that uniquely identifies the user who owns the account.
Secret Key
Key that plays the role of a password.
Try Default AWS Credentials Provider Chain
Checkbox that controls whether temporary credentials should be used.
S3 Compatible Storage URL
Optional URL of S3 compatible storage if Amazon S3 is not used.
In the image, the placeholder values refer to a configuration file in the
src
folder of your project. See Learn How to Configure Properties.
You can either enter your credentials into the global configuration properties, or reference a configuration file that contains these values. For simpler maintenance and better re-usability of your project, Mule recommends that you use a configuration file. Keeping these values in a separate file is useful if you need to deploy to different environments, such as production, development, and QA, where your access credentials differ. See Deploying to Multiple Environments for instructions on how to manage this.
-
Keep the Pooling Profile and the Reconnection tabs with their default entries.
-
Click Test Connection to confirm that the parameters of your global configuration are accurate, and that Mule is able to successfully connect to your instance of Amazon S3. Read more about this in Testing Connections.
-
Click OK to save the global connector configurations.
Configure Using an XML Editor
First, ensure that you have included the Amazon S3 namespaces in your configuration file.
<mule xmlns:s3="http://www.mulesoft.org/schema/mule/s3"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking"
xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans"
http://www.springframework.org/schema/beans/spring-beans-current.xsd
<!-- Put your flows and configuration elements here -->
</mule>
Follow these steps to configure Amazon S3 Connector in your app: Create a global Amazon S3 configuration outside and above your flows, using the following global configuration code.
<!-- simple configuration -->
<s3:config name="Amazon_S3_Configuration" accessKey="${s3.accessKey}" secretKey="${s3.secretKey}" doc:name="Amazon S3: Configuration"/>
Parameter | Description |
---|---|
Name |
Enter a name for the configuration with which it can be referenced later by config-ref. The name in this example is |
accessKey |
The access key provided by Amazon. It is required for non-anonymous operations. |
secretKey |
The secret key provided by Amazon. It is required for non-anonymous operations. |
Connect to S3-Compatible Storage
If you need to connect to a different storage than the default AWS S3, you must specify its URL in the S3 Compatible Storage URL
field.
An example of S3 compatible storage is the minio project. For instance, if you configure it locally you may need to set the S3 Compatible Storage URL
to http://127.0.0.1:9000
Use the Connector
Amazon S3 connector is an operation-based connector, which means that when you add the connector to your flow, you need to configure a specific operation for the connector to perform. The Amazon S3 connector currently supports the following list of operations:
See the full list of operations for the latest version of the connector in the Amazon S3 Technical Reference.
For the operations to work, you need to enable or update the subset of the overall list of Amazon S3 actions on the bucket to specify that the AWS account has access to the subset actions on the bucket.
-
Abort Multipart Upload
-
Complete Multipart Upload
-
Copy Object
-
Create Bucket
-
Create Object
-
Create Object Presigned URI
-
Delete Bucket
-
Delete Bucket Cross Origin Configuration
-
Delete Bucket Lifecycle Configuration
-
Delete Bucket Policy
-
Delete Bucket Tagging Configuration
-
Delete Bucket Website Configuration
-
Delete Object
-
Delete Objects
-
Get Bucket ACL
-
Get Bucket Cross Origin Configuration
-
Get Bucket Lifecycle Configuration
-
Get Bucket Location
-
Get Bucket Logging Configuration
-
Get Bucket Notification Configuration
-
Get Bucket Policy
-
Get Bucket Tagging Configuration
-
Get Bucket Versioning Configuration
-
Get Bucket Website Configuration
-
Get Object
-
Get Object ACL
-
Get Object Content
-
Get Object Metadata
-
Initiate Multipart Upload
-
List Buckets
-
List Multipart Uploads
-
List Next Batch of Objects
-
List Next Batch of Versions
-
List Objects
-
List Parts
-
List Versions
-
Set Bucket ACL
-
Set Bucket Cross Origin Configuration
-
Set Bucket Lifecycle Configuration
-
Set Bucket Logging Configuration
-
Set Bucket Notification Configuration
-
Set Bucket Policy
-
Set Bucket tagging Configuration
-
Set Bucket Versioning Configuration
-
Set Bucket Website Configuration
-
Set Object ACL
-
Set Object Storage Class
-
Upload Part
-
Upload Part Copy
Use the AWS KMS Customer Key
If you need to encrypt the objects that you are going to store to S3 buckets using customer managed keys, then you must specify the customer key ID in the KMS Master Key
field in the Create Object configuration.
Use AWS Credentials Provider Chain in CloudHub
With Default AWS Credentials Provider Chain the user can specify the access key and secret in the CloudHub environment. Following are the steps with which this can be done:
-
Use the following configuration to prepare a mule app.
<s3:config name="Amazon_S3__Configuration" accessKey="dummy" secretKey="dummy" doc:name="Amazon S3: Configuration" tryDefaultAWSCredentialsProviderChain="true"/>
-
Export this to get a deployable zip archive.
-
Deploy to cloudhub and set the properties
aws.accessKeyId
andaws.secretKey
throughRuntime Manager > Settings > Properties
. -
Finish deployment and test.
Observe that access key and secret key are not mentioned in the connector config and the correct values are used from the values specified in the settings. More information about Default AWS Credentials Provider Chain can be found here at using the default provider credential chain
Connector Namespace and Schema
When designing your app in Studio, the act of dragging the connector from the Mule Palette view onto the Anypoint Studio canvas should automatically populate the XML code with the connector namespace and schema location.
Namespace: http://www.mulesoft.org/schema/mule/connector
Schema Location: http://www.mulesoft.org/schema/mule/s3/current/mule-connector.xsd
If you are manually coding the Mule app in Studio’s XML editor or other text editor, paste these into the header of your configuration XML, inside the <mule>
tag:
<mule xmlns:connector="http://www.mulesoft.org/schema/mule/connector"
...
xsi:schemaLocation="http://www.mulesoft.org/schema/mule/connector
http://www.mulesoft.org/schema/mule/connector/current/mule-connector.xsd">
...
<flow name="yourFlow">
...
</flow>
</mule>
Use the Connector in a Mavenized Mule App
If you are coding a Mavenized Mule app, include this XML snippet in your pom.xml
file:
<dependency>
<groupId>org.mule.modules</groupId>
<artifactId>mule-module-s3</artifactId>
<version>x.x.x</version>
</dependency>
Replace x.x.x
with the version that corresponds to the connector you are using.
Use Cases
The following are the common use cases for the Amazon S3 connector:
-
Store an image from a URL on Amazon S3, then retrieve and display the image.
-
Create an image link in Amazon S3 and update the status in twitter along with the image link.
Add the Connector to a Mule Flow
-
Create a new Mule project in Anypoint Studio.
-
Drag the Amazon S3 connector onto the canvas, then select it to open the properties editor.
-
Configure the connector’s parameters:
Field Description Example Display Name
Enter a unique label for the connector in your app.
Amazon S3
Connector Configuration
Select a global Amazon S3 connector element from the drop-drown.
N/A
Operation
Select the action this component must perform.
Create bucket
Bucket Name
Select a parameter for the operation.
#[payload] or
${bucketName}
to pick the value using MEL expression. -
Save your configurations.
Connector Performance
To define the pooling profile for the connector manually, access the Pooling Profile tab in the applicable global element for the connector.
For background information about pooling, see Tuning Performance.
Demo: Store an Image from a URL
Create a Mule app that stores an image from a URL on Amazon S3, then retrieve and display the image.
Demo Using the Studio Visual Editor
-
Begin the flow by sending a message to a bucket.
-
Create a new Mule project in Anypoint Studio.
-
Drag an HTTP connector into the canvas, then select it to open the properties editor console.
-
Add a new HTTP Listener Configuration global element:
-
In General Settings, click the + button:
-
Configure the following HTTP parameters, and retain the default values for the other fields:
Field Value Name
HTTP_Listener_Configuration
Port
8081
-
Reference the HTTP Listener Configuration global element.
-
-
Drag an Amazon S3 connector into the flow, and double-click the connector to open its Properties Editor.
-
If you do not have an existing Amazon S3 connector global element to choose, click the plus sign next to Connector Configuration.
-
Configure the global element properties, then click OK.
-
Configure the remaining parameters of the connector:
Field Value Display Name
Enter a name for the connector instance.
Connector Configuration
Select a global configuration for the connector.
Operation
Create bucket
Bucket Name
${config.bucket}
Canned ACL
PUBLIC_READ
Add a HTTP Connector to request the MuleSoft logo from MuleSoft.
Field Value Display Name
Enter a name for the connector instance.
Connector Configuration
Create a new default configuration with Host as mulesoft.org and Port as 80.
Path
Set the path as
sites/all/themes/mulesoft_community/logo.png
Method
GET
-
Drag another Amazon S3 connector to create the above requested MuleSoft logo in the selected Amazon S3 Bucket.
Field Value Display Name
Enter a name of your choice.
Connector Configuration
Select the global configuration that you created.
Operation
Create object
Bucket Name
${config.bucket}
Key
mulesoft.png
Content Reference
#[payload]
-
Add another Amazon S3 connector to get the newly created MuleSoft logo image object from the bucket:
Field Value Display Name
Enter a name for the connector instance.
Connector Configuration
Select the global configuration you create.
Operation
Get object content
Bucket Name
${config.bucket}
Key
mulesoft.png
-
Finally, add another Amazon S3 connector to delete the bucket. Since delete bucket operation’s return type is void, the payload contains the object returned by the get image operation.
Demo XML Code
For this code to work in Anypoint Studio, you must provide Amazon Web Services credentials. You can either replace the variables with their values in the code, or you can provide the values for each variable in the src/main/app/mule-app.properties file.
<?xml version="1.0" encoding="UTF-8" ?>
<mule xmlns:s3="http://www.mulesoft.org/schema/mule/s3"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking"
xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation=" http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core
http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http
http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/tracking
http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd
http://www.mulesoft.org/schema/mule/s3
http://www.mulesoft.org/schema/mule/s3/current/mule-s3.xsd" >
<http:listener-config
name="HTTP_Listener_Configuration"
host="0.0.0.0"
port="8081" doc:name="HTTP Listener Configuration" />
<http:request-config
name="HTTP_Request_Configuration"
host="mulesoft.org"
port="80" doc:name="HTTP Request Configuration" />
<s3:config
name="Amazon_S3_Configuration"
accessKey="${config.accessKey}"
secretKey="${config.secretKey}" doc:name="Amazon S3: Configuration" />
<flow name="s3-example-flow" >
<http:listener
config-ref="HTTP_Listener_Configuration"
path="/"
doc:name="HTTP" />
<s3:create-bucket
config-ref="Amazon_S3_Configuration"
bucketName="${config.bucket}"
acl="PUBLIC_READ"
doc:name="Create S3 Bucket" />
<http:request
config-ref="HTTP_Request_Configuration"
path="sites/all/themes/mulesoft_community/logo.png"
method="GET"
doc:name="Get MuleSoft logo" />
<s3:create-object
config-ref="Amazon_S3_Configuration"
doc:name="Create logo object in S3 bucket"
acl="PUBLIC_READ" bucketName="${config.bucket}"
key="mulesoft.png" />
<s3:get-object-content
config-ref="Amazon_S3_Configuration"
bucketName="${config.bucket}"
key="mulesoft.png"
doc:name="Get Image" />
<s3:delete-bucket
config-ref="Amazon_S3_Configuration"
bucketName="${config.bucket}"
force="true"
doc:name="Delete S3 Bucket" />
</flow>
</mule>