Nav

Amazon S3 Connector

Select

The Anypoint Amazon S3 connector provides connectivity to the Amazon S3 API, enabling you to interface with Amazon S3 to store objects, download and use data with other AWS services, and build applications that call for internet storage.

Instant access to the Amazon S3 API enables seamless integrations between Amazon S3 and other databases, CMS applications such as Drupal, and CRM applications such as Salesforce. You can use Amazon S3 to store and retrieve any amount of data at any time, from anywhere on the web. You can accomplish these tasks using the simple and intuitive web interface of the AWS Management Console.

The AWS SDK for Java provides a Java API for AWS infrastructure services. The Amazon S3 connector is built using the SDK for Java.

Prerequisites

To use the Amazon S3 connector, you must have the following:

  • Access to Amazon Web Services. 

  • AWS Identity and Access Management (IAM) credentials.

  • Anypoint Studio Enterprise edition.

This document assumes that you are familiar with Amazon S3, Mule, Anypoint Connectors, Anypoint Studio, elements in a Mule flow, and global elements.

Compatibility

Amazon S3 connector is compatible with:

Application/Service Version

Mule Runtime

4.0.0 or later

AWS SDK for Java

1.11.79

To Install this Connector

  1. In Anypoint Studio, click the Exchange icon in the Studio taskbar.

  2. Click Login in Anypoint Exchange.

  3. Search for the connector and click Install.

  4. Follow the prompts to install the connector.

When Studio has an update, a message displays in the lower right corner, which you can click to install the update.

To Configure Using the Studio Visual Editor

To use the Amazon S3 connector in your Mule application, configure a global Amazon S3 element that can be used by all the Amazon S3 connectors in your application.

  1. Click the Global Elements tab at the base of the canvas, then click Create.

  2. In the Choose Global Type window, expand Connector Configuration, and click Amazon S3 Configuration.

  3. Click OK.

  4. Specify the global element properties:

    Global Element Properties

    Parameter Description

    Name

    Enter a name for the configuration to reference the connector instance.

    Access Key

    Alphanumeric text string that uniquely identifies the user who owns the AWS account.

    Secret Key

    Key that plays the role of a password.

    Try Default AWS Credentials Provider Chain

    Checkbox that controls whether temporary credentials should be used.

    In the above dialog, values can refer to a configuration file placed in the  src folder of your project.

    You can either enter your credentials into the global configuration properties, or reference a configuration file that contains these values. For simpler maintenance and better reuse of your project, Mule recommends that you use a configuration file.

  5. Keep the Proxy and the Advanced tabs with their default entries.

  6. Click Test Connection to confirm that the parameters of your global configuration are accurate, and that Mule is able to successfully connect to your instance of Amazon S3. Access to operations on Amazon S3 is controlled through policies for that user. It may not always be possible to validate the credentials before the exact operation for which the user has access is done. This can vary for instance on the bucket name and other parameters. Hence the test connection may fail in case the credentials have a restricted policy such as access to a specific folder within a specific bucket.

  7. Click OK to save the global connector configurations.

To Connect to S3 Compatible Storage

If you need to connect to a different storage than the default AWS S3, you must specify its URL in the S3 Compatible Storage URL field. This field is present in the Advanced tab.

An example of S3 compatible storage is the Minio Project. For instance, if you configure it locally you may need to set the S3 Compatible Storage URL to http://127.0.0.1:9000.

To Configure Using the XML Editor

  1. Ensure that you have included the Amazon S3 namespaces in your configuration file.

    
                
             
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    
    <mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
    xmlns:s3="http://www.mulesoft.org/schema/mule/s3"
      xmlns:http="http://www.mulesoft.org/schema/mule/http"
      xmlns="http://www.mulesoft.org/schema/mule/core"
      xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
      xmlns:spring="http://www.springframework.org/schema/beans"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.springframework.org/schema/beans
      http://www.springframework.org/schema/beans/spring-beans-current.xsd
    http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
    http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
    http://www.mulesoft.org/schema/mule/s3 http://www.mulesoft.org/schema/mule/s3/current/mule-s3.xsd
    http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
    
      <!-- Put your flows and configuration elements here -->
    
    </mule>
  2. Create a global Amazon S3 configuration outside and above your flows, using the following global configuration code.

    
                
             
    1
    2
    
    <!-- simple configuration -->
    <s3:config name="Amazon_S3_Configuration" accessKey="${s3.accessKey}" secretKey="${s3.secretKey}" doc:name="Amazon S3: Configuration"/>
    Parameter Description

    Name

    Enter a name for the configuration with which it can be referenced later by config-ref. The name in this example is Amazon_S3_Configuration.

    accessKey

    The access key provided by Amazon, which is required for non-anonymous operations.

    secretKey

    The secrete key provided by Amazon, which is required for non-anonymous operations.

To Specify the Connector Namespace and Schema

In Studio, when you drag a connector from the palette to the Canvas, Studio automatically populates the XML code with the connector namespace and schema location. When you manually code the Mule application in Studio’s XML editor or another text editor, paste the following into the header of your configuration XML, inside the <mule> tag:

Namespace: http://www.mulesoft.org/schema/mule/s3 Schema Location: http://www.mulesoft.org/schema/mule/s3/current/mule-s3.xsd

For example:


          
       
1
2
3
4
5
6
7
8
9
<mule xmlns:s3="http://www.mulesoft.org/schema/mule/s3"
  ...
  xsi:schemaLocation="http://www.mulesoft.org/schema/mule/s3
   http://www.mulesoft.org/schema/mule/s3/current/mule-s3.xsd">
  ...
  <flow name="yourFlow">
  ...
  </flow>
</mule>

To Reference a Connector in the pom.xml File

The pom.xml is a Maven file that Anypoint Studio generates that contains dependencies for a Mule application. When you code a Mule application manually, include this XML snippet in your pom.xml file.


          
       
1
2
3
4
5
6
<dependency>
  <groupId>org.mule.connectors</groupId>
  <artifactId>mule-s3-connector</artifactId>
  <version>5.0.0</version>
  <classifier>mule-plugin</classifier>
</dependency>

To Use this Connector

For the following operations to work, you need to enable or update the subset of the overall list of Amazon S3 actions in the bucket to specify that the AWS account has access to the subset actions on the bucket.

The Amazon S3 connector supports the following list of operations:

  • Abort Multipart Upload

  • Complete Multipart Upload

  • Copy Object

  • Create Bucket

  • Create Object

  • Create Object Presigned URI

  • Delete Bucket

  • Delete Bucket Cross Origin Configuration

  • Delete Buckt Lifecycle Configuration

  • Delete Bucket Policy

  • Delete Bucket Tagging Configuration

  • Delete Bucket Website Configuration

  • Delete Object

  • Delete Objects

  • Get Bucket ACL

  • Get Bucket Cross Origin Configuration

  • Get Bucket Lifecycle Configuration

  • Get Bucket Location

  • Get Bucket Logging Configuration

  • Get Bucket Notification Configuration

  • Get Bucket Policy

  • Get Bucket Tagging Configuration

  • Get Bucket Versioning Configuration

  • Get Bucket Website Configuration

  • Get Object

  • Get Object ACL

  • Get Object Content

  • Get Object Metadata

  • Initiate Multipart Upload

  • List Buckets

  • List Multipart Uploads

  • List Next Batch of Objects

  • List Next Batch of Versions

  • List Objects

  • List Parts

  • List Versions

  • Set Bucket ACL

  • Set Bucket Cross Origin Configuration

  • Set Bucket Lifecycle Configuration

  • Set Bucket Logging Configuration

  • Set Bucket Notification Configuration

  • Set Bucket Policy

  • Set Bucket tagging Configuration

  • Set Bucket Versioning Configuration

  • Set Bucket Website Configuration

  • Set Object ACL

  • Set Object Storage Class

  • Upload Part

  • Upload Part Copy

Note: In the Create Object operation, the Content Length must be set to a value greater than zero. If the Content Length is set to zero (0), the Create Object operation creates a zero-byte object.

To Use a AWS KMS Master Key

If you need to encrypt objects that you are going to store in S3 buckets using customer managed master keys, specify a Customer Master Key ID in the KMS Master Key field in the Create Object configuration.

To Use the AWS Credentials Provider Chain in CloudHub

With the Default AWS Credentials Provider Chain, you can specify the access key and secret in the CloudHub environment:

  1. Use the following configuration to prepare a Mule app.

    
                 
              
    1
    2
    3
    4
    
     <s3:config name="Amazon_S3_S3_configuration" doc:name="Amazon S3 S3 configuration" doc:id="DOC_ID" >
      <s3:basic-connection accessKey="${aws.accessKeyId}" secretKey="${aws.secretKey}"
      tryDefaultAWSCredentialsProviderChain="true"/>
     </s3:config>
  2. Export this to get a deployable zip archive.

  3. Deploy to CloudHub and set the properties aws.accessKeyId and aws.secretKey through Runtime Manager > Settings > Properties.

  4. Finish deployment and test. Observe that the access key and secret key are not mentioned in the connector configuration and that the correct values are used from the values specified in the settings.

Use Cases

The following are common use cases for the Amazon S3 connector:

  • Store an image from a URL on Amazon S3, then retrieve and display the image.

  • Create an image link in Amazon S3 and update the status in Twitter along with the image link (not shown in this document).

Use Case: Store and Retrieve an Image from a URL to Amazon S3

Store an image from a URL in Amazon S3, and retrieve and display the image. For this example to work in Anypoint Studio, you must provide Amazon Web Services credentials. You can either replace the variables with their values in the code, or you can provide the values for each variable in the src/main/resources/mule-artifact.properties file.

Studio 7 Visual Studio Icon Flow

  1. Create a new Mule project in Anypoint Studio.

  2. Drag an HTTP connector into the canvas, and select it to open the properties editor console.

  3. Add a new HTTP Listener Config global element:

    1. In General Settings, click the + button:

    2. Configure the following HTTP parameters, and retain the default values for the other fields, and click OK after that.

      Field Value

      Name

      HTTP_Listener_Configuration

      Host

      127.0.0.1

      Port

      8081

    3. Reference the HTTP Listener Configuration global element. In the General tab, specify the / Path.

  4. Drag an Amazon S3 connector into the flow, and double-click the connector to open its Properties Editor.

  5. If you do not have an existing Amazon S3 connector global element to choose, click the plus sign next to Extension Configuration.

  6. Configure the global element properties, then click OK.

  7. Configure the parameters of the connector:

    Connector Properties - General tab

    Field Value

    Display Name

    Enter a name for the connector instance, such as Create bucket.

    Extension Configuration

    Select a global configuration for the connector.

    Bucket Name

    ${config.bucket}

    Region

    Select the region, for example US_STANDARD (default).

    Canned ACL

    PRIVATE (default) access control list. A canned ACL is a predefined grant.

  8. Add a HTTP Connector to request the MuleSoft logo from MuleSoft.

    HTTP Connector - General Tab

    Field Value

    Display Name

    Enter a name for the connector instance. In this example, the HTTP connector is named Get MuleSoft logo.

    Configuration

    Click the green plus symbol and create a new configuration with Host as developer.mulesoft.com, Protocol as HTTPS, and Port as 80.

    URL or Path

    Set to the /sites/all/themes/muletheme/images/mulesoft_dev_logo_v2.svg path.

    Body

    #[payload]

    Target Value

    #[payload]

  9. Drag another Amazon S3 connector to create the requested MuleSoft logo in the selected Amazon S3 Bucket.

    S3 Connector Properties General Tab

    Field Value

    Display Name

    Enter a name of your choice. In this example, the S3 connector is named Create logo object in S3 bucket

    Extension Configuration

    Select the global configuration that you created.

    Bucket Name

    ${config.bucket}

    Key

    muledevlogo

    Content

    #[payload]

    Canned ACL

    PRIVATE (default)

    Storage Class

    STANDARD (default)

  10. Add another Amazon S3 connector to get the newly created MuleSoft logo image object from the bucket:

    S3 Connector Get Image General Tab

    Field Value

    Display Name

    Enter a name for the connector instance. In this example the S3 connector is called Get Image.

    Extension Configuration

    Select the global configuration you create.

    Bucket Name

    ${config.bucket}

    Key

    muledevlogo

  11. Add another Amazon S3 connector to delete the bucket. Because the delete bucket operation’s return type is void, the payload contains the object returned by the get image operation.

    S3 Connector Delete Bucket General Tab

    Field Value

    Bucket Name

    ${config.bucket}

    Force

    True

Use Case: XML Code


         
      
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:s3="http://www.mulesoft.org/schema/mule/s3"
xmlns:http="http://www.mulesoft.org/schema/mule/http"
xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core
http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http
http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/s3
http://www.mulesoft.org/schema/mule/s3/current/mule-s3.xsd">
  <configuration-properties file="mule-artifact.properties"/>
  <http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="DOC_ID">
    <http:listener-connection host="127.0.0.1" port="8081"/>
  </http:listener-config>
  <s3:config name="Amazon_S3_Configuration" doc:name="Amazon S3 Configuration" doc:id="DOC_ID">
    <s3:basic-connection accessKey="${config.accessKey}" secretKey="${config.secretKey"/>
  </s3:config>
  <http:request-config name="HTTPS_Request_Configuration" doc:name="HTTP Request configuration" doc:id="DOC_ID">
    <http:request-connection protocol="HTTPS" host="www.mulesoft.com"/>
  </http:request-config>
  <flow name="s3docuFlow" doc:id="DOC_ID">
    <http:listener config-ref="HTTP_Listener_config" path="/" doc:name="Listener" doc:id="DOC_ID"/>
    <s3:create-bucket config-ref="Amazon_S3_Configuration" bucketName="${config.bucket}"
    doc:name="Create bucket" doc:id="DOC_ID"/>
    <http:request method="GET" path="/sites/default/files/new-application_network_diagram-01.svg"
    doc:name="Request" doc:id="DOC_ID" config-ref="HTTPS_Request_Configuration"/>
    <s3:create-object config-ref="Amazon_S3_Configuration" bucketName="${config.bucket}" key="muledevelogo"
    doc:name="Create object" doc:id="DOC_ID"/>
    <s3:get-object config-ref="Amazon_S3_Configuration" bucketName="${config.bucket}" key="muledevelogo"
    doc:name="Get object" doc:id="DOC_ID"/>
    <s3:delete-bucket config-ref="Amazon_S3_Configuration" bucketName="${config.bucket}"
    doc:name="Delete bucket" doc:id="DOC_ID" force="true"/>
  </flow>
</mule>