Amazon S3 Connector 5.8 Additional Configuration

Create Object Operation

In the Create Object operation, set the Content Length to a value greater than zero. If the Content Length is set to zero (0), the Create Object operation creates a zero-byte object.

Use an AWS KMS Master Key

To encrypt objects that you want to store in S3 buckets using customer-managed master keys, specify a Customer Master Key ID in the KMS Master Key field in the Create Object configuration.

Before You Begin for Source Operations

Amazon S3 source operations use Amazon SQS for notification of when a new object is created or deleted in an S3 bucket.

There are multiple scenarios for the On New Object and On Deleted Object Amazon S3 source operations:

The queue is already created and specified
The queue is created by the source
The event notification configuration is created by the source

Queue Is Already Created

To correctly use a source for a queue that already exists, the user must have these actions enabled for the Amazon SQS queue:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:GetQueueUrl",
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes"
            ],
            "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:QUEUE_NAME"
        }
    ]
}

Additionally, to enable the queue to receive messages when a new object is created or deleted in an S3 bucket, create a policy for the queue in this format:

{
  "Version": "2012-10-17",
  "Id": "arn:aws:sqs:REGION:ACCOUNT_ID:QUEUE_NAME/SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "Sid1593770403887",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:QUEUE_NAME",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::BUCKET_NAME"
        }
      }
    }
  ]
}

Queue Is Created by Source

If the queue is created by the source operation, the queue name has the prefix MULE-S3-TRIGGER.

When the queue is created by the source, the user must have permission to both create a queue and set the queue attributes:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:ReceiveMessage",
                "sqs:GetQueueAttributes",
                "sqs:CreateQueue",
                "sqs:SetQueueAttributes",
                "sqs:GetQueueUrl"
            ],
            "Resource": "arn:aws:sqs:REGION:ACCOUNT_ID:MULE-S3-TRIGGER-*"
        }
    ]
}

When the queue is created by the source operation, an SQS policy is added to the queue when the S3 object is created or deleted in the bucket in the following format:

{
  "Version": "2012-10-17",
  "Id": "example-ID",
  "Statement": [
    {
      "Sid": "example-statement-ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
    },
      "Action": "SQS:SendMessage",
      "Resource":QUEUE_ARN,
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "BUCKET_ARN"
        }
      }
    }
  ]
}

Notification configuration for S3 bucket events is already created

To correctly use a source for a notification configuration that already exists, user just needs to specify notification configuration name in the ADVANCED tab of the source:

   <s3:new-object-trigger doc:name="On New Object" config-ref="${config}" bucketName="example-bucket-name"
                               notificationConfigurationName="on-new-object-configuration">
            <scheduling-strategy>
                <fixed-frequency/>
            </scheduling-strategy>
        </s3:new-object-trigger>

S3 Policy Bucket

For the S3 policy, use this format for all cases:

{
    "Version": "2012-10-17",
    "Id": "Policy1593761427184",
    "Statement": [
        {
            "Sid": "Stmt1593760119344",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT_ID:user/test"
            },
            "Action": "SPECIFIC_ACTION",
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        },
        {
            "Sid": "Stmt1593760259223",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::ACCOUNT_ID:user/test"
            },
            "Action": ["s3:PutBucketNotification", "s3:GetBucketNotification"],
            "Resource": "arn:aws:s3:::BUCKET_NAME"
        }
    ]
}

In this example, the SPECIFIC_ACTION is s3:PutObject or s3:DeleteObject, depending on the specified source operation.

Next Step

After you understand how to configure a master key and provide credentials, you can try out the Example.