<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:sftp="http://www.mulesoft.org/schema/mule/sftp"z
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core
http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/sftp
http://www.mulesoft.org/schema/mule/sftp/current/mule-sftp.xsd">
<spring:beans>
<spring:import resource="sftp-properties.xml"/>
</spring:beans>
<sftp:connector name="sftpConnector" />
<sftp:connector name="sftpConnectorWithKnownHosts"
knownHostsFile="src/test/resources/my_known_hosts" />
<flow name="knownHostsInConnector">
<sftp:outbound-endpoint
address="sftp://${USER1_NAME}@${SFTP_HOST}:${SFTP_PORT}/~/tempdir2-test-outbound"
connector-ref="sftpConnectorWithKnownHosts" name="outboundEndpoint1">
</sftp:outbound-endpoint>
</flow>
<flow name="knownHostsInEndpoint">
<sftp:outbound-endpoint
address="sftp://${USER1_NAME}@${SFTP_HOST}:${SFTP_PORT}/~/tempdir2-test-outbound"
connector-ref="sftpConnector" name="outboundEndpoint2"
knownHostsFile="src/test/resources/my_known_hosts">
</sftp:outbound-endpoint>
</flow>
</mule>
SFTP Transport Reference
The Secure Shell (SSH) File Transfer Protocol (SFTP) transport allows files to be read and written to and from directories over SFTP. Unlike the Virtual File System (VFS) transport, it can handle large files because it streams message payloads. The SFTP transport can be used as an inbound or an outbound endpoint. Files can be retrieved from an SFTP server and processed as Mule messages, or Mule messages can be uploaded as files to an SFTP server. Mule uses the JCraft library for SFTP SSH.
SFTP Accepts SSH Known Hosts File
In Mule 3.8 and newer, an SSH known_hosts file may be passed to SFTP. The format for the SSH known_hosts
file is standard to your operating system. Refer to your OS documentation for details. The system property mule.sftp.knownHostsFile
has been removed in Mule 3.8 and newer. Instead, the name of the file with the known hosts must now be referenced from the mule-config.xml
file, using the knownHostsFile
attribute of the connector or the endpoints.​ The mule-config.xml
file is in the $MULE_HOME/apps/default
directory.
The following example uses outbound SFTP to ensure that the hosts are known in connectors and endpoints:
This example sets the USER1_NAME, SFTP_HOST, and SFTP_PORT values in the sftp-properties.xml
file.
Overview
-
Mule 3.7 and newer supports SFTP retries.
-
In code examples in this guide,
spring-beans-current.xsd
is a placeholder. To locate the correct version, seehttp://www.springframework.org/schema/beans/
Transport | |
---|---|
Doc |
|
Inbound |
|
Outbound |
|
Request |
|
Transactions |
|
Streaming |
|
Retries |
|
MEPs |
|
Default MEP |
|
Maven Artifact |
org.mule.transport:mule-transport-sftp |
SFTP |
Namespace and Syntax
This section lists namespace, schema, and connector syntax information.
XML Schema Location
http://www.mulesoft.org/schema/mule/sftp http://www.mulesoft.org/schema/mule/sftp/3.8/mule-sftp.xsd
Schema
You can view the full schema for the SFTP transport here.
Connector Syntax
<sftp:connector name="sftp-default" archiveDir="" archiveTempReceivingDir="/tmp/get"
archiveTempSendingDir="/tmp/send" autoDelete="true"
duplicateHandling="throwException" fileAge="60000"
identityFile="/home/test/.ssh/id_rsa" keepFileOnError="false"
outputPattern="#[message.inboundProperties.originalFilename]" passphrase="passPhrase"
pollingFrequency="10000" sizeCheckWaitTime="30000"
tempDirInbound="/tmp/inbound" tempDirOutbound="/tmp/outbound"
useTempFileTimestampSuffix="true" />
Endpoint Syntax
You can define your endpoints two different ways:
-
Prefixed endpoint:
<sftp:inbound-endpoint user="mule" password="test123" path="/tmp/sftp" host="myhost.com" pollingFrequency="500" name="inboundEndpoint1"/> <sftp:outbound-endpoint user="mule" password="test123" path="/tmp/sftp" host="myhost.com" name="outboundEndpoint2" keepFileOnError="false"/>
-
Non-prefixed URI:
<inbound-endpoint address="sftp://mule:test123@myhost.com/tmp/sftp"/>
<outbound-endpoint address="sftp://mule:test123@myhost.com/tmp/sftp"/>
Considerations
You can use the SFTP transport to download from or upload to a secured resource accessible via SFTP. This transport does not currently support transactions policies. Some uses for the SFTP transport are downloading data into a database and picking up files and uploading them via SFTP. You can use this transport to implement the file transfer Enterprise Integration Pattern. As explained in the EIP book, the file transfer pattern allows you to loosely couple two applications together, with delays in processing time. If your integration is time-sensitive, you may want to look at implementing the messaging pattern with the JMS transport which can give you closer to real-time processing.
Note: You need to have the proper permissions for the folder and files that the connector points to. If not, an exception is raised and no more files are processed after the first failed attempt.
Using the SFTP transport allows you to optionally use streaming support for larger files and asynchronous and synchronously chain other endpoints with an SFTP endpoint. It also allows you to use Mule’s robust error handling in your Mule application.
The examples on this page show how to define SFTP inbound and outbound endpoints in your Mule application.
Features
-
Streaming support of resources
-
For inbound endpoints, poll the resource at a specified interval
-
For outbound endpoints, choices on how to handle duplicate files: throw and exception, overwrite, append a sequence number to the file name
-
In Mule 3.8 and later, the knownHostsFile parameter was added, which if provided, the client validates the server’s key against the one in the referenced file. If the server key doesn’t match the one in the file, the connection is aborted.
Usage
To include the SFTP transport in your configuration:
-
Define these namespaces:
<?xml version="1.0" encoding="utf-8"?> <mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:sftp="http://www.mulesoft.org/schema/mule/sftp" xsi:schemaLocation=" http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd http://www.mulesoft.org/schema/mule/sftp http://www.mulesoft.org/schema/mule/sftp/current/mule-sftp.xsd">
-
Define a connector:
<sftp:connector name="sftp-default"/>
-
Define an inbound and/or outbound endpoint:
-
Use an inbound endpoint if you want new files found on the SFTP site to trigger a Mule flow.
-
Use an outbound endpoint if you want to upload files to an SFTP site. These files typically start as Mule messages and are converted to files.
<sftp:inbound-endpoint name="inboundEndpoint1" connector-ref="sftp" address="sftp://user:password@host/~/data1"/> <sftp:outbound-endpoint address="sftp://user:password@host/~/data" outputPattern="#[function:count]-#[function:systime].dat"/>
-
Rules for Using the Transport
On the connector, you define the connection pool size, and your inbound and outbound temporary directories. The endpoint is where you define the authentication information, polling frequency, file name patterns, etc. See below for the full list of configuration options.
One-way and request-response exchange patterns are supported. If an exchange pattern is not defined, 'one-way' is the default.
This is a polling transport. The inbound endpoint for SFTP uses polling to look for new files. The default is to check every second, but it can be changed via the 'pollingFrequency' attribute on the inbound endpoint.
Streaming is supported by the SFTP transport and is enabled by default.
Example Configurations
The following example saves any files found on a remote SFTP server to a local directory. This demonstrates using an SFTP inbound endpoint and a file outbound endpoint.
Important: Before running this example, create an SFTP properties file:
-
Create the sftp.properties properties file in your Classpath or set your PATH variable to the file’s location. For information on specifying SFTP server access information for a username, password, host, and port, using Anypoint Studio, see SFTP Connector.
-
Provide these parameters:
sftp.user=user sftp.host=host sftp.port=port sftp.password=password
Substitute each value to the right of the equal sign with SFTP access information. For example:
sftp.user=memyselfandi sftp.host=localhost sftp.port=8081 sftp.password=icannottellyou
Example SFTP-to-File Flow
Downloading files from SFTP using a Flow
<mule xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:sftp="http://www.mulesoft.org/schema/mule/sftp"
xmlns:file="http://www.mulesoft.org/schema/mule/file"
xmlns:spring="http://www.springframework.org/schema/beans"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/sftp
http://www.mulesoft.org/schema/mule/sftp/current/mule-sftp.xsd
http://www.mulesoft.org/schema/mule/file
http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/core
http://www.mulesoft.org/schema/mule/core/current/mule.xsd">
<!-- This placeholder bean lets you import the properties from the sftp.properties file. -->
<spring:bean id="property-placeholder"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<spring:property name="location" value="classpath:sftp.properties"/> (1)
</spring:bean>
<flow name="sftp2file">
<sftp:inbound-endpoint host="${sftp.host}" port="${sftp.port}"
path="/home/test/sftp-files" user="${sftp.user}" password="${sftp.password}"> (2)
<file:filename-wildcard-filter pattern="*.txt,*.xml"/> (3)
</sftp:inbound-endpoint>
<file:outbound-endpoint path="/tmp/incoming"
outputPattern="#[message.inboundProperties.originalFilename]"/> (4)
</flow>
</mule>
1 | A properties file which holds the SFTP server login credentials. |
2 | An SFTP inbound endpoint is declared that checks the /home/test/sftp-files directory for new files every one second by default. |
3 | Defines a file filter which only sends files ending with .txt or .xml to the outbound endpoint. |
4 | Any conforming files found on the inbound endpoint are then written to the /tmp/incoming local directory with the same file name it had on the SFTP server. |
Using a File Inbound Endpoint and an SFTP Outbound Endpoint
The following example uploads files found in a local directory to an SFTP server.
Uploading Files Via SFTP Using a Flow
<mule xmlns="http://www.mulesoft.org/schema/mule/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:sftp="http://www.mulesoft.org/schema/mule/sftp"
xmlns:file="http://www.mulesoft.org/schema/mule/file"
xmlns:spring="http://www.springframework.org/schema/beans"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/sftp
http://www.mulesoft.org/schema/mule/sftp/current/mule-sftp.xsd
http://www.mulesoft.org/schema/mule/file
http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/core
http://www.mulesoft.org/schema/mule/core/current/mule.xsd">
<!-- This placeholder bean lets you import the properties from the sftp.properties file. -->
<spring:bean id="property-placeholder"
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<spring:property name="location" value="classpath:sftp.properties"/> (1)
</spring:bean>
<flow name="file2sftp">
<file:inbound-endpoint path="/tmp/outgoing"> (2)
<file:filename-wildcard-filter pattern="*.txt,*.xml"/> (3)
</file:inbound-endpoint>
<sftp:outbound-endpoint host="${sftp.host}"
port="${sftp.port}"
path="/home/test/sftp-files"
user="${sftp.user}"
password="${sftp.password}"/> (4)
</flow>
</mule>
1 | A properties file which holds the SFTP server login credentials. |
2 | A file inbound endpoint is declared on which checks the /tmp/outgoing directory for new files every one second by default. |
3 | Defines a file filter which only sends files ending with .txt or .xml to the outbound endpoint. |
4 | Any conforming files found on the inbound endpoint are then written to the /home/test/sftp-files remote SFTP directory with the same file name it had on the local filesystem. |
Exchange Patterns and Features of the Transport
See transport matrix.
Configuration Reference
All the nodes of the cluster try to consume files from a source by first applying a distributed lock on each file. You can change this behavior so that only one node polls from the source, making a distributed lock unnecessary. Changing this causes a message to have to be processed by only one node unless a distributed queue (such as a VM queue) is used.
To enable this behavior:
-
Define the following system property:
mule.transport.sftp.singlepollinstance=true
-
Add a statement in the wrapper.conf file, for example:
wrapper.java.additional.<n>=-Dmule.transport.sftp.singlepollinstance=true
Replace
<n>
with the instance number.
Connector
SFTP connectivity
Attributes of the connector Element
Name | Description |
---|---|
preferredAuthenticationMethods |
Comma-separated list of authentication methods used by the SFTP client. Valid values are: gssapi-with-mic, publickey, keyboard-interactive and password. Type: String |
maxConnectionPoolSize |
If the number of active connections is specified, then a connection pool is used with active connections up to this number. Use a negative value for no limit. If the value is zero no connection pool is used. Type: Integer |
pollingFrequency |
The frequency in milliseconds that the read directory should be checked. Note that the read directory is specified by the endpoint of the listening component. Type: Long |
autoDelete |
Whether to delete the file after successfully reading it. Type: Boolean |
fileAge |
Minimum age (in ms) for a file to be processed. This can be useful when consuming large files. It tells Mule to wait for a period of time before consuming the file, allowing the file to be completely written before the file is processed. Warning: The fileAge attribute only works properly if the servers where Mule and the sftp-server runs have synchronized time. Note: See attribute sizeCheckWaitTime for an alternate method of determining if a incoming file is ready for processing. Type: Long |
|
Type: long. Required: no. Default: disabled. Wait time (in ms) between size-checks to determine if a file is ready to be processed. Disabled if not set or set to a negative value. This feature can be useful to avoid processing not yet completely written files (such as, consuming large files). It tells Mule to do two size checks waiting the specified time between the two size calls. If the two size calls return the same value Mule consider the file ready for processing. Note: See attribute fileAge for an alternate method of determining if a incoming file is ready for processing. Type: Long |
archiveDir |
Archives a copy of the file in the specified directory on the file system where mule is running. The archive folder must have been created before Mule is started and the user Mule runs under must have privileges to read and write to the folder. Type: String |
archiveTempReceivingDir |
If specified, the file to be archived is received in this folder and then moved to the archiveTempSendingDir while sent further on to the outbound endpoint. This folder is created as a subfolder to the archiveDir. Note: Must be specified together with the archiveTempSendingDir and archiveDir attributes. Type: String |
archiveTempSendingDir |
If specified, the file to be archived is sent to the outbound endpoint from this folder. This folder is created as a subfolder to the archiveDir. After the file is consumed by the outbound endpoint or the component itself (that is, when the underlying InputStream is closed) it is moved to the archive folder. Note: Must be specified together with the archiveTempReceivingDir and archiveDir attributes. Type: String |
outputPattern |
The pattern to use when writing a file to disk. This can use the patterns supported by the filename-parser configured for this connector. By default the File Transport Reference is used. See this same document section for information on how to override the default parser. Type: String |
keepFileOnError |
If true, the file on the inbound-endpoint is not deleted if an error occurs when writing to the outbound-endpoint. Note: This assumes that both the inbound and outbound endpoints are using the SFTP-Transport. Type: Boolean |
duplicateHandling |
Determines what to do if a file already exist on the outbound endpoint with the specified name.
Type: duplicateHandlingType |
identityFile |
An identityFile location for a PKI private key. Type: String |
passphrase |
The passphrase (password) for the identityFile if required.
Type: String |
tempDirInbound |
If specified, Mule tries to create the temp-directory in the endpoint folder if it doesn’t already exist. Ensure that the user Mule is configured to use to access the SFTP server has privileges to create a temp folder if required! For inbound endpoints: A temporary directory on the ftp-server from where the download takes place. The file is moved (locally on the sftp-server) to the tempDir, to mark that a download is taking place, before the download starts. Note: A file in the tempDir of an inbound endpoint is always correct (has only been moved locally on the sftp-server) and can therefore be used to restart a failing file transfer. Type: String |
tempDirOutbound |
If specified, Mule tries to create the temp-directory in the endpoint folder if it doesn’t already exist. Ensure that the user Mule configured to use to access the SFTP server has privileges to create a temp folder if required. For outbound endpoints: A temporary directory on the sftp-server to first upload the file to. When the file is fully uploaded the file is moved to its final destination. The tempDir is created as a sub directory to the endpoint. Note: A file in the tempDir of an outbound endpoint might not be correct (since the upload takes place to this folder) and can therefore NOT be used to restart a failing file transfer. Type: String |
useTempFileTimestampSuffix |
Used together with the tempDir - attribute to give the files in the tempDir a guaranteed unique name based on the local time when the file was moved to the tempDir. Type: boolean |
knownHostsFile |
If provided, the client validates the server’s key against the one in the referenced file. If the server key doesn’t match the one in the file, the connection is aborted. knownHostsFile specifies the path to a SSH known hosts file. This file contains public key fingerprints that ensure an SSH client is connecting to a known SSH server and not a malicious SSH server. Type: String |
Proxy Config
Attributes of the proxy-config Element
Name | Description |
---|---|
host |
Proxy host name or internet address. Type: String |
port |
Proxy port number. Type: substitutableInt |
username |
User name for proxy authentication. Type: String |
password |
Password for proxy authentication. Type: String |
protocol |
Proxy type. Type: proxyProtocol
|
Inbound Endpoint
Attributes of the inbound-endpoint Element
Name | Description |
---|---|
path |
A file location. Type: String |
user |
A username. Type: String |
password |
A password. Type: String |
host |
An IP address (for example, 0.0.0.0). Type: String |
port |
A port number. Type: Port Number |
pollingFrequency |
The frequency in milliseconds that the read directory should be checked. Note that the read directory is specified by the endpoint of the listening component. Type: Long |
fileAge |
Age (in ms) for a file to be processed. This can be useful when consuming large files. It tells Mule to wait for a period of time before consuming the file, allowing the file to be completely written before the file is processed. Warning: The fileAge attribute only works properly if the servers where Mule and the sftp-server runs have synchronized time. Note: See attribute sizeCheckWaitTime for an alternate method of determining if a incoming file is ready for processing. Type: Long |
sizeCheckWaitTime |
Wait time (in ms) between size-checks to determine if a file is ready to be processed. Disabled if not set or set to a negative value. This feature can be useful to avoid processing not yet completely written files (such as when consuming large files). It tells Mule to do two size checks waiting the specified time between the two size calls. If the two size calls return the same value Mule consider the file ready for processing. Note: See attribute fileAge for an alternate method of determining if a incoming file is ready for processing. Type: Long |
archiveDir |
Archives a copy of the file in the specified directory on the file system where mule is running. The archive folder must have been created before Mule is started and the user Mule runs under must have privileges to read and write to the folder. Type: String |
archiveTempReceivingDir |
If specified then the file to be archived is received in this folder and then moved to the archiveTempSendingDir while sent further on to the outbound endpoint. This folder is created as a subfolder to the archiveDir. Note: Must be specified together with the archiveTempSendingDir and archiveDir attributes. Type: String |
archiveTempSendingDir |
If specified then the file to be archived is sent to the outbound endpoint from this folder. This folder is created as a subfolder to the archiveDir. After the file is consumed by the outbound endpoint or the component itself (that is, when the underlying InputStream is closed) it is moved to the archive folder. Note: Must be specified together with the archiveTempReceivingDir and archiveDir attributes. Type: String |
identityFile |
An identityFile location for a PKI private key. Type: String |
passphrase |
The passphrase (password) for the identityFile if required. Type: String |
tempDir |
If specified then Mule tries to create the temp-directory in the endpoint folder if it doesn’t already exist. Ensure that the user Mule is configured to use to access the sftp server has privileges to create a temp folder if required! For inbound endpoints: A temporary directory on the sftp-server from where the download takes place. The file is moved (locally on the sftp-server) to the tempDir, to mark that a download is taking place, before the download starts. Note: A file in the tempDir of an inbound endpoint is always correct (has only been moved locally on the sftp-server) and can therefore be used to restart a failing file transfer. For outbound endpoints: A temporary directory on the sftp-server to first upload the file to. When the file is fully uploaded the file is moved to its final destination. The tempDir is created as a sub directory to the endpoint. Note: A file in the tempDir of an outbound endpoint might not be correct (since the upload takes place to this folder) and can therefore NOT be used to restart a failing file transfer. Type: String |
useTempFileTimestampSuffix |
Used together with the tempDir - attribute to give the files in the tempDir a guaranteed unique name based on the local time when the file was moved to the tempDir. Type: Boolean |
No child elements for inbound-endpoint
.
Outbound Endpoint
Attributes of outbound-endpoint
Name | Description |
---|---|
path |
A file location. Type: String |
user |
A username. Type: String |
password |
A password. Type: String |
host |
An IP address (for example, 0.0.0.0). Type: String |
|
Type: port number. Required: no. A port number. Type |
outputPattern |
The pattern to use when writing a file to disk. This can use the patterns supported by the filename-parser configured for this connector. By default the File Transport Reference is used. See this same document section for information on how to override the default parser. Type: String |
keepFileOnError |
If true, the file on the inbound-endpoint is not deleted if an error occurs when writing to the outbound-endpoint. Note: This assumes that both the inbound and outbound endpoints are using the SFTP-Transport. Type: Boolean |
duplicateHandling |
Determines what to do if a file already exist on the outbound endpoint with the specified name.
Type: duplicateHandlingType |
identityFile |
Type: string. Required: no. Default: disabled. An Type: String |
passphrase |
The passphrase (password) for the identityFile if required. Type: String |
tempDir |
If specified, Mule tries to create the temp-directory in the endpoint folder if it doesn’t already exist. Ensure that the user Mule is configured to use to access the SFTP server has privileges to create a temp folder if required! For inbound endpoints: A temporary directory on the ftp-server from where the download takes place. The file is moved (locally on the sftp-server) to the tempDir, to mark that a download is taking place, before the download starts. Note: A file in the tempDir of an inbound endpoint is always correct (has only been moved locally on the sftp-server) and can therefore be used to restart a failing file transfer. For outbound endpoints: A temporary directory on the sftp-server to first upload the file to. When the file is fully uploaded the file is moved to its final destination. The tempDir is created as a sub directory to the endpoint. Note: A file in the tempDir of an outbound endpoint might not be correct (since the upload takes place to this folder) and can therefore NOT be used to restart a failing file transfer. Type: String |
useTempFileTimestampSuffix |
Used together with the tempDir - attribute to give the files in the tempDir a guaranteed unique name based on the local time when the file was moved to the tempDir. Type: Boolean |
knownHostsFile |
If provided, the client validates the server’s key against the one in the referenced file. If the server key doesn’t match the one in the file, the connection is aborted. knownHostsFile specifies the path to a SSH known hosts file. This file contains public key fingerprints that ensure an SSH client is connecting to a known SSH server and not a malicious SSH server. Type: String |
No child elements for outbound-endpoint
.
Maven
This transport is part of the following Maven module:
<dependency>
<groupId>org.mule.transports</groupId>
<artifactId>mule-transport-sftp</artifactId>
</dependency>
Best Practices
Place your SFTP login credentials in a file and reference them in the Mule configuration.