Binary Streaming
In binary streaming use cases, the stream is a flux of bytes that are treated in a generic way regardless of what they represent. Examples include the response of an HTTP request or retrieving and writing a file in a FTP server.
The SDK supports this abstraction by using the Java InputStream class.
Binary Streaming in Operations
Operations can receive stream parameters, return streams, or do both.
The Write operation in the File connector is an example of an operation that operates over a stream:
public class FileOperations {
public void write(@Content InputStream content, String path) {
// do the writing
}
}If you have parameter of type InputStream, the SDK will provide all the necessary support for using Mule’s streaming capabilities. The content parameter in the example above will always be assigned to a repeatable stream that points to the beginning of the stream, treating it as if this is the first time the stream is getting read, whether that is true or not.
Another possibility is that your operation returns a binary stream instead, like the Read operation does:
public class FileOperations {
public InputStream read(String path) {
// do the reading
}
}If you return an InputStream, the SDK will automatically add all the necessary magic to make that stream repeatable.
Configuring Streaming
By allowing elements such as <repeatable-file-store-stream />, <repeatable-in-memory-stream /> and <non-repeatable-stream />, the SDK automatically adds support for configuring how streaming should behave. Here is an example that configures custom file store streaming for the Read example above:
<file:read path="myFile.json">
<repeatable-file-store-stream inMemorySize="10" bufferUnit="KB"/>
</file:read>Stream from Existing Files
Available since version 1.3.
When the <repeatable-file-store-stream> strategy is used, a file temporarily holds information that doesn’t fit into the in-memory buffer. There are particular cases in which using this strategy is inefficient and leads to suboptimal I/O and disk space requirements.
The most common cases of this include:
- Scenario 1
-
You are building an Operation or Source that returns the content of a file. The component that you are developing opens a file and returns an InputStream that contains the file content. If the file is bigger than the memory buffer, the <repeatable-file-store-stream> starts building a temporary file with the stream content. That file becomes an exact duplicate of the original file that was already opened.
- Scenario 2
-
In a variation of Scenario 1, an Operation or Source obtains an InputStream and needs to pre-process it before it is returned. Because the original stream is not yet pushed to the flow, it hasn’t been made repeatable. As a result, the source needs to implement its own buffering by storing the processed contents into a temporary file as it processes the original stream. When pre-processing finishes, the source then pushes the contents of the file, going back to Scenario 1.
These two scenarios articulate a need to get a repeatable stream from the content of an existing file, without the repeatable streaming framework creating a new file of its own, since the contents already originate from a file.
In these scenarios, use the ImmutableFileRepeatableInputStream class, which is similar to the standard java.io.FileInputStream but prevents Mule runtime engine (Mule) from generating a new buffering file and instructs Mule to reuse the existing file.
For example, this class can be used in an operation:
@MediaType(value = ANY, strict = false)
public InputStream read(String path) {
File file = new File(path);
if (!file.exists()) {
// throw ModuleException
}
return new ImmutableFileRepeatableInputStream(file);
}An important aspect of working with files of this nature is understanding when to delete them. Depending on the use case, you might need to delete them through your own code. In simpler cases, you can delete files when the stream is closed. If you want to delete the file when the stream is closed, you can use the following convenience constructor, which enables you to instruct Mule to automatically delete the file when the stream is closed:
new ImmutableFileRepeatableInputStream(file, true);
The file must be immutable for you to use this feature. That means that once the ImmutableFileRepeatableInputStream instance is created, the contents of that file cannot change, at least until ImmutableFileRepeatableInputStream is closed. Changing the file’s content while the flow is reading it can lead to inconsistencies, dirty reads, and other types of race conditions.
|