Automatic App Restarts
CloudHub 2.0 restarts your Mule applications during infrastructure failure or maintenance, security patching, replica crashes, or replica rebalancing.
The CloudHub 2.0 infrastructure gracefully shuts down by draining connections, transitioning traffic to a new instance of your application replica before terminating the older instance. If your application configuration sets the shutdownTimeout attribute, the grace period is twice the configured value, up to 12 hours maximum. By default, shutdownTimeout is five seconds. See Global Configurations Reference.
This grace period applies only to application restarts triggered by infrastructure maintenance. For a user-triggered stop or restart, if your application configuration sets the shutdownTimeout attribute to a value greater than 2 minutes and 30 seconds, the total shutdown grace period is 5 minutes. Otherwise, the grace period is twice the configured shutdownTimeout value.
<mule ...>
<configuration shutdownTimeout="150000" /> <!-- This sets shutdownTimeout to 2 minutes, 30 seconds. Default is 5 seconds. -->
</mule>To ensure business resiliency, design your application to be idempotent and deploy across multiple replicas with clustering enabled with a layer of persistence (Object Store v2).
Considerations:
Review these details about how each replica strategy manages application optimization:
- Single Replica with Recreate Strategy
-
The sole replica is terminated first, allowing it to gracefully shut down. A new replica launches as soon as the previous replica enters Stopping status. Incoming traffic experiences a brief downtime. If the application schedule triggers during this downtime, the trigger is missed.
- Single Replica with Rolling Restart
-
A new replica launches before the old replica begins its graceful shutdown. Two replicas run concurrently until the old replica is fully terminated. Once in Stopping status, the old replica stops receiving new connections or triggering application schedules.
- Multiple Replicas with Recreate or Rolling Restart
-
At least two thirds of the replicas remain available throughout the optimization process. Some replicas are gracefully shut down and restarted on new infrastructure.
Infrastructure Failure and Maintenance
If the compute infrastructure suffers a failure, CloudHub 2.0 moves your application to a healthy infrastructure and restarts the application replicas. Likewise, during infrastructure maintenance, the platform moves application replicas running on a compute infrastructure that becomes impaired or end-of-life onto a new infrastructure to ensure availability.
Security Patching
To mitigate security vulnerabilities, CloudHub 2.0 patches and restarts applications every month. See the Critical Security Vulnerabilities policy.
Replica Crash
If an application replica crashes for any reason, CloudHub 2.0 automatically restarts it. If it continues to fail to start, the platform attempts to restart it using an exponential backoff policy.
Replica Rebalancing
The CloudHub 2.0 infrastructure optimizes application efficiency through replica rebalancing. This process gracefully moves and restarts replicas, provided there are at least two replicas available.
During rebalancing, a maximum of 30% of an application’s replicas are rebalanced concurrently. The system rounds up this percentage to the number of replicas that can be disrupted. For example, if your application has two replicas, a disruption affects one replica, exceeding the configured 30%.
This table shows the potential impact of rebalancing operations on your applications based on the number of replicas.
| Total Application Replicas | Maximum Replicas Disrupted |
|---|---|
1 |
1 |
2 |
1 |
3 |
1 |
4 |
2 |
5 |
2 |
6 |
2 |
7 |
3 |
8 |
3 |
9 |
3 |
10 |
3 |
11 |
4 |
12 |
4 |
13 |
4 |
14 |
5 |
15 |
5 |
16 |
5 |
This method ensures that the application’s configuration, including the Mule runtime version and the infrastructure, remains consistent after all restarts, except for restarts occurring after a security patching. For more details about security patching, see CloudHub 2.0 Operating System Patch Updates.