Nav

Worker Monitoring

logo cloud active logo hybrid disabled logo server disabled logo pcf disabled

Worker Monitoring is a feature that is only available for applications deployed to CloudHub workers. It is not available for applications that you deploy via the Runtime Manager to local servers. See Deployment Strategies and Monitoring Applications for more details.

CloudHub supports worker monitoring and automatic restarts for unresponsive applications. Monitoring allows you the option to receive notifications and diagnostic information if your application becomes unresponsive. Optionally, enable automatic restarts so that your application can recover automatically without your intervention. CloudHub gives you the flexibility to conduct monitoring with or without enabling automatic restarts. 

Monitoring is enabled by default on all applications deployed on CloudHub. (Applications already deployed before the monitoring feature was introduced need to be redeployed for monitoring to take effect.) You can activate the automatic restart feature on a per-application basis using the Runtime Manager console when you deploy a new application or redeploy an existing application.

Prerequisites

This document assumes that you are familiar with Deploying to CloudHub and that you are deploying to runtime 3.4.n, 3.5.n, or newer. 

Enabling Automatic Restarts

During deployment, you have the option to enable automatic restarts for your application. 

Click Settings > Automatically restart application when not responding, or you can click the same setting from the Deploy Application menu:

WorkerMonitoring

If you are setting this from the Deploy Application menu, click Deploy Application.

If you are setting this from the Manage Application menu, click Apply Changes.

Upon deployment, CloudHub inserts a monitoring ping flow into your application. When the application starts, the logs display a message to indicate that monitoring is enabled and that the ping flow has been injected into your application.

LogPingFlowInjector

After your application fully starts, the log displays an additional message to indicate that monitoring has started, "Mule system health monitoring started for your application."

CHMuleSystemHealthMsg

How Monitoring Works

Every thirty seconds, CloudHub sends an HTTP get request to the ping flow within the application. If the application is running normally, the ping flow sends a response. If CloudHub receives no response from the ping three times in a row, the Logs print a warning and display a full thread dump.

CH-monitoring-threaddump

In addition, CloudHub triggers a notification to the console to alert you that a worker for this application stopped responding.

If you have configured email alerts, the nonresponsive worker also triggers an email alert based on the "Worker Not Responding" event. See  Alerts for more information about configuring these alerts.

How Automatic Restarts Work

If you check the box labeled Automatically restart application when not responding, CloudHub follows the same procedure described above, but also automatically triggers redeployment of the application. 

If you do not have this box checked, CloudHub only produces a notification and a log message and takes no further action.

CH-monitoring-notification2

CH-monitoring-logs5

The monitoring script exits and the application is left unchanged until you manually intervene.

If you have enabled automatic restart, CloudHub produces a notification to signal that the restart has been triggered and another to report the success or failure of the restart: 

CH-monitoring-notifications

The logs report the details of the restart procedure. If the restart succeeds, the ping script is reinitialized so that monitoring continues.

CH-successfulrestart

If the restart fails, CloudHub continues to attempt restarting up to five times. Once the maximum number of restart attempts is exceeded, CloudHub takes no further action. The monitoring script exits and the application is left unchanged until you manually intervene.