Configure Reports of Test Failures

You can configure BAT for use with these reporting systems.

  • New Relic

  • PagerDuty

  • Slack

  • Sumo Logic

  • Email

You can also configure custom reports and reports that are sent to a location on your computer.

New Relic

Using New Relic with API Functional Monitoring, you can determine the times of each test, detect performance loss, and schedule alerts.

New Relic has a plugin configured for BAT that can capture the metrics from test suites.

To Configure Reporting

  1. Obtain your New Relic license key.

    1. Go to the New Relic Account Settings.

    2. Copy your license key from the right sidebar under Account Summary.

  2. In your bat.yaml file, create the reporters section, if one doesn’t already exist in the file. On a blank line, type reporters:.

  3. If you want to keep your license key secret and you are running tests from a private location:

    1. In Anypoint Secrets Manager, create a shared secret as a symmetric key. If the info that you want to store as a shared secret is currently in plain text, then you must encode it in Base64 before you create the secret.

      1. Open a secrets group or create a new one. The group must be created in the same environment that you are using in the BAT CLI. To find out which environment you are currently using in the BAT CLI, run the command bat whoami. In the output, there is the ID for the environment. Run the command bat environment ls to list the environments that you have access to. Match the ID from the bat whoami command with one of the environments listed. If you need to switch to the environment that your secrets group is in, run the command bat environment switch name, where name is the name of the environment.

      2. Select Shared Secret on the left side of the screen.

      3. In the Type field, select Symmetric Key.

      4. In the Key field, paste your license key encoded as a Base64 string.

      5. Paste the same Base64 string into the Confirm Key field.

    2. Copy the name of the new shared secret.

    3. At a command prompt, run the bat grant command, specifying an alias for the shared secret. When you run this command, the BAT CLI creates a section named secrets in your test suite’s bat.yaml file, if the section does not already exist. In that section, the BAT CLI adds these indented lines:

        alias:
    secretId: "secret-ID"
      alias

      This is the alias that you specified in the bat grant command.

      secret-ID

      This is the ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your bat.yaml file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.

    4. In your bat.yaml file, add New Relic as a reporter by adding these indented lines to the reporters section:

       - type: NewRelic
   outFile: filename.json
   options:
     NEW_RELIC_LICENSE_KEY: <alias_of_the_shared_secret>
     DISABLED_FOR_CONFIGS: dev,qa
      filename.json

      Specify a name for the output file to create.

      alias_of_the_shared_secret

      Specify the alias that you used for the secret for the license key when you ran the bat grant command. The alias must be between back ticks.

      DISABLED_FOR_CONFIGS

      Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

  4. If you do not want to keep your license key secret, in your bat.yaml file add New Relic as a reporter by adding these indented lines to the reporters section:

     - type: NewRelic
   outFile: filename.json
   options:
    NEW_RELIC_LICENSE_KEY: "license_key"
    DISABLED_FOR_CONFIGS: dev,qa
    filename.json

    Specify a name for the output file to create.

    license_key

    Specify the license key.

    DISABLED_FOR_CONFIGS

    Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

What to Do Next

After a test runs, go to New Relic and click Plugins. The BAT plugin information appears.

Click the test suite name to see the result of the test.

PagerDuty

The integration of BAT with PagerDuty generates an incident when a test fails and triggers configured PagerDuty alerts. If a monitor always fails with the same error, BAT does not generate a new issue, but instead ignores these errors.

To Configure Reporting

  1. In PagerDuty, configure an integration:

    1. Select Configure > Services.

    2. Select the integration.

  2. Copy the routing key.

  3. In your bat.yaml file, create the reporters section, if one doesn’t already exist in the file. On a blank line, type reporters:.

  4. If you want to keep your routing key secret and you are running tests from a private location:

    1. In Anypoint Secrets Manager, create a shared secret as a symmetric key. If the info that you want to store as a shared secret is currently in plain text, then you must encode it in Base64 before you create the secret.

      1. Open a secrets group or create a new one. The group must be created in the same environment that you are using in the BAT CLI. To find out which environment you are currently using in the BAT CLI, run the command bat whoami. In the output, there is the ID for the environment. Run the command bat environment ls to list the environments that you have access to. Match the ID from the bat whoami command with one of the environments listed. If you need to switch to the environment that your secrets group is in, run the command bat environment switch name, where name is the name of the environment.

      2. Select Shared Secret on the left side of the screen.

      3. In the Type field, select Symmetric Key.

      4. In the Key field, paste your routing key encoded as a Base64 string.

      5. Paste the same Base64 string into the Confirm Key field.

    2. Copy the name of the new shared secret.

    3. At a command prompt, run the bat grant command, specifying an alias for the shared secret. When you run this command, the BAT CLI creates a section named secrets in your test suite’s bat.yaml file, if the section does not already exist. In that section, the BAT CLI adds these indented lines:

        alias:
    secretId: "secret-ID"
      alias

      This is the alias that you specified in the bat grant command.

      secret-ID

      This is the ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your bat.yaml file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.

    4. In your bat.yaml file, add PagerDuty as a reporter by adding these indented lines to the reporters section:

       - type: PagerDuty
   outFile: filename.json
   options:
    PAGER_DUTY_ROUTING_KEY: <alias_of_the_shared_secret>
    DISABLED_FOR_CONFIGS: dev,qa
      filename.json

      Specify a name for the output file to create.

      alias_of_the_shared_secret

      Specify the alias that you used for the secret for the routing key when you ran the bat grant command. The alias must be between carets.

      DISABLED_FOR_CONFIGS

      Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

  5. If you do not want to keep your routing key secret, in your bat.yaml file add PagerDuty as a reporter by adding these indented lines to the reporters section:

     - type: PagerDuty
   outFile: filename.json
   options:
    PAGER_DUTY_ROUTING_KEY: "service_key"
    DISABLED_FOR_CONFIGS: dev,qa
    filename.json

    Specify a name for the output file to create.

    service_key

    Specify the routing key.

    DISABLED_FOR_CONFIGS

    Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

What to Do Next

If a test fails, look in PagerDuty for the report.

Slack

You can integrate BAT and Slack to send notifications to a Slack channel when tests fail. A notification provides the name of the suite in which the error occurs.

To Configure Failure Notifications

  1. Create a webhook:

    1. In Slack, go to your organization’s app directory page.

    2. Search for "Incoming WebHooks" and add a configuration.

  2. Choose or create a channel to post notifications to.

  3. Copy the webhook URL of your new configuration.

  4. In your bat.yaml file, create the metadata and reporters sections, if they do not already exist in the file. On a blank line, type metadata:. A few blank lines below that, type reporters:`.

  5. If you want to keep your webhook URL secret and you are running tests from a private location:

    1. In Anypoint Secrets Manager, create a shared secret as a symmetric key. If the info that you want to store as a shared secret is currently in plain text, then you must encode it in Base64 before you create the secret.

      1. Open a secrets group or create a new one. The group must be created in the same environment that you are using in the BAT CLI. To find out which environment you are currently using in the BAT CLI, run the command bat whoami. In the output, there is the ID for the environment. Run the command bat environment ls to list the environments that you have access to. Match the ID from the bat whoami command with one of the environments listed. If you need to switch to the environment that your secrets group is in, run the command bat environment switch name, where name is the name of the environment.

      2. Select Shared Secret on the left side of the screen.

      3. In the Type field, select Symmetric Key.

      4. In the Key field, paste your webhook URL encoded as a Base64 string.

      5. Paste the same Base64 string into the Confirm Key field.

    2. Copy the name of the new shared secret.

    3. At a command prompt, run the bat grant command, specifying an alias for the shared secret. When you run this command, the BAT CLI creates a section named secrets in your test suite’s bat.yaml file, if the section does not already exist. In that section, the BAT CLI adds these indented lines:

        alias:
    secretId: "secret-ID"
      alias

      This is the alias that you specified in the bat grant command.

      secret-ID

      This is the ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your bat.yaml file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.

    4. In your bat.yaml file, add the Slack channel to the metadata section by adding this indented line:

        slack_channel: slack_channel
      slack_channel

      Specify the name of the Slack channel that you want to send notifications to.

    5. In your bat.yaml file, add Slack as a reporter by adding these indented lines to the reporters section:

        - type: Slack
    outFile: filename.json
    options:
      SLACK_WEBHOOK: <alias_of_the_shared_secret>
      SEND_NOTIFICATION_IF_PASSED: true
      AT_HERE_ENABLED: true
      DISABLED_FOR_CONFIGS: dev,qa.
      filename.json

      Specify a name for the output file to create.

      alias_of_the_shared_secret

      Specify the alias that you used for the secret for the webhook URL when you ran the bat grant command. The alias must be between carets.

      SEND_NOTIFICATION_IF_PASSED

      Setting this option to true causes a notification to be sent if a test passes.

      AT_HERE_ENABLED

      Setting this option to true adds @here to a notification if a test fails.

      DISABLED_FOR_CONFIGS

      Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

  6. If you do not want to keep your webhook URL secret, add Slack as a reporter in your bat.yaml file by adding these lines to the reporter section:

      - type: Slack
    outFile: filename.json
    options:
      SLACK_WEBHOOK: "webhook_URL"
      SEND_NOTIFICATION_IF_PASSED: true
      AT_HERE_ENABLED: true
      DISABLED_FOR_CONFIGS: dev,qa.
    filename.json

    Specify a name for the output file to create.

    webhook_URL

    Specify the URL of the webhook.

    SEND_NOTIFICATION_IF_PASSED

    Setting this option to true causes a notification to be sent if a test passes.

    AT_HERE_ENABLED

    Setting this option to true adds @here to a notification if a test fails.

    DISABLED_FOR_CONFIGS

    Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

Sumo Logic

To use Sumo Logic, you create a collection and specify it in the bat.yaml. The collection can be shared with other tests or used only as a monitor. At the end of each execution, BAT generates a post to SUMO that sends the trade-off information of requests and responses that were executed during the test. This enables you to see expected results and how the test behaves.

You must provide the URL of the endpoint of the collection.

You can use BAT to post a log in the specified collection, and create dashboards.

To Configure Reporting

  1. In Sumo Logic, select Set Up Streaming Data in the setup wizard.

  2. Select Your Custom App > HTTP Source or All Other Sources > HTTP Source and specify the source category.

  3. Copy the endpoint URL.

  4. In your bat.yaml file, create the reporters section, if one doesn’t already exist in the file. On a blank line, type reporters:.

  5. If you want to keep your endpoint URL secret and you are running tests from a private location:

    1. In Anypoint Secrets Manager, create a shared secret as a symmetric key. If the info that you want to store as a shared secret is currently in plain text, then you must encode it in Base64 before you create the secret.

      1. Open a secrets group or create a new one. The group must be created in the same environment that you are using in the BAT CLI. To find out which environment you are currently using in the BAT CLI, run the command bat whoami. In the output, there is the ID for the environment. Run the command bat environment ls to list the environments that you have access to. Match the ID from the bat whoami command with one of the environments listed. If you need to switch to the environment that your secrets group is in, run the command bat environment switch name, where name is the name of the environment.

      2. Select Shared Secret on the left side of the screen.

      3. In the Type field, select Symmetric Key.

      4. In the Key field, paste your endpoint URL encoded as a Base64 string.

      5. Paste the same Base64 string into the Confirm Key field.

    2. Copy the name of the new shared secret.

    3. At a command prompt, run the bat grant command, specifying an alias for the shared secret. When you run this command, the BAT CLI creates a section named secrets in your test suite’s bat.yaml file, if the section does not already exist. In that section, the BAT CLI adds these indented lines:

        alias:
    secretId: "secret-ID"
      alias

      This is the alias that you specified in the bat grant command.

      secret-ID

      This is the ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your bat.yaml file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.

    4. In your bat.yaml file, add Sumo Logic as a reporter by adding these indented lines to the reporters section:

        - type: SumoLogic
    outfile: filename.json
    options:
      SUMO_ENDPOINT: <alias_of_the_shared_secret>
      DISABLED_FOR_CONFIGS: dev,qa.
      filename.json

      Specify a name for the output file to create.

      alias_of_the_shared_secret

      Specify the alias that you used for the secret for the endpoint URL when you ran the bat grant command. The alias must be between carets.

  6. If you do not want to keep your endpoint URL secret, add these lines to the reporters section in your bat.yaml file:

    DISABLED_FOR_CONFIGS

    Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

      - type: SumoLogic
    outfile: filename.json
    options:
      SUMO_ENDPOINT: endpoint-URL
      DISABLED_FOR_CONFIGS: dev,qa.
    filename.json

    Specify a name for the output file to create.

    endpoint-URL

    The URL of the endpoint for the collection.

    DISABLED_FOR_CONFIGS

    Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension. == Email

To email test reports that indicate failures, specify any email addresses that you want to send reports to. Each report includes the name of the suite in which the error occurs and a list of the statements.

Reports are emailed only for tests created in API Functional Monitoring in Anypoint Platform, not for tests created with the BAT CLI.

  1. In your bat.yaml file, create the reporters section, if one doesn’t already exist in the file. On a blank line, type reporters:.

  2. If you want to keep an email address secret and you are running tests from a private location:

    1. In Anypoint Secrets Manager, create a shared secret as a symmetric key. If the info that you want to store as a shared secret is currently in plain text, then you must encode it in Base64 before you create the secret.

      1. Open a secrets group or create a new one. The group must be created in the same environment that you are using in the BAT CLI. To find out which environment you are currently using in the BAT CLI, run the command bat whoami. In the output, there is the ID for the environment. Run the command bat environment ls to list the environments that you have access to. Match the ID from the bat whoami command with one of the environments listed. If you need to switch to the environment that your secrets group is in, run the command bat environment switch name, where name is the name of the environment.

      2. Select Shared Secret on the left side of the screen.

      3. In the Type field, select Symmetric Key.

      4. In the Key field, paste your the email address encoded as a Base64 string.

      5. Paste the same Base64 string into the Confirm Key field.

    2. Copy the name of the new shared secret.

    3. At a command prompt, run the bat grant command, specifying an alias for the shared secret. When you run this command, the BAT CLI creates a section named secrets in your test suite’s bat.yaml file, if the section does not already exist. In that section, the BAT CLI adds these indented lines:

        alias:
    secretId: "secret-ID"
      alias

      This is the alias that you specified in the bat grant command.

      secret-ID

      This is the ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your bat.yaml file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.

    4. In your bat.yaml file, add Email as a reporter by adding these indented lines to the reporters section:

        - type: Email
    options:
      EMAILS: <alias_of_the_shared_secret>
      DISABLED_FOR_CONFIGS: dev,qa
      DISABLED_FOR_CONFIGS

      Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

      alias_of_the_shared_secret

      Specify the alias that you used for the secret for the email address when you ran the bat grant command. The alias must be between carets.

      If you want to add aliases for additional email addresses, put each alias between carets and separate the aliases with semicolons:

      EMAILS: <alias_1>;<alias_2>;<alias_3>

  3. If you do not want to keep an email address secret, add these lines to the reporters section in your bat.yaml file:

      - type: Email
    options:
      EMAILS: email_address
      DISABLED_FOR_CONFIGS: dev,qa
    email_address

    Specify the email address to send notifications to. If you want to use more than one email address, separate them with semicolons:

    EMAILS: email_address1;email_address2;email_address3
    DISABLED_FOR_CONFIGS

    Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

Custom Reports

BAT publishes the result.json with a post to the URL that you specify in the bat.yaml file. You can add headers if necessary.

To publish custom reports for a test suite, add the following indented lines to the reporters section of the bat.yaml file, replacing the value for URL with your own URL.

reporters:
  - type: Custom
    options:
      URL: "http://www.httpbin.org/post"
      HEADERS: "'Authorization':'Bearer token','Content-Type':'application/json'"
      DISABLED_FOR_CONFIGS: dev,qa
DISABLED_FOR_CONFIGS

Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

Local Reports

BAT generates basic types of reports locally in JSON and HTML. When you execute tests, the path to the reports appears in the output.

Example:

Reporter: /var/folders/vz/56jp75d941592x_1d8mfy9f40000gn/T/bat_report_20180320121736.json
Reporter: /var/folders/vz/56jp75d941592x_1d8mfy9f40000gn/T/bat_report_20180320121736.html

To Configure Local Report Generation

In the bat.yaml file, specify a type field to generate local reports. For example:

reporters:
  - type: JSON
    outFile: JSON.json
  - type: HTML
    outFile: HTML.html
  - type: JUnit
    outFile: JUnit.xml

To Create a Local Custom Report

You can generate a custom report based on a DataWeave transformation that is specified in the bat.yaml using the file keyword.

reporters:
  - type: Local
    file: reporter/transform.dwl
    outFile: local.json

The transform.dwl file contains the following code:

%dw 2.0
output application/json
var result = if (payload.result.pass[0]) " is ok" else " failed"
---
{
    result: "The suite <" ++ payload.name ++ "> with assertion <" ++ payload.result[0].name ++ ">" ++ result
}

And the result is:

{
  "result": "The suite <Hello world suite> with assertion <answer 200> is ok"
}

Was this article helpful?

💙 Thanks for your feedback!

Edit on GitHub