salesforce Dreamforce On-Demand
Contact Us 1-800-596-4880
  1. Homepage
  2. API Functional Monitoring
  3. API Functional Monitoring with BAT CLI

  4. Configuring and Viewing Test Reports

Configuring and Viewing Reports

You can configure BAT for use with these reporting systems.

  • New Relic

  • PagerDuty

  • Slack

  • Sumo Logic

  • Email

You can also configure custom reports and reports that are sent to a location on your computer.

New Relic

Using New Relic with API Functional Monitoring, you can view metrics from test suites in New Relic. Metrics include the times of each test, performance losses detected, and alert schedules.

Configure Reporting in New Relic

  1. Obtain your New Relic license key. For details on how to do this, see the New Relic documentation.

  2. In your bat.yaml file, create the reporters section, if one doesn’t already exist in the file. On a blank line, type reporters:.

  3. In your bat.yaml file, add New Relic as a reporter by adding these indented lines to the reporters section:

     - type: NewRelic
   options:
     NEW_RELIC_LICENSE_KEY: <new-relic-license-key>
     DISABLED_FOR_CONFIGS: dev,qa

    • new-relic-license-key: Specify the New Relic license key enclosed in double quotation marks.

      You can configure a secret to securely pass the license key. If you do so, use the secret’s alias in new-relic-license-key. The alias must be between backticks. See Configure a Secret in Anypoint Secret Manager.

    • DISABLED_FOR_CONFIGS: Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

What to Do Next

After a test runs, go to New Relic and click Metrics. The BAT test metrics appear.

Click the test suite name to see the result of the test.

PagerDuty

The integration of BAT with PagerDuty generates an incident when a test fails and triggers configured PagerDuty alerts. If a monitor always fails with the same error, BAT does not generate a new issue, but instead ignores these errors.

Configure Reporting in PagerDuty

  1. In PagerDuty, configure an integration:

    1. Select Configure > Services.

    2. Select the integration.

  2. Copy the routing key.

  3. In your bat.yaml file, create the reporters section, if one doesn’t already exist in the file. On a blank line, type reporters:.

  4. In your bat.yaml file, add PagerDuty as a reporter by adding these indented lines to the reporters section:

     - type: PagerDuty
   outFile: filename.json
   options:
    PAGER_DUTY_ROUTING_KEY: "service_key"
    DISABLED_FOR_CONFIGS: dev,qa

    • filename.json: Specify a name for the output file to create.

    • service_key: Specify the routing key.

      You can configure a secret to securely pass the service key. If you do so, use the alias that you used for the secret for the service key when you ran the bat grant command. The alias must be between carets. See Configure a Secret in Anypoint Secret Manager.

    • DISABLED_FOR_CONFIGS: Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

What to Do Next

If a test fails, look in PagerDuty for the report.

Slack

You can integrate BAT and Slack to send notifications to a Slack channel when tests fail. A notification provides the name of the suite in which the error occurs.

Configure Failure Notifications in Slack

  1. Create a webhook:

    1. In Slack, go to your organization’s app directory page.

    2. Search for "Incoming WebHooks" and add a configuration.

  2. Choose or create a channel to post notifications to.

  3. Copy the webhook URL of your new configuration.

  4. In your bat.yaml file, create the metadata and reporters sections, if they do not already exist in the file. On a blank line, enter metadata:. A few blank lines after that, enter reporters:.

  5. In your bat.yaml file, add the Slack channel to the metadata section by adding this indented line:

      slack_channel: slack_channel

    • slack_channel: Specify the name of the Slack channel that you want to send notifications to.

  6. In your bat.yaml file, add Slack as a reporter by adding these indented lines to the reporters section:

      - type: Slack
    outFile: filename.json
    options:
      SLACK_WEBHOOK: "webhook_URL"
      SEND_NOTIFICATION_IF_PASSED: true
      AT_HERE_ENABLED: true
      DISABLED_FOR_CONFIGS: dev,qa.

    • filename.json: Specify a name for the output file to create.

    • webhook_URL: Specify the URL of the webhook.

      You can configure a secret to securely pass the webhook URL. If you do so, use the alias that you used for the secret for the webhook URL when you ran the bat grant command. The alias must be between carets. See Configure a Secret in Anypoint Secret Manager.

    • SEND_NOTIFICATION_IF_PASSED: Setting this option to true causes a notification to be sent if a test passes.

    • AT_HERE_ENABLED: Setting this option to true adds @here to a notification if a test fails.

    • DISABLED_FOR_CONFIGS: Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

Sumo Logic

To use Sumo Logic, you create a collection and specify it in the bat.yaml. The collection can be shared with other tests or used only as a monitor. At the end of each execution, BAT generates a post to SUMO that sends the trade-off information of requests and responses that were executed during the test. This enables you to see expected results and how the test behaves.

You must provide the URL of the endpoint of the collection.

You can use BAT to post a log in the specified collection, and create dashboards.

Configure Reporting in Sumo Logic

  1. In Sumo Logic, select Set Up Streaming Data in the setup wizard.

  2. Select Your Custom App > HTTP Source or All Other Sources > HTTP Source and specify the source category.

  3. Copy the endpoint URL.

  4. In your bat.yaml file, create the reporters section, if one doesn’t already exist in the file. On a blank line, type reporters:.

      - type: SumoLogic
    outfile: filename.json
    options:
      SUMO_ENDPOINT: endpoint-URL
      DISABLED_FOR_CONFIGS: dev,qa.

    • filename.json: Specify a name for the output file to create.

    • endpoint-URL: The URL of the endpoint for the collection.

      You can configure a secret to securely pass the endpoint URL key. If you do so, use the alias that you used for the secret for the endpoint URL key when you ran the bat grant command. The alias must be between carets. See Configure a Secret in Anypoint Secret Manager.

    • DISABLED_FOR_CONFIGS: Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

Email

To email test reports that indicate failures, specify any email addresses that you want to send reports to. Each report includes the name of the suite in which the error occurs and a list of the statements.

Because reports are emailed only for tests created in or uploaded to the API Functional Monitoring in Anypoint Platform, after you add the reporters to your test in BAT CLI, upload the test. You can alternatively add the email reporting tool in the monitor’s Notifications tab after you upload the test.

  1. In your bat.yaml file, create the reporters section, if one doesn’t already exist in the file. On a blank line, type reporters:.

    EMAILS: <alias_1>;<alias_2>;<alias_3>

  2. If you do not want to keep an email address secret, add these lines to the reporters section in your bat.yaml file:

      - type: Email
    options:
      EMAILS: email_address
      DISABLED_FOR_CONFIGS: dev,qa

    • email_address: Specify the email address to send notifications to. If you want to use more than one email address, separate them with semicolons:

      EMAILS: email_address1;email_address2;email_address3

      You can configure a secret to securely pass the email addresses. If you do so, use the alias that you used for the secret for each email address when you ran the bat grant command. The alias must be between carets. To specify multiple email address aliases, put each alias between carets and separate the aliases with semicolons. See Configure a Secret in Anypoint Secret Manager.

    • DISABLED_FOR_CONFIGS: Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

Custom Reports

BAT publishes the result.json with a post to the URL that you specify in the bat.yaml file. You can add headers if necessary.

To publish custom reports for a test suite, add the following indented lines to the reporters section of the bat.yaml file, replacing the value for URL with your own URL.

reporters:
  - type: Custom
    options:
      URL: "http://www.httpbin.org/post"
      HEADERS: "'Authorization':'Bearer token','Content-Type':'application/json'"
      DISABLED_FOR_CONFIGS: dev,qa

  • DISABLED_FOR_CONFIGS: Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.

Local Reports

BAT generates basic types of reports locally in JSON and HTML. When you execute tests, the path to the reports appears in the output.

Example:

Reporter: /var/folders/vz/56jp75d941592x_1d8mfy9f40000gn/T/bat_report_20180320121736.json
Reporter: /var/folders/vz/56jp75d941592x_1d8mfy9f40000gn/T/bat_report_20180320121736.html

To Configure Local Report Generation

In the bat.yaml file, specify a type field to generate local reports. For example:

reporters:
  - type: JSON
    outFile: JSON.json
  - type: HTML
    outFile: HTML.html
  - type: JUnit
    outFile: JUnit.xml

To Create a Local Custom Report

You can generate a custom report based on a DataWeave transformation that is specified in the bat.yaml using the file keyword.

reporters:
  - type: Local
    file: reporter/transform.dwl
    outFile: local.json

The transform.dwl file contains the following code:

%dw 2.0
output application/json
var result = if (payload.result.pass[0]) " is ok" else " failed"
---
{
    result: "The suite <" ++ payload.name ++ "> with assertion <" ++ payload.result[0].name ++ ">" ++ result
}

And the result is:

{
  "result": "The suite <Hello world suite> with assertion <answer 200> is ok"
}

Configure a Secret in Anypoint Secret Manager

To keep sensitive information secret in tests you run from a private location, create a shared secret in Secrets Manager.

To configure a shared secret:

  1. Open a secrets group or create a new one. Create the group in the same environment that you are using in BAT CLI.

    To find out which environment you are using in BAT CLI:

    1. Run the command bat whoami.

      The output has the ID for the environment.

    2. Run the command bat environment ls to list the environments to which you have access.

    3. Match the ID from the bat whoami command with one of the environments listed.

    4. If you need to switch to the environment that your secrets group is in, run the command bat environment switch name, where name is the name of the environment.

  2. Select Shared Secret.

  3. In the Type field, select Symmetric Key.

  4. In the Key field, paste the sensitive information encoded as a Base64 string.

  5. Paste the Base64 string into the Confirm Key field.

Grant Your Monitor Access to the Secret

After you modify your monitor to add the secret information, grant the monitor access to the secret. At runtime, BAT CLI looks up the shared secrets using the aliases.

To grant a monitor access to the secret:

  1. Copy the name of the new shared secret obtained in Configure a Secret in Anypoint Secret Manager.

  2. At a command prompt, run the bat grant command, specifying an alias for the shared secret. When you run this command, BAT CLI creates a section named secrets in your test suite’s bat.yaml file, if the section does not already exist. In that section, BAT CLI adds these indented lines:

    alias:
 secretId: "secret-ID"

    • alias: The alias that you specified in the bat grant command.

    • secret-ID: The ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your bat.yaml file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.

See Also

Creating Monitors Using Secrets to Mask Sensitive Information in Monitors