Contact Free trial Login

To Configure and View Test Reports

You can configure BAT for use with a number of reporting systems. For example:

  • New Relic

  • PagerDuty

  • Slack

  • SUMO

  • Custom

  • Local

  • Email

Specify report generation in the reporters section of the bat.yaml file.

New Relic

Using New Relic with API functional monitoring, you can determine the times of each test, detect performance loss, and schedule alerts.

New Relic has a plugin configured for BAT that can capture the metrics from test suites. To use the plugin to capture metrics, you need to configure your bat.yaml file by adding the following information:

  • An output directory

  • Your NEW_RELIC_LICENSE_KEY

To View New Relic Reports

  1. Obtain a New Relic license key.

    • Go to the New Relic Account Settings.

    • Copy your license key from the right sidebar under Account Summary.

  2. Add the license key to your bat.yaml file:

    reporters:
  - type: NewRelic
    outFile: outputNewRelic.json
    options:
      NEW_RELIC_LICENSE_KEY: "xxxxxxxxxxxxxxx"

  3. Schedule a test and create a monitor.

  4. Go to New Relic, and click Plugins.

    The BAT plugin information appears.

  5. Click the test suite name.

    bat plugin

    New test result data appears in New Relic after you run each test.

    bat new relic metrics

PagerDuty

The integration of BAT with PagerDuty generates an incident when a test fails and triggers configured PagerDuty alerts. If a monitor always fails with the same error, BAT does not generate a new issue, but instead ignores these errors.

To Configure PagerDuty Reporting

  1. In PagerDuty, configure an integration:

    • Configure > Services.

      bat pager duty

    • Select the integration. For example, select bat-monitoring.

      bat integrations pagerduty

  2. Copy the integration key and add it to your bat.yaml file:

    Bat.yaml example

    reporters:
  - type: PagerDuty
    outFile: PagerDuty.json
    options:
      SERVICE_KEY: "xxxxxxxxxxxxxxxxxxx"

    Reports about failed tests appear in PagerDuty.

bat pager duty report

Slack

You can integrate BAT and Slack to generate alerts in Slack. You send messages to a specific channel through a webhook.

  1. Create the webhook:

    • In Slack, go to your organization’s app directory page at yourorg.slack.com/apps.

    • Search for Incoming WebHooks and add a configuration.

      bat incoming webhooks

  2. Choose or create a channel to post notifications to.

    bat post to channel

    • Copy the webhook URL of your new configuration.

      bat webhook url

  3. In the bat.yaml file, add the URL of the webhook as a parameter.

    metadata:
  slack_channel: bat-bot
...
reporters:
  - type: Slack
    outFile: outputSlack.json
    options:
      SLACK_WEBHOOK: "yourwebhook-URL"

    The alert notification occurs in the case of failure and provides the name of the suite in which the error occurs.

bat demo failed

Sumo Logic

To use Sumo Logic, you create a collection and specify it in the bat.yaml. The collection can be shared with other tests or used only as a monitor. At the end of each execution, BAT generates a post to SUMO that sends the trade-off information of requests and responses that were executed during the test. This enables you to see expected results and how the test behaves.

You must provide the endpoint of the collection.

bat sumo report

You can use BAT to post a log in the specified collection, and create dashboards.

bat sumo collection

To Integrate Sumo and BAT

  1. In Sumo Logic, use the Setup Wizard to Set Up Streaming Data.

    bat sumo setup

  2. Choose Your Custom App > HTTP Source or All Other Sources > HTTP Source and fill out the source category.

  3. Copy the HTTP source and implement the Sumo Logic integration in your bat.yaml file:

    reporters:
  - type: SumoLogic
    outfile: SumoLogic.json
    options:
      SUMO_ENDPOINT: <your endpoint URL>

    The SUMO_ENDPOINT is, for example:

    https://endpoint1.collection.us2.sumologic.com/...

    In a few moments, the test results appear in Sumo Logic under Log Search.

    bat sumo results

Custom Post Report

BAT publishes the result.json with a post to the URL specified in the bat.yaml file. You can add headers if necessary.

reporters:
  - type: Custom
    options:
      URL: "http://www.httpbin.org/post"
      HEADERS: "'Authorization':'Bearer token','Content-Type':'application/json'"

About Local Reports

BAT generates basic types of reports locally in JSON and HTML. When you execute tests, the path to the reports appears in the output.

Example:

Reporter: /var/folders/vz/56jp75d941592x_1d8mfy9f40000gn/T/bat_report_20180320121736.json
Reporter: /var/folders/vz/56jp75d941592x_1d8mfy9f40000gn/T/bat_report_20180320121736.html

To Configure Local Report Generation

In the bat.yaml file, specify a type field to generate local reports. For example:

reporters:
  - type: JSON
    outFile: JSON.json
  - type: HTML
    outFile: HTML.html
  - type: JUnit
    outFile: JUnit.xml

To Create a Local Custom Report

You can generate a custom report based on a DataWeave transformation that is specified in the bat.yaml using the file keyword.

reporters:
  - type: Local
    file: reporter/transform.dwl
    outFile: local.json

The transform.dwl file contains the following code:

%dw 2.0
output application/json
var result = if (payload.result.pass[0]) " is ok" else " failed"
---
{
    result: "The suite <" ++ payload.name ++ "> with assertion <" ++ payload.result[0].name ++ ">" ++ result
}

And the result is:

{
  "result": "The suite <Hello world suite> with assertion <answer 200> is ok"
}

Email

To email test reports that indicate failures, specify any email addresses that you want to send reports to. You can send reports to one or more email addresses, separating multiple email addresses with semicolons. Each report includes the name of the suite in which the error occurs and a list of the statements. Reports are emailed only for tests created in Anypoint Functional Monitoring in Anypoint Platform, not for tests created with the BAT CLI.

reporters:
  - type: Email
    options:
      EMAILS: test@example.com;other@example.com;youremail@example.com