alias: secretId: "secret-ID"
Configure Reports of Test Failures
You can configure BAT for use with these reporting systems.
-
New Relic
-
PagerDuty
-
Slack
-
Sumo Logic
-
Email
You can also configure custom reports and reports that are sent to a location on your computer.
New Relic
Using New Relic with API Functional Monitoring, you can determine the times of each test, detect performance loss, and schedule alerts.
New Relic has a plugin configured for BAT that can capture the metrics from test suites.
To Configure Reporting
-
Obtain your New Relic license key.
-
Go to the New Relic Account Settings.
-
Copy your license key from the right sidebar under Account Summary.
-
-
In your
bat.yaml
file, create thereporters
section, if one doesn’t already exist in the file. On a blank line, typereporters:
. -
If you want to keep your license key secret and you are running tests from a private location:
-
In Anypoint Secrets Manager, create a shared secret as a symmetric key. If the info that you want to store as a shared secret is currently in plain text, then you must encode it in Base64 before you create the secret.
-
Open a secrets group or create a new one. The group must be created in the same environment that you are using in the BAT CLI. To find out which environment you are currently using in the BAT CLI, run the command
bat whoami
. In the output, there is the ID for the environment. Run the commandbat environment ls
to list the environments that you have access to. Match the ID from thebat whoami
command with one of the environments listed. If you need to switch to the environment that your secrets group is in, run the commandbat environment switch name
, wherename
is the name of the environment. -
Select Shared Secret on the left side of the screen.
-
In the Type field, select Symmetric Key.
-
In the Key field, paste your license key encoded as a Base64 string.
-
Paste the same Base64 string into the Confirm Key field.
-
-
Copy the name of the new shared secret.
-
At a command prompt, run the
bat grant
command, specifying an alias for the shared secret. When you run this command, the BAT CLI creates a section namedsecrets
in your test suite’sbat.yaml
file, if the section does not already exist. In that section, the BAT CLI adds these indented lines:- alias
-
This is the alias that you specified in the
bat grant
command. - secret-ID
-
This is the ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your
bat.yaml
file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.
-
In your
bat.yaml
file, add New Relic as a reporter by adding these indented lines to thereporters
section:- type: NewRelic outFile: filename.json options: NEW_RELIC_LICENSE_KEY: <alias_of_the_shared_secret> DISABLED_FOR_CONFIGS: dev,qa
- filename.json
-
Specify a name for the output file to create.
- alias_of_the_shared_secret
-
Specify the alias that you used for the secret for the license key when you ran the
bat grant
command. The alias must be between back ticks. - DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.
-
-
If you do not want to keep your license key secret, in your
bat.yaml
file add New Relic as a reporter by adding these indented lines to thereporters
section:- type: NewRelic outFile: filename.json options: NEW_RELIC_LICENSE_KEY: "license_key" DISABLED_FOR_CONFIGS: dev,qa
- filename.json
-
Specify a name for the output file to create.
- license_key
-
Specify the license key.
- DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.
PagerDuty
The integration of BAT with PagerDuty generates an incident when a test fails and triggers configured PagerDuty alerts. If a monitor always fails with the same error, BAT does not generate a new issue, but instead ignores these errors.
To Configure Reporting
-
In PagerDuty, configure an integration:
-
Select Configure > Services.
-
Select the integration.
-
-
Copy the routing key.
-
In your
bat.yaml
file, create thereporters
section, if one doesn’t already exist in the file. On a blank line, typereporters:
. -
If you want to keep your routing key secret and you are running tests from a private location:
-
In Anypoint Secrets Manager, create a shared secret as a symmetric key. If the info that you want to store as a shared secret is currently in plain text, then you must encode it in Base64 before you create the secret.
-
Open a secrets group or create a new one. The group must be created in the same environment that you are using in the BAT CLI. To find out which environment you are currently using in the BAT CLI, run the command
bat whoami
. In the output, there is the ID for the environment. Run the commandbat environment ls
to list the environments that you have access to. Match the ID from thebat whoami
command with one of the environments listed. If you need to switch to the environment that your secrets group is in, run the commandbat environment switch name
, wherename
is the name of the environment. -
Select Shared Secret on the left side of the screen.
-
In the Type field, select Symmetric Key.
-
In the Key field, paste your routing key encoded as a Base64 string.
-
Paste the same Base64 string into the Confirm Key field.
-
-
Copy the name of the new shared secret.
-
At a command prompt, run the
bat grant
command, specifying an alias for the shared secret. When you run this command, the BAT CLI creates a section namedsecrets
in your test suite’sbat.yaml
file, if the section does not already exist. In that section, the BAT CLI adds these indented lines:alias: secretId: "secret-ID"
- alias
-
This is the alias that you specified in the
bat grant
command. - secret-ID
-
This is the ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your
bat.yaml
file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.
-
In your
bat.yaml
file, add PagerDuty as a reporter by adding these indented lines to thereporters
section:- type: PagerDuty outFile: filename.json options: PAGER_DUTY_ROUTING_KEY: <alias_of_the_shared_secret> DISABLED_FOR_CONFIGS: dev,qa
- filename.json
-
Specify a name for the output file to create.
- alias_of_the_shared_secret
-
Specify the alias that you used for the secret for the routing key when you ran the
bat grant
command. The alias must be between carets. - DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.
-
-
If you do not want to keep your routing key secret, in your
bat.yaml
file add PagerDuty as a reporter by adding these indented lines to thereporters
section:- type: PagerDuty outFile: filename.json options: PAGER_DUTY_ROUTING_KEY: "service_key" DISABLED_FOR_CONFIGS: dev,qa
- filename.json
-
Specify a name for the output file to create.
- service_key
-
Specify the routing key.
- DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.
Slack
You can integrate BAT and Slack to send notifications to a Slack channel when tests fail. A notification provides the name of the suite in which the error occurs.
To Configure Failure Notifications
-
Create a webhook:
-
In Slack, go to your organization’s app directory page.
-
Search for "Incoming WebHooks" and add a configuration.
-
-
Choose or create a channel to post notifications to.
-
Copy the webhook URL of your new configuration.
-
In your
bat.yaml
file, create themetadata
andreporters
sections, if they do not already exist in the file. On a blank line, typemetadata:
. A few blank lines below that, type reporters:`. -
If you want to keep your webhook URL secret and you are running tests from a private location:
-
In Anypoint Secrets Manager, create a shared secret as a symmetric key. If the info that you want to store as a shared secret is currently in plain text, then you must encode it in Base64 before you create the secret.
-
Open a secrets group or create a new one. The group must be created in the same environment that you are using in the BAT CLI. To find out which environment you are currently using in the BAT CLI, run the command
bat whoami
. In the output, there is the ID for the environment. Run the commandbat environment ls
to list the environments that you have access to. Match the ID from thebat whoami
command with one of the environments listed. If you need to switch to the environment that your secrets group is in, run the commandbat environment switch name
, wherename
is the name of the environment. -
Select Shared Secret on the left side of the screen.
-
In the Type field, select Symmetric Key.
-
In the Key field, paste your webhook URL encoded as a Base64 string.
-
Paste the same Base64 string into the Confirm Key field.
-
-
Copy the name of the new shared secret.
-
At a command prompt, run the
bat grant
command, specifying an alias for the shared secret. When you run this command, the BAT CLI creates a section namedsecrets
in your test suite’sbat.yaml
file, if the section does not already exist. In that section, the BAT CLI adds these indented lines:alias: secretId: "secret-ID"
- alias
-
This is the alias that you specified in the
bat grant
command. - secret-ID
-
This is the ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your
bat.yaml
file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.
-
In your
bat.yaml
file, add the Slack channel to themetadata
section by adding this indented line:slack_channel: slack_channel
- slack_channel
-
Specify the name of the Slack channel that you want to send notifications to.
-
In your
bat.yaml
file, add Slack as a reporter by adding these indented lines to thereporters
section:- type: Slack outFile: filename.json options: SLACK_WEBHOOK: <alias_of_the_shared_secret> SEND_NOTIFICATION_IF_PASSED: true AT_HERE_ENABLED: true DISABLED_FOR_CONFIGS: dev,qa.
- filename.json
-
Specify a name for the output file to create.
- alias_of_the_shared_secret
-
Specify the alias that you used for the secret for the webhook URL when you ran the
bat grant
command.The alias must be between carets.
- SEND_NOTIFICATION_IF_PASSED
-
Setting this option to
true
causes a notification to be sent if a test passes. - AT_HERE_ENABLED
-
Setting this option to
true
adds@here
to a notification if a test fails. - DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.
-
-
If you do not want to keep your webhook URL secret, add Slack as a reporter in your
bat.yaml
file by adding these lines to thereporter
section:- type: Slack outFile: filename.json options: SLACK_WEBHOOK: "webhook_URL" SEND_NOTIFICATION_IF_PASSED: true AT_HERE_ENABLED: true DISABLED_FOR_CONFIGS: dev,qa.
- filename.json
-
Specify a name for the output file to create.
- webhook_URL
-
Specify the URL of the webhook.
- SEND_NOTIFICATION_IF_PASSED
-
Setting this option to
true
causes a notification to be sent if a test passes. - AT_HERE_ENABLED
-
Setting this option to
true
adds@here
to a notification if a test fails. - DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.
Sumo Logic
To use Sumo Logic, you create a collection and specify it in the bat.yaml. The collection can be shared with other tests or used only as a monitor. At the end of each execution, BAT generates a post to SUMO that sends the trade-off information of requests and responses that were executed during the test. This enables you to see expected results and how the test behaves.
You must provide the URL of the endpoint of the collection.
You can use BAT to post a log in the specified collection, and create dashboards.
To Configure Reporting
-
In Sumo Logic, select Set Up Streaming Data in the setup wizard.
-
Select Your Custom App > HTTP Source or All Other Sources > HTTP Source and specify the source category.
-
Copy the endpoint URL.
-
In your
bat.yaml
file, create thereporters
section, if one doesn’t already exist in the file. On a blank line, typereporters:
. -
If you want to keep your endpoint URL secret and you are running tests from a private location:
-
In Anypoint Secrets Manager, create a shared secret as a symmetric key. If the info that you want to store as a shared secret is currently in plain text, then you must encode it in Base64 before you create the secret.
-
Open a secrets group or create a new one. The group must be created in the same environment that you are using in the BAT CLI. To find out which environment you are currently using in the BAT CLI, run the command
bat whoami
. In the output, there is the ID for the environment. Run the commandbat environment ls
to list the environments that you have access to. Match the ID from thebat whoami
command with one of the environments listed. If you need to switch to the environment that your secrets group is in, run the commandbat environment switch name
, wherename
is the name of the environment. -
Select Shared Secret on the left side of the screen.
-
In the Type field, select Symmetric Key.
-
In the Key field, paste your endpoint URL encoded as a Base64 string.
-
Paste the same Base64 string into the Confirm Key field.
-
-
Copy the name of the new shared secret.
-
At a command prompt, run the
bat grant
command, specifying an alias for the shared secret. When you run this command, the BAT CLI creates a section namedsecrets
in your test suite’sbat.yaml
file, if the section does not already exist. In that section, the BAT CLI adds these indented lines:alias: secretId: "secret-ID"
- alias
-
This is the alias that you specified in the
bat grant
command. - secret-ID
-
This is the ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your
bat.yaml
file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.
-
In your
bat.yaml
file, add Sumo Logic as a reporter by adding these indented lines to thereporters
section:- type: SumoLogic outfile: filename.json options: SUMO_ENDPOINT: <alias_of_the_shared_secret> DISABLED_FOR_CONFIGS: dev,qa.
- filename.json
-
Specify a name for the output file to create.
- alias_of_the_shared_secret
-
Specify the alias that you used for the secret for the endpoint URL when you ran the
bat grant
command. The alias must be between carets.
-
-
If you do not want to keep your endpoint URL secret, add these lines to the
reporters
section in yourbat.yaml
file:- DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.
- type: SumoLogic outfile: filename.json options: SUMO_ENDPOINT: endpoint-URL DISABLED_FOR_CONFIGS: dev,qa.
- filename.json
-
Specify a name for the output file to create.
- endpoint-URL
-
The URL of the endpoint for the collection.
- DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension. == Email
To email test reports that indicate failures, specify any email addresses that you want to send reports to. Each report includes the name of the suite in which the error occurs and a list of the statements.
Reports are emailed only for tests created in API Functional Monitoring in Anypoint Platform, not for tests created with the BAT CLI. |
-
In your
bat.yaml
file, create thereporters
section, if one doesn’t already exist in the file. On a blank line, typereporters:
. -
If you want to keep an email address secret and you are running tests from a private location:
-
In Anypoint Secrets Manager, create a shared secret as a symmetric key. If the info that you want to store as a shared secret is currently in plain text, then you must encode it in Base64 before you create the secret.
-
Open a secrets group or create a new one. The group must be created in the same environment that you are using in the BAT CLI. To find out which environment you are currently using in the BAT CLI, run the command
bat whoami
. In the output, there is the ID for the environment. Run the commandbat environment ls
to list the environments that you have access to. Match the ID from thebat whoami
command with one of the environments listed. If you need to switch to the environment that your secrets group is in, run the commandbat environment switch name
, wherename
is the name of the environment. -
Select Shared Secret on the left side of the screen.
-
In the Type field, select Symmetric Key.
-
In the Key field, paste your the email address encoded as a Base64 string.
-
Paste the same Base64 string into the Confirm Key field.
-
-
Copy the name of the new shared secret.
-
At a command prompt, run the
bat grant
command, specifying an alias for the shared secret. When you run this command, the BAT CLI creates a section namedsecrets
in your test suite’sbat.yaml
file, if the section does not already exist. In that section, the BAT CLI adds these indented lines:alias: secretId: "secret-ID"
- alias
-
This is the alias that you specified in the
bat grant
command. - secret-ID
-
This is the ID of the secret within Anypoint Secrets Manager. This ID does not appear in ASM, so there is no way for someone looking in your
bat.yaml
file to associate the ID with any particular secret. The BAT CLI uses this ID to look up the secret that you associated with the alias.
-
In your
bat.yaml
file, addEmail
as a reporter by adding these indented lines to thereporters
section:- type: Email options: EMAILS: <alias_of_the_shared_secret> DISABLED_FOR_CONFIGS: dev,qa
- DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.
- alias_of_the_shared_secret
-
Specify the alias that you used for the secret for the email address when you ran the
bat grant
command. The alias must be between carets.If you want to add aliases for additional email addresses, put each alias between carets and separate the aliases with semicolons:
EMAILS: <alias_1>;<alias_2>;<alias_3>
-
-
If you do not want to keep an email address secret, add these lines to the
reporters
section in yourbat.yaml
file:- type: Email options: EMAILS: email_address DISABLED_FOR_CONFIGS: dev,qa
- email_address
-
Specify the email address to send notifications to. If you want to use more than one email address, separate them with semicolons:
EMAILS: email_address1;email_address2;email_address3
- DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.
Custom Reports
BAT publishes the result.json with a post to the URL that you specify in the bat.yaml file. You can add headers if necessary.
To publish custom reports for a test suite, add the following indented lines to the reporters
section of the bat.yaml
file, replacing the value for URL
with your own URL.
reporters:
- type: Custom
options:
URL: "http://www.httpbin.org/post"
HEADERS: "'Authorization':'Bearer token','Content-Type':'application/json'"
DISABLED_FOR_CONFIGS: dev,qa
- DISABLED_FOR_CONFIGS
-
Disables the reporter for one or more configurations. The value is the name of one configuration or multiple names that are separated by commas. Names do not include the .dwl file extension.
Local Reports
BAT generates basic types of reports locally in JSON and HTML. When you execute tests, the path to the reports appears in the output.
Example:
Reporter: /var/folders/vz/56jp75d941592x_1d8mfy9f40000gn/T/bat_report_20180320121736.json
Reporter: /var/folders/vz/56jp75d941592x_1d8mfy9f40000gn/T/bat_report_20180320121736.html
To Configure Local Report Generation
In the bat.yaml file, specify a type
field to generate local reports. For example:
reporters:
- type: JSON
outFile: JSON.json
- type: HTML
outFile: HTML.html
- type: JUnit
outFile: JUnit.xml
To Create a Local Custom Report
You can generate a custom report based on a DataWeave transformation that is specified in the bat.yaml using the file
keyword.
reporters:
- type: Local
file: reporter/transform.dwl
outFile: local.json
The transform.dwl file contains the following code:
%dw 2.0
output application/json
var result = if (payload.result.pass[0]) " is ok" else " failed"
---
{
result: "The suite <" ++ payload.name ++ "> with assertion <" ++ payload.result[0].name ++ ">" ++ result
}
And the result is:
{
"result": "The suite <Hello world suite> with assertion <answer 200> is ok"
}