Amazon CodeCatalyst is no longer open to new customers. Existing customers can continue to use the service as normal. For more information, see [How to migrate from CodeCatalyst](migration.md). # Configuring quality reports in an action This section describes how to configure a quality report in an action. **Topics** + [Auto-discovery and manual reports](#test.auto-discovery) + [Configuring success criteria for reports](#test.success-criteria) + [Quality reports YAML example](#test.success-criteria-example) ## Auto-discovery and manual reports When auto-discovery is enabled, CodeCatalyst searches all inputs passed into the action, and all files generated by the action itself, looking for test, code coverage, software composition analysis (SCA), and static analysis (SA) reports. You can view and manipulate each of these reports in CodeCatalyst. You can also manually configure which reports are generated. You can specify the type of report you'd like to generate as well as the file format. For more information, see [Quality report types](test-workflow-actions.md#test-reporting). ## Configuring success criteria for reports You can set the values that determine the success criteria for a test, code coverage, software composition analysis (SCA), or static analysis (SA) report. Success criteria are thresholds that determine whether a report passes or fails. CodeCatalyst first generates your report, which can be a test, code coverage, SCA, or SA report, and then applies the success criteria to the generated reports. It then shows whether the success criteria were met, and to what extent. If any report does not meet the specified success criteria, the CodeCatalyst action that specified the success criteria fails. For example, when you set the success criteria for your SCA report, the valid vulnerability values ranging from most to least severe are: `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, `INFORMATIONAL`. If you set the criteria to scan for one vulnerability at `HIGH` severity, the report will fail if there is either at least one vulnerability at `HIGH` severity or no vulnerabilities at `HIGH` severity, but at least one vulnerability at a higher severity level, such as one vulnerability at `CRITICAL` severity. If you do not specify success criteria, then: + The CodeCatalyst report that is generated based on your raw reports will not display success criteria. + Success criteria will not be used to determine whether the associated workflow action passes or fails. ------ #### [ Visual ] **To configure success criteria** 1. In the navigation pane, choose **CI/CD**, and then choose **Workflows**. 1. Choose a workflow containing an action that generates a report. This is the report for which you want to apply success criteria. You can filter by the source repository or branch name where the workflow is defined, or filter by workflow name or status. 1. Choose **Edit**. 1. Choose **Visual**. 1. In the workflow diagram, choose the action that you have configured to generate CodeCatalyst reports. 1. Choose the **Outputs** tab. 1. Under **Auto-discover reports** or under **Manually configure reports**, choose **Success criteria**. Success criteria appear. Depending on your previous selections, you may see any or all of these options: **Pass rate** Specify the percentage of tests in a test report that must pass for the associated CodeCatalyst report to be marked as passed. Valid values include decimal numbers. For example: `50`, `60.5`. The pass rate criteria are applied only to test reports. For more information about test reports, see [Test reports](test-workflow-actions.md#test-reports). **Line coverage** Specify the percentage of lines in a code coverage report that must be covered for the associated CodeCatalyst report to be marked as passed. Valid values include decimal numbers. For example: `50`, `60.5`. Line coverage criteria are applied only to code coverage reports. For more information about code coverage reports, see [Code coverage reports](test-workflow-actions.md#test-code-coverage-reports). **Branch coverage** Specify the percentage of branches in a code coverage report that must be covered for the associated CodeCatalyst report to be marked as passed. Valid values include decimal numbers. For example: `50`, `60.5`. Branch coverage criteria are applied only to code coverage reports. For more information about code coverage reports, see [Code coverage reports](test-workflow-actions.md#test-code-coverage-reports). **Vulnerabilities (SCA)** Specify the maximum number and severity of vulnerabilities permitted in the SCA report for the associated CodeCatalyst report to be marked as passed. To specify vulnerabilities, you must specify: + The minimum severity of the vulnerabilities you want to include in the count. Valid values, from most to least severe, are: `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, `INFORMATIONAL`. For example, if you choose `HIGH`, then `HIGH` and `CRITICAL` vulnerabilities will be tallied. + The maximum number of vulnerabilities of the specified severity you want permit. Exceeding this number causes the CodeCatalyst report to be marked as failed. Valid values are whole numbers. Vulnerabilities criteria are applied only to SCA reports. For more information about SCA reports, see [Software composition analysis reports](test-workflow-actions.md#test-sca-reports). **Bugs** Specify the maximum number and severity of bugs permitted in the SA report for the associated CodeCatalyst report to be marked as passed. To specify bugs, you must specify: + The minimum severity of the bugs you want to include in the count. Valid values, from most to least severe, are: `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, `INFORMATIONAL`. For example, if you choose `HIGH`, then `HIGH` and `CRITICAL` bugs will be tallied. + The maximum number of bugs of the specified severity you want permit. Exceeding this number causes the CodeCatalyst report to be marked as failed. Valid values are whole numbers. Bugs criteria are applied only to PyLint and ESLint SA reports. For more information about SA reports, see [Static analysis reports](test-workflow-actions.md#test-static-analysis-reports). **Security vulnerabilities** Specify the maximum number and severity of security vulnerabilities permitted in the SA report for the associated CodeCatalyst report to be marked as passed. To specify security vulnerabilities, you must specify: + The minimum severity of the security vulnerabilities you want to include in the count. Valid values, from most to least severe, are: `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, `INFORMATIONAL`. For example, if you choose `HIGH`, then `HIGH` and `CRITICAL` security vulnerabilities will be tallied. + The maximum number of security vulnerabilities of the specified severity you want permit. Exceeding this number causes the CodeCatalyst report to be marked as failed. Valid values are whole numbers. Security vulnerabilities criteria are applied only to PyLint and ESLint SA reports. For more information about SA reports, see [Static analysis reports](test-workflow-actions.md#test-static-analysis-reports). **Quality issues** Specify the maximum number and severity of quality issues permitted in the SA report for the associated CodeCatalyst report to be marked as passed. To specify quality issues, you must specify: + The minimum severity of the quality issues you want to include in the count. Valid values, from most to least severe, are: `CRITICAL`, `HIGH`, `MEDIUM`, `LOW`, `INFORMATIONAL`. For example, if you choose `HIGH`, then `HIGH` and `CRITICAL` quality issues will be tallied. + The maximum number of quality issues of the specified severity you want permit. Exceeding this number causes the CodeCatalyst report to be marked as failed. Valid values are whole numbers. Quality issues criteria are applied only to PyLint and ESLint SA reports. For more information about SA reports, see [Static analysis reports](test-workflow-actions.md#test-static-analysis-reports). 1. Choose **Commit**. 1. Run your workflow to have CodeCatalyst apply success criteria to your raw reports, and regenerate the associated CodeCatalyst reports with success criteria information included. For more information, see [Starting a workflow run manually](workflows-manually-start.md). ------ #### [ YAML ] **To configure success criteria** 1. In the navigation pane, choose **CI/CD**, and then choose **Workflows**. 1. Choose a workflow containing an action that generates a report. This is the report for which you want to apply success criteria. You can filter by the source repository or branch name where the workflow is defined, or filter by workflow name or status. 1. Choose **Edit**. 1. Choose **YAML**. 1. In the workflow diagram, choose the action that you have configured to generate CodeCatalyst reports. 1. In the details pane, choose the **Outputs** tab. 1. In the action, in `AutoDiscoverReports` section, or in the `Reports` section, add a **SuccessCriteria** property, along with `PassRate`, `LineCoverage`, `BranchCoverage`, `Vulnerabilities`, `StaticAnalysisBug`, `StaticAnalysisSecurity`, and `StaticAnalysisQuality` properties. For an explanation of each of these properties, consult the [Build and test actions YAML](build-action-ref.md). 1. Choose **Commit**. 1. Run your workflow to have CodeCatalyst apply success criteria to your raw reports, and regenerate the associated CodeCatalyst reports with the success criteria information included. For more information on starting a workflow, see [Starting a workflow run manually](workflows-manually-start.md). ------ ## Quality reports YAML example The following example shows how to manually configure four reports: a test report, a code coverage report, a software composition analysis report, and a static analysis report. ``` Reports: MyTestReport: Format: JUNITXML IncludePaths: - "*.xml" ExcludePaths: - report1.xml SuccessCriteria: PassRate: 90 MyCoverageReport: Format: CLOVERXML IncludePaths: - output/coverage/jest/clover.xml SuccessCriteria: LineCoverage: 75 BranchCoverage: 75 MySCAReport: Format: SARIFSCA IncludePaths: - output/sca/reports.xml SuccessCriteria: Vulnerabilities: Number: 5 Severity: HIGH MySAReport: Format: ESLINTJSON IncludePaths: - output/static/eslint.xml SuccessCriteria: StaticAnalysisBug: Number: 10 Severity: MEDIUM StaticAnalysisSecurity: Number: 5 Severity: CRITICAL StaticAnalysisQuality: Number: 0 Severity: INFORMATIONAL ```