Configuring quality reports in an action - Amazon CodeCatalyst
Auto-discovery and manual reportsConfiguring success criteria for reportsQuality reports YAML example

Configuring quality reports in an action

This section describes how to configure a quality report in an action.

Auto-discovery and manual reports

When auto-discovery is enabled, CodeCatalyst searches all inputs passed into the action, and all files generated by the action itself, looking for test, code coverage, software composition analysis (SCA), and static analysis (SA) reports. You can view and manipulate each of these reports in CodeCatalyst.

You can also manually configure which reports are generated. You can specify the type of report you'd like to generate as well as the file format. For more information, see Quality report types.

Configuring success criteria for reports

You can set the values that determine the success criteria for a test, code coverage, software composition analysis (SCA), or static analysis (SA) report.

Success criteria are thresholds that determine whether a report passes or fails. CodeCatalyst first generates your report, which can be a test, code coverage, SCA, or SA report, and then applies the success criteria to the generated reports. It then shows whether the success criteria were met, and to what extent. If any report does not meet the specified success criteria, the CodeCatalyst action that specified the success criteria fails.

For example, when you set the success criteria for your SCA report, the valid vulnerability values ranging from most to least severe are: CRITICAL, HIGH, MEDIUM, LOW, INFORMATIONAL. If you set the criteria to scan for one vulnerability at HIGH severity, the report will fail if there is either at least one vulnerability at HIGH severity or no vulnerabilities at HIGH severity, but at least one vulnerability at a higher severity level, such as one vulnerability at CRITICAL severity.

If you do not specify success criteria, then:

  • The CodeCatalyst report that is generated based on your raw reports will not display success criteria.

  • Success criteria will not be used to determine whether the associated workflow action passes or fails.

Visual
To configure success criteria

  1. In the navigation pane, choose CI/CD, and then choose Workflows.

  2. Choose a workflow containing an action that generates a report. This is the report for which you want to apply success criteria. You can filter by the source repository or branch name where the workflow is defined, or filter by workflow name.

  3. Choose Edit.

  4. Choose Visual.

  5. In the workflow diagram, choose the action that you have configured to generate CodeCatalyst reports.

  6. Choose the Outputs tab.

  7. Under Auto-discover reports or under Manually configure reports, choose Success criteria.

    Success criteria appear. Depending on your previous selections, you may see any or all of these options:

    Pass rate

    Specify the percentage of tests in a test report that must pass for the associated CodeCatalyst report to be marked as passed. Valid values include decimal numbers. For example: 50, 60.5. The pass rate criteria are applied only to test reports. For more information about test reports, see Test reports.

    Line coverage

    Specify the percentage of lines in a code coverage report that must be covered for the associated CodeCatalyst report to be marked as passed. Valid values include decimal numbers. For example: 50, 60.5. Line coverage criteria are applied only to code coverage reports. For more information about code coverage reports, see Code coverage reports.

    Branch coverage

    Specify the percentage of branches in a code coverage report that must be covered for the associated CodeCatalyst report to be marked as passed. Valid values include decimal numbers. For example: 50, 60.5. Branch coverage criteria are applied only to code coverage reports. For more information about code coverage reports, see Code coverage reports.

    Vulnerabilities (SCA)

    Specify the maximum number and severity of vulnerabilities permitted in the SCA report for the associated CodeCatalyst report to be marked as passed. To specify vulnerabilities, you must specify:

    • The minimum severity of the vulnerabilities you want to include in the count. Valid values, from most to least severe, are: CRITICAL, HIGH, MEDIUM, LOW, INFORMATIONAL.

      For example, if you choose HIGH, then HIGH and CRITICAL vulnerabilities will be tallied.

    • The maximum number of vulnerabilities of the specified severity you want permit. Exceeding this number causes the CodeCatalyst report to be marked as failed. Valid values are whole numbers.

    Vulnerabilities criteria are applied only to SCA reports. For more information about SCA reports, see Software composition analysis reports.

    Bugs

    Specify the maximum number and severity of bugs permitted in the SA report for the associated CodeCatalyst report to be marked as passed. To specify bugs, you must specify:

    • The minimum severity of the bugs you want to include in the count. Valid values, from most to least severe, are: CRITICAL, HIGH, MEDIUM, LOW, INFORMATIONAL.

      For example, if you choose HIGH, then HIGH and CRITICAL bugs will be tallied.

    • The maximum number of bugs of the specified severity you want permit. Exceeding this number causes the CodeCatalyst report to be marked as failed. Valid values are whole numbers.

    Bugs criteria are applied only to PyLint and ESLint SA reports. For more information about SA reports, see Static analysis reports.

    Security vulnerabilities

    Specify the maximum number and severity of security vulnerabilities permitted in the SA report for the associated CodeCatalyst report to be marked as passed. To specify security vulnerabilities, you must specify:

    • The minimum severity of the security vulnerabilities you want to include in the count. Valid values, from most to least severe, are: CRITICAL, HIGH, MEDIUM, LOW, INFORMATIONAL.

      For example, if you choose HIGH, then HIGH and CRITICAL security vulnerabilities will be tallied.

    • The maximum number of security vulnerabilities of the specified severity you want permit. Exceeding this number causes the CodeCatalyst report to be marked as failed. Valid values are whole numbers.

    Security vulnerabilities criteria are applied only to PyLint and ESLint SA reports. For more information about SA reports, see Static analysis reports.

    Quality issues

    Specify the maximum number and severity of quality issues permitted in the SA report for the associated CodeCatalyst report to be marked as passed. To specify quality issues, you must specify:

    • The minimum severity of the quality issues you want to include in the count. Valid values, from most to least severe, are: CRITICAL, HIGH, MEDIUM, LOW, INFORMATIONAL.

      For example, if you choose HIGH, then HIGH and CRITICAL quality issues will be tallied.

    • The maximum number of quality issues of the specified severity you want permit. Exceeding this number causes the CodeCatalyst report to be marked as failed. Valid values are whole numbers.

    Quality issues criteria are applied only to PyLint and ESLint SA reports. For more information about SA reports, see Static analysis reports.

  8. Choose Commit.

  9. Run your workflow to have CodeCatalyst apply success criteria to your raw reports, and regenerate the associated CodeCatalyst reports with success criteria information included. For more information, see Starting a workflow run manually.

YAML
To configure success criteria

  1. In the navigation pane, choose CI/CD, and then choose Workflows.

  2. Choose a workflow containing an action that generates a report. This is the report for which you want to apply success criteria. You can filter by the source repository or branch name where the workflow is defined, or filter by workflow name.

  3. Choose Edit.

  4. Choose YAML.

  5. In the workflow diagram, choose the action that you have configured to generate CodeCatalyst reports.

  6. In the details pane, choose the Outputs tab.

  7. In the action, in AutoDiscoverReports section, or in the Reports section, add a SuccessCriteria property, along with PassRate, LineCoverage, BranchCoverage, Vulnerabilities, StaticAnalysisBug, StaticAnalysisSecurity, and StaticAnalysisQuality properties.

    For an explanation of each of these properties, consult the Build and test action YAML definition.

  8. Choose Commit.

  9. Run your workflow to have CodeCatalyst apply success criteria to your raw reports, and regenerate the associated CodeCatalyst reports with the success criteria information included. For more information on starting a workflow, see Starting a workflow run manually.

Quality reports YAML example

The following example shows how to manually configure four reports: a test report, a code coverage report, a software composition analysis report, and a static analysis report.

Reports:
  MyTestReport:
    Format: JUNITXML
    IncludePaths:
      - "*.xml"
    ExcludePaths:
      - report1.xml
      SuccessCriteria:
        PassRate: 90
  MyCoverageReport:
    Format: CLOVERXML
    IncludePaths:
      - output/coverage/jest/clover.xml
      SuccessCriteria:
        LineCoverage: 75
        BranchCoverage: 75
  MySCAReport:
    Format: SARIFSCA
    IncludePaths:
      - output/sca/reports.xml
      SuccessCriteria:
        Vulnerabilities:
          Number: 5
          Severity: HIGH
  MySAReport:
    Format: ESLINTJSON
    IncludePaths:
      - output/static/eslint.xml
      SuccessCriteria:
        StaticAnalysisBug:
          Number: 10
          Severity: MEDIUM
        StaticAnalysisSecurity:
          Number: 5
          Severity: CRITICAL
        StaticAnalysisQuality:
          Number: 0
          Severity: INFORMATIONAL