Best practices for testing - Amazon CodeCatalyst
Auto-discoverySuccess criteriaInclude/exclude paths

Best practices for testing

When using the testing features provided by CodeCatalyst, we recommend that you follow these best practices.

Auto-discovery

When configuring actions in CodeCatalyst, auto-discovery lets you automatically discover outputs of various tools, such as JUnit test reports, and generate relevant CodeCatalyst reports from them. Auto-discovery helps ensure that reports continue to be generated even if names or paths to discovered outputs change. When new files are added, CodeCatalyst automatically discovers them and produces relevant reports. However, if you use auto-discovery, it is important to factor in some of the following aspects of this feature:

  • When you activate auto-discovery in your action, all automatically discovered reports of the same type will share the same success criteria. For example, a shared criteria such as minimum pass rate would apply to all auto-discovered test reports. If you need different criteria for reports of the same type, you must explicitly configure each of these reports.

  • Auto-discovery can also find reports that are produced by your dependencies and, if success criteria are configured, might fail the action on these reports. This issue can be addressed by updating the exclude path configuration.

  • Auto-discovery is not guaranteed to produce the same list of reports every time, because it scans the action at runtime. In the case where you want a particular report to always be produced, you should configure reports explicitly. For example, if tests were to stop running as part of your build, the test framework would not produce any outputs and, as a result, no test report would be produced and the action might succeed. If you want the success of your action to depend on that particular test, then you must explicitly configure that report.

Tip

When getting started on a new or existing project, use auto-discovery for the entire project directory (include **/*). This invokes report generation across all files in your project, including those within subdirectories.

For more information, see Configuring quality reports in an action.

Success criteria

You can enforce quality thresholds on your reports by configuring success criteria. For example, if two code coverage reports were auto-discovered, one with a line coverage of 80% and the other with a line coverage of 60%, you have the following options:

  • Set the auto-discovery success criteria for line coverage at 80%. This would cause the first report to pass and the second report to fail, which would result in the overall action failing. To unblock the workflow, add new tests to your project until the line coverage for the second report exceeds 80%.

  • Set the auto-discovery success criteria for line coverage at 60%. This would cause both reports to pass, which would result in the action succeeding. You could then work on increasing the code coverage in the second report. However, with this approach, you cannot guarantee that the coverage in the first report is not dropping below 80%.

  • Explicitly configure one or both of the reports by using the visual editor or adding an explicit YAML section and path for each report. This would allow you to configure separate success criteria and custom names for each report. However, with this approach, the action could fail if the report paths change.

For more information, see Configuring success criteria for reports.

Include/exclude paths

When reviewing action results, you can adjust the list of reports that are generated by CodeCatalyst by configuring IncludePaths and ExcludePaths.

  • Use IncludePaths to specify the files and file paths you want CodeCatalyst to include when searching for reports. For example, if you specify "/test/report/*", CodeCatalyst searches the entire build image used by the action looking for the /test/report/ directory. When it finds that directory, CodeCatalyst then looks for reports in that directory.

    Note

    For manually configured reports, IncludePaths must be a glob pattern that matches a single file.

  • Use ExcludePaths to specify the files and file paths you want CodeCatalyst to exclude when searching for reports. For example, if you specify "/test/reports/**/*", CodeCatalyst will not search for files in the /test/reports/ directory. To ignore all files in a directory, use the **/* glob pattern.

The following are examples of possible glob patterns.

Pattern Description

*.*

Matches all object names in the current directory that contain a dot

*.xml

Matches all object names in the current directory ending with .xml

*.{xml,txt}

Matches all object names in the current directory ending with .xml or .txt

**/*.xml

Matches object names across all directories ending with .xml

testFolder

Matches an object called testFolder, treating it as a file

testFolder/*

Matches objects in one level of the subfolder from testFolder, such as testFolder/file.xml

testFolder/*/*

Matches objects in two levels of the subfolder from testFolder, such as testFolder/reportsFolder/file.xml

testFolder/**

Matches subfolder testFolder as well as files below testFolder, such as testFolder/file.xml and testFolder/otherFolder/file.xml

CodeCatalyst interprets the glob patterns as follows:

  • The slash (/) character separates directories in file paths.

  • The asterisk (*) character matches zero or more characters of a name component without crossing folder boundaries.

  • A double asterisk (**) matches zero or more characters of a name component across all directories.

Note

ExcludePaths takes precedence over IncludePaths. If both IncludePaths and ExcludePaths include the same folder, that folder is not scanned for reports.