Matrix outcome in Github workflows

At Lottie, we use Datadog extensively. From instrumenting our services to synthetic testing. These tests are then referenced in our CI/CD pipelines to verify that new deployments are without degradation

A bunch of these end-to-end tests were running sequentially as part of a single job in a GitHub Actions workflow, invoked on deployments, etc. It worked okay until the number of tests started to increase.

Issues started to surface when tests failed.

Since the invocation of all the tests was a single job, upon failure, all the tests had to be re-run, causing blocked pipelines, frustrated colleagues, and wasted compute minutes both on GitHub and Datadog.

A task was created to make use of a matrix job, to utilize parallel running of the tests and provide the ability to re-run only the failed jobs in cases when the test needed updates, etc.

This worked fine, but we noticed that since we "matrixified" our tests, none of the subsequent steps ran once the test step of the job finished, cutting off Slack notifications and, more importantly, our semaphore system.

I consulted GitHub Docs, which did not prove useful. Then, a quick Google search for GitHub Actions matrix job status landed me on a GitHub Community Discussion page, where I quickly found my answer.

Simply set up a new job within the workflow that always runs and evaluates the outcomes of the matrix job, exiting with code 1 if any of the previous jobs were:

• Cancelled.
• Skipped.
• Failed.

Something like this could be a good base:

e2e_test:
...
e2e_test_results:
  if: ${{ always() }}
  needs:
    - e2e_test
  runs-on: ubuntu-22.04
  steps:
    - name: Slack Notification on failure
      if: >-
        ${{
              contains(needs.*.result, 'failure')
          || contains(needs.*.result, 'cancelled')
          || contains(needs.*.result, 'skipped')
        }}
      uses: rtCamp/action-slack-notify@v2
    - name:
        Check all job status
        # see https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#needs-context
        # see https://stackoverflow.com/a/67532120/4907315
      if: >-
        ${{
              contains(needs.*.result, 'failure')
          || contains(needs.*.result, 'cancelled')
          || contains(needs.*.result, 'skipped')
        }}
      run: exit 1