Check that your new Playwright test is robust before letting it loose on the CICD pipeline

Boyana

Cover Image for Check that your new Playwright test is robust before letting it loose on the CICD pipeline

Boyana

As hard as we try to prevent it, flaky tests happen sometimes. It is important to make extra sure that a test is robust before releasing it on the CICD pipeline, where it can potentially cause bottlenecks.

One way to surface failures is to run the tests multuple times. When using Playwright, you can run the below command. This will repeat the selected test 20 times.

npx playwright test  {{insert test name here}} –repeat–each=20

However, sometimes the test might behave differently in the pipeline. Nothing more annoying than merging your new test and it start failing in the pipeline

If you use Github Actions, you can create a custom github action workflow to run the Playwright command above. That would allow you to run the new test multiple times in the pipeline against any relevant environments.

The workflow

This is creating a workflow, which can be manually run on Github Actions. It takes two inputs: the number of runs i.e. how many times you want your test to run and the test path i.e. which test to run.

If you'll run the test on different environments, for example productions / staging etc. You could also add an input to change the base url.

name: Playwright Tests
on:
  workflow_dispatch:
    inputs:
        number-of-runs:
          required: true
          type: string
        test-path:
          required: true
          type: string
          description: Needs to be in the format todo-1.spec.js  
jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
      with:
        node-version: 16
    - name: Set env variables
      run: |
          echo "NUMBER_OF_RUNS=${{ inputs.number-of-runs }}" >> "$GITHUB_ENV"
          echo "TEST_PATH=${{ github.event.inputs.test-path }}" >> "$GITHUB_ENV"
    - name: Install dependencies
      run: npm ci
    - name: Install Playwright Browsers
      run: npx playwright install --with-deps
    - name: Run Playwright tests
      run: npx playwright test "${{ env.TEST_PATH }}"
          --workers=1 --repeat-each=${{ env.NUMBER_OF_RUNS}}
    - uses: actions/upload-artifact@v3
      if: always()
      with:
        name: playwright-report
        path: playwright-report/
        retention-days: 30

boyana.dev

Check that your new Playwright test is robust before letting it loose on the CICD pipeline

The workflow