Automated Testing in a CI/CD Pipeline with TestCollab

On this page

Integrating automated testing into a CI/CD pipeline means running tests as part of the build or deployment workflow and recording their results. TestCollab connects those automated checks to the test plans and manual testing your team already manages. Build traceability and deployment decisions are optional extensions.

This guide gives you a working first integration with the TestCollab CLI, then points you to provider-specific and framework-specific guides.

What TestCollab Does in a Pipeline

Your existing test framework still runs the tests. TestCollab does not provide a hosted test runner and does not start your CI pipeline. Instead, @testcollab/cli runs inside your pipeline and adds test management around the tools you already use.

• Creates or selects a Test Plan for the version under test

• Uploads JUnit XML or Mochawesome JSON and maps results to TestCollab cases

• Optionally records build context when you want version traceability

• Optionally evaluates a quality gate when CI should control a later deployment

The Three-Step Reporting Workflow

The standard reporting workflow creates a TestCollab plan, runs your tests, and reports the result. It does not require createBuild or gate:

tc createTestPlan
      |
      v
your test runner creates JUnit XML or Mochawesome JSON
      |
      v
tc report

1. tc createTestPlan creates a plan from CI-tagged cases and writes its ID for later steps.

2. Your existing test framework runs on the CI agent and writes JUnit XML or Mochawesome JSON.

3. tc report reads that file and updates the matching executions in the plan.

Prerequisites

For the standard reporting workflow below, you need:

1. A TestCollab API token from your profile settings

2. Your numeric project ID

3. A tag applied to the cases that belong in the CI plan, plus that tag's ID

4. The user ID that will own the automated execution. Use the owner of the API token

5. Test names containing an explicit TestCollab ID such as [TC-42] Login should succeed

6. A test framework configured to write JUnit XML or Mochawesome JSON

Add the GitHub Actions Secrets

In your repository, open Settings > Secrets and variables > Actions and add these repository secrets:

Your First Integration in About 10 Minutes

Create .github/workflows/testcollab.yml. This example uses Playwright's built-in JUnit reporter, but the TestCollab steps are the same for any framework that writes JUnit XML.

name: Test and report to TestCollab

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    env:
      TESTCOLLAB_TOKEN: ${{ secrets.TESTCOLLAB_TOKEN }}
      TC_PROJECT_ID: ${{ secrets.TC_PROJECT_ID }}
      TC_CI_TAG_ID: ${{ secrets.TC_CI_TAG_ID }}
      TC_ASSIGNEE_ID: ${{ secrets.TC_ASSIGNEE_ID }}
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '22'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps

      - name: Create a TestCollab plan
        run: |
          npx @testcollab/cli createTestPlan \
            --project "$TC_PROJECT_ID" \
            --ci-tag-id "$TC_CI_TAG_ID" \
            --assignee-id "$TC_ASSIGNEE_ID"
          cat tmp/tc_test_plan >> $GITHUB_ENV

      - name: Run Playwright tests
        run: PLAYWRIGHT_JUNIT_OUTPUT_NAME=results.xml npx playwright test --reporter=junit

      - name: Upload results to TestCollab
        if: always()
        run: |
          npx @testcollab/cli report \
            --project "$TC_PROJECT_ID" \
            --test-plan-id "$TESTCOLLAB_TEST_PLAN_ID" \
            --format junit \
            --result-file ./results.xml

The report step uses if: always(), so it runs after a failed Playwright step. Because the test step does not use continue-on-error, a failed test still leaves the job failed after its result is uploaded.

How the Workflow Works

1. createTestPlan selects cases with the CI tag, assigns them, and writes the new plan ID to tmp/tc_test_plan.

2. The workflow loads that ID into GITHUB_ENV, so the report step can use TESTCOLLAB_TEST_PLAN_ID.

3. Playwright writes results.xml. report maps names such as [TC-42] to executions in the plan.

Optional Build Traceability and Quality Gates

After the three-step reporting workflow works, you can independently add either of these extensions:

• tc createBuild records or reuses the deployed version and lets createTestPlan --build link the plan to it.

• tc gate evaluates the reported plan when CI should block a later deployment.

Neither extension is required to create a plan or report results, and you do not need to adopt both.

Pick Your CI Provider

The reporting workflow works in any system that can run Node.js and shell commands. Use the provider guide that matches your pipeline:

• GitHub Actions

• GitLab CI

• Azure Pipelines

• Bitbucket Pipelines

• Jenkins

• CircleCI

Other CI providers can run the same three reporting steps. If you separately opt into build traceability, pass the provider's version and URL values explicitly to tc createBuild.

Pick Your Test Framework and Result Format

TestCollab accepts two result formats:

• JUnit XML - the broadest option. Playwright, pytest, Jest, JUnit, TestNG, PHPUnit, Robot Framework, Selenium-based suites, and many other runners can generate it. See JUnit XML Format Explained.

• Mochawesome JSON - a direct option for Cypress and Mocha-based suites.

Use --format junit for JUnit XML or --format mochawesome for Mochawesome JSON. The --result-file path must point to the file your test command actually created.

Test Case ID Mapping

For an existing TestCollab plan, put an explicit case ID in each automated test name. The recommended form is:

[TC-42] Login should succeed

The CLI also recognizes TC-42, id-42, and testcase-42. Explicit markers are checked before any fallback, so names containing technical numbers such as UTF-8 or SHA-256 still map to the stated [TC-42] ID.

Where Manual Testing Fits

A TestCollab plan can contain automated cases and cases executed by people. This is useful when a release needs both fast regression checks and human review of usability, exploratory risk, hardware, or business acceptance.

tc report updates the automated cases, while testers execute the remaining cases in TestCollab. If you choose to add a production gate, --require-complete can make that optional gate wait for both groups to have a result.

Authentication and the EU Region

Store the API token in your CI system's secret manager and expose it as TESTCOLLAB_TOKEN. Do not put the token directly in pipeline YAML or pass it as a visible command-line value.

The default API is https://api.testcollab.io. For an EU-hosted account, add this flag to every TestCollab CLI command:

--api-url https://api-eu.testcollab.io

Troubleshooting

The result file is missing

Make the reporter write to a known file and pass the same path to --result-file. Playwright writes JUnit XML to standard output unless you set PLAYWRIGHT_JUNIT_OUTPUT_NAME or configure an output file.

Cases are not matched

Confirm every test name has a supported ID marker such as [TC-42], the case belongs to the plan, and the API token owner is the assigned executor.

The gate passes before manual testing is done

Add --require-complete. Without it, unexecuted cases are excluded from the pass-rate denominator and do not fail the default gate.

Frequently Asked Questions

Can TestCollab trigger my pipeline?

No. Your CI provider starts the workflow based on its normal triggers. The core workflow calls @testcollab/cli to create a plan and send results to TestCollab. Optional build and gate commands do not give TestCollab control of your CI system.

Which CI tools are supported?

Azure DevOps, GitHub Actions, GitLab CI, Bitbucket Pipelines, CircleCI, and Jenkins all have provider guides for the same reporting workflow. Any other CI tool that can run Node.js and shell commands can use it too. Optional createBuild provenance detection covers those six named providers.

Does TestCollab run my automated tests?

No. Playwright, Cypress, pytest, Jest, JUnit, or another framework runs them on your CI agent. TestCollab records and organizes the resulting evidence.

Do I need to create all TestCollab records first?

Not always. The standard workflow uses a CI tag and tc createTestPlan. For a zero-setup import, tc report --auto-create can create missing suites, cases, a folder, and a plan directly from the result file.