Quality Gates in CI/CD: Fail a Pipeline on Test Results with tc gate

On this page

A CI quality gate is an automated decision point that allows a pipeline to continue only when defined quality conditions are met. With TestCollab, tc gate reads a Test Plan's live execution results and returns a process exit code, so the same gate can stop a build, release, or production deployment in any CI system.

Quality gating is optional. The default reporting workflow is tc createTestPlan, run tests, then tc report, and it can stop there. Add tc gate only when CI should make a later deployment decision. tc createBuild is an independent traceability option and is not required for reporting or gating.

This guide assumes your pipeline already creates or selects a plan and uploads automated results with the TestCollab CLI. For the default reporting workflow, see Automated Testing in a CI/CD Pipeline with TestCollab.

How tc gate Decides

The command validates the plan and project, selects the latest Test Plan run by default, and reads that run's executed test cases directly from TestCollab. It does not depend on a cached plan summary, so a gate run immediately after tc report sees the newly uploaded results.

Every criterion you supply must pass. The command can count selected failure statuses, enforce a maximum number of those cases, require a minimum pass rate, and reject any run that still has unexecuted cases.

Exit Codes

Treat every nonzero result as a blocked pipeline. The command-line parser can also return a nonzero status before tc gate runs if a required option is omitted, so do not write pipeline logic that allows an unknown nonzero code to deploy.

Prerequisites

1. Install the CLI with npm install -g @testcollab/cli.

2. Generate a TestCollab API token and store it as the secret environment variable TESTCOLLAB_TOKEN.

3. Know the numeric TestCollab project ID and Test Plan ID.

4. Make sure the plan has a run with executed cases. A pipeline-created plan can be filled by tc report; human testers can execute other cases in the same run.

5. For automated mapping, use test names such as [TC-42] Login should succeed.

Start with a Basic Gate

The default gate fails when at least one case has the failed status:

tc gate \
  --project 45 \
  --test-plan-id 123

The explicit equivalent is:

tc gate \
  --project 45 \
  --test-plan-id 123 \
  --fail-on failed \
  --max-failed 0

Gate Criteria

Examples:

# Failed or blocked cases count against the gate.
# Exactly two are tolerated; three cause exit code 1.
tc gate \
  --project 45 \
  --test-plan-id 123 \
  --fail-on failed,blocked \
  --max-failed 2

# Require every case to have a result and at least 95% of executed cases to pass.
tc gate \
  --project 45 \
  --test-plan-id 123 \
  --require-complete \
  --min-pass-rate 95

Why --require-complete Matters

The default gate only checks the statuses named by --fail-on. An unexecuted case is not failed, and it is excluded from the pass-rate denominator. A plan with 5 passed cases and 20 unexecuted cases can therefore pass a default gate or show a 100% pass rate.

Add --require-complete whenever every selected case must be executed before a release can move forward:

tc gate \
  --project 45 \
  --test-plan-id 123 \
  --fail-on failed,blocked \
  --require-complete

Pattern: Staging Deploys, Production Waits for QA

A useful release-gate pattern deploys the build to staging immediately, then creates one TestCollab plan containing both automated and manual cases. The pipeline uploads automated results, people execute the remaining checks against staging, and production becomes available only after the complete plan passes.

Build
  -> Deploy to staging
  -> Create a hybrid TestCollab plan
  -> Upload automated results
  -> Human executes the remaining cases
  -> Run the production stage
  -> tc gate --require-complete
  -> Deploy to production only after exit code 0

The automated tests might cover fast regression checks while people cover exploratory behavior, visual quality, hardware, business approval, or other work that cannot be represented by the automated result file. Both groups contribute to the same gate.

Azure DevOps Deployment Gate

For a human-sign-off workflow on Microsoft-hosted agents, use a manually triggered production stage. It consumes no agent while the TestCollab plan is being executed. When QA is ready, a person selects Run stage; the gate runs first and the deployment command runs only after exit code 0.

This copyable stage assumes TC_PROJECT and TC_PLAN are pipeline variables, and TESTCOLLAB_TOKEN is a secret variable:

- stage: Deploy_Production
  displayName: 'Gate and deploy to production'
  trigger: manual
  jobs:
    - job: GateAndDeploy
      pool:
        vmImage: ubuntu-latest
      steps:
        - task: NodeTool@0
          inputs:
            versionSpec: '22.x'

        - script: npm install -g @testcollab/cli
          displayName: 'Install TestCollab CLI'

        - script: |
            tc gate \
              --project "$(TC_PROJECT)" \
              --test-plan-id "$(TC_PLAN)" \
              --fail-on failed,blocked \
              --require-complete \
              --min-pass-rate 95
          displayName: 'TestCollab production quality gate'
          env:
            TESTCOLLAB_TOKEN: $(TESTCOLLAB_TOKEN)

        - script: ./scripts/deploy-production.sh
          displayName: 'Deploy to production'

A manually triggered stage cannot declare dependsOn. If each pipeline run creates a new plan, publish tmp/tc_test_plan as a small pipeline artifact in the preparation stage and download it in the manual stage, or store the resulting plan ID in a pipeline variable through your existing release process.

Why Not Use --wait on a Hosted Agent?

--wait <seconds> polls until no case is unexecuted or the timeout is reached. It is useful for short waits and self-hosted agents, but a long human review holds the hosted agent and can hit the provider's job timeout.

If you do use it, pair it with --require-complete. After the wait expires, the command evaluates whatever results exist; without --require-complete, remaining unexecuted cases do not fail the gate.

tc gate \
  --project 45 \
  --test-plan-id 123 \
  --wait 600 \
  --poll-interval 30 \
  --require-complete

GitHub Actions Quality Gate

Place this after the step that uploads results. GitHub Actions fails the step automatically when tc gate returns 1 or 2:

- name: TestCollab quality gate
  env:
    TESTCOLLAB_TOKEN: ${{ secrets.TESTCOLLAB_TOKEN }}
  run: |
    tc gate \
      --project "${{ secrets.TC_PROJECT_ID }}" \
      --test-plan-id "$TESTCOLLAB_TEST_PLAN_ID" \
      --fail-on failed,blocked \
      --require-complete \
      --min-pass-rate 95

Do not add continue-on-error: true to the gate or test-runner step. Keep the test failure and make the reporting step run after it, such as with if: always() in GitHub Actions. Allowing the gate to fail without failing the job would neutralize the deployment decision.

GitLab CI Quality Gate

Store TESTCOLLAB_TOKEN as a masked CI/CD variable. This job runs after a result-reporting job and blocks later stages on any nonzero gate result. The report_results job must publish a dotenv artifact containing TESTCOLLAB_TEST_PLAN_ID, which needs makes available here:

quality_gate:
  stage: gate
  needs:
    - report_results
  script:
    - |
      tc gate \
        --project "$TC_PROJECT_ID" \
        --test-plan-id "$TESTCOLLAB_TEST_PLAN_ID" \
        --fail-on failed,blocked \
        --require-complete \
        --min-pass-rate 95

deploy_production:
  stage: deploy
  needs:
    - quality_gate
  script:
    - ./scripts/deploy-production.sh

Gate a Configuration or a Specific Run

By default, tc gate evaluates every configuration in the latest run of the plan.

• --config <id> filters the result to one numeric Test Plan configuration ID, such as the Chrome and Windows combination with ID 9. It is an ID, not a configuration file path.

• --regression <id> selects a specific Test Plan run or regression record ID instead of the latest run.

# Gate configuration ID 9 in the latest run.
tc gate \
  --project 45 \
  --test-plan-id 123 \
  --config 9 \
  --require-complete

# Gate run/regression record ID 7001 instead of the latest run.
tc gate \
  --project 45 \
  --test-plan-id 123 \
  --regression 7001 \
  --require-complete

EU Region

For an EU-hosted TestCollab account, add the EU API URL to tc gate:

tc gate \
  --project 45 \
  --test-plan-id 123 \
  --require-complete \
  --api-url https://api-eu.testcollab.io

Use the same --api-url https://api-eu.testcollab.io flag on whichever preceding TestCollab commands your pipeline uses. In the default reporting workflow, those commands are tc createTestPlan and tc report. tc createBuild is optional and is not required by tc gate. See Creating Builds from Your CI/CD Pipeline if you want build traceability.

Troubleshooting

The plan has no runs or no executed cases

The command returns exit code 2 because there is nothing to evaluate. Start the plan's first run, upload results with tc report, or execute at least one case before applying the gate.

The gate evaluates the wrong results

The default is the latest run. Pass the desired run record ID with --regression when your pipeline must gate an earlier or explicitly selected run.

A partially executed plan passes

Add --require-complete. Also pair it with --min-pass-rate, because unexecuted cases are excluded from the pass-rate denominator.

A configuration gate finds no cases

Confirm --config is the numeric TestCollab configuration ID used by the execution records. It is not a zero-based matrix position and not a local file.

Authentication or project validation fails

Check that TESTCOLLAB_TOKEN is present in the gate step, the token owner can access the project, the plan belongs to that project, and EU accounts use the EU API URL.

The pipeline continues after a failed gate

Remove shell constructs such as || true, provider settings such as continue-on-error, or job rules that allow failure. The CI step must preserve the exit code returned by tc gate.