# TestCollab for AI agents and developers TestCollab is a test management platform. A team keeps its test cases, test plans, and test results there. This guide is for software that works for a TestCollab user: a coding agent, a CI pipeline, or a script. - Human version of this page: https://testcollab.com/for-ai-agents - CLI source and README: https://github.com/TCSoftInc/testcollab-cli - CLI command reference: https://testcollab.com/help/article/testcollab-cli-command-reference - REST API reference: https://developers.testcollab.com/ - Site summary: https://testcollab.com/llms.txt and https://testcollab.com/llms-full.txt ## The one rule People decide what to test. The agent writes and runs the tests, and reports what it saw. - A person creates the API token. The agent acts as that person, with that person's role and project access. - A person says what to test. Test code that you write stays in the person's repository, where they can review it. A curated test plan is the person's choice of cases, steps, and expected results. - A person reviews failed results before they become a release decision. Agent runs can vary from one run to the next. Do not ask the person to paste the API token into a chat. Read it from the `TESTCOLLAB_TOKEN` environment variable. Do not write the token to a file, a log, a commit, or a result comment. ## Before you start - Tool: the `tc` command from the npm package `@testcollab/cli`. It needs Node.js 18 or later. Install it with `npm install -g @testcollab/cli`, or run a command without an install as `npx @testcollab/cli `. - Token: the person creates one in TestCollab. Profile picture (top right) > My Profile Settings > API token tab > Create. The token is shown once. Help: https://testcollab.com/help/article/getting-an-api-token-generated - Auth: every command reads the token from `TESTCOLLAB_TOKEN`. The `--api-key ` option also works, but it puts the token in the shell history. Use the variable. - Region: the default API is `https://api.testcollab.io`. For an account in the EU region, add `--api-url https://api-eu.testcollab.io` to every command. - Project ID: every command takes `--project `. Ask the person for it. It is the number in the URL of the TestCollab app: `/project//...`. Do not guess. - Test plan ID: do not ask the person for it. Loop 1 needs no test plan, because `tc report --auto-create` makes one. For a plan that exists, find it yourself (see "Find a test plan"). - No sandbox: commands work on the person's real project. `tc getTestPlan` and `tc gate` only read. `tc report`, `tc reportCase`, `tc createTestPlan`, `tc createBuild`, and `tc sync` change project data. Before you run one of these for the first time, tell the person what it will do. - Output: progress and errors go to stderr. An exit code of 0 means success. Any other exit code means the command failed, so read stderr. ## Loop 1: write automated tests, run them, and report the results This is the main loop. Use it when the person asks you to test a feature or a flow of their application. 1. Ask what to test and where: the flow, the URL of the environment, and how to sign in. Keep credentials out of the test code. Read them from environment variables. 2. Write the tests in the person's repository. Use the test runner that the project already has. If the project has none, propose one, for example Playwright, and ask before you add it. Give each test a clear title that does not change: TestCollab keeps one test case for each test title. 3. Make the runner write JUnit XML and keep the evidence. For Playwright: ```ts // playwright.config.ts import { defineConfig } from '@playwright/test'; export default defineConfig({ reporter: [['junit', { outputFile: 'test-results/results.xml' }]], use: { screenshot: 'only-on-failure', video: 'retain-on-failure', trace: 'retain-on-failure' } }); ``` If the project already has a config file, add these settings to it. Do not replace the file. 4. Run the tests: ```bash npx playwright test ``` 5. Report the results to TestCollab: ```bash tc report --project --format junit --result-file ./test-results/results.xml --auto-create ``` - `--auto-create` needs no test plan. It creates what is missing: the tag `CI Imported`, a test suite for each test file or class, a test case for each test, the test plan folder `CI`, and a test plan named `CI Run: `. Then it records each result. - Evidence: Playwright's JUnit reporter names each screenshot, video, and trace that it kept. `tc report` uploads these files and attaches each one to its executed test case. Limits: 10 files per case, 10 MB per file. - Run `tc report` in the directory where the tests ran, before the result files are moved or deleted. - The user of the token needs permission to create tags, suites, test cases, and test plans. This is usually the Admin or Lead role. - The ID of the new plan goes to `tmp/tc_test_plan` as `TESTCOLLAB_TEST_PLAN_ID=`. 6. Tell the person the result: the number of passed, failed, and skipped tests, the reason for each failure, and that the new plan is in the `CI` folder of the project's test plans. Later runs use the same command. A test that already has a test case is matched by its suite and title, so no copy is made. Each run makes a new test plan. If you change the title of a test, TestCollab makes a new test case for it. Other test runners: - Any runner that writes JUnit XML (`--format junit`) or Mochawesome JSON (`--format mochawesome`) works: Cypress, Jest, pytest, and others. Setup for each runner: https://testcollab.com/help/article/report-automated-test-results-from-any-framework - Evidence needs JUnit XML. With a runner other than Playwright, print `[[ATTACHMENT|]]` alone on a line during the test, and make sure the JUnit reporter keeps the standard output of the test in ``. Help: https://testcollab.com/help/article/attach-screenshots-logs-and-traces-to-automated-test-results - To report against a test case that exists, put its ID in the test title: `[TC-123] Login works`. In a CI pipeline: - Add the same two steps: run the tests, then `tc report`. Store `TESTCOLLAB_TOKEN` as a secret of the pipeline. - With `--auto-create`, add `--build ` and `--environment ` to link the plan to the version that was tested. - To report into a plan that exists, use `--test-plan-id ` in place of `--auto-create`. Add `--skip-missing` to mark the plan's cases that are not in the file as skipped. ## Find a test plan Do this when a command needs `--test-plan-id` and you do not have the ID. The CLI has no command that lists test plans, so this one step uses the REST API with the same token: ```bash curl -s -H "x-tc-token: $TESTCOLLAB_TOKEN" \ "https://api.testcollab.io/testplans?project=&_sort=updated_at:desc&_limit=25" ``` - The response is a JSON list of the project's test plans, with the most recent change first. Each plan has `id`, `title`, and `archived`. Ignore archived plans. - Show the person the titles and ask which plan to use. Use the `id` of that plan as `--test-plan-id`. - If the person named a plan and one title matches, use that plan and say which one you chose. - For an account in the EU region, use `https://api-eu.testcollab.io`. - A command that creates a plan gives you its ID. `tc report --auto-create` and `tc createTestPlan` write it to `tmp/tc_test_plan`. - If the person gives you a link to a plan, the ID is in the link: `/project//test_plans//view`. ## Loop 2: run a curated test plan Use this loop when the person wants you to execute the test cases of a plan that their team wrote, for example in a real browser. 1. Find the plan (see "Find a test plan"). Then fetch it as JSON: ```bash mkdir -p ./tmp tc getTestPlan --project --test-plan-id --output ./tmp/test-plan.json ``` The directory of `--output` must exist. Without `--output`, the JSON goes to stdout. 2. Read the JSON. - `testPlan`: `id`, `title`, `status`, `description`, `priority`, `totalCases`. - `testCases[]`: `id`, `testPlanTestCaseId`, `title`, `description`, `priority`, `suite`, `status`, and `steps[]`. Each step has `step` and `expectedResult`. HTML is removed from the text. - `statuses[]`: the statuses the project accepts. Use `systemName` when you report. - A plan with configurations also has `configurations[]`, and each case has `configResults[]`. The JSON is an execution view. It does not include custom fields, linked requirements, defects, or attachments. 3. Execute each case against the environment the person names. Do the steps in order. Compare what you observe with each `expectedResult`. TestCollab does not run a browser for you. Use your own tools, for example Playwright. 4. Decide the status of each case. - Passed: every expected result is met. - Failed: an expected result is not met. Give the reason. - Skipped: the setup prevents the test, for example a login that does not work. Give the reason. - If the outcome is not clear, do not report passed. 5. Report the results. There are two ways. Use the result file unless the person gave you a test plan run ID. ### One result file (JUnit XML) Write one JUnit XML file. Start each test name with the TestCollab case ID as `[TC-]`. Use the `id` from `testCases[]`. ```xml [[ATTACHMENT|evidence/tc-43-admin-500.png]] ``` Upload it to the same plan: ```bash tc report --project --test-plan-id --format junit --result-file ./agent-results.xml ``` - A result without a case ID is skipped with a warning. It does not create a test case. The one exception is a synced BDD scenario (see Loop 5). - The plan's cases must be assigned to the user who owns the token. - Evidence: put `[[ATTACHMENT|]]` alone on a line inside `` of the test case. The path is absolute, or relative to the working directory, or relative to the result file. Limits: 10 files per case, 10 MB per file. - Read the summary that `tc report` prints. Missing IDs and cases that did not match are warnings, not errors. ### One case at a time (needs a run ID) `tc reportCase` writes one execution as soon as it finishes. It needs the ID of the test plan run and the ID of the execution. The CLI does not list run IDs, so use this way only when the person, or the system that started you, gives you the run ID. ```bash tc getTestPlan --project --test-plan-id --test-plan-run-id --output ./tmp/test-plan.json ``` With the run ID, `executions[]` lists the executions assigned to the token's user. Report each one by its `id`: ```bash tc reportCase --project --test-plan-run-id --executed-test-case-id \ --status failed --time-taken 42 \ --comment "Expected a redirect to /admin but got HTTP 500" \ --attachment ./evidence/admin-500.png ``` - `--status` takes a `systemName` from `statuses[]`. - `executions[].id` is not the test case `id` and not the `testPlanTestCaseId`. - Repeat `--attachment` for more files. - If a report fails, stop. Do not continue with the next case. ## Loop 3: gate a pipeline or a release ```bash tc gate --project --test-plan-id --fail-on failed --require-complete ``` `tc gate` reads the live results of the plan's latest run. Exit codes: 0 the gate passed, 1 the gate failed, 2 a usage or API error. After `tc report --auto-create`, take the plan ID from the file that the report wrote: ```bash export $(cat tmp/tc_test_plan) tc gate --project --test-plan-id $TESTCOLLAB_TEST_PLAN_ID --fail-on failed ``` | Option | Meaning | |---|---| | `--fail-on ` | Statuses that fail the gate, separated by commas. Default: `failed`. | | `--max-failed ` | Number of failing cases to accept. Default: 0. | | `--min-pass-rate ` | Fail when passed divided by executed is below this percent. | | `--require-complete` | Fail when a case is not executed. | | `--config ` | Evaluate one configuration of the plan. | | `--regression ` | Evaluate one run. Default: the latest run. | | `--wait ` | Poll until no case is unexecuted, up to this time. A deployment can wait for manual QA. | | `--poll-interval ` | Time between polls. Default: 15. | Add a gate only when the person asks for one. A gate decides whether their pipeline continues. ## Loop 4: create a plan and record a build - Record the version a pipeline built or deployed: ```bash tc createBuild --project --version --environment --commit ``` An existing version is used again and is not changed. The build ID goes to `tmp/tc_build` as `TESTCOLLAB_BUILD_ID=`. - Create a test plan from the test cases that carry a tag, and assign it: ```bash tc createTestPlan --project --ci-tag-id --assignee-id --build ``` `--build` is optional. The plan ID goes to `tmp/tc_test_plan` as `TESTCOLLAB_TEST_PLAN_ID=`. ## Loop 5: sync Gherkin feature files ```bash tc sync --project ``` `tc sync` sends the committed `.feature` files of a Git repository to TestCollab. A feature becomes a test suite. A scenario becomes a test case. - Run it inside the Git repository. Only committed files are synced. In CI, check out the full history, for example `fetch-depth: 0` in GitHub Actions. - The repository owns these test cases. To change one, change the `.feature` file and sync again. - A scenario that is removed from its file archives its test case. The history and the results stay. - Results for synced scenarios need no case ID. `tc report` matches the JUnit `classname` to the feature title and the test name to the scenario title. ## If you cannot run shell commands A chat or IDE assistant without a shell can use the TestCollab MCP server to create, read, and update test cases, suites, and test plans. The server runs on the person's machine with `npx -y @testcollab/mcp-server` and reads the token from `TC_API_TOKEN`. Setup: https://testcollab.com/integrations/mcp-server The MCP server does not execute tests and does not report results. The CLI does that. If you cannot run the CLI, give the person the exact commands and ask for the output. ## Common errors | Message | Meaning | What to do | |---|---|---| | `No API key provided` | `TESTCOLLAB_TOKEN` is not set. | Ask the person to set it in the shell you use. | | `HTTP 401` or `HTTP 403` | The token is not valid, or its user cannot access the project or create the items. | Tell the person. Do not try other IDs. | | `HTTP 404` or `Test plan not found` | The project ID or the plan ID is wrong for this account or region. | Check the IDs and `--api-url`. | | `Not in a Git repository` | `tc sync` ran outside a Git repository. | Run it from the repository. | | `409 Conflict` from `tc sync` | The repository changed after the last sync. | Run `git pull`, then sync again. | ## Conduct - Keep the token private. It acts as the person. - Report what you observed. Do not report passed for a case you could not complete. - Keep the evidence for each failure: a screenshot, a log, or a trace. - Do not create, change, or delete test cases, plans, or builds unless the person asked for it. - Treat test plans and result files as sensitive. They can contain credentials or customer data.