Chapter 15: Automation & CI/CD

The open-source **`mcp-tester`** is designed to work not only for manual debugging, but as an integral part of your software lifecycle. In this chapter we learn how to automate tests.

Exit Codes: The Language of Pipelines

A test tool in a CI/CD environment (such as GitHub Actions or GitLab CI) must communicate success or failure. mcp-tester uses standard exit codes for this:

Exit code 0: Success. All steps in the script ran without errors.
Exit code 1: Failure. A tool call failed or an assertion was violated.

Example shell script:

./bin/mcp-tester test --profile staging --script tests/smoke_test.mcp
if [ $? -eq 0 ]; then
  echo "Deployment validated!"
else
  echo "Critical error in the MCP server!"
  exit 1
fi

Test Summaries

At the end of every script run, the tester prints a compact statistic: Test Summary: 12 commands executed, 12 passed, 0 failed

This lets developers see at a glance how extensive the test coverage is.

Best Practices for Stable Pipelines

Isolated environments: Use separate profiles in mcp-tester.yml for dev, staging, and production.
Variables for dynamism: Use set_var to extract IDs from creation tools and reuse them in follow-up tests. This avoids hardcoded test data.
Guardrail tests: Create scripts that deliberately exercise the strategic prompts (Chapter 6). Does the server respond correctly to disallowed requests?
Raw mode for logs: When a pipeline fails, --raw mode in the CI logs can help you see the exact JSON response of the server.

Conclusion

By automating your MCP tests with mcp-tester, you build trust in your AI infrastructure. You ensure that changes to your server's code do not lead to unpredictable behavior in the LLM chat.

← Chapter 14: Quality Assurance | Table of Contents | Next Chapter: Transports in Detail →