product: maestro audience: test-developer authority: normative

End-to-End Development Workflow

The journey from idea to production

1. DESIGN    → Describe what needs to be tested (DUT, measurements, limits)
2. SCAFFOLD  → TAT UI (/packages → New Package) — creates directory, package.json,
               VS Code schema, validation script, and all docs
3. AUTHOR    → Write YAML test definitions in VS Code with schema autocomplete
4. VALIDATE  → Run py validate.py before every commit (or use MCP validate_yaml)
5. RUN       → Execute locally with mock runner or real hardware
6. REVIEW    → Check results in Test Results page or Swagger
7. COMMIT    → Git commit + push to package repository
8. REFRESH   → Call trigger_package_refresh so the station detects the new commit
9. PUBLISH   → Add to registry catalog.json (or use UI)

MCP-driven workflow (AI-assisted development)

When developing with an AI assistant connected via MCP, the full sequence from first commit to verified test result involves these steps in order. Three of them are not obvious and are called out explicitly below.

Step-by-step MCP workflow

⚠️ Step 7 — trigger_package_refresh is mandatory after git push

After pushing to the registry the station does not automatically detect the new commit. You must call trigger_package_refresh first, then wait a few seconds before calling install_package. Without this step install_package will download the previously cached version.

⚠️ Step 10 — list_available_tests before start_test

start_test requires the absolute path to the YAML file on the API server (e.g. /data/tat-packages/MyPackage/tests/my-test.yaml). This path is not predictable from the package name alone. Always call list_available_tests (or get_available_test_files) first and use the filePath value from the response.

⚠️ Runner warm-up — first run after cold install may time out

After calling install_package the .NET and Python runner containers have not yet loaded your new assembly. The first start_test call against a freshly installed package may fail with runner unavailable or a gRPC timeout while the runner process initialises. This is expected and not a bug.

Recommended approach:

1. After install_package, wait ~5 seconds before the first start_test.
2. If the first run returns runner unavailable, call get_system_health to confirm the runner is listed as healthy: true in version.runners[].
3. If the runner is not healthy after 15 seconds, call get_service_logs (service: dotnet-runner) to check for startup errors, then get_system_events to look for OOM kills or crash loops.
4. Once get_system_health reports healthy, retry start_test — subsequent runs will succeed without delay.

⚠️ YAML validation — expect two passes

The scaffold generates structurally valid YAML for the schema it targets. If you edit a step and validate_yaml returns errors, fix them one at a time and re-validate before committing. The most common scaffolding errors are:

• type: runner (legacy) — use runner: dotnet or runner: python instead
• steps: at top level — all step arrays must be nested inside test:

Quick commands

Note: The stack scripts below are for infrastructure operators / CI, not for test developers. As a test developer you author YAML locally and validate it with py validate.py. A running stack is only needed for level-2 (semantic) validation and for actually executing tests.

Validate YAML

# All files in the package (level 1 always; level 2 requires stack)
py validate.py

# Specific file
py validate.py tests/my-test.yaml

Validate via API

$yaml = Get-Content "tests/my-test.yaml" -Raw
$body = @{ yamlContent = $yaml } | ConvertTo-Json
Invoke-RestMethod -Uri "http://localhost:7000/api/yaml/validate" `
    -Method POST -Body $body -ContentType "application/json"

Run a test via API

$body = @{
    yamlFilePath = "/data/tat-packages/my-package/tests/my-test.yaml"
    serialNumber = "DEV-001"
} | ConvertTo-Json
Invoke-RestMethod -Uri "http://localhost:7000/api/testexecution/run" `
    -Method POST -Body $body -ContentType "application/json"

Note: Use the absolute server-side path returned by GET /api/packages/test-files, not a relative path. Call that endpoint (or the MCP list_available_tests tool) to discover the correct value.

Run a test via UI

Open http://localhost:7001/test-monitor, select the YAML file, enter the serial number, click Start Test.

View results

Open http://localhost:7001/test-results and search by serial number.

Capturing developer feedback

The Maestro team continuously improves the platform based on developer experience. If you encounter missing documentation, unexpected behaviour, or workflow friction, add it to a feedback.md file in your package repository. Your AI assistant can help capture these as you work — it can recognise friction points (failed validations, undocumented steps, incorrect scaffolding) and record them automatically without interrupting the developer.

Send feedback.md to the Maestro team when you have useful observations.