product: assert audience: test-developer, operator, ai-assistant authority: normative

MCP Server — Statistics Tools

The statistics tools let an AI assistant (or a human operator via the AI chat interface) answer yield, quality, and traceability questions about test data without writing SQL or opening a dashboard.

Architecture overview

AI assistant
    │  get_yield / get_measurement_statistics / …
    ▼
WorkflowEngine.McpServer  ──HTTP──▶  WorkflowEngine.Api
                                         │  GET /api/statistics/*
                                         ▼
                                    stat_cache table  (PostgreSQL)
                                         ▲
                               StatisticsBackgroundService
                               (WorkflowEngine.Orchestra, 60 s poll)
                                         │  reads
                                    TestExecutions / StepExecutions /
                                    MeasurementValues tables

Hot path: The API controller checks the stat_cache table first (valid for 60 minutes). If the cache is warm it returns in milliseconds. If it is stale or absent it falls back to a live query over the execution tables.

Cache writer: StatisticsBackgroundService runs in Orchestra, polls every 60 seconds, and writes pre-computed yield and Cp/Cpk rows for every test definition whose executions changed since the last run. It only processes executions that completed more than 5 minutes ago to avoid racing with in-flight writes.

Tools reference

See tools-reference.md for the full parameter table. Quick summary:

Yield (get_yield)

Returns pass rate for a test or a single step within a test.

get_yield(testName="FinalTest")
get_yield(testName="FinalTest", stepName="RF Calibration", from="2025-01-01T00:00:00Z")
get_yield(testName="FinalTest", stationId="station-A")

Response fields:

Measurement name discovery (list_measurement_names)

Before querying statistics you need the exact measurementName as it is registered in the test definition. Use list_measurement_names to discover what names exist.

Results are deduplicated by name — each distinct name appears once.

list_measurement_names()                           # all names (up to 50)
list_measurement_names(search="VOUT")              # all names containing "VOUT"
list_measurement_names(search="resistance", limit=100)

Response fields per entry:

The sampleCount, cp, and cpk fields come from the stat_cache and allow callers to identify poor-capability measurements without making N follow-up calls. They are null for measurements with no executions yet. For measurements with definitionVersions > 1 these values reflect the most-recently-computed cache entry rather than pooled statistics — always follow up with get_measurement_statistics before acting on Cp/Cpk in that case.

To find all measurements below a Cpk threshold in one call, use get_capability_summary instead.

What definitionVersions tells you:

Every time the low_limit, high_limit, or unit fields are changed in the test YAML and the package is redeployed, a new definition row is inserted with the same name but a different content hash. definitionVersions counts those rows.

• definitionVersions = 1 — the specification has never changed. Statistics are straightforward: all samples were recorded against the same limits, and Cp/Cpk is directly comparable across the entire date range.

• definitionVersions > 1 — the specification was changed at least once. This has consequences for how statistics should be interpreted. Read the Caveat: limit changes section before proceeding.

Capability summary (get_capability_summary)

Returns all measurement points whose Cpk is below a given threshold, sorted worst-first. Served entirely from the pre-computed stat_cache — one call replaces the N+1 pattern of calling list_measurement_names followed by get_measurement_statistics for every entry when scanning a large population for quality problems.

get_capability_summary()                                       # all with Cpk < 1.33
get_capability_summary(maxCpk=1.00)                           # only actively defective
get_capability_summary(maxCpk=1.33, testName="FinalTest")     # scoped to one test
get_capability_summary(maxCpk=1.67, stationId="station-A")    # scoped to one station

When to use vs get_measurement_statistics:

Response fields per item:

Important: When a measurement name appears in multiple tests, only the row with the lowest Cpk (worst case) is returned. The testName field identifies which test produced it.

definitionVersions caveat: The stat_cache stores per-definition statistics rather than pooled. For definitionVersions > 1, the Cp/Cpk here is for one definition version only. Use get_measurement_statistics (which pools all versions) for the authoritative figure before concluding the process is or is not capable.

Measurement statistics (get_measurement_statistics)

Returns descriptive statistics and process capability indices for a named numeric measurement point. Non-numeric measurement points (boolean, string) are ignored.

Samples are always pooled across all specification versions of the name. This means sampleCount always reflects the complete historical population — there is no silent truncation when limits changed.

Typical workflow:

1. Call list_measurement_names(search="...") to find the exact name and check definitionVersions.
2. If definitionVersions > 1, read the caveat below before interpreting results.
3. Call get_measurement_statistics(measurementName="...") to get statistics.
4. If limitsNote is non-null, show it to the user verbatim before presenting Cp/Cpk.
5. If poolingNote is non-null, show it too, and prefer re-querying with stepName.
6. Optionally narrow the sample set with testName, stationId, stepName, from, to.

get_measurement_statistics(measurementName="OutputVoltage_mV")
get_measurement_statistics(measurementName="OutputVoltage_mV", testName="PowerSupplyTest", from="2025-06-01T00:00:00Z")
get_measurement_statistics(measurementName="VOUT_5V", stationId="station-A")
get_measurement_statistics(measurementName="INL_MAX_MA", stepName="CH3 DAC Linearity")

Two other ways the population is wider than it looks

The specification-version caveat below is not the only way this response pools things that are not alike. Both of the others are disclosed in the response rather than left for you to notice.

One name, several steps. A package may record the same measurement name inside several steps — eight channels of one board, each with its own CH1 DAC Linearity .. CH8 DAC Linearity step. Every sample from all of them is pooled here. stepNames lists the contributors and poolingNote is set whenever there is more than one. A Cpk computed across eight physically distinct channels describes none of them: it is perfectly normal for it to read as an incapable process while every individual channel is healthy and near-identical. Re-query with stepName to get the figure that means something, and check get_yield per step as a sanity test.

Sentinel values. A measurement can declare an invalid_value in YAML — the number the package uses to mean "no result", typically a -999 seeded into a variable so an unrun step fails rather than recording a midpoint. Samples equal to it are excluded from mean, sigma and Cp/Cpk, counted in excludedInvalidSamples, and explained in dataQualityNote. They remain in the report and still count against yield, because the failure is real; they are just not measurements of the quantity.

Steps that never ran. A sample recorded by a step whose verdict is not PASS or FAIL — aborted, skipped, not started, undetermined — is excluded and counted in excludedNonMeasurementSamples. A step that did not run did not measure anything, whatever value was left in the variable it read. This needs nothing declared in the package, so unlike invalid_value it applies to data already in the database. A FAILING step keeps its sample: an out-of-limit reading is a real measurement, and usually the most important one in the set.

When nothing was declared

The two exclusions above are by provenance. Neither can catch a sentinel written by a step that ran, compared the stale variable against its limits and failed honestly — which is the common case.

For that, dataQualityNote also carries a WARNING, with nothing excluded: when one exact value repeats AND sits grossly outside the specification, the response names it and reports how far out it is, measured in specification widths.

Both signals are required, because each alone is wrong in a way that matters:

Measured in specification widths the two populations are far apart, which is why the combination works: a real out-of-limit reading sits around 9 widths outside its limits, an actual sentinel between 24 and 10,000.

It never filters, on either signal. The note tells you to look. The decision is yours, and the fix is to declare invalid_value on the measurement if that is what it turns out to be.

Two limits on the warning, worth knowing before you rely on it:

• It needs both specification limits present and a non-zero window. A measurement whose current definition carries no limits has no Cp/Cpk to poison, but its mean and sigma can still be dragged by a sentinel and nothing here will say so.
• It looks at the population you queried. Narrowing with stepName concentrates a sentinel and makes the warning louder, so a pooled query that says nothing is worth re-running per step.

Analysing data that is already contaminated (plausibleMin / plausibleMax)

invalid_value is forward-only. It helps samples recorded after somebody edits the YAML and re-runs, and it repairs nothing already in the database. For a population that is already contaminated, supply plausibility bounds:

get_measurement_statistics(
    measurementName = "PHOENIX_CH0_IMEAS_A",
    testName        = "Sparrow Fixture Electronics Sub-Test",
    plausibleMin    = 0,
    plausibleMax    = 10,
    reason          = "-999 A is not a physically possible current on a 1 A rail")

Samples outside the bounds are excluded from mean, sigma, Cp and Cpk for that query only. Nothing is mutated and nothing is written; this is a view over stored data.

Ask whether a value is physically possible, not whether it is convenient. That is the whole discipline of the feature. Minus 999 amperes on a one amp rail is impossible and is not a measurement. Minus one hundred percent gain error is entirely possible and is usually the most important sample in the set. Bounds chosen from the physics of the unit and the specification are defensible; bounds narrowed until the number looks acceptable are not, and the response is built to make the difference visible.

Four guarantees:

A capability index computed under caller-chosen bounds is a different claim from one computed over everything recorded. Both are legitimate; confusing them is not.

Finding which measurements need bounds (get_capability_summary)

The capability scan does NOT take plausibility bounds, on purpose. A bound is per-measurement physics, and one pair of numbers cannot mean anything across a scan that mixes amps, hertz and percent: minus 999 is impossible for a one amp rail and unremarkable for a temperature offset.

Instead each row can carry suspectedSentinelNote, set when that row's Cpk looks bad for a reason that is not the process. Use the scan to find where to point bounds, then point them with get_measurement_statistics.

Cp and Cpk explained

Process capability indices answer the question: "Given how much this measurement naturally varies, how likely is it to stay within specification?"

Cp measures how wide the specification window is relative to the natural process spread:

Cp = (USL − LSL) / (6σ)

A Cp of 1.0 means the specification window is exactly 6σ wide — in a perfectly centred process, 99.73 % of parts would pass. A Cp of 1.33 gives a safety margin of ±4σ on each side (standard production target).

Cpk additionally penalises a process that is off-centre — it tells you the actual margin to the nearest limit:

Cpk = min( (USL − μ) / (3σ),  (μ − LSL) / (3σ) )

If Cp > Cpk the process is off-centre. A process can have a high Cp (wide spec) but a low Cpk (mean drifted close to one limit), which will produce failures even though the tolerance is generous.

When Cp/Cpk are applicable:

• Both lowerLimit and upperLimit must be defined in the test YAML (measurement: block).
• At least a few dozen samples are needed for meaningful indices (σ becomes unstable below ~30 samples).
• σ must be > 0 (all-identical samples indicate a stuck sensor, not a real process).

Both values are null when any of these conditions are not met.

How to use the results:

Cp and Cpk are null when:

• The measurement definition has no lower or upper limit, or
• σ = 0 (all samples are identical — likely a fixture or software issue)

Response fields:

Yield trend (get_yield_trend)

Returns yield bucketed by day, week, or month.

get_yield_trend(testName="FinalTest", bucket="week", from="2025-01-01T00:00:00Z")

Each point in trend.points[]:

Worst steps (get_worst_steps)

Returns the steps with the lowest pass rate, ranked ascending.

get_worst_steps(testName="FinalTest", top=5)
get_worst_steps(testName="FinalTest", stationId="station-A", from="2025-06-01T00:00:00Z")

Use this tool to focus engineering attention: a step with 60 % yield is a stronger cost-reduction candidate than improving a 95 % step.

Unit history (get_unit_history)

Returns all test runs for a serial number, newest first.

get_unit_history(serialNumber="SN-1042")
get_unit_history(serialNumber="SN-1042", page=2, pageSize=10)

Use this tool to answer "why did this unit fail?" or to reconstruct the full test history for a warranty return.

Caveat: specification limit changes split the population

This section explains what happens when lowerLimit or upperLimit are changed in the test YAML and the package is redeployed.

What happens at the database level

Assert uses content-hash identity for measurement definitions. Every time a test package is loaded, the YAML content of each measurement block is hashed. If the hash already exists in MeasurementDefinitions, the existing row is reused. If the hash is new (because limits changed), a new row is inserted with the same name but a different Id and different lowerLimit/upperLimit.

Values recorded before the change point at the old definition Id. Values recorded after point at the new definition Id. Both rows have the same name, but querying by Id returns non-overlapping populations.

What this means for statistics

Before this fix was implemented, get_measurement_statistics used FirstOrDefault when resolving the definition — it returned a database-order-dependent row, silently querying only the samples attached to that one definition. A programme with limits tightened at month 3 of 6 could return half the true sample population with no warning.

The endpoint now:

1. Loads all definition rows for the name.
2. Queries values from all of them (the full population).
3. Identifies the most recently active definition (the one whose values carry the latest timestamp) and uses its limits for lowerLimit, upperLimit, Cp, and Cpk.
4. Returns definitionVersions (the count) and limitsNote (a plain-English explanation when definitionVersions > 1).

How to use the response correctly

Step 1 — check definitionVersions

{ "definitionVersions": 2, "sampleCount": 1924, ... }

If definitionVersions is 1, nothing special. If it is 2 or more, proceed to step 2.

Step 2 — read and show limitsNote verbatim

{
  "limitsNote": "Specification limits for 'VOUT_5V' changed 1 time(s) during the programme,
    creating 2 definition versions. All 1924 samples are pooled across all versions.
    The lowerLimit, upperLimit, and Cp/Cpk values reflect the most recently active
    specification only. Samples recorded under older specifications may have been
    accepted or rejected against different limits."
}

Always present this text to the user before showing Cp/Cpk. Do not paraphrase or omit it. The user needs to know that some historical samples were gated against different limits before deciding whether the pooled Cp/Cpk is meaningful for their purpose.

Step 3 — decide whether the pooled statistics answer the question

Ask: "Was the limit change intentional process knowledge (tighter spec, same process) or a correction of an error (wrong limits deployed for a period)?"

Step 4 — narrow the sample set if needed

If the user needs statistics for the current specification only, filter by date:

get_measurement_statistics(
  measurementName="VOUT_5V",
  from="<date the new spec was deployed>T00:00:00Z"
)

This restricts TestExecution.StartTime to after the change, effectively isolating the current-spec population.

How to present data from a multi-version measurement to users

When reporting on a measurement where definitionVersions > 1, a well-formed response looks like this:

VOUT_5V — 1924 samples pooled across 2 specification versions.

Mean: 4.972 V | StdDev: 0.031 V | Min: 4.891 V | Max: 5.048 V

Current specification: 4.75–5.25 V → Cp: 1.07, Cpk: 0.96 (marginally capable)

⚠️ Note: Specification limits changed once during the programme. These statistics cover the full population (both before and after the change). Cp/Cpk reflect the current specification only. If you need statistics for the current spec period alone, filter using the from parameter with the date the new spec was deployed.

If the user asks a direct question like "Is VOUT_5V capable?", the correct answer is:

"Based on the pooled population of 1924 samples, Cpk is 0.96 against the current spec — marginally not capable. However, the limits were changed once during the programme, so some of these samples were gated against a different specification. To get a precise answer for the current spec, provide the date from which the current limits have been in use."

Never answer "Yes, it is capable" or "No, it is not capable" without disclosing definitionVersions > 1 when it applies.

Example AI prompts

The following prompts work well with Claude, GitHub Copilot, or any MCP-compatible client connected to a running Assert MCP server.

Yield analysis

What was the yield for FinalTest last month?

Which step in FinalTest failed most often in the last 30 days?

Show me the yield trend for FinalTest week by week since January.

Measurement name discovery

What measurement points are available on this station?

List all voltage-related measurements I can analyse.

Measurement quality

Is OutputVoltage_mV in control for the PowerSupplyTest? Give me Cp and Cpk.

What were the min, max, and mean for ContactResistance_Ohm last week?

Which measurements have a Cpk below 1.33? Start by listing all measurement names.

Show me all measurements that are not process capable (use get_capability_summary).

Which measurements are actively producing defects (Cpk < 1.00)?

Unit traceability

What tests has serial number SN-1042 completed? Did it ever fail?

Background service

StatisticsBackgroundService (in WorkflowEngine.Orchestra) continuously pre-computes statistics so that API responses are fast even for large datasets.

The service writes the following row types per test definition:

Troubleshooting

get_yield returns totalRuns: 0

• Check that testName matches the name field in the test YAML exactly (case-sensitive).
• The stat_cache may not yet have data — wait up to 65 seconds after the first execution completes.

sampleCount is lower than expected

• Check definitionVersions. If it is > 1, the pooling is working and the count is correct — all versions are included. Re-read limitsNote and consider whether a from date filter is appropriate.
• If definitionVersions is 1 and the count still seems low, check the from/to date filter and verify the testName / stationId filters are not over-restricting.

cp and cpk are null

• The current measurement definition has no lowerLimit / upperLimit in the YAML.
• Add low_limit and high_limit to the measurement definition and re-run a test.
• If definitionVersions > 1: prior versions may have had limits but the current one does not. Check the YAML and confirm limits are present in the deployed version.

Statistics are stale

• Check that WorkflowEngine.Orchestra is running and healthy.
• Inspect Orchestra logs for StatisticsBackgroundService error messages.
• Measurements from executions completed less than 5 minutes ago will not appear yet.

passCount seems wrong when limits changed

• passCount and failCount reflect the verdict recorded at the time of each measurement — i.e., evaluated against the limits that were active when the test ran.
• They are not re-evaluated when limits change. This is by design: the verdict is a historical fact, not a projection.
• If you need to re-evaluate historical samples against the current spec, that requires a custom query against the raw MeasurementValues table.

Log search (search_execution_logs)

Finds log text from past executions using PostgreSQL full-text search. For full details see live-events.md.

search_execution_logs(query="timeout error")
search_execution_logs(query="calibration failure", testExecutionId="<uuid>")
search_execution_logs(query="drift", stepName="RF Calibration")

Indexing latency: Logs are indexed by EmbeddingBackgroundService in Orchestra 5–65 seconds after execution completion. If totalHits is 0 for a very recent execution, wait and retry.