Instrument comparison

Run this study#

What you are checking: whether each analyzer agrees closely enough with your chosen reference analyzer on the same specimens.

Suggested starting plan: 20 specimens, each measured once on every analyzer. With three analyzers that is 60 results in 60 rows. Choose the final count from the precision you need for the mean difference, using earlier paired differences or a pilot. For example, with a paired-difference SD of about 2 units, 20 specimens give a 95% interval of about ±0.94 units around the mean, by the NIST mean-interval formula. Plan more for wider scatter or separate concentration groups. Repeats on one specimen still count as one specimen. The calculation minimum is reference results on 2 specimens, and 5 specimens to decide specimen agreement.

1. Decide first. Choose the reference analyzer, the other analyzers and the allowable difference for each specimen: in the study unit, as a percentage of the reference result, or both. Use a CLIA / CAP limit adopts a CLIA or CAP limit if it fits your analyte, fluid, method and unit. Plan concentration groups across your intended range, with enough specimens in each to show whether differences change with concentration. Create the study and enter these on Set up (see Study setup).
2. Measure every specimen on every analyzer. Check QC and calibration first. Use the same specimen IDs and handling, within each specimen's stability time. Keep repeats under the same ID.
3. Enter one result per row: specimen ID, analyzer name, result and concentration group. Spell analyzer names exactly as on Set up. On Data, choose Import a file and choose your file (start from Blank template (CSV) or comparability.csv). Answer any question the dialog asks, then choose Import into this study.
4. Calculate and review each analyzer. Choose Calculate results. Results opens. Check each analyzer's count within limits under Why, then its mean difference, interval and per-level bias under Differences vs the reference analyzer. On Report, choose Download PDF. Its Specimen decisions name every specimen that failed or couldn't be evaluated; check zero reference results under combined limits.

An analyzer meets the limit when at least 95% of specimens, and at least 5, are within their limits. Mean difference and limits of agreement are supporting detail.

Purpose#

An instrument comparison shows how far each instrument reads from your reference analyzer on the same specimens, and whether the difference is acceptable. It suits a main analyzer and its backups, or analyzers on different sites. Enter one row per result, with an instrument column.

Every difference is instrument minus reference analyzer.

When to use it#

For regression (slope, intercept, decision-level bias) with two results side by side, use Method comparison. For a new reagent lot, use Reagent lot comparison. For categorical results, use Qualitative agreement.

Study setup#

You need specimens measured on every instrument including the reference, one analyte in one exact unit, the same instrument names on every row, and one concentration level per specimen, for example 1, 2 or 3.

The experimental unit is the specimen. Replicates on one instrument are averaged. An instrument's mean difference uses the specimens with an included result on both it and the reference. For the specimen rule, a specimen missing either result counts as not evaluated.

1. Choose New study. Enter Analyte and Unit, select Instrument comparison under Comparison and bias, and fill in Reference analyzer, spelled exactly as in your file. Choose Create study, or Create and import data if you already have results.
2. On Set up, under Instruments and reference analyzer, check the Instrument panel (comma-separated). It lists every instrument with results, up to 30. Set the Instrument setting to Same model · same method or Different methods · method-pair agreement.
3. Under Acceptance limits, choose Enter your own limit. Under What to limit, pick Specimen difference, then enter the limit in the study unit, as a percentage, or both (see both parts). Set Applies to to All analyzers, or pick one analyzer. More statistics adds an Instrument bias vs reference analyzer limit on the mean difference. See Acceptance limits.

Entering data#

On Data, choose Import a file, or Add row and type or paste into the grid. The import fills the instrument panel from your data. Blank template (CSV) downloads the columns this study needs.

See Importing a file and Missing and censored values.

These problems stop the calculation or the save:

Statistics#

• Differences: d = instrument cell mean − reference cell mean, for specimens with both.
• Specimen agreement: the allowance at reference result X is A, or P × X / 100, or with both limits the larger, max(A, P × X / 100) (Pass when within either (the larger allowance)) or the smaller, min(A, P × X / 100) (Pass only when within both), as chosen under With both limits. The specimen is within its limit when |d| ≤ allowance, inclusive. The comparison uses exact decimal arithmetic on unrounded values.
• Summary: mean difference with a Student t CI on n − 1 df, and the SD of differences. 95% limits of agreement = mean ± 1.96 SD, each with a CI using SE = √(3s²/n) and t(n − 1) (Giavarina 2015).
• Per-level bias: mean and SD of the differences within each level.

Sources: NIST/SEMATECH e-Handbook. See Methods and sources.

Acceptance limits for this study#

The specimen rule, Individual specimen agreement, decides agreement. Each specimen is checked against your limit at its reference result. With 5–19 specimens, all must be within their limits. With 20 specimens 19 must be, and with 25 specimens 24. A percentage limit needs a reference result above zero on a ratio scale. A specimen that can't be evaluated leaves the limit Undecided unless enough specimens already fail. Each instrument other than the reference gets an outcome. The reference analyzer shows Not applicable.

Instrument bias vs reference analyzer (under More statistics) adds a check on the mean difference, in the study unit. Without a specimen rule, a built-in Specimen agreement check stays Undecided for each analyzer and the study can't pass. See how outcomes are decided.

Worked example#

Download comparability.csv: six synthetic glucose specimens, S01 to S06, in three levels, each measured once on Analyzers A, B and C. That is 18 rows.

specimen_id,instrument,result,level,replicate
S01,Analyzer A,98.4,1,1
S01,Analyzer B,101.2,1,1
S01,Analyzer C,99.1,1,1

Per-level bias for Analyzer B, 2 specimens per level:

C agrees with A. B reads higher, and its bias grows with concentration. B's mean difference of +3.77 mg/dL is within ±4 mg/dL, but three specimens are outside the limit: S03 +4.50, S05 +5.50 and S06 +4.70. B fails, so the study status is Criteria not met. If the bias looks proportional, run a Method comparison of B against A.

A missing result. Excluding S06 on Analyzer C (with a reason) leaves C with 5 compared specimens and a mean difference of +0.60 mg/dL. S06 counts as not evaluated, so C's limit shows 5/6 (83.333%) and Undecided. The study status is still Criteria not met because B fails.

Reading the results#

The top of the page gives the verdict. Why, under it, gives each analyzer's count within limits, for example 3/6 (50%), and its outcome (Met, Not met or Undecided). With Different methods · method-pair agreement, a note says the differences include method bias as well as instrument bias.

Instrument summaries: mean and largest difference lists each analyzer's comparable specimens, mean difference, largest absolute difference and Result. The line under the table names any missing results.

Differences vs the reference analyzer shows one instrument at a time. Pick it with Instrument.

• 95% CI for the mean difference: how well the average bias is known.
• 95% limits of agreement: where about 95% of this instrument's specimen differences fall.
• Per-level bias: whether bias changes with concentration.
• Specimen differences: each specimen's mean, difference and replicate count, shown when there is no specimen rule.

The report's Specimen decisions lists each specimen's reference and analyzer result, difference, allowed difference, error index and outcome. Its plot shows each specimen's difference, with each analyzer's mean bias and CI. A single absolute specimen limit is shaded. Percentage and greater-of limits are in the specimen tables.

If an instrument fails or shows warning signs (specimens outside their limit or not evaluated, a mean difference near your limit, wide limits of agreement, one large specimen difference, bias that climbs with concentration, many missing results), check for inconsistent instrument names, the wrong reference and typing errors. If the data are right, remeasure, run QC on both instruments and recalculate.

To compare two instruments other than the reference, create a study with one of them as the reference.

Common mistakes#