How to Interpret the Operating Characteristics of a Simon Two-Stage Design

What Are Operating Characteristics?

A Simon two-stage design is often described using a few key numbers: the stage 1 sample size, the stage 1 stopping boundary, the total sample size, and the final success boundary. But these numbers alone do not tell the whole story.

Operating characteristics are performance measures of a statistical design evaluated under different assumptions about the true response probability, usually denoted by p. They answer questions such as how often the trial will stop early, how many patients will be treated on average, how likely the trial is to proceed to stage 2, and how the probability of success changes as the true response rate increases.

For a standard Simon two-stage design with futility stopping after stage 1, the important operating characteristics are the probability of early termination (PET), the probability of reaching stage 2, the expected sample size (ESS), the probability of declaring the treatment promising, and the power function.

Rather than looking at these measures only at p₀ and p₁, it is often useful to examine them across a range of possible response probabilities. This gives a much more complete picture of how the design behaves.

Probability of Early Termination

The probability of early termination (PET) is the probability that the trial stops after stage 1.

Suppose a design has a stage 1 sample size of n₁ and a stage 1 futility boundary of r₁. The trial stops for futility when the number of responses in stage 1 is at most r₁. Therefore:

Suppose n₁ = 15 and r₁ = 2. If the true response probability is p = 0.20, then PET(0.20) = P(X₁ ≤ 2), which using the binomial distribution is approximately 0.398.

So there is approximately a 39.8% probability of stopping after stage 1 when the true response probability is 20%. In other words, under this particular value of p, roughly 4 out of every 10 trials would be expected to stop after the first 15 patients.

PET Is Not the Probability of Final Failure

An important distinction is that PET and the probability of ultimately failing to declare the treatment promising are not the same thing. PET describes only whether the trial stops at the interim stage. A trial that does not stop at stage 1 can still fail at the final analysis.

Therefore 1 − PET(p) is the probability of reaching stage 2, not the probability of final success.

For example, if PET(0.20) = 0.398, then the probability of reaching stage 2 at p = 0.20 is 1 − 0.398 = 0.602. Approximately 60.2% of trials would therefore proceed to stage 2.

The final probability of success is a different quantity because it also depends on the final decision boundary.

Expected Sample Size

Another important operating characteristic is the expected sample size (ESS). The maximum sample size of a Simon design is N = n₁ + n₂, but the trial does not always enroll all N patients because it may stop after stage 1. The expected sample size is therefore smaller than the maximum whenever early stopping is possible.

For a two-stage design:

The interpretation is straightforward: every trial enrols n₁ patients, and only trials that pass the stage-1 futility boundary enrol the additional n₂ patients.

Continuing the example, suppose n₁ = 15 and n₂ = 15, so that N = 30. At p = 0.20, PET(0.20) is approximately 0.398, so ESS(0.20) = 15 + (1 − 0.398)(15), which is approximately 24.0.

So although the maximum sample size is 30 patients, the expected sample size is approximately 24 patients when the true response probability is 20%.

PET, Continuation, and ESS Should Be Interpreted Together

PET and ESS are closely related, because the probability of reaching stage 2 is exactly 1 − PET(p) and ESS is built directly from it. A high PET generally means a low ESS, because more trials terminate after stage 1. A low PET generally means a higher ESS, because more trials proceed to stage 2.

The Power Function

The power function gives the probability that the design ultimately declares the treatment promising, for each possible true response probability.

A single power value, such as "80% power," describes the design at one particular response probability, usually p₁. The power function provides much more information because it shows how the probability of success changes as the true response probability changes.

For a typical Simon design the power function is increasing: low when p is small, rising as p increases, and high when p is sufficiently large. When plotted against p, it is often called the power curve.

Why p₀ and p₁ Are Not the Whole Story

Simon designs are constructed using two important response probabilities: p₀, the response probability considered uninteresting, and p₁, the response probability considered sufficiently promising. For example, a design might use p₀ = 0.20 and p₁ = 0.40, and be constructed to have type I error at most 0.05 at p = 0.20 and power at least 0.80 at p = 0.40.

But what happens at p = 0.25, 0.30, or 0.35? The operating characteristics tell us.

This is important because the true response probability in a real trial is rarely known to be exactly p₀ or p₁.

PET Is Not the Type I Error

The relationship between PET and type I error deserves particular attention. At p₀, a Simon design controls the probability of incorrectly declaring the treatment promising. For a standard two-stage design with futility stopping, the type I error is a joint probability involving both stages:

It is therefore not equal to 1 − PET(p₀). Instead, 1 − PET(p₀) is the probability of reaching stage 2 under p₀, and only some of the trials that reach stage 2 will ultimately meet the final success criterion.

Two inequalities are worth remembering: 1 − PET(p₀) is not α, and 1 − PET(p₀) is not Power(p₀).

A Complete Numerical Simon-Design Example

Consider the following illustrative design with p₀ = 0.20 and p₁ = 0.40, using n₁ = 18, r₁ = 4, n₂ = 15, and r = 10, so that the maximum sample size is N = 33.

The stage-1 rule is that four or fewer responses in the first 18 patients stops the trial for futility. If the trial continues, a final success is declared when the total number of responses exceeds 10.

For this illustrative design the type I error is approximately 0.0458 at p = p₀ = 0.20, and the power is approximately 0.8011 at p = p₁ = 0.40. The design therefore approximately satisfies a 5% type I error requirement and an 80% power requirement. These quantities, together with PET and ESS, are computed directly from the relevant binomial probabilities.

What Happens Between p₀ and p₁?

The region between p₀ and p₁ is often particularly informative. For the illustrative design above, consider three true response probabilities.

Several patterns are visible. At p = 0.20 the trial frequently stops early. At p = 0.30 the behaviour changes substantially: PET has fallen to about 33%, about two-thirds of trials reach stage 2, ESS has increased to about 28 patients, and the probability of success has risen to about 37%. At p = 0.40 the trial usually proceeds to stage 2 and has approximately 80% power.

This illustrates why the region between p₀ and p₁ matters. The design does not suddenly change from "low power" at p₀ to "high power" at p₁. Its behaviour changes continuously across the response-probability range.

The power function is also typically S-shaped rather than linear: its rate of increase changes across the range of p. This is why values between p₀ and p₁ should be calculated from the binomial model rather than obtained by linear interpolation.

Operating Characteristics Are Interdependent

PET, expected sample size, probability of reaching stage 2, and power are not independent properties. They are all derived from the same underlying probability model, which is why the probability of reaching stage 2 is the complement of PET and ESS follows directly from it.

Power is also affected by the stage-1 stopping rule, because a trial that stops for futility cannot subsequently achieve final success.

The stage-1 boundary therefore creates a trade-off. A more aggressive futility boundary can cause more trials to stop early, reducing ESS, but the boundary cannot be chosen independently of the type I error and power requirements. Conversely, a less aggressive stopping rule allows more trials to proceed to stage 2, increasing ESS and potentially changing the probability of final success.

The important point is that operating characteristics should be interpreted as a system, rather than as isolated numbers.

Operating Characteristics Depend on the Chosen Simon Design

Different Simon designs can satisfy the same type I error and power requirements while having different operating characteristics.

For example, an optimal design minimizes the expected sample size under the null response probability, whereas a minimax design minimizes the maximum sample size. Consequently the designs may differ in their stage-1 sample sizes, stopping boundaries, maximum sample sizes, and their PET, ESS, and power functions.

This is why selecting a Simon design is not simply a matter of finding a design that satisfies the type I error and power requirements. The way the design behaves across plausible response probabilities should also be considered.

ESS Is Not a Fixed Property of the Design

A common source of confusion is treating ESS as if it were one fixed number. It is not. ESS is a function of p, and it changes with the true response probability. The same applies to PET.

Therefore a statement such as "this design has an expected sample size of 24 patients" is incomplete unless the response probability used to calculate that value is specified.

How to Read an Operating-Characteristics Plot

A useful way to visualize a Simon design is to plot its operating characteristics against the true response probability. A particularly informative figure has three panels: PET versus p, ESS versus p, and power versus p.

The same response-probability scale should be used in all three panels, with p₀ and p₁ marked as reference points. When interpreting the figure, focus on how the three curves describe the trial’s behaviour as the true response probability changes.

What Operating Characteristics Cannot Tell You

Operating characteristics are powerful, but they do not answer every design question. They do not by themselves determine whether p₀ is clinically appropriate, whether p₁ represents a meaningful improvement, whether the response endpoint is appropriate, whether the planned sample size is feasible, whether the treatment effect is clinically relevant, or whether the trial population is appropriate.

Those are scientific and clinical decisions. Operating characteristics tell you how the statistical design behaves given those assumptions.

How Can You Calculate Operating Characteristics?

Operating characteristics can be calculated directly from the binomial distributions underlying the Simon design. Statistical software can also calculate Simon designs and their operating characteristics; for example, the clinfun package in R provides functionality for Simon two-stage designs.

For practical design work, software is particularly useful for evaluating PET, ESS, and the power function over a range of response probabilities rather than only at p₀ and p₁.

The important point is not simply to calculate one number, but to examine how the design behaves across the response-probability range that is clinically relevant.

A Practical Checklist

When evaluating a Simon two-stage design, work through the following checks in order.

One Important Scope Note

This article describes the standard Simon two-stage framework with futility stopping after stage 1.

Some other two-stage or adaptive designs may include additional interim decision rules, such as early efficacy stopping or adaptive sample-size modifications. Those designs have different operating characteristics and should not automatically be interpreted using the formulas presented here.

The Key Idea

A Simon two-stage design is more than a collection of four integers. Its operating characteristics describe the behaviour of the entire decision process, and three relationships capture most of it:

As the true response probability increases, a typical Simon design moves from early stopping, low ESS, and a low probability of success toward continuation, higher ESS, and a high probability of success.

Looking at these characteristics across a range of plausible response probabilities provides a much richer understanding of the design than looking only at its maximum sample size, type I error, and power at p₁.

Frequently Asked Questions