Stats

How should I validate N F P T across sequential constraint histories?

Step-by-step statistics solution: How should I validate N F P T across sequential constraint histories?

As an Amazon Associate, I earn from qualifying purchases. For more practice problems like this, see Schaum’s Outline of Statistics, 6th Edition.

Validation of N‑F‑P‑T Across Sequential Constraint Histories

Below is a step‑by‑step worked solution that shows how to test the four component hypotheses (N, F, P, T) for each ordered‑history condition (GA, GB, GC) and how to combine them with an intersection‑union test (IUT).

1. Plain‑language restatement of the problem

You have a series of independent experimental sessions (one model run per session).
Within a session the model goes through a pre‑treatment (acquisition) phase, a post‑treatment (persistence) phase, and a transfer phase.

Six experimental groups differ in which constraints are shown during the acquisition phase:

Group Acquisition phase Post‑treatment phase Transfer phase
G0 none (session starts directly at post‑treatment) – –
G1 same number of trials as treatment groups, but no constraint – –
G2 one constant constraint (any constraint vs. ordered history) – –
GA, GB, GC three different constraints presented in a different order (the ordered‑history conditions) – –

For each ordered‑history group we must show four things:

Letter What it means (in plain language)
N (Novel organization) The model uses a pre‑specified “advantageous” strategy more often (binary outcome per trial) than both G0 and G1.
F (Functionality) A continuous performance metric (e.g., regret) is not worse (non‑inferior) than the appropriate control.
P (Persistence) The N‑pattern from the post‑treatment phase remains across the later post‑treatment trials (no meaningful decline; a non‑inferiority/equivalence test on the trend).
T (Transfer) The same N‑pattern shows up in the different transfer environment (binary outcome).

A condition (e.g., GA) is declared positive only if all four component tests are significant at the pre‑specified α‑level (α = 0.05 unless otherwise stated). This is an intersection‑union test:

[ E^{*}= N\;\cap\;F\;\cap\;P\;\cap\;T . ]

Our task is to specify the statistical procedures that will let us test each component while respecting the dependence of trials within a session, and then to combine the four p‑values into the final decision rule.

2. Step‑by‑step solution

2.1. Organise the data

Variable Type Description
session_id factor Unique identifier for each independent model run.
group factor (6 levels) G0, G1, G2, GA, GB, GC.
phase factor (3 levels) acq (acquisition), post, transfer.
trial integer Trial number within the phase (1–12 for acquisition/post, 1–9 for transfer).
strategy_used binary (0/1) Did the model use the pre‑specified advantageous strategy on this trial?
regret continuous Continuous performance metric (lower = better).
time numeric Real‑time or trial index, useful for trend analysis.

All observations from the same session_id belong to the same experimental unit and are correlated.

2.2. Choose the appropriate mixed‑effects models

Because trials are nested inside sessions we use random‑effects to capture within‑session correlation.

Component Outcome Model type Fixed‑effects of interest
N (binary) strategy_used Logistic mixed model group (GA/GB/GC vs. reference G0 & G1) × phase (post)
F (continuous) regret Linear mixed model (LMM) group (GA/GB/GC) vs. control (G0 or G1) in post phase
P (trend) strategy_used Logistic mixed model with time Interaction group × time in post phase (test that slope for GA/GB/GC is not more negative than control)
T (binary) strategy_used Logistic mixed model group (GA/GB/GC) vs. control in transfer phase

All models contain a random intercept for session_id (and, optionally, a random slope for time when testing persistence).

2.3. Fit the models (example in R)

library(lme4)
library(multcomp)   # for multiple comparisons
library(emmeans)    # for equivalence / non‑inferiority tests

# 1. N – post‑treatment strategy use
mod_N <- glmer(strategy_used ~ group * phase +
               (1|session_id),
               data = d,
               subset = phase %in% c("post"),
               family = binomial)

# 2. F – post‑treatment regret
mod_F <- lmer(regret ~ group * phase +
              (1|session_id),
              data = d,
              subset = phase %in% c("post"))

# 3. P – persistence (trend over post‑treatment trials)
mod_P <- glmer(strategy_used ~ group * time +
               (1 + time|session_id),
               data = d,
               subset = phase %in% c("post"),
               family = binomial)

# 4. T – transfer phase strategy use
mod_T <- glmer(strategy_used ~ group * phase +
               (1|session_id),
               data = d,
               subset = phase %in% c("transfer"),
               family = binomial)

Replace group contrasts so that each ordered‑history group (GA, GB, GC) is tested against the pooled control (G0 + G1). One convenient way is to create a new factor group_cmp with three levels: ordered (GA/GB/GC), control (G0/G1), and other (G2).

2.4. Conduct the four component hypothesis tests

2.4.1. N – “more frequent use of the strategy”

Null: The odds of using the strategy for an ordered‑history group ≤ the odds for the control.
Alternative: Greater odds (one‑sided).

# Extract the GA vs. control contrast
contrast_N <- glht(mod_N,
                   linfct = mcp(group_cmp = "Dunnett"))  # Dunnett controls family‑wise error
p_N <- summary(contrast_N)$test$pvalues["ordered_vs_control"]

Reject H₀ if p_N < α. (One‑sided test: use alternative = "greater" if using emmeans.)

2.4.2. F – non‑inferiority of regret

Define a non‑inferiority margin Δ (e.g., Δ = 0.05 regret units).

Null: Mean regret for ordered ≥ (control mean + Δ).
Alternative: Mean regret ≤ (control + Δ).

# Estimated marginal means
emm_F <- emmeans(mod_F, ~ group_cmp | phase, at = list(phase = "post"))
# Contrast with equivalence margin
ni_test <- contrast(emm_F, method = "consec",
                    infer = c(TRUE, FALSE),   # one‑sided
                    null = Δ)                  # specify Δ
p_F <- ni_test$p.value

Reject H₀ if p_F < α.

2.4.3. P – persistence of the N‑pattern

We test that the slope of the strategy‑use probability over post‑treatment trials for an ordered group is not more negative than the control slope (non‑inferiority on the trend).

Null: slope_ordered ≤ slope_control - Δ_slope (i.e., a larger decline).
Alternative: slope_ordered ≥ slope_control - Δ_slope.

# Extract slopes
slopes <- emtrends(mod_P, ~ group_cmp, var = "time")
# Compute the difference in slopes
diff_slope <- contrast(slopes, method = "pairwise")["ordered - control", ]
# One‑sided test with margin Δ_slope (e.g., 0.02 per trial)
p_P <- pnorm((diff_slope$estimate - Δ_slope) / diff_slope$SE, lower.tail = FALSE)

Reject H₀ if p_P < α.

2.4.4. T – transfer of the N‑pattern

Same structure as N, but now the data are from the transfer phase.

contrast_T <- glht(mod_T,
                   linfct = mcp(group_cmp = "Dunnett"))
p_T <- summary(contrast_T)$test$pvalues["ordered_vs_control"]

Reject H₀ if p_T < α.

2.5. Combine the four p‑values with an Intersection‑Union Test (IUT)

For an IUT the overall null hypothesis is the union of the four component nulls; the alternative is the intersection (all four must hold). The test is conservative: we reject the overall null only if every individual test rejects at the pre‑specified α.

A convenient way to implement this is to take the maximum p‑value:

[ p_{\text{IUT}} = \max(p_N,\;p_F,\;p_P,\;p_T). ]

The overall decision rule is

[ \text{Declare the ordered‑history condition positive } \Longleftrightarrow p_{\text{IUT}} < \alpha . ]

Because we use the same α for each component, the overall type‑I error is guaranteed to be ≤ α (the test is exact for independent components; with dependence it is still conservative).

If you want a formal family‑wise error rate (FWER) control you may also apply a Bonferroni correction (α/4) to each component, but for a pure IUT the max‑p rule is standard.

2.6. Apply the procedure to each ordered‑history group

Because GA, GB, and GC differ only in the order of the three constraints, we repeat the whole set of four tests separately for each group:

Group p_N p_F p_P p_T max(p) Decision (α = 0.05)
GA … … … … … Positive / Negative
GB … … … … … Positive / Negative
GC … … … … … Positive / Negative

(Insert the actual numeric results once the data are analyzed.)

If any of the four component p‑values for a given group fails to reach significance, that group is not declared to have satisfied the N‑F‑P‑T criteria.

2.7. Summarise the final answer

  • Step 1 – Fit mixed‑effects models that respect trial‑level dependence.
  • Step 2 – Perform a one‑sided test for N, a non‑inferiority test for F, a non‑inferiority trend test for P, and a one‑sided test for T.
  • Step 3 – Compute the four p‑values for each ordered‑history group.
  • Step 4 – Form the IUT statistic p_IUT = max(p_N, p_F, p_P, p_T).
  • Step 5 – Declare the group positive if p_IUT < α (e.g., 0.05).

Thus the validation of N‑F‑P‑T reduces to four mixed‑effects hypothesis tests per ordered‑history condition, followed by a simple max‑p rule to enforce the conjunctive decision requirement.

3. Final answer (concise)

To validate that an ordered‑history condition (GA, GB, GC) satisfies the N‑F‑P‑T criteria, fit four mixed‑effects models (logistic for binary N, P, T; linear for continuous F) that include a

Original question: How should I validate N F P T across sequential constraint histories? on Cross Validated (Stats Stack Exchange), licensed CC BY-SA.

More Stats Problems