S3U

Reading / Compare and contrast / Graduate / rce02

Compare Effects Rather Than Significance Labels

Evaluate a moderation claim using the actual contrast and its uncertainty.

All worksheets
5 questions0m 0s

Prerequisites

Effect estimates, standard errors, normal intervals, and covariance.

Learn the skill

To compare two effects, estimate their difference directly. A significant estimate in one group and a nonsignificant estimate in another do not establish a significant difference. For independent estimates, variances of their difference add.

Worked example

Estimates 3 and 3 can receive different significance labels if their standard errors differ. The estimated contrast is still zero; the labels are not a test of that contrast.

Model and Assumptions

Fictional independent group-effect estimates on the same scale: A=4 with SE=1.5; B=3 with SE=2. A normal 95% interval is estimate +/- 1.96*SE. The abstract argues that the intervention works differently because only A's interval excludes zero.

1. What is the estimated effect contrast A-B?
2. What is the standard error of A-B under independence?
3. What normal 95% interval follows for that contrast?
4. What conclusion is justified by this contrast interval?
5. If the estimates share participants, what term must be considered?

Further inquiry

Reconstruct both separate intervals and the contrast interval. Then repeat the contrast calculation with covariance 1 and explain why neither nonsignificance nor an interval containing zero proves practical equivalence.

Review criteria

  • Distinguish separate null tests from a contrast test.
  • Use variance 4.25 when covariance is 1.
  • State a substantive equivalence margin before making an equivalence claim.