9. Replication and reporting

Learning goals

You will learn to:

  • distinguish computational reproduction from empirical replication;
  • rerun the same analysis on a new sample;
  • compare estimates and confidence intervals;
  • create an auditable output table;
  • write a concise quantitative-results paragraph.

Reproduction and replication

  • Reproduction: use the same data and code to recover the reported result.
  • Replication: use a new sample or setting to examine whether the result is similar.

This lesson uses a second simulated employee sample as a simple replication.

Research question: Is the association of engagement and workload with performance similar in an independent teaching sample?

Prepare both datasets

library(tidyverse)
library(broom)

prepare_employee_data <- function(file) {
  read_csv(file, show_col_types = FALSE) |>
    mutate(
      engagement_4 = 6 - engagement_4_reverse,
      engagement = rowMeans(
        across(c(
          engagement_1,
          engagement_2,
          engagement_3,
          engagement_4
        )),
        na.rm = TRUE
      )
    )
}

original <- prepare_employee_data("data/employee_survey.csv")
replication <- prepare_employee_data("data/employee_replication.csv")

The small function prevents us from applying different preparation steps to the two samples. Functions are optional for beginners, but this example shows why they can improve consistency.

Estimate the same model twice

original_model <- lm(
  performance ~ engagement + workload + training_group,
  data = original
)

replication_model <- lm(
  performance ~ engagement + workload + training_group,
  data = replication
)

Build a comparison table

comparison <- bind_rows(
  tidy(original_model, conf.int = TRUE) |>
    mutate(sample = "Original"),
  tidy(replication_model, conf.int = TRUE) |>
    mutate(sample = "Replication")
) |>
  select(sample, term, estimate, conf.low, conf.high, p.value)

comparison
# A tibble: 8 × 6
  sample      term                   estimate conf.low conf.high  p.value
  <chr>       <chr>                     <dbl>    <dbl>     <dbl>    <dbl>
1 Original    (Intercept)               43.9    40.5      47.3   1.91e-89
2 Original    engagement                 4.86    4.31      5.42  4.43e-52
3 Original    workload                  -1.54   -2.16     -0.930 1.07e- 6
4 Original    training_groupTraining     2.45    1.22      3.67  9.60e- 5
5 Replication (Intercept)               45.4    41.0      49.8   3.45e-61
6 Replication engagement                 4.11    3.41      4.82  2.70e-26
7 Replication workload                  -1.43   -2.23     -0.636 4.57e- 4
8 Replication training_groupTraining     2.46    0.917     4.01  1.88e- 3

Compare:

  • direction of each estimate;
  • approximate magnitude;
  • confidence-interval overlap and width;
  • whether substantive conclusions are similar;
  • sample and measurement differences.

A coefficient need not have the same p-value in both samples to show a broadly similar pattern.

Save the output

write_csv(
  comparison,
  "replication_model_comparison.csv"
)

A reproducible project should preserve the code that generated a table, not only the copied table.

Report the result

A compact results paragraph can follow this structure:

We estimated linear regressions predicting employee performance from engagement, workload, and training status. In the original sample, engagement was [positively/negatively] associated with performance (b = …, 95% CI […, …]), controlling for the other predictors. The estimate in the replication sample was [similar/different] in direction and magnitude (b = …, 95% CI […, …]). These observational results indicate an association and do not by themselves establish a causal effect.

Reproducibility checklist

  • Use an R Project and relative file paths.
  • Keep raw data separate from derived outputs.
  • Record package requirements.
  • Set a seed for random procedures.
  • Preserve scripts and data-processing decisions.
  • Label simulated and real data accurately.
  • Avoid editing numerical output by hand.
  • Document deviations from the original analysis.

Practice

  1. Compare descriptive statistics across the samples.
  2. Add age and tenure to both models.
  3. Identify one estimate that changes most.
  4. Complete the results paragraph with the actual numbers.
  5. Render the report template in templates/research-report.qmd.

Common mistakes

  • Calling a rerun on the same data a replication.
  • Changing the model after seeing the replication result without disclosure.
  • Treating a non-significant replication estimate as proof of no relationship.
  • Comparing only p-values.
  • Copying output without preserving the generating code.

Takeaway

Replication is a comparison of evidence, not a binary pass/fail test. Clear code, consistent preparation, and transparent reporting make the comparison credible.