4. Reliability and scale construction

Learning goals

You will learn to:

distinguish an observed survey item from an underlying construct;
reverse-score negatively worded items;
calculate Cronbach’s alpha;
inspect item-level diagnostics;
create scale scores transparently;
explain why reliability is not the same thing as validity.

Measurement question: We cannot observe “leadership quality” directly. Do several survey items behave coherently enough to be summarized as a scale?

From construct to items

Concept / theory→Survey items→Observed responses→Measurement evidence→Scale score

Management constructs such as leadership, engagement, trust, or organizational support are often latent: they are not directly observed in one column.

Reverse coding

For a 1–5 item, a negatively worded response can be reversed with:

reversed score = 6 - original score

library(tidyverse)
library(psych)

employees <- read_csv(
  "data/employee_survey.csv",
  show_col_types = FALSE
) |>
  mutate(
    leadership_4 = 6 - leadership_4_reverse,
    engagement_4 = 6 - engagement_4_reverse
  )

Select the leadership items

leadership_items <- employees |>
  select(
    leadership_1,
    leadership_2,
    leadership_3,
    leadership_4
  )

Cronbach’s alpha

leadership_alpha <- alpha(leadership_items)
leadership_alpha


Reliability analysis   
Call: alpha(x = leadership_items)

  raw_alpha std.alpha G6(smc) average_r S/N    ase mean  sd median_r
      0.87      0.87    0.84      0.63 6.8 0.0095    3 1.2     0.64

    95% confidence boundaries 
         lower alpha upper
Feldt     0.85  0.87  0.89
Duhachek  0.85  0.87  0.89

 Reliability if an item is dropped:
             raw_alpha std.alpha G6(smc) average_r S/N alpha se   var.r med.r
leadership_1      0.82      0.82    0.76      0.60 4.6    0.014 0.00152  0.63
leadership_2      0.84      0.84    0.78      0.65 5.5    0.012 0.00032  0.65
leadership_3      0.85      0.85    0.79      0.65 5.6    0.012 0.00034  0.66
leadership_4      0.83      0.83    0.77      0.62 4.9    0.013 0.00290  0.65

 Item statistics 
               n raw.r std.r r.cor r.drop mean  sd
leadership_1 480  0.87  0.87  0.82   0.76  3.0 1.3
leadership_2 472  0.84  0.84  0.76   0.71  3.0 1.4
leadership_3 480  0.83  0.83  0.75   0.70  3.0 1.4
leadership_4 472  0.86  0.86  0.79   0.74  2.9 1.4

Non missing response frequency for each item
                1    2    3    4    5 miss
leadership_1 0.17 0.22 0.23 0.20 0.18 0.00
leadership_2 0.20 0.21 0.21 0.19 0.19 0.02
leadership_3 0.19 0.22 0.21 0.19 0.19 0.00
leadership_4 0.19 0.22 0.24 0.17 0.18 0.02

Look at:

raw alpha;
corrected item-total correlations;
what happens if an item is removed;
whether an item appears to run in the opposite direction.

Alpha is evidence about internal consistency. It does not prove that the scale is one-dimensional, theoretically valid, or appropriate for another population.

Create a scale score

A transparent rule might require at least three answered items out of four.

employees_scored <- employees |>
  mutate(
    leadership_valid_items = rowSums(!is.na(leadership_items)),
    leadership = if_else(
      leadership_valid_items >= 3,
      rowMeans(leadership_items, na.rm = TRUE),
      NA_real_
    )
  )

The missing-data rule should be defined and reported, not improvised after seeing results.

Why the next lesson matters

Reliability asks whether items behave consistently.

The next lesson asks a different set of questions:

Can we reduce many variables to fewer components? → PCA
What latent factor structure is suggested by the correlations? → EFA
Does a pre-specified theoretical structure fit another sample? → CFA

Practice

Reverse-code engagement_4_reverse.
Calculate alpha for the engagement items.
Compare alpha before and after correct reverse coding.
Create an engagement mean score.
Explain in one sentence why a high alpha does not prove construct validity.

Takeaway

Scale construction is a measurement decision, not a mechanical averaging operation. Reliability is one piece of evidence in a larger measurement argument.