Factor Analysis

In Lesson 1, PCA took Rosa's four flower measurements and compressed them into a couple of readable directions. It answered one question: which directions capture the most spread? Factor analysis answers a different, deeper one: what hidden things could have produced these measurements in the first place?

Meet Priya, a school counselor. At the end of term she has percentage scores for 250 students in six subjects: reading, vocabulary and essay writing (English), and arithmetic, algebra and geometry (maths). When she lines the columns up and looks at how they move together, a pattern jumps out, shown in the grid below. The three English subjects rise and fall together. The three maths subjects rise and fall together. But an English score tells you almost nothing about a maths score.

Priya never gave a test called "language ability" or "number ability". Yet it is as if two such hidden abilities are quietly steering all six columns. Factor analysis is the tool that finds them.

By the end of this lesson you will be able to:

• Say what a latent (hidden) factor is, and how the common-factor model explains a set of correlations
• Run factor analysis in R, then read the loadings, communalities and uniquenesses
• Decide how many factors to keep, rotate them so you can name them, and say how all this differs from PCA

Prerequisites: Lesson 1, PCA in R (correlation, loadings, standardizing, variance explained). You can run R and read its output. No linear algebra is assumed; every term is defined as it appears.

Two green blocks, and cold blue-white everywhere else. That two-block shape is the fingerprint we are about to explain.

The clue is in the correlations

Let us make Priya's data concrete so you can compute on it. Each lesson runs in a fresh R session, so we build the marks right here (run this once). Two hidden abilities, one number per student, are turned into six subject scores, each with its own random noise. You never see the two abilities; you only see the six columns they produce.

Now look at how the six columns move together:

Inside the English block, correlations sit around 0.5. Inside the maths block, around 0.6. Across the two blocks, they collapse to near zero. Six columns, but really only two "clumps" of shared movement.

The hidden ability you never tested

Two kinds of variable are in play here, and keeping them straight is the key to the whole lesson.

• An observed variable (also called a manifest variable, or an indicator) is something you actually measured. Priya has six of them: the six subject scores sitting in her spreadsheet.
• A latent factor is a hidden variable you did not measure, but that influences several observed variables at once. Priya suspects two: a language ability and a number ability. She never scored a student on "language ability" directly, yet its fingerprints are all over the reading, vocabulary and essay columns.

Why would a hidden ability create correlation? Because it is a common cause. A student who happens to be strong in language will tend to score high on reading and vocabulary and essays, all together. A student who is weak in language dips on all three together. That shared push and pull is exactly what makes the three columns rise and fall in step, which is exactly the correlation you saw.

The common-factor model

Here is the machinery, stated plainly, then in symbols. Factor analysis assumes each observed score is built from the hidden factors, plus a leftover that belongs to that subject alone. Read the arrows as "produces": the hidden factors produce the scores you see. The flow below traces one student's marks being generated.

Now the formula. For one subject \(j\) (say, reading) and two factors, the model is:

\[ X_j = \lambda_{j1} F_1 + \lambda_{j2} F_2 + \varepsilon_j \]

Every symbol, in words:

• \(X_j\) is the observed, standardized score on subject \(j\).
• \(F_1, F_2\) are the two hidden factors (the number ability and the language ability), the same two for every student.
• \(\lambda_{j1}, \lambda_{j2}\) are the loadings: how strongly subject \(j\) responds to each factor. A big loading means that factor drives that subject hard.
• \(\varepsilon_j\) is the unique factor: everything about subject \(j\) that the common factors do not explain, its own quirks plus measurement noise.

Notice the direction. In PCA a component was a blend built out of the observed columns. Here it is the reverse: the observed columns are built out of the hidden factors. Factor analysis is a model of where the data came from, and its promise is that a couple of hidden factors can reproduce the whole web of correlations you measured.

What is a factor, really?

Priya never gave a test called "language ability", yet it appears in the model as \(F\). In the common-factor model, what is that language ability?

Fit the model in R

R has factor analysis built in: the function is factanal(). Give it the data frame and the number of factors to look for. Priya has two hunches, a language ability and a number ability, so ask for two factors. Fill in the blank.

fa <- factanal(scores, factors = 2)
fa
#> Uniquenesses:
#>    reading vocabulary      essay arithmetic    algebra   geometry
#>      0.416      0.430      0.544      0.341      0.372      0.397
#>
#> Loadings:
#>            Factor1 Factor2
#> reading            0.760
#> vocabulary         0.755
#> essay              0.671
#> arithmetic  0.808
#> algebra     0.792
#> geometry    0.775
#>
#>                Factor1 Factor2
#> SS loadings      1.892   1.607
#> Proportion Var   0.315   0.268
#> Cumulative Var   0.315   0.583
#>
#> Test of the hypothesis that 2 factors are sufficient.
#> The p-value is 0.678

Reading the loadings, and naming the factors

The loadings table is the heart of the result. Each number is close to the correlation between that subject and that factor, so it runs from -1 to +1, and a large size (either sign) means a strong tie. Look at the pattern:

• Factor1 loads high on arithmetic (0.81), algebra (0.79) and geometry (0.78), and near zero on the English subjects.
• Factor2 loads high on reading (0.76), vocabulary (0.76) and essay (0.67), and near zero on the maths subjects.

That clean split is called simple structure: each subject leans on essentially one factor. And now the factors are nameable. Factor1 is the hidden number ability; Factor2 is the hidden language ability, exactly the two abilities Priya suspected. The bars below show Factor2's loadings across all six subjects, so you can see the language factor pick out its three subjects and ignore the rest.

You can pull the loadings out as a plain table too:

Which factor is which?

Factor1 has loadings near 0.80 on arithmetic, algebra and geometry, and near 0 on the three English subjects. What is Factor1?

Communality and uniqueness

Each subject's variance splits cleanly into two parts: the share the common factors explain, and the share that is the subject's own. Because the columns are standardized, the two shares add up to exactly 1.

The part the factors explain is the communality, written \(h_j^2\). It is just the sum of that subject's squared loadings:

\[ h_j^2 = \lambda_{j1}^2 + \lambda_{j2}^2 \]

Whatever is left over is the uniqueness, \(\psi_j = 1 - h_j^2\), the Uniquenesses line R printed. Let us not just assert it; let us watch it add up for reading, whose loadings are 0.077 and 0.760:

0.584 plus 0.416 is 1: the two hidden abilities account for 58% of reading's ups and downs, and the remaining 42% is reading's own. The bars below rank every subject by communality, how much of it the two factors capture:

Essay has the lowest communality (0.46), so it is the most "its own thing": more of an essay mark is personal style and luck than the two abilities can reach.

The model rebuilds the correlations

Here is the promise made good. If the factors really produced the scores, then the correlation between any two subjects should be recoverable from their loadings, specifically, the sum of their paired loadings. Watch the model reconstruct a correlation it was never handed directly:

0.573 versus 0.575, a near-perfect match, from two subjects the model treated separately. And across the two blocks, where there should be almost no shared cause:

Both tiny. This is the difference from PCA in one picture: factor analysis is judged by how well a few hidden factors reproduce the correlation matrix. That is its whole objective.

Rotation: spinning to simple structure

There is one catch. The raw factanal solution does not always come out neatly aligned to "maths" and "language". The naked, unrotated loadings can be muddy, with the first factor loading a little on everything:

Here Factor1 loads on reading (0.28) as well as the maths subjects, so it is hard to name. Rotation fixes this. A rotation spins the factor axes to a spot where each subject loads strongly on one factor and near zero on the others, without changing how well the model fits (the communalities stay identical). The most common choice, varimax, is what factanal() applies by default:

How many factors should Priya keep?

We told factanal() to look for two. How would she have known? Three tools, and they agree here.

1. The Kaiser rule: keep factors whose eigenvalue exceeds 1. An eigenvalue of the correlation matrix is how much total variance a direction captures; one bigger than 1 means it beats a single original column. Count them:

Exactly two values top 1, then a steep drop. That drop, plotted, would be the scree plot elbow you met in Lesson 1: keep the factors before the curve flattens.

2. The likelihood test. factanal() runs a formal test whose null hypothesis is "this many factors is enough". A small p-value (below 0.05) rejects that number as too few:

One factor is crushed (p far below 0.05, so one is not enough). Two factors give p = 0.68, comfortably above 0.05, so we do not reject two. Both signals point to the same answer Priya suspected: two.

Compute a communality yourself

A subject's communality is the sum of its squared loadings, the share of its ups and downs the two abilities explain. Compute it for algebra (its loadings are in fa$loadings). Fill in the exponent that makes the loadings squared.

L <- fa$loadings
round(sum(L["algebra", ] ^ 2), 3)
#> [1] 0.628

Reading the evidence

Two eigenvalues of the correlation matrix exceed 1, the one-factor model has p far below 0.05, and the two-factor model has p = 0.68. How many factors should Priya keep?

Factor scores: where each student lands

Loadings describe the subjects. Priya also wants a number per student: how strong is each student in language, and in number ability? Those are the factor scores, estimated for every row. Ask factanal() for them:

Each student now has two numbers, roughly standardized (mean 0, so positive is above average). Student 4 sits high on Factor2 (strong language, +1.60) but slightly below average on Factor1 (number). Student 6 is the mirror image: strong in number (+1.03), weak in language (-1.11). Priya has turned six raw columns into two interpretable abilities she can actually act on.

Factor analysis is not PCA

They look similar (both take correlated columns and hand back a few loadings) but they answer opposite questions. PCA re-expresses the data; factor analysis models where it came from.

You can feel the difference by running PCA on the very same data. It reports variance captured by components, not a model of hidden causes:

Useful, but it never separates a subject's shared variance from its own, and it makes no claim about hidden abilities. Rule of thumb: reach for PCA to compress or plot, and factor analysis to test and name the hidden structure behind a correlation pattern.

PCA or factor analysis?

Priya's colleague insists factor analysis and PCA are the same thing. What is the key difference?

Where factor analysis can go wrong

Factor analysis is powerful precisely because it commits to a model, and a model can be wrong. Keep these in mind:

• It needs genuine latent structure. Feed it columns with no real shared causes and it will still hand back factors, but they will be noise dressed as insight. Blocks in the correlation matrix are your sanity check that factors are worth seeking.
• Heywood cases. Sometimes the fit produces a communality above 1 or a negative uniqueness, which is impossible for real variance. It signals too many factors, too small a sample, or a variable that does not fit the model.
• Factors are inferred, not proven. The maths and language labels are your interpretation of the loadings, not a fact the data guarantees. A different analyst might name them differently.
• Assumptions. It expects roughly interval-scale, linearly related variables and enough rows (a common rule of thumb is several times more students than subjects). Too few rows and the loadings wobble.

References

Four trustworthy places to take factor analysis further:

• An Introduction to Statistical Learning, ch. 12 (free PDF) - the gentle, canonical treatment of unsupervised methods, with PCA alongside for contrast.
• factanal: the R function documentation - every argument and every value the function you used returns.
• William Revelle, psych package and "An Introduction to Psychometric Theory" - the standard R toolkit (fa(), parallel analysis) and a deep, readable text on factor analysis.
• Spearman (1904), General Intelligence, Objectively Determined and Measured - the founding paper that started factor analysis from exactly Priya's kind of table.

Lesson 2 complete

You can now look at a wall of correlated columns and ask a modeler's question: what few hidden factors could have produced all this? You met the latent factor and the common-factor model, ran factanal() and read its loadings, split each variable into communality and uniqueness, watched the model rebuild the correlations, rotated to nameable simple structure, chose the number of factors three different ways, scored each student, and drew the sharp line between factor analysis and PCA.

Next, Lesson 3: k-Means and choosing k. PCA and factor analysis compressed the columns; now you will group the rows, sorting students (or flowers, or customers) into clusters, and face the central puzzle of how many clusters the data really has.