MDM Reference • Density matrices
Multivariate Density and Matrix Notation
When several quantities are observed together, the single-variable account no longer suffices. The joint distribution records not only how each quantity varies but how the quantities vary with one another, and the natural home for that record is a square arrangement of numbers.
From one scale to many
A joint density over several variables gives the weight attached to each small box of the space. Its single-variable margins are recovered by integrating away the remaining variables, and the conditional densities are recovered by dividing the joint by a margin. The whole grammar of the single-variable case carries over, with integrals taken over several coordinates at once.
Matrix notation
A list of observations is written as a column, its expectations as a column of the same height, and the second moments as a square arrangement whose entry in row i and column j is the covariance of the i-th and j-th quantities. The diagonal of that arrangement holds the variances. The notation is compact, and the compactness is not idle: many operations on the joint distribution take the form of operations on that arrangement.
The off-diagonal entries
An off-diagonal entry records how two quantities move together. A positive entry marks a tendency to rise and fall in step; a negative entry marks a tendency to move in opposite directions; a zero entry marks the absence of a linear tendency, which is weaker than independence. The reader should resist the temptation to read independence into a zero entry, since non-linear dependence may remain.
Symmetry, definiteness and invertibility
- The arrangement is symmetric, since covariance does not depend on the order of the pair.
- It is positive semidefinite, because any linear combination has a variance that cannot be negative.
- It is invertible exactly when no quantity is a linear combination of the others, a condition that fails under perfect collinearity.
These three facts respect the geometry of the space and govern which operations are available. The inverse, when it exists, is the natural companion of the joint density and sits in the exponent of the multivariate Gaussian.
Reading the arrangement
A well-ordered display of the matrix, with the diagonal shaded, lets the reader see at a glance which quantities carry the most variation and which pairs move most closely together. In the study of wave forms the same arrangement governs the joint behaviour of readings taken at different instants, and it becomes the entry point to the theory of stationary processes.
Working caution
Scale affects the reading. A large covariance may reflect nothing more than the large units in which one quantity is measured. For that reason the standardised version of the arrangement, in which each entry is divided by the two relevant standard deviations, is often the more informative object and is treated in the cell on correlation and coherence.
A caution on the arrangement
A well-formed arrangement of covariances is symmetric and carries no negative spread in any direction. An arrangement that fails either condition cannot describe real dispersion, and its presence usually marks an arithmetic slip rather than a genuine finding. The reader is urged to check the two conditions before building on such an arrangement.