Four Fundamental Subspaces

Name: Four Fundamental Subspaces
Uploaded: 2025-01-30T11:08:03.000Z
Duration: 1 h 4 min 37 s

Machine Learning Foundations: Week 3 Overview

Introduction to Linear Algebra in Machine Learning

The lecture introduces the focus on linear algebra over the next four weeks, emphasizing its relevance to understanding machine learning concepts.

Key topics include the four fundamental subspaces associated with any matrix: column space, null space, row space, and left null space.

Column Space of a Matrix

The column space (C of A) is defined as the span of the columns of matrix A, which includes all linear combinations of these vectors.

Understanding the role of column space is crucial for solving equations in the form Ax = b; solutions exist if b belongs to this column space.

An example illustrates that not every vector in R^4 can be represented within the column space of A due to dimensional constraints (4 equations in 3 unknowns).

The specific case shows that some vectors do not belong to C(A), highlighting limitations based on dimensions.

Null Space of a Matrix

The null space (N of A) consists of all vectors x such that Ax = 0. It is confirmed as a subspace by verifying closure under addition and scalar multiplication.

If two vectors are in N(A), their sum also belongs to N(A), demonstrating one property required for subspaces.

Another property states that scaling a vector in N(A) by a scalar results in another vector still within N(A).

Example Calculation for Null Space

An example using matrix A encourages viewers to find an x such that Ax = 0 through linear combinations leading to zero vectors.

Demonstrates how combining certain columns can yield valid solutions for null spaces; specifically, adding columns 1 and 2 while subtracting column 3 results in zero.

Dimensionality and Properties

The dimensionality discussion clarifies that while there are three columns in matrix A, its null space only forms a line within R^3.

If matrix A is invertible, then its null space contains only the zero vector. Conversely, if it has non-trivial solutions, Ax = b can be expressed as x = xp + xn.

Gaussian Elimination Methodology

Gaussian elimination serves as an effective method for finding the null space by solving Ax = 0.

An illustrative example demonstrates steps taken during Gaussian elimination on a specific matrix setup.

Understanding Null Space and Rank-Nullity Theorem

Exploring Free Variables and Null Space Entries

The discussion begins with identifying pivot columns (column 1 and column 3) versus free variables (column 2 and column 4).

Setting free variables to basis entries, such as x2 = 1 and x4 = 0, leads to the equation 2x_3 + 4x_4 = 0, implying x_3 = 0.

From the equation x_1 + 2x_2 + 2x_3 + 2x_4 = 0, it follows that x_1 = -2, yielding the first null space entry: (-2, 1, 0, 0).

An alternative choice of setting x_2 = 0 and x_4 = 1 results in another null space entry: (2, 0, -2, 1).

The conclusion is that the null space of matrix A is spanned by these two vectors.

Rank and Nullity Concepts

The rank of a matrix corresponds to the number of pivot columns; for this example, rank(A) equals to 2.

Nullity is defined as the number of free variables; here, nullity(A) also equals 2.

According to the rank-nullity theorem: if A has n columns, then rank + nullity equals n. This can be expressed as rank(A)=r and nullity(A)=n-r.

Subspaces Related to Matrices

Key subspaces include column space (dimension related to rank), row space (equivalent to column space of A transpose), and their respective dimensions.

It’s noted that column rank equals row rank; thus both dimensions are equal.

Left Null Space Definition

The left null space refers to the null space of A transpose. It consists of all y such that A^T y = 0.

This implies finding linear combinations of rows resulting in a zero vector.

Dimensions Relation in Matrices

For an m x n matrix A: dimension(column space of A)+dimension(null space of A)=number of columns=n.

Applying this concept to A transpose gives dimension(column space)+dimension(null space)=number of rows=m.

Example Calculation for Clarity

An example illustrates calculating linear combinations without Gaussian elimination using a specific matrix configuration.

Further Simplified Example Analysis

Another simple example with a smaller matrix demonstrates how one can determine its rank through visual inspection—showing that one column is a multiple of another.

Understanding the Four Fundamental Subspaces of a Matrix

Exploring Nullity and Row Space

The discussion begins with the concept of nullity, where it is stated that for a matrix A , if the line passes through the point (-2, 1), then the nullity of A can be calculated as n - r = 1 .

The row space dimension is examined, revealing that it equals the rank ( r = 1 ), indicating a one-dimensional quantity represented by a line through (1, 2).

Left Null Space and Its Dimension

The left null space is introduced as the null space of A^T . It is noted that this can be verified by observing that multiplying -3 times the first row and once times the second row results in a zero entry. The dimension of this left null space is also found to equal 1.

Homework Assignment on Fundamental Subspaces

An assignment is given to work out the four fundamental subspaces: column space of A , null space of A , column space of A^T , and null space of A^T . Students are instructed to perform Gaussian elimination to reduce it to row equivalent form.

Recap of Key Concepts Discussed

In summary, key concepts covered include understanding what constitutes the four fundamental subspaces associated with a matrix: column space, null space, row space, and left null space.

The relationship between dimensions was emphasized: specifically, how rank (dimension of column/row spaces) plus nullity (dimension of corresponding null spaces) equals the number of columns in matrix A .

Relationship Between Row Space and Column Space