- Understanding matrix-vector multiplications
- Special matrix types
- How we can “measure” vectors
- How we can “measure” matrices

A ** vector space** is a set \(V\) of vectors and a field \(F\) (elements of F are called scalars) with the following two operations:

- Vector addition: \(\forall \mathbf{v},\mathbf{w} \in V\), \(\mathbf{v} + \mathbf{w} \in V\)
- Scalar multiplication: \(\forall \alpha \in F, \mathbf{v} \in V\), \(\alpha \mathbf{v} \in V\)

which satisfiy the following conditions:

- Associativity (vector): \(\forall \mathbf{u}, \mathbf{v}, \mathbf{w} \in V\), \((\mathbf{u} + \mathbf{v}) + \mathbf{w} = \mathbf{u} + (\mathbf{v}+\mathbf{w})\)
- Zero vector: There exists a vector \(\mathbf{0} \in V\) such that \(\forall \mathbf{u} \in V, \mathbf{0} + \mathbf{u} = \mathbf{u}\)
- Additive inverse (negatives): For every \(\mathbf{u} \in V\), there exists \(\mathbf{-u} \in V\), such that \(\mathbf{u} + \mathbf{-u} = \mathbf{0}\).
- Associativity (scalar): \(\forall \alpha, \beta \in F, \mathbf{u} \in V\), \((\alpha \beta) \mathbf{u} = \alpha (\beta \mathbf{u})\)
- Distributivity: \(\forall \alpha, \beta \in F, \mathbf{u} \in V\), \((\alpha + \beta) \mathbf{u} = \alpha \mathbf{u} + \beta \mathbf{u}\)
- Unitarity: \(\forall \mathbf{u} \in V\), \(1 \mathbf{u} = \mathbf{u}\)

If there exist a set of vectors \(\mathbf{v}_1,\mathbf{v}_2\dots, \mathbf{v}_n\) such that any vector \(\mathbf{x}\in V\) can be written as a *linear combination*

with uniquely determined scalars \(c_1,\dots,c_n\), the set \({\mathbf{v}_1,\dots, \mathbf{v}_n}\) is called a ** basis** for \(V\). The size of the basis \(n\) is called the

The standard example of a vector space is \(V=\mathbb{R}^n\) with \(F=\mathbb{R}\). Vectors in \(\mathbb{R}^n\) are written as an array of numbers:

The dimension of \(\mathbb{R}^n\) is \(n\). The standard basis vectors of \(\mathbb{R}^n\) are written as

A set of vectors \(\mathbf{v}_1,\dots,\mathbf{v}_k\) is called ** linearly independent** if the equation \(\alpha_1\mathbf{v}_1 + \alpha_2\mathbf{v}_2 + \dots + \alpha_k\mathbf{v}_k = \mathbf{0}\) in the unknowns \(\alpha_1,\dots,\alpha_k\), has only the

Let \(V\) be a real vector space. Then, an ** inner product** is a function \(\langle\cdot, \cdot \rangle: V \times V \rightarrow \mathbb{R}\) (i.e., it takes two vectors and returns a real number) which satisfies the following four properties, where \(\mathbf{u}, \mathbf{v}, \mathbf{w} \in V\) and \(\alpha, \beta \in \mathbb{R}\):

- Positivity: \(\langle \mathbf{u}, \mathbf{u} \rangle \geq 0\)
- Definiteness: \(\langle \mathbf{u}, \mathbf{u} \rangle = 0\) if and only if \(\mathbf{u} = 0\)
- Symmetric: \(\langle \mathbf{u}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{u} \rangle\)
- Linearity: \(\langle \alpha \mathbf{u} + \beta \mathbf{v}, \mathbf{w} \rangle = \alpha \langle \mathbf{u}, w \rangle + \beta \langle \mathbf{v}, \mathbf{w} \rangle\)

The inner product intuitively represents the similarity between two vectors. Two vectors \(\mathbf{u}, \mathbf{v} \in V\) are said to be ** orthogonal** if \(\langle \mathbf{u}, \mathbf{v} \rangle = 0\).

The standard inner product on \(\mathbb{R}^n\) is the dot product :

To read more about Inner Product Definition

A function \(f: V \to W\) between two vector spaces \(V\) and \(W\) is called ** linear** if

- \(f(\mathbf{u} + \mathbf{v}) = f(\mathbf{u}) + f(\mathbf{v})\), for any \(\mathbf{u},\mathbf{v} \in V\)
- \(f(c\mathbf{v}) = cf(\mathbf{v})\), for all \(\mathbf{v} \in V\) and all scalars \(c\)

\(f\) is commonly called a ** linear transformation**.

If \(n\) and \(m\) are the dimension of \(V\) and \(W\), respectively, then \(f\) can be represented as an \(m\times n\) rectangular array or *matrix*

The numbers in the matrix \(\mathbf{A}\) are determined by the basis vectors for the spaces \(V\) and \(W\). To see how, we first review matrix vector multiplication.

Let \(\mathbf{A}\) be an \(m\times n\) matrix of real numbers. We can also write \(\mathbf{A}\in\mathbb{R}^{m\times n}\) as shorthand. If \(\mathbf{x}\) is a vector in \(\mathbb{R}^n\) then the matrix-vector product \(\mathbf{A}\mathbf{x} = \mathbf{b}\) is a vector in \(\mathbf{R}^m\) defined by:

We can interpret matrix-vector multiplications in two ways. Throughout this online textbook reference, we will use the notation to refer to the column of the matrix and to refer to the row of the matrix .

1) Writing a matrix-vector multiplication as inner products of the rows :

2) Writing a matrix-vector multiplication as linear combination of the columns of :

It is this representation that allows us to express any linear transformation between finite-dimensional vector spaces with matrices.

Let be the standard basis of . If we define the vector , then using the interpretation of matrix-vector products as linear combinations of the column of , we have that:

where we have written the standard basis of as .

In other words, if is written as a linear combination of the basis vectors of , the element is the coefficient corresponding to .

Let’s try an example. Suppose that \(V\) is a vector space with basis \(\mathbf{v}_1,\mathbf{v}_2,\mathbf{v}_3\), and \(W\) is a vector space with basis \(\mathbf{w}_1,\mathbf{w}_2\). Then \(V\) and \(W\) have dimension 3 and 2, respectively. Thus any linear transformation \(f: V \to W\) can be represented by a \(2\times 3\) matrix. We can introduce column vector notation, so that vectors \(\mathbf{v} = \alpha_1\mathbf{v}_1 + \alpha_2\mathbf{v}_2 + \alpha_3\mathbf{v}_3\) and \(\mathbf{w} = \beta_1\mathbf{w}_1 + \beta_2\mathbf{w}_2\) can be written as

We have not specified what the vector spaces \(V\) and \(W\), but it is fine if we treat them like elements of \(\mathbb{R}^3\) and \(\mathbb{R}^2\).

Suppose that the following facts are known about the linear transformation \(f\):

- \(f(\mathbf{v}_1) = \mathbf{w}_1\)
- \(f(\mathbf{v}_2) = 5\mathbf{w}_1 - \mathbf{w}_2\)
- \(f(\mathbf{v}_3) = 2\mathbf{w}_1 + 2\mathbf{w}_2\)

This is enough information to completely determine the matrix representation of \(f\). The first equation tells us

So we know . The second equation tells us that

So we know . Finally, the third equation tells us

Thus, \(a_{13} = 2,\ a_{23} = 2\), and the linear transformation \(f\) can be represented by the matrix:

It is important to note that the matrix representation not only depends on \(f\), but also our choice of basis. If we chose different bases for the vector spaces \(V\text{ and } W\), the matrix representation of \(f\) would change as well.

The \(m \times n\) ** zero matrix** is denoted by \({\bf 0}_{mn}\) and has all entries equal to zero. For example, the \(3 \times 4\) zero matrix is

The \(n \times n\) ** identity matrix** is denoted by \({\bf I}_n\) and has all entries equal to zero except for the diagonal, which is all 1. For example, the \(4 \times 4\) identity matrix is

A \(n \times n\) ** diagonal matrix** has all entries equal to zero except for the diagonal entries. We typically use \({\bf D}\) for diagonal matrices. For ecample, \(4 \times 4\) diagonal matrices have the form:

A ** lower-triangular matrix** is a square matrix that is entirely zero above the diagonal. We typically use \({\bf L}\) for lower-triangular matrices. For example, \(4 \times 4\) lower-triangular matrices have the form:

An ** upper triangular matrix** is a square matrix that is entirely zero below the diagonal. We typically use \({\bf U}\) for upper-triangular matrices. For example, \(4 \times 4\) upper-triangular matrices have the form:

Properties of triangular matrices:

- An \(n \times n\) triangular matrix has \(n(n-1)/2\) entries that must be zero, and \(n(n+1)/2\) entries that are allowed to be non-zero.
- Zero matrices, identity matrices, and diagonal matrices are all both lower triangular and upper triangular.

A ** permuation matrix** is a square matrix that is all zero, except for a single entry in each row and each column which is 1. We typically use \({\bf P}\) for permutation matrices. An example of a \(4 \times 4\) permutation matrix is

The properties of a permutation matrix are:

- Exactly \(n\) entries are non-zero.
- Multiplying a vector with a permutation matrix permutes (rearranges) the order of the entries in the vector. For example, using \({\bf P}\) above and \(x = [1, 2, 3, 4]^T\), the product is \({\bf Px} = [2, 4, 1, 3]^T\).
- If \(P_{ij} = 1\) then \(({\bf Px})_i = x_j\).
- The inverse of a permutation matrix is its transpose, so \({\bf PP}^T = {\bf P}^T{\bf P} = {\bf I}\).

A matrix in ** block form** is a matrix partitioned into blocks. A block is simply a submatrix. For example, consider

where \({\bf A}\), \({\bf B}\), \({\bf C}\), and \({\bf D}\) are submatrices.

There are special matrices in block form as well. For instance, a ** block diagonal** matrix is a block matrix whose off-diagonal blocks are zero matrices.

The ** rank** of a matrix is the number of linearly independent columns of the matrix. It can also be shown that the matrix has the same number of linearly indendent rows, as well. If \(\mathbf{A} \text{ is an } m \times n\) matrix, then

- \(\text{rank}(\mathbf{A}) \leq \text{min}(m,n)\).
- If \(\text{rank}(\mathbf{A}) = \text{min}(m,n)\), then \(\mathbf{A}\) is
. Otherwise, \(\mathbf{A}\) is*full rank*.*rank deficient*

A square \(n\times n\) matrix \(\mathbf{A}\) is ** invertible** if there exists a square matrix \(\mathbf{B}\) such that \(\mathbf{AB} = \mathbf{BA} = \mathbf{I}\), where \(\mathbf{I}\) is the \(n\times n\) identity matrix. The matrix \(\mathbf{B}\) is denoted by \(\mathbf{A}^{-1}\). A square matrix is invertible if and only if it has full rank. A square matrix that is not invertible is called a

A ** vector norm** is a function (i.e., it takes a vector and returns a nonnegative real number) that satisfies the following properties, where and :

- Positivity:
- Definiteness: if and only if
- Homogeneity:
- Triangle inequality:

A norm is a generalization of “absolute value” and measures the “magnitude” of the input vector.

The ** p-norm** is defined as

.

The definition is a valid norm when . If then it is not a valid norm because it violates the triangle inequality.

When (2-norm), this is called the ** Euclidean norm** and it corresponds to the length of the vector.

Consider the case of , in this part we will show how to calculate the 1, 2, and norm of .

For the 1-norm:

For the 2-norm:

For the -norm:

A ** general matrix norm** is a real valued function that satisfies the following properties:

- Positivity:
- Definiteness: if and only if
- Homogeneity: for all scalars
- Triangle inequality:

** Induced (or operator) matrix norms** are associated with a specific vector norm and are defined as:

An induced matrix norm is a particular type of a general matrix norm. Induced matrix norms tell us the maximum amplification of the norm of any vector when multiplied by the matrix. Note that the definition above is equivalent to

In addition to the properties above of general matrix norms, induced matrix norms also satisfy the submultiplicative conditions:

The Frobenius norm is simply the sum of every element of the matrix squared, which is equivalent to applying the vector -norm to the flattened matrix,

The Frobenius norm is an example of a general matrix norm that is not an induced norm.

The matrix p-norm is induced by the p-norm of a vector. It is . There are three special cases:

For the 1-norm, this reduces to the maximum absolute column sum of the matrix, i.e.,

For the 2-norm, this reduces the maximum singular value of the matrix.

For the -norm this reduces to the maximum absolute row sum of the matrix.

Now we will go through a few examples with a matrix , defined below.

For the 1-norm:

For the 2-norm:

The singular values are the square roots of the eigenvalues of the matrix . You can also find the maximum singular values by calculating the Singular Value Decomposition of the matrix.

For the -norm:

- See this review link

- 2020-04-27 Mariana Silva mfsilva@illinois.edu: updated notation and mat-vec section
- 2020-02-01 Peter Sentz: added more text from current slide deck
- 2018-3-14 Adam Stewart adamjs5@illinois.edu: clarifying definition of Frobenius norm
- 2017-11-10 Erin Carrier ecarrie2@illinois.edu: fixing index range for example, adds linear function definition
- 2017-10-29 Erin Carrier ecarrie2@illinois.edu: changing inner product notation, additional comments on induced vs general norms
- 2017-10-29 Erin Carrier ecarrie2@illinois.edu: adds review questions, completes vector space definition, rewords matrix norm section, minor other revisions
- 2017-10-29 Erin Carrier ecarrie2@illinois.edu: adds block form
- 2017-10-28 John Doherty jjdoher2@illinois.edu: first complete draft
- 2017-10-16 Matthew West mwest@illinois.edu: first complete draft