Vectors, matrices and norms

---

Learning Objectives

• Understanding matrix-vector multiplications
• Special matrix types
• How we can “measure” vectors
• How we can “measure” matrices

Vector Spaces

A vector space is a set $V$ of vectors and a field $F$ (elements of F are called scalars) with the following two operations:

1. Vector addition: $\mathrm{\forall }\mathbf{v},\mathbf{w}\in V$, $\mathbf{v}+\mathbf{w}\in V$
2. Scalar multiplication: $\mathrm{\forall }\alpha \in F,\mathbf{v}\in V$, $\alpha \mathbf{v}\in V$

which satisfiy the following conditions:

1. Associativity (vector): $\mathrm{\forall }\mathbf{u},\mathbf{v},\mathbf{w}\in V$, $(\mathbf{u}+\mathbf{v})+\mathbf{w}=\mathbf{u}+(\mathbf{v}+\mathbf{w})$
2. Zero vector: There exists a vector $\mathbf{0}\in V$ such that $\mathrm{\forall }\mathbf{u}\in V,\mathbf{0}+\mathbf{u}=\mathbf{u}$
3. Additive inverse (negatives): For every $\mathbf{u}\in V$, there exists $\mathbf{-}\mathbf{u}\in V$, such that $\mathbf{u}+\mathbf{-}\mathbf{u}=\mathbf{0}$.
4. Associativity (scalar): $\mathrm{\forall }\alpha ,\beta \in F,\mathbf{u}\in V$, $(\alpha \beta )\mathbf{u}=\alpha (\beta \mathbf{u})$
5. Distributivity: $\mathrm{\forall }\alpha ,\beta \in F,\mathbf{u}\in V$, $(\alpha +\beta )\mathbf{u}=\alpha \mathbf{u}+\beta \mathbf{u}$
6. Unitarity: $\mathrm{\forall }\mathbf{u}\in V$, $1\mathbf{u}=\mathbf{u}$

If there exist a set of vectors ${\mathbf{v}}_1,{\mathbf{v}}_2\dots ,{\mathbf{v}}_n$ such that any vector $\mathbf{x}\in V$ can be written as a linear combination

$$\mathbf{x}=c_1{\mathbf{v}}_1+c_2{\mathbf{v}}_2+\cdots +c_n{\mathbf{v}}_n$$

with uniquely determined scalars $c_1,\dots ,c_n$, the set ${\mathbf{v}}_1,\dots ,{\mathbf{v}}_n$ is called a basis for $V$. The size of the basis $n$ is called the dimension of $V$.

The standard example of a vector space is $V={\mathbb{R}}^n$ with $F=\mathbb{R}$. Vectors in ${\mathbb{R}}^n$ are written as an array of numbers:

$$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]={\left[\begin{array}{cccc}{x}_1& x_2& \cdots & x_n\end{array}\right]}^T$$

The dimension of ${\mathbb{R}}^n$ is $n$. The standard basis vectors of ${\mathbb{R}}^n$ are written as

$${\mathbf{e}}_1=\left[\begin{array}{c}1\\ 0\\ ⋮\\ 0\end{array}\right]{\mathbf{e}}_2=\left[\begin{array}{c}0\\ 1\\ ⋮\\ 0\end{array}\right]\dots {\mathbf{e}}_n=\left[\begin{array}{c}0\\ 0\\ ⋮\\ 1\end{array}\right].$$

A set of vectors ${\mathbf{v}}_1,\dots ,{\mathbf{v}}_k$ is called linearly independent if the equation ${\alpha }_1{\mathbf{v}}_1+{\alpha }_2{\mathbf{v}}_2+\cdots +{\alpha }_k{\mathbf{v}}_k=\mathbf{0}$ in the unknowns ${\alpha }_1,\dots ,{\alpha }_k$, has only the trivial solution ${\alpha }_1={\alpha }_2=\cdots ={\alpha }_k=0$. Otherwise the vectors are linearly dependent, and at least one of the vectors can be written as a linear combination of the other vectors in the set. A basis is always linearly independent.

Inner Product

Let $V$ be a real vector space. Then, an inner product is a function $⟨\cdot ,\cdot ⟩:V\times V\to \mathbb{R}$ (i.e., it takes two vectors and returns a real number) which satisfies the following four properties, where $\mathbf{u},\mathbf{v},\mathbf{w}\in V$ and $\alpha ,\beta \in \mathbb{R}$:

1. Positivity: $⟨\mathbf{u},\mathbf{u}⟩\ge 0$
2. Definiteness: $⟨\mathbf{u},\mathbf{u}⟩=0$ if and only if $\mathbf{u}=0$
3. Symmetric: $⟨\mathbf{u},\mathbf{v}⟩=⟨\mathbf{v},\mathbf{u}⟩$
4. Linearity: $⟨\alpha \mathbf{u}+\beta \mathbf{v},\mathbf{w}⟩=\alpha ⟨\mathbf{u},w⟩+\beta ⟨\mathbf{v},\mathbf{w}⟩$

The inner product intuitively represents the similarity between two vectors. Two vectors $\mathbf{u},\mathbf{v}\in V$ are said to be orthogonal if $⟨\mathbf{u},\mathbf{v}⟩=0$.

The standard inner product on ${\mathbb{R}}^n$ is the dot product :$⟨\mathbf{x},\mathbf{y}⟩={\mathbf{x}}^T\mathbf{y}=\sum _{i=1}^n{x}_i{y}_i.$

To read more about Inner Product Definition

Linear Transformations and Matrices

A function $f:V\to W$ between two vector spaces $V$ and $W$ is called linear if

1. $f(\mathbf{u}+\mathbf{v})=f(\mathbf{u})+f(\mathbf{v})$, for any $\mathbf{u},\mathbf{v}\in V$
2. $f(c\mathbf{v})=cf(\mathbf{v})$, for all $\mathbf{v}\in V$ and all scalars $c$

$f$ is commonly called a linear transformation.

If $n$ and $m$ are the dimension of $V$ and $W$, respectively, then $f$ can be represented as an $m\times n$ rectangular array or matrix

$$\mathbf{A}=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ a_{21}& a_{22}& \dots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \dots & a_{mn}\end{array}\right].$$

The numbers in the matrix $\mathbf{A}$ are determined by the basis vectors for the spaces $V$ and $W$. To see how, we first review matrix vector multiplication.

Matrix-vector multiplication

Let $\mathbf{A}$ be an $m\times n$ matrix of real numbers. We can also write $\mathbf{A}\in {\mathbb{R}}^{m\times n}$ as shorthand. If $\mathbf{x}$ is a vector in ${\mathbb{R}}^n$ then the matrix-vector product $\mathbf{A}\mathbf{x}=\mathbf{b}$ is a vector in ${\mathbf{R}}^m$ defined by:

$$b_i=\sum _{j=1}^n{a}_{ij}{x}_j\text{for }i=1,2,\dots ,m.$$

We can interpret matrix-vector multiplications in two ways. Throughout this online textbook reference, we will use the notation ${\mathbf{a}}_i$ to refer to the $i^{th}$ column of the matrix $\mathbf{A}$ and ${\mathbf{a}}_i^T$ to refer to the $i^{th}$ row of the matrix $\mathbf{A}$.

1) Writing a matrix-vector multiplication as inner products of the rows $\mathbf{A}$:

$$\mathbf{A}\mathbf{x}=\left[\begin{array}{c}{\mathbf{a}}_1^T\cdot \mathbf{x}\\ {\mathbf{a}}_2^T\cdot \mathbf{x}\\ ⋮\\ {\mathbf{a}}_m^T\cdot \mathbf{x}\end{array}\right]$$

2) Writing a matrix-vector multiplication as linear combination of the columns of $\mathbf{A}$:

$\mathbf{A}\mathbf{x}=x_1{\mathbf{a}}_1+x_2{\mathbf{a}}_2+\dots x_n{\mathbf{a}}_n=x_1\left[\begin{array}{c}{a}_{11}\\ a_{21}\\ ⋮\\ a_{m1}\end{array}\right]+x_2\left[\begin{array}{c}{a}_{12}\\ a_{22}\\ ⋮\\ a_{m2}\end{array}\right]+\cdots +x_n\left[\begin{array}{c}{a}_{1n}\\ a_{2n}\\ ⋮\\ a_{mn}\end{array}\right]$

It is this representation that allows us to express any linear transformation between finite-dimensional vector spaces with matrices.

Matrix Representation of Linear Transformations

Let ${\mathbf{e}}_1,{\mathbf{e}}_2,\dots ,{\mathbf{e}}_n$ be the standard basis of ${\mathbb{R}}^n$. If we define the vector ${\mathbf{z}}_j=\mathbf{A}{\mathbf{e}}_j$, then using the interpretation of matrix-vector products as linear combinations of the column of $\mathbf{A}$, we have that:

$${\mathbf{z}}_j=\mathbf{A}{\mathbf{e}}_j=\left[\begin{array}{c}{a}_{1j}\\ a_{2j}\\ ⋮\\ a_{mj}\end{array}\right]=a_{1j}\left[\begin{array}{c}1\\ 0\\ ⋮\\ 0\end{array}\right]+a_{2j}\left[\begin{array}{c}0\\ 1\\ ⋮\\ 0\end{array}\right]+\cdots +a_{mj}\left[\begin{array}{c}0\\ 0\\ ⋮\\ 1\end{array}\right]=\sum _{i=1}^m{a}_{ij}{\hat{\mathbf{e}}}_i,$$

where we have written the standard basis of ${\mathbb{R}}^m$ as ${\hat{\mathbf{e}}}_1,{\hat{\mathbf{e}}}_2,\dots ,{\hat{\mathbf{e}}}_m$.

In other words, if ${\mathbf{z}}_j=\mathbf{A}{\mathbf{e}}_j$ is written as a linear combination of the basis vectors of ${\mathbb{R}}^m$, the element $a_{ij}$ is the coefficient corresponding to ${\hat{\mathbf{e}}}_i$.

Example

Let’s try an example. Suppose that $V$ is a vector space with basis ${\mathbf{v}}_1,{\mathbf{v}}_2,{\mathbf{v}}_3$, and $W$ is a vector space with basis ${\mathbf{w}}_1,{\mathbf{w}}_2$. Then $V$ and $W$ have dimension 3 and 2, respectively. Thus any linear transformation $f:V\to W$ can be represented by a $2\times 3$ matrix. We can introduce column vector notation, so that vectors $\mathbf{v}={\alpha }_1{\mathbf{v}}_1+{\alpha }_2{\mathbf{v}}_2+{\alpha }_3{\mathbf{v}}_3$ and $\mathbf{w}={\beta }_1{\mathbf{w}}_1+{\beta }_2{\mathbf{w}}_2$ can be written as

$$\mathbf{v}=\left[\begin{array}{c}{\alpha }_1\\ {\alpha }_2\\ {\alpha }_3\end{array}\right],\mathbf{w}=\left[\begin{array}{c}{\beta }_1\\ {\beta }_2\end{array}\right].$$

We have not specified what the vector spaces $V$ and $W$, but it is fine if we treat them like elements of ${\mathbb{R}}^3$ and ${\mathbb{R}}^2$.

Suppose that the following facts are known about the linear transformation $f$:

• $f({\mathbf{v}}_1)={\mathbf{w}}_1$
• $f({\mathbf{v}}_2)=5{\mathbf{w}}_1-{\mathbf{w}}_2$
• $f({\mathbf{v}}_3)=2{\mathbf{w}}_1+2{\mathbf{w}}_2$

This is enough information to completely determine the matrix representation of $f$. The first equation tells us

$${\mathbf{w}}_1=f({\mathbf{v}}_1)⟹\left[\begin{array}{c}1\\ 0\end{array}\right]=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\end{array}\right]\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]=\left[\begin{array}{c}{a}_{11}\\ a_{21}\end{array}\right].$$

So we know $a_{11}=1,a_{21}=0$. The second equation tells us that

$$5{\mathbf{w}}_1-{\mathbf{w}}_2=f({\mathbf{v}}_2)⟹\left[\begin{array}{c}5\\ -1\end{array}\right]=\left[\begin{array}{ccc}1& a_{12}& a_{13}\\ 0& a_{22}& a_{23}\end{array}\right]\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right]=\left[\begin{array}{c}{a}_{12}\\ a_{22}\end{array}\right].$$

So we know $a_{12}=5,a_{22}=-1$. Finally, the third equation tells us

$$2{\mathbf{w}}_1+2{\mathbf{w}}_2=f({\mathbf{v}}_2)⟹\left[\begin{array}{c}2\\ 2\end{array}\right]=\left[\begin{array}{ccc}1& 5& a_{13}\\ 0& -1& a_{23}\end{array}\right]\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]=\left[\begin{array}{c}{a}_{13}\\ a_{23}\end{array}\right].$$

Thus, $a_{13}=2,a_{23}=2$, and the linear transformation $f$ can be represented by the matrix:

$$\left[\begin{array}{ccc}1& 5& 2\\ 0& -1& 2\end{array}\right].$$

It is important to note that the matrix representation not only depends on $f$, but also our choice of basis. If we chose different bases for the vector spaces $V\text{ and }W$, the matrix representation of $f$ would change as well.

Special Matrices

Zero Matrices

The $m\times n$ zero matrix is denoted by ${\mathbf{0}}_{mn}$ and has all entries equal to zero. For example, the $3\times 4$ zero matrix is

$${\mathbf{0}}_{34}=\left[\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right].$$

Identity Matrices

The $n\times n$ identity matrix is denoted by ${\mathbf{I}}_n$ and has all entries equal to zero except for the diagonal, which is all 1. For example, the $4\times 4$ identity matrix is

$${\mathbf{I}}_4=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right].$$

Diagonal Matrices

A $n\times n$ diagonal matrix has all entries equal to zero except for the diagonal entries. We typically use $\mathbf{D}$ for diagonal matrices. For ecample, $4\times 4$ diagonal matrices have the form:

$$\left[\begin{array}{cccc}{d}_{11}& 0& 0& 0\\ 0& d_{22}& 0& 0\\ 0& 0& d_{33}& 0\\ 0& 0& 0& d_{44}\end{array}\right].$$

Triangular Matrices

A lower-triangular matrix is a square matrix that is entirely zero above the diagonal. We typically use $\mathbf{L}$ for lower-triangular matrices. For example, $4\times 4$ lower-triangular matrices have the form:

$$\mathbf{L}=\left[\begin{array}{cccc}{\ell }_{11}& 0& 0& 0\\ {\ell }_{21}& {\ell }_{22}& 0& 0\\ {\ell }_{31}& {\ell }_{32}& {\ell }_{33}& 0\\ {\ell }_{41}& {\ell }_{42}& {\ell }_{43}& {\ell }_{44}\end{array}\right].$$

An upper triangular matrix is a square matrix that is entirely zero below the diagonal. We typically use $\mathbf{U}$ for upper-triangular matrices. For example, $4\times 4$ upper-triangular matrices have the form:

$$\mathbf{U}=\left[\begin{array}{cccc}{u}_{11}& u_{12}& u_{13}& u_{14}\\ 0& u_{22}& u_{23}& u_{24}\\ 0& 0& u_{33}& u_{34}\\ 0& 0& 0& u_{44}\end{array}\right].$$

Properties of triangular matrices:

1. An $n\times n$ triangular matrix has $n(n-1)/2$ entries that must be zero, and $n(n+1)/2$ entries that are allowed to be non-zero.
2. Zero matrices, identity matrices, and diagonal matrices are all both lower triangular and upper triangular.

Permutation Matrices

A permuation matrix is a square matrix that is all zero, except for a single entry in each row and each column which is 1. We typically use $\mathbf{P}$ for permutation matrices. An example of a $4\times 4$ permutation matrix is

$$\mathbf{P}=\left[\begin{array}{cccc}0& 1& 0& 0\\ 0& 0& 0& 1\\ 1& 0& 0& 0\\ 0& 0& 1& 0\end{array}\right].$$

The properties of a permutation matrix are:

1. Exactly $n$ entries are non-zero.
2. Multiplying a vector with a permutation matrix permutes (rearranges) the order of the entries in the vector. For example, using $\mathbf{P}$ above and $x=[1,2,3,4{]}^T$, the product is $\mathbf{P}\mathbf{x}=[2,4,1,3{]}^T$.
3. If $P_{ij}=1$ then $(\mathbf{P}\mathbf{x}{)}_i=x_j$.
4. The inverse of a permutation matrix is its transpose, so ${\mathbf{P}\mathbf{P}}^T={\mathbf{P}}^T\mathbf{P}=\mathbf{I}$.

Matrices in Block Form

A matrix in block form is a matrix partitioned into blocks. A block is simply a submatrix. For example, consider

$$\mathbf{M}=\left[\begin{array}{cc}\mathbf{A}& \mathbf{B}\\ \mathbf{C}& \mathbf{D}\end{array}\right]$$

where $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, and $\mathbf{D}$ are submatrices.

There are special matrices in block form as well. For instance, a block diagonal matrix is a block matrix whose off-diagonal blocks are zero matrices.

Matrix Rank

The rank of a matrix is the number of linearly independent columns of the matrix. It can also be shown that the matrix has the same number of linearly indendent rows, as well. If $\mathbf{A}\text{ is an }m\times n$ matrix, then

1. $\text{rank}(\mathbf{A})\le \text{min}(m,n)$.
2. If $\text{rank}(\mathbf{A})=\text{min}(m,n)$, then $\mathbf{A}$ is full rank. Otherwise, $\mathbf{A}$ is rank deficient.

A square $n\times n$ matrix $\mathbf{A}$ is invertible if there exists a square matrix $\mathbf{B}$ such that $\mathbf{AB}=\mathbf{BA}=\mathbf{I}$, where $\mathbf{I}$ is the $n\times n$ identity matrix. The matrix $\mathbf{B}$ is denoted by ${\mathbf{A}}^{-1}$. A square matrix is invertible if and only if it has full rank. A square matrix that is not invertible is called a singular matrix.

Matrices as operators

Rotation operator

$$(\genfrac{}{}{0ex}{}{y_1}{y_2})=\left[\begin{array}{cc}\cos \mathbf{(}\theta \mathbf{)}& \mathbf{-}\sin \mathbf{(}\theta \mathbf{)}\\ \sin \mathbf{(}\theta \mathbf{)}& \cos \mathbf{(}\theta \mathbf{)}\end{array}\right](\genfrac{}{}{0ex}{}{x_1}{x_2})$$

Scale operator

$$(\genfrac{}{}{0ex}{}{y_1}{y_2})=\left[\begin{array}{cc}a& 0\\ 0& b\end{array}\right](\genfrac{}{}{0ex}{}{x_1}{x_2})$$

Reflection operator

$$(\genfrac{}{}{0ex}{}{y_1}{y_2})=\left[\begin{array}{cc}-a& 0\\ 0& -b\end{array}\right](\genfrac{}{}{0ex}{}{x_1}{x_2})$$

Translation operator

$$(\genfrac{}{}{0ex}{}{y_1}{y_2})=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right](\genfrac{}{}{0ex}{}{x_1}{x_2})+(\genfrac{}{}{0ex}{}{a}{b})$$

Vector Norm

A vector norm is a function $\Vert \mathbf{u}\Vert :V\to {\mathbb{R}}_0^{+}$ (i.e., it takes a vector and returns a nonnegative real number) that satisfies the following properties, where $\mathbf{u},\mathbf{v}\in V$ and $\alpha \in \mathbb{R}$:

1. Positivity: $\Vert \mathbf{u}\Vert \ge 0$
2. Definiteness: $\Vert \mathbf{u}\Vert =0$ if and only if $\mathbf{u}=\mathbf{0}$
3. Homogeneity: $\Vert \alpha \mathbf{u}\Vert =|\alpha |\Vert \mathbf{u}\Vert$
4. Triangle inequality: $\Vert \mathbf{u}+\mathbf{v}\Vert \le \Vert \mathbf{u}\Vert +\Vert \mathbf{v}\Vert$

A norm is a generalization of “absolute value” and measures the “magnitude” of the input vector.

The p-norm

The p-norm is defined as

$\Vert \mathbf{w}{\Vert }_p=(\sum _{i=1}^N|w_i{|}^p{)}^{\frac{1}{p}}$.

The definition is a valid norm when $p\ge 1$. If $0\le p<1$ then it is not a valid norm because it violates the triangle inequality.

When $p=2$ (2-norm), this is called the Euclidean norm and it corresponds to the length of the vector.

Vector Norm Examples

Consider the case of $\mathbf{w}=[-3,5,0,1]$, in this part we will show how to calculate the 1, 2, and $\mathrm{\infty }$ norm of $\mathbf{w}$.

For the 1-norm:

$$\Vert \mathbf{w}{\Vert }_1=(\sum _{i=1}^N|w_i{|}^1{)}^{\frac{1}{1}}$$ $$\Vert \mathbf{w}{\Vert }_1=\sum _{i=1}^N|w_i|$$ $$\Vert \mathbf{w}{\Vert }_1=|-3|+|5|+|0|+|1|$$ $$\Vert \mathbf{w}{\Vert }_1=3+5+0+1$$ $$\Vert \mathbf{w}{\Vert }_1=9$$

For the 2-norm:

$$\Vert \mathbf{w}{\Vert }_2=(\sum _{i=1}^N|w_i{|}^2{)}^{\frac{1}{2}}$$ $$\Vert \mathbf{w}{\Vert }_2=\sqrt{\sum _{i=1}^N{w}_i^2}$$ $$\Vert \mathbf{w}{\Vert }_2=\sqrt{(-3{)}^2+(5{)}^2+(0{)}^2+(1{)}^2}$$ $$\Vert \mathbf{w}{\Vert }_2=\sqrt{9+25+0+1}$$ $$\Vert \mathbf{w}{\Vert }_2=\sqrt{35}\approx 5.92$$

For the $\mathrm{\infty }$-norm:

$$\Vert \mathbf{w}{\Vert }_{\mathrm{\infty }}=\underset{p\to \mathrm{\infty }}{lim}(\sum _{i=1}^N|w_i{|}^p{)}^{\frac{1}{p}}$$ $$\Vert \mathbf{w}{\Vert }_{\mathrm{\infty }}=\underset{i=1,\dots ,N}{max}|w_i|$$ $$\Vert \mathbf{w}{\Vert }_{\mathrm{\infty }}=max(|-3|,|5|,|0|,|1|)$$ $$\Vert \mathbf{w}{\Vert }_{\mathrm{\infty }}=max(3,5,0,1)$$ $$\Vert \mathbf{w}{\Vert }_{\mathrm{\infty }}=5$$

Norms and Errors

To calculate the error when computing a vector result, you can apply a norm.

Absolute error = ||true value - approximate value||

Relative error = Absolute error / ||true value||

Matrix Norm

A general matrix norm is a real valued function $\Vert \mathbf{A}\Vert$ that satisfies the following properties:

1. Positivity: $\Vert \mathbf{A}\Vert \ge 0$
2. Definiteness: $\Vert \mathbf{A}\Vert =0$ if and only if $\mathbf{A}=0$
3. Homogeneity: $\Vert \lambda \mathbf{A}\Vert =|\lambda |\Vert \mathbf{A}\Vert$ for all scalars $\lambda$
4. Triangle inequality: $\Vert \mathbf{A}+\mathbf{B}\Vert \le \Vert \mathbf{A}\Vert +\Vert \mathbf{B}\Vert$

Induced (or operator) matrix norms are associated with a specific vector norm $\Vert \cdot \Vert$ and are defined as:

$$\Vert \mathbf{A}\Vert :=\underset{\Vert \mathbf{x}\Vert =1}{max}\Vert \mathbf{A}\mathbf{x}\Vert .$$

An induced matrix norm is a particular type of a general matrix norm. Induced matrix norms tell us the maximum amplification of the norm of any vector when multiplied by the matrix. Note that the definition above is equivalent to

$$\Vert \mathbf{A}\Vert =\underset{\Vert \mathbf{x}\Vert \ne 0}{max}\frac{\Vert \mathbf{A}\mathbf{x}\Vert }{\Vert x\Vert }.$$

In addition to the properties above of general matrix norms, induced matrix norms also satisfy the submultiplicative conditions:

$$\Vert \mathbf{A}\mathbf{x}\Vert \le \Vert \mathbf{A}\Vert \Vert \mathbf{x}\Vert$$ $$\Vert \mathbf{A}\mathbf{B}\Vert \le \Vert \mathbf{A}\Vert \Vert \mathbf{B}\Vert$$

Frobenius norm

The Frobenius norm is simply the sum of every element of the matrix squared, which is equivalent to applying the vector $2$-norm to the flattened matrix,

$$\Vert \mathbf{A}{\Vert }_F=\sqrt{\sum _{i,j}{a}_{ij}^2}.$$

The Frobenius norm is an example of a general matrix norm that is not an induced norm.

The matrix p-norm

The matrix p-norm is induced by the p-norm of a vector. It is $\Vert \mathbf{A}{\Vert }_p:=\underset{\Vert \mathbf{x}{\Vert }_p=1}{max}\Vert \mathbf{A}\mathbf{x}{\Vert }_p$. There are three special cases:

For the 1-norm, this reduces to the maximum absolute column sum of the matrix, i.e.,

$$\Vert \mathbf{A}{\Vert }_1=\underset{j}{max}\sum _{i=1}^n|a_{ij}|.$$

For the 2-norm, this reduces the maximum singular value of the matrix.

$$\Vert \mathbf{A}{\Vert }_2=\underset{k}{max}{\sigma }_k$$

For the $\mathrm{\infty }$-norm this reduces to the maximum absolute row sum of the matrix.

$$\Vert \mathbf{A}{\Vert }_{\mathrm{\infty }}=\underset{i}{max}\sum _{j=1}^n|a_{ij}|.$$

Matrix Norm Examples

Now we will go through a few examples with a matrix $\mathbf{C}$, defined below.

$$\mathbf{C}=\left[\begin{array}{cc}3& -2\\ -1& 3\end{array}\right]$$

For the 1-norm:

$$\Vert \mathbf{C}{\Vert }_1=\underset{\Vert \mathbf{x}{\Vert }_1=1}{max}\Vert \mathbf{C}\mathbf{x}{\Vert }_1$$ $$\Vert \mathbf{C}{\Vert }_1=\underset{1\le j\le 3}{max}\sum _{i=1}^3|C_{ij}|$$ $$\Vert \mathbf{C}{\Vert }_1=max(|3|+|-1|,|-2|+|3|)$$ $$\Vert \mathbf{C}{\Vert }_1=max(4,5)$$ $$\Vert \mathbf{C}{\Vert }_1=5$$

For the 2-norm:

The singular values are the square roots of the eigenvalues of the matrix ${\mathbf{C}}^T\mathbf{C}$. You can also find the maximum singular values by calculating the Singular Value Decomposition of the matrix.

$$\Vert \mathbf{C}{\Vert }_2=\underset{\Vert \mathbf{x}{\Vert }_2=1}{max}\Vert \mathbf{C}\mathbf{x}{\Vert }_2$$ $$det({\mathbf{C}}^T\mathbf{C}-\lambda \mathbf{I})=0$$ $$det(\left[\begin{array}{cc}3& -1\\ -2& 3\end{array}\right]\left[\begin{array}{cc}3& -2\\ -1& 3\end{array}\right]-\lambda \mathbf{I})=0$$ $$det(\left[\begin{array}{cc}9+1& -6-3\\ -3-6& 4+9\end{array}\right]-\lambda \mathbf{I})=0$$ $$det(\left[\begin{array}{cc}10-\lambda & -9\\ -9& 13-\lambda \end{array}\right])=0$$ $$(10-\lambda )(13-\lambda )-81=0$$ $${\lambda }^2-23\lambda +130-81=0$$ $${\lambda }^2-23\lambda +49=0$$ $$(\lambda -\frac{1}{2}(23+3\sqrt{37}))(\lambda -\frac{1}{2}(23-3\sqrt{37}))=0$$ $$\Vert \mathbf{C}{\Vert }_2=\sqrt{{\lambda }_{max}}=\sqrt{\frac{1}{2}(23+3\sqrt{37})}\approx 4.54$$

For the $\mathrm{\infty }$-norm:

$$\Vert \mathbf{C}{\Vert }_{\mathrm{\infty }}=\underset{\Vert \mathbf{x}{\Vert }_{\mathrm{\infty }}=1}{max}\Vert \mathbf{C}\mathbf{x}{\Vert }_{\mathrm{\infty }}$$ $$\Vert \mathbf{C}{\Vert }_{\mathrm{\infty }}=\underset{1\le i\le 3}{max}\sum _{j=1}^3|C_{ij}|$$ $$\Vert \mathbf{C}{\Vert }_{\mathrm{\infty }}=max(|3|+|-2|,|-1|+|3|)$$ $$\Vert \mathbf{C}{\Vert }_{\mathrm{\infty }}=max(5,4)$$ $$\Vert \mathbf{C}{\Vert }_{\mathrm{\infty }}=5$$

Review Questions

• See this review link

ChangeLog

• 2022-03-01 Arnav Shah arnavss2@illinois.edu: added norms and errors section
• 2022-02-22 Arnav Shah arnavss2@illinois.edu: added matrices as operators section
• 2020-04-27 Mariana Silva mfsilva@illinois.edu: updated notation and mat-vec section
• 2020-02-01 Peter Sentz: added more text from current slide deck
• 2018-3-14 Adam Stewart adamjs5@illinois.edu: clarifying definition of Frobenius norm
• 2017-11-10 Erin Carrier ecarrie2@illinois.edu: fixing index range for example, adds linear function definition
• 2017-10-29 Erin Carrier ecarrie2@illinois.edu: changing inner product notation, additional comments on induced vs general norms
• 2017-10-29 Erin Carrier ecarrie2@illinois.edu: adds review questions, completes vector space definition, rewords matrix norm section, minor other revisions
• 2017-10-29 Erin Carrier ecarrie2@illinois.edu: adds block form
• 2017-10-28 John Doherty jjdoher2@illinois.edu: first complete draft
• 2017-10-16 Matthew West mwest@illinois.edu: first complete draft