find a homogeneous linear system of two equations in three unknowns whose solution space consists of those vectors in R3 that are orthogonal to a=(1,1,1) and b=(-2,3,0)

Verified Answer :

\(\mathbf a=(1,1,1)^\top=1\,\mathbf i+1\,\mathbf j+1\,\mathbf k\)
\(\mathbf b=(-2,3,0)^\top=-2\,\mathbf i+3\,\mathbf j\)
\(\mathbf a\times\mathbf b=\begin{vmatrix}\mathbf i&\mathbf j&\mathbf k\\1&1&1\\-2&3&0\end{vmatrix}=-3\,\mathbf i-2\,\mathbf j+5\,\mathbf k=(-3,-2,5)^\top\)

Basically, you're looking for a matrix \(\mathbf A\) such that

\(\mathbf A(\mathbf a\times\mathbf b)=\mathbf 0\)

i.e. a matrix \(\mathbf A\) whose nullspace with basis vector \(\mathbf a\times\mathbf b\).

By the rank-nullity theorem, the rank of \(\mathbf A\) and the dimension of its nullspace must add up to the number of columns, so

\(\mathrm{rank}\mathbf A+\underbrace{\mathrm{null}\mathbf A}_1=3\implies\mathrm{rank}\mathbf A=2\)

One easy choice for a row would be \(\begin{bmatrix}1&1&1\end{bmatrix}\), since

\((1,1,1)(-3,-2,5)^\top=0\)

Now you only need to find another combination such that the second row of \(\mathbf A\) is independent of the first. An easy choice for this is to let the first element be 0, and the next be 1. Then the last element must be \(\dfrac25\), as

\(\left(0,1,\dfrac25\right)(-3,-2,5)^\top=0\)

So,

\(\underbrace{\begin{bmatrix}1&1&1\\0&1&\frac25\end{bmatrix}}_{\mathbf A}\begin{bmatrix}-3\\-2\\5\end{bmatrix}=\begin{bmatrix}0\\0\end{bmatrix}\)

is one possible solution.