First, the characteristic solution. The homogeneous ODE
\(y''-3y'+2y=0\)
has characteristic equation
\(r^2-3r+2=(r-2)(r-1)=0\)
which has roots at \(r=2\) and \(r=1\), which means the characteristic solution is
\(y_c=C_1e^{2x}+C_2e^x\)
To find the particular solution, the method of undetermined coefficients will probably be the easiest to use. Suppose the particular solution and its derivative are
\(y_p=ae^{-x}+b\cos3x+c\sin3x\)
\({y_p}'=-ae^{-x}-3b\sin3x+3c\cos3x\)
\({y_p}''=ae^{-x}-9b\cos3x-9c\sin3x\)
Substituting into the ODE gives
\(\left(ae^{-x}-9b\cos3x-9c\sin3x\right)-3\left(-ae^{-x}-3b\sin3x+3c\cos3x\right)+2\left(ae^{-x}+b\cos3x+c\sin3x\right)=3e^{-x}-10\cos3x\)
\(6ae^{-x}+(-9b-9c+2b)\cos3x+(-9c+9b+2c)\sin3x=3e^{-x}-10\cos3x\)
\(6ae^{-x}+(-7b-9c)\cos3x+(-7c+9b)\sin3x=3e^{-x}-10\cos3x\)
It follows that
\(\begin{cases}6a=3\\-7b-9c=-10\\-7c+9b=0\end{cases}\implies a=\dfrac12,b=\dfrac7{13},c=\dfrac9{13}\)
So the solution is
\(y=y_c+y_p\)
\(y=C_1e^{2x}+C_2e^x+\dfrac12e^{-x}+\dfrac7{13}\cos3x+\dfrac9{13}\sin3x\)
