\(\displaystyle\int\tan^5x\,\mathrm dx=\int\tan^4x\tan x\)
\(=\displaystyle\int(\sec^2x-1)^2\tan x\,\mathrm dx\)
\(=\displaystyle\int(\sec^4x-2\sec^2x+1)\tan x\,\mathrm dx\)
\(=\displaystyle\int\sec^3x\sec x\tan x\,\mathrm dx-2\int\sec^2x\tan x\,\mathrm dx+\int\tan x\,\mathrm dx\)
For the first integral, let \(a=\sec x\) so that \(\mathrm da=\sec x\tan x\,\mathrm dx\). For the second, \(b=\tan x\) so that \(\mathrm db=\sec^2x\,\mathrm dx\). The last integral is standard, but if you're not familiar with it, you could write \(\tan x=\dfrac{\sin x}{\cos x}\) and set \(c=\cos x\) so that \(\mathrm dc=-\sin x\,\mathrm dx\). Now
\(=\displaystyle\int a^3\,\mathrm da-2\int b\,\mathrm db-\int\frac{\mathrm dc}c\)
\(=\displaystyle\frac{a^4}4-b^2-\ln|c|+C\)
\(=\displaystyle\frac{\sec^4x}4-\tan^2x-\ln|\cos x|+C\)
