If we use the substitution \(u = \tan x\), then \(du = \sec^2 {x}\ dx\). If you try substituting just \(u\) and \(du\) into the integrand, though, you'll notice that there's a \(\sec^2x\) left over that we have to deal with.
To get rid of this problem, use the identity \(\tan^2 x + 1 = \sec^2 x\) and substitute in the left side of the identity for the extra \(\sec^2x\), as shown:
\(\int\limits^{\pi/6}_0 {tan^2 x \ sec^4 x} \, dx\)
\(\int\limits^{\pi/6}_0 {tan^2 x \ (tan^2 x + 1) \ sec^2 x} \, dx\)
From there, we can substitute in \(u\) and \(du\), and then evaluate:
\(\int\limits^{\pi/6}_0 {tan^2 x \ (tan^2 x + 1) \ sec^2 x} \, dx\)
\(\int\limits^{\frac{1}{\sqrt{3}}}_0 {u^2(u^2 + 1)} \, du\)
\(\int\limits^{\frac{1}{\sqrt{3}}}_0 {u^4 + u^2} \, du\)
\(= \left.\frac{u^5}{5} + \frac{u^3}{3}\right|_0^\frac{1}{\sqrt{3}}\)
\(= (\frac{(\frac{1}{\sqrt{3}})^5}{5} + \frac{(\frac{1}{\sqrt{3}})^3}{3}) - (\frac{(0)^5}{5} + \frac{(0)^3}{3})\)
\(= \frac{1}{45\sqrt{3}} + \frac{1}{9\sqrt{3}} = \frac{6}{45\sqrt{3}} = \bf \frac{2}{15\sqrt{3}}\)
