How would I at least start out those two problems?
It looks to me like you already did. Now just follow through with it.
If the length of each side of the frame's inner dimension (i.e. the glass) is x, and the length of each side of the frame's outer dimension is x^2 - 3, then the area of the frame is:
A = (outer dimension)^2 - (inner dimension)^2
27 = (x^2 - 3)^2 - (x)^2
27 = x^4 - 6x^2 + 9 - x^2
27 = x^4 - 7x^2 + 9 <----- This is where you got to already.
0 = x^4 - 7x^2 - 18
0 = (x^2 - 9)(x^2 + 2)
Therefore, either x^2 = 9 or x^2 = -2... and you know it can't be the latter.
Thus, x^2 = 9 and therefore x = 3. The glass is 3" on each side.
Each (outer) side of the frame is x^2 - 3 = 9 - 3 = 6"
