-512 divided by 9 [itex]\approx[/itex] -56 (but not exactly)
But the ninth root is not division.
Anyway, following on from that mistake, when you used the formula:
[tex]\frac{a\left(1-r^n\right)}{1-r}[/tex]
You substituted incorrectly. a=1/4, r=-56 (your mistake), n=5 (since it asks for the 5th term)
This is what you had:
[tex]\frac{\frac{1}{4}\left(1-(-56)^9\right)}{1-(-9)}[/tex]
This is what you should have:
[tex]\frac{\frac{1}{4}\left(1-(-56)^5\right)}{1-(-56)}[/tex]
But anyway, following on from that mistake also, whatever happened to the [itex]1-r^n[/itex] part?
[itex]1-(-56)^9[/itex] all of a sudden became something like 5x4 from the looks of it, but since you end up with the answer of -5.7 from there, it means that part must've been equal to 182.4, but it doesn't seem like it is.
Honestly, you need to go back and revise your algebra. You will struggle to solve all these harder questions if you don't have a firm foundation with algebra.