a(t) = at² + b
A(t) = A(0)a(t)
at time t = 0:
A(0) = A(0)(0 + b)
b = 1

Now is it correct to say the following:
172 = 100(a*3² + 1)
thus a = 0.08
and then the accumulated value at time 10 of $100 invested at time 5 is:
A(10) = 100(0.08(10²)+1) - 100(0.08(5²)+1)
A(10) = 900 - 300 = $600?

Your question makes no sense. You say that a(t)= at^{2}+ b. How can you use "a" inside an equation for "a"? Did you mean A(t)? But then you say A(t)= A(0)a(t).

If you meant A(t)= at^{2}+ b with a and b constants, then
A(0)= a(0)+ b= b= 100 so b= 100. Then A(3)= a(3^{2})+ 100= 100+ 9a= 172 so 9a= 72 and a= 8.
That is, A(t)= 8t^{2}+ 100.

I'm not sure what is meant here. Are we to recalculate b, so that A(5)= 100, assuming the same a, and then calculate A(10)?