I'd like to make it clear to you what you have to do, in order to prove this by induction.
First, we ASSUME it is true for $n$, that is we assume:
$n^2(n+5) = 3k$, for some integer $k$ (we don't have to really know which one).
Let's re-write this in a form that will be more easy to "spot" later on:
$n^3 + 5n = 3k$.
We need to USE this somehow, to prove that under these circumstances for $n+1$:
$(n+1)^2((n+1) + 5) = 3k'$ (again, all we need to do is show $k'$ exists, we do not need to find it specifically).
Let's re-write this in a form that will be more helpful to us:
$(n+1)^3 + 5(n+1) = 3k'$ <---this is what we want to prove.
I suggest multiplying out the left-hand side, and see if you "recognize" some part of it (like maybe the part we use in our assumption (induction hypothesis)). What can you say about what's "left over"?