Based on personal experience, I think it can. When I was in 8th grade, I taught myself how to program in APL. I became acquainted with a wide variety of topics like matrix multiplication, matrix inverses, binomial coefficients, trig functions, modulo arithmetic, etc., because they were part of APL (which was a math-based programming language). When I took Algebra II, I was primed to learn more fully about these topics than I normally would have been. Having already given some thought to these topics, I was filling in the blanks rather than starting from scratch like my classmates.
If your goal is teach math as opposed to programming (how to implement algorithms on a computer), I would suggest you stay away from languages like C or FORTRAN. Students will waste too much time fighting with irrelevant details, like real variables vs. integer variables, and tracking down stupid bugs instead of focusing on the higher-level concepts which are presumably your focus. I'd look for a rich, interactive language rather than a compiled one. It's much more satisfying and conducive to learning to test out an idea and get an immediate result rather than having to type up a program in an editor, compile it, go back to the editor to fix the inevitable typos, compile it again, and then finally being able to run it to see what the result of your idea was.
@jedishrfu also raises an excellent point. The computer is going to do exactly what the student tells it to do. When students get an unexpected result, they'll typically want to figure out why. The ability to figure out where the error in their thinking is by checking their assumptions and paying attention to details is crucial to being able to learn effectively.
Using packages like Mathematica and Matlab can help students learn math simply because it can remove the drudgery of algebraic manipulations. It can let them focus on the concept they're trying to learn instead of on the calculations. Also, with subjects like statistics, I want to avoid forcing students to do mind-numbing calculations by hand with tons of numbers.