Mathematics is often called the universal language, so the question is nonsensical as stated. Most animals have been shown to be capable of counting up to five. This is thought to be a hardwired phenomenon related to the neural networks in animal brains. Whether or not this constitutes "mathematics" is another issue. The capacity for more complex and abstract mathematics and languages evidently comes after the ability to emote and reason.
Well, we can prove a weaker fact, that mathematics is prior to writing. There exist tallies scratched on bone from the paleolithic era, some of which may be calendars. So people were counting back then.
Wait.. we can prove that mathematics is prior to writing based on ancient mathematical writings?
Anyway, it's clear that we are born with very basic mathematical concepts such as counting and addition. Even the counting is very limited, as evidence has been found (IIRC) of primitive cultures whose number systems consisted of "1," "2," "3," and "many."
In modern cognitive theories, language development is typically associated with the ability to form high-level abstractions. This is backed up by evidence-- for example, they ran an experiment where chimps were rewarded if they could determine if a pair of objects exhibited the relationship of "sameness" or "difference." One group of chimps was given tokens to represent this relationship-- the chimps would label the pair with a green token if the objects were the same, red if different-- and the other group of chimps did not use tokens. (The tokens basically function like words-- they are labels used to represent abstract concepts.) It turns out that both groups of chimps did equally well in this initial experiment. But the experimenters then ran a higher-level form of the experiment, where the chimps would have to determine if the relationships between 2 pairs of objects were the same or different. (For instance, a pair consisting of two bananas and a pair consisting of two shoes would have higher-order sameness [both pairs have the same internal relationship: sameness], as would a pair consisting of a shoe and a crayon and another pair consisting of a ball and a stick [both pairs have the same internal relationship: difference]; whereas a pair consisting of two bananas and a pair consisting of a shoe and a crayon would have higher-order difference.) This time, the chimps with the tokens were able to understand the more abstract task whereas the chimps without the tokens were clueless. The explanation for this outcome is really elegant and goes a long way towards demonstrating exactly how intimately our capacity for intelligent abstract thought is bound up with our capacity for language. It goes like this: the chimps who had been using the tokens naturally created a simple mental association-- same = green, different = red. So what happens when they are presented with pair A consisting of a banana and a ball, and pair B consisting of a stick and a crayon? They recognize that the objects in A are different, and mentally label A with a red token. Same for B. So the abstract higher-level problem involving pairs of pairs has been reduced to one pair comparison: compare red token to red token. The two tokens are the same, so the system exhibits higher-level sameness and is labelled with a green token.
So anyway, language clearly seems to play a major role in capacity for abstract thought. Since mathematics is entirely abstract thought, I would say it's pretty conclusive that language needs to be developed to a sufficient degree before an understanding of mathematics (beyond basic counting, addition, and subtraction) becomes possible.