Yes, the definition of "species" poses many problems, such as the one you just described. I'm sure there's an entire wikipedia entry on "species" with all the varying definitions.
In any case, since you're trying to use math, let me give you a more realistic model of how biology works. Suppose...