Again, this is just your opinion. There are many expressions in common usage that don't withstand literal logical scrutiny, but we use them anyway. Even the phrasing you prefer, "I couldn't care less", is strictly illogical because it's almost vanishingly unlikely that the speaker is being accurate about caring less about the topic of conversation than *anything else* in the entire Universe. For one thing, there are many things that the speaker is actually not consciously considering at the time. It's easy to make the argument that by expressing a strong opinion, even a disdainful one, about the subject under discussion, the speaker is showing he cares more about the subject than all those things he's not consciously thinking about right then.
So the statement is inaccurate if taken literally. But we accept it because it's just another trope - in this case, it's obvious hyperbole.
Now which speaker sounds silly?