It just refers to the fact that it's a carbonate ion with an H+, essentially a proton, attached to it. I'm not sure where 'bi' comes from, but that's the reason why bicarbonate is also sometimes called hydrogen carbonate.
I think it predates the modern understanding of molecules, and was because a bicarbonate contains twice as much carbon (ie it produced twice as much CO2) as the carbonate of the same metal.
eg NaHCO3 contains twice as much C per Na as does Na2CO3 …
since the latter is obviously a carbonate, the former must be a bicarbonate!