Yes, it is reimann integrable. Well after a bit of thinking, i figured it out. The trick is that for any m, there are finite number of Xs such that f(x) < 1/m. So define the step functions at those points as 1, and the rest as 1/m. But you can make the step functions width arbitrarily small at the points f(x) > 1/m, so they don't contribute to the step function integral. Then obviously, the infemum goes to 0.