In fact, the things I experience are the only reason I can (perhaps erroneously, but I think not) infer there is a physical world. That is, for me, qualia, have a better claim to being "real" than the inferred physical world. I experience them directly, I do not infer their existance.