What do you think you would like to study in grad school? It would make sense to make this discipline one of your majors, but if you decided you wanted to study physics with a physics minor and a math major, you still probably present a compelling application. It sounds like you're leaning toward CS too (natural language processing, AI?)
One of my friends has a degree in biophysics with minors in Chinese and Philosophy - his undergrad degree took six years. I have an almost-major in english lit. I can't say that my knowledge of Shakespeare comes in handy as a physics researcher (although I am known as the Office Grammar Person). On the other hand, I really appreciate the English coursework that I did. English was a lot different from physics, I read some awesome books and I was introduced to some really head-spinning ideas that I would never have encountered in a physics class. I also had some really good professors and I met a lot of people I would never have met in physics or math.
If you feel you can afford the time and the tuition, go for it! It also might be worth investigating whether the courses required for a minor are really courses that you're interested in. Otherwise, there's nothing stopping you from just taking the courses that you're keen on.