The specifics of your undergraduate research experience tend not to matter that much. People on admissions committees are well aware that not everyone will know precisely what they want to do in graduate school as an undergraduates, nor will they necessarily have the opportunity to get involved with the specific sub-field of their interest if they do. It will look a lot better to do well in the opportunities that you have
And in your case, it's not like the research work is completely unrelated to the kind of experience you would be getting doing research in a physics lab. If your work leads to something like a publication for example, that's going to look great.
The other thing to keep in mind is that if you have something that's working well for you - i.e. you're feel like you're learning a lot, you like the people you're working with, you enjoy the work, etc. then it's generally a good idea to keep going with a good thing.
On the other hand, if your current position isn't teaching you much, you feel like you're just turning the crank, you don't like the people you're working with, then exploring other options might be a good idea. Another good reason to change is if you really have a desire to explore another opportunity. So if there's a physics professor who is doing something a lot closer to what you envision yourself doing for graduate school and really want to try that - it's okay to change. Undergraduate years are a time for exploration.