No, as long as you understand and appreciate maths I don't think there's a need for you to be particularly passionate about it. I personally do enjoy maths very much, which is helpful because I like to play with equations which sometimes leads to simplifications, or to me noticing interesting parallels between areas. But that doesn't really mean I have a great advantage over those who use it as a tool alone.
I am not very interested in computers, and I'm not very knowledgeable on the subject except for being able to use the necessary programmes adequately. It's swings and roundabouts, different people's interests give them advantages in different areas, but as long as you cover all the bases in all of them, you really don't need to have an outside interest in them.
Incidentally, whilst maths is crucial for physics, you say you want to be a neuroscientist. I have a friend with a degree in Biomedicine and her maths skills are really quite poor, she only learns things as she absolutely needs to know them and has no interest in furthering them more than necessary. So I think that as you get away from physics and more towards the biological end of the science spectrum, it becomes less and less important for you to regularly use mathematical tools in your work and studies.