Hey Dremmer.
Physical things are usually defined to be tangible, measurable, and finite. If something doesn't have these properties then it is usually not considered to be physical.
Mathematics can come up with things outside this description and thus are considered outside the physical realm.
Mathematics in many ways was invented as a way to try make sense of the finite class of objects and how relations can be formed between them: in other words, the natural numbers were the first kinds of things considered because these referred to objects that had a correspondence with something physical whatever that physical thing is.
As time went on though, generalizations started to appear that considered things that were not fixed. Pythagoras and his "cult" were one group where one person discovered that you could get a square root of the hypotenuse when the other two sides were of unit 1 (of a right angled triangle) and they drowned the guy that discovered it.
Eventually this kind of thing has matured a little better with the idea of real numbers and the work started by Cantor on dealing with infinities and how they can be classified.
But the thing is that real numbers are not tangible: you can't measure things that correspond to things that are uncountable: physically, it doesn't make sense to think about such a thing: you can either measure something or you can't and measurement requires a form of quantization and this requires language.
language is important because what language does is it takes something and turns it into some descriptive capacity and more importantly, the descriptive capacity is finite. in other words, all linguistic expressions in some language have to have a fixed number of symbols and the number of symbols themselves in the alphabet must also be finite as well.
But here is the thing: the language can map infinitely many things to one finite expression and this what analysis (analysis in general, not only mathematical) does: it turns something into a description in some language that has a finite sized expression, then it breaks it up, then uses that to do analysis, then it takes the result and brings it all together.
In the entire process, all elements of the descriptive capacity (i.e. all the sentences used to describe all the information used that is broken up and put back together) are all finite.
Now to get back to a response to your question: is everything physical?
I'm only going to give my opinion and the opinion is that due to analysis, we force things to be internally categorized as physical because we need to have a finite description of something to analyze it.
If we were not able to produce a languistic version of some measurement or description that wasn't finite in some way, then it could not be analyzed. Analysis requires things to be finite: what that finite description refers to is always in the context of the language, but it is always finite in the linguistic representation sense (i.e the sentence corresponding to the actual description/measurement/thing in the language).
So whether or not there are things that don't correspond to the physical things we sense either through ourselves or other physical instruments, we will always try and quantify things which means making them finite and this means that at least in this sense, everything in some sense is always physical.
The key thing though is to look at what the language corresponds to: in mathematics we can construct things that map infinite and uncountable things to countable ones linguistically and ultimately analyzing the nature of generic languages will give a further insight into your question because everything is descriptive.