Infinite = in-finite = not finite = not bounded.
Typically you see an "infinity" (in application) as a place holder for a boundary which does not exist or which we wish to leave ambiguous.
For example, in considering intervals (a,b) = {x: a<x<b}
Then "infinite intervals" are intervals lacking a boundary e.g.
{x: x<b} = {x: -infinity < x < b} = (-infinity, b).
Similarly when measuring we in actuality measure to finite resolution and with practical upper and lower limits so our set of measurable values is necessarily finite (e.g. the set of marks on your measuring tape or meter scale or the number of values your digital meter can resolve). But we wish to work with different resolutions and bounds in the same context so we invent the concept of all the rational numbers or all the real numbers to express the lack of a boundary to scale or resolution. (We also need it to consider averages of arbitrarily large numbers of measurements.)
Mathematically of course we can define anything we like as long as we are consistent and rigorous. Note there is a distinction between cardinal and ordinal infinities. (Cardinality = count, ordinality = position in a sequence)
It is instructive to see the sequence of definitions leading to the extended real numbers (the reals plus + and - infinity).
(Note what follows is one of many variations.)
First we define the whole numbers as finite
cardinal numbers (counts of elements of constructed sets). The entire set of whole numbers is defined by defining a set upon which one may iterate to yield successors and postulating that there is no largest element. (0 is in the set, given n is in the set then so is n++ = n+1).
In essence the whole numbers as a set is the container of all the nested containers of the form:
{}=0 subset of{0}=1 subset of {0,1}=2 subset of {0,1,2}=3 ... subset of N.
Next we define the negatives to get the whole set of integers. Typically you can define the integers as acts of translation on the whole numbers, i.e. as discrete vectors. With this we get the definition of addition as the composition of acts of translation.
Next we define multiplication (iterated addition) and quotients (equivalence classes of ratios) to define the rational numbers. We can order these rationals, larger to smaller and so draw a "rational number line".
To get the real numbers the classic method is to define http://en.wikipedia.org/wiki/Dedekind_cut" , pairs of sets containing together all the rationals, and segregating them so all the elements of one set is greater than each element of the other. Picture a cut in the ordered sequence of rational numbers. These cuts define the real numbers. (A Dedekind cut is "rational" if the set of larger elements has a smallest element or if the set of smaller elements has a largest element.)
Finally, noting that one can map the points on the real number line to the points on an open line segment one defines the "end points" as + and - infinity to yield the extended real number line. We however loose the normal arithmetic properties we had with the real number line. This is a nice way to express the calculus concept of a limit.
One may also map the real line to a circle so that instead of +/- infinity one has a single point at infinity (the point opposite 0 on the circle). This carries nicely into higher dimensions (see http://en.wikipedia.org/wiki/Stereographic_projection" )
In nearly all cases the infinite objects emerge as boundaries tacked on where no boundary previously existed. (biggest container of natural numbers, boundary between rationals, endpoints for the reals).