The first thing to realize is this:
"infinity" is a noun.
"infinite" is an adjective.
I think one of the problems the layman has is that whenever people talk about things that are "infinite", the layman mentally substitutes "infinity" and gets all confused.
This is unfortunate, because "infinite" is much more common than "infinity".
The only common use of infinity in mathematics is topological. One often likes to construct new topological spaces from old topological spaces by adding additional points. For example, suppose you're working with the open interval, (0, 1). One day, you grow weary of the fact your topological space has "open ends", so you decide to add two additional points to your space, giving you the closed interval [0, 1].
We can do a similar thing with the real line. It has two "open ends". So, as we sometimes like to do, we can add two "endpoints" to the real line. (In fact, in topological terms, this process is virtually the same as adding the endpoints to the aforementioned open interval) Because these points are "farther away" from zero than any other point we originally had, mathematicians opted to call them "points at infinity". In particular, we name them -∞ and +∞. Topologically speaking, we have this perfect analogy:
0 and 1 are to (0, 1) as -∞ and +∞ are to the real line
Incidentally, sometimes we only want to add a single point, but it's at both ends of the real line. We can intuit that we looped the real line into a "circle", because we "go off" towards infinity in one direction, and "come back" from the other side. This intuition is reasonable, because this situation is perfectly analogous to starting with a circle missing a point, then adding that point back. In this case, the "one-point compactification" of our topological space, we simply call the point ∞.
(Incidentally, there are reasonable ways to add much more than two points to the real line, but it becomes more complicated to describe)
This becomes more involved in higher dimensions. There are several reasonable ways to add "points at infinity" to the Euclidean plane. For example, we can add a "circle at infinity" to turn the plane into something that acts like a disk. We can add a "line at infinity" to get something extremely geometrically useful, called the projective plane. We can also add a single point at infinity, which makes it just like the surface of a sphere.
(And we can do other, much uglier, things)
Enough about the noun, let's get to the adjective. Infinite simply means "not finite".
In one context, (we use "infinite" in many circumstances) we would say "the universe is finite" would mean that there is an upper bound to the "distance" between two points.
So, saying "the universe is infinite" would mean that there is no upper bound to the "distance" between two points.