The assumption that every sightline should terminate on a luminous object is indeed based on the premise of a static Universe of infinite age, not just infinite spatial extent. With non-static Universe of finite age, two things make this assumption untrue
- Finite age combined with finite speed of light creates a horizon (maximum distance out to which we can see). Just because there may be infinitely-many objects now doesn't mean that light from all of them has had time to reach us.
- Finite age and changing state with time (non-static Universe) means that discrete luminous objects haven't always existed in the first place, so even if you can look outward really far, there may be nothing to see at the time you're looking back to. (But this also depends on what we mean by "luminous": see below).
EDIT: I guess the second bullet point relies on the finiteness of the speed of light as well. If light propagated instantaneously, then this effect of "lookback time" would not be an issue -- we'd always see things as they are
now.
Extending to all wavelengths, a facetious resolution to Olbers' Paradox that I sometimes like to trot out is that the sky is
indeed bright everywhere, provided you can see microwaves (or far IR, or 21 cm or...) :)
But I think there's a danger in taking this resolution at face value and just concluding that Olbers was right (no paradox -- issue resolved!). Olbers was right that every sightline should terminate on matter that is "luminous" in some band, but he was not right about the
reason. The reason is not that there have always been infinitely-many visible-light-emitting objects, and you can see all of them now. Both of those things are untrue. The actual reason is that if you look back far enough, you reach an epoch where there were no discrete objects, and instead all the matter (neutral gas or primordial plasma) was distributed nearly homogeneously (initially down to 1 part in 10^5). I think this explanation falls under the second "state change" bullet point above, and is an example of the continuous state change eventually making the "bright everywhere" assumption true again, albeit in a wavelength-dependent way.