What it takes is for light to go through stars so that the brightness adds instead of stars blocking light from other stars (or, for the temperature to increase without bound due to all of the extra radiation bouncing around). The standard setups of Olbers' paradox result in a universe where every line of sight ends on the surface of a star and therefore the brightness is that of the surface of a star.Well, on second thought, maybe it takes too specific configuration to get infinitely bright everywhere.
Yes, it can make for a brighter universe depending on the assumptions made about the large scale dynamics of the universe (infinite or finite, expanding or not, etc.).But still, it makes sense that at the very least, it would be much brighter if speed of light is infinite, which was essentially my main point in the OP. I mean, infinite light speed implies we would be getting light from far away stars that we wouldn't have otherwise on a regular basis. This high brightness should be the case in a finite universe or an infinite eternal universe when there is no speed limit.