The so called permanent magnets eventually do wear out, i.e., their strength slowly lessens. How fast this happens depends on how it has been treated. If you heat it, or drop it, or keep it near a strong electrical current, the faster it wears out. That is why you have "keeper" of a magnet, which is nothing but a piece of ferromagnetic material placed so as to connect the poles of the magnet, and keeping the "magnetic lines of force" closed.
Random thermal motion of the molecules tend to destroy the alignment of the domains. A powerful magnet is in a highly ordered state, and the thermal motion tends to destroy that. As with every other process in nature, the entropy increases. In fact, above the Curie point of a metal, a magnet cannot be formed at all due to thermal motion of the molecules.