gnurf has hit on all the major points. Anyplace you can think of with resistive loss is something to look at. You are right, the higher the voltages the less effect a diode drop has on overall energy efficiency.
Low ESR capacitors are good to have. It's good to select the right transformer core material.
The lower the frequency the less crossover (vi-product) in the switching transistors, less transformer loss and less capacitor loss.
Cut the frequency in half and the current in and out of the capacitor occurs half as often. So as gnurf suggested, the capacitor has to double in capacitance to have the same output ripple.
For the transformer you have both core and copper losses. For half the frequency you get about half the core loss, but you now need a bigger core to add more windings to keep the peak flux density the same. In opposition, core losses are proportional to the volume of core material.
Selecting the best core geometry, size and material meeting price and efficiency constraints can be a consuming process. If volume price is not a constraint for a one-off (you just want to make one), this greatly simplifies things.
Also, core size effects winding window area (more area is better) but also the length of a wind (longer wind length = not better).
Copper losses: Doubling the frequency means the wind count goes down by a factor of 1/sqrt(2) to obtain the same peak flux [Is my factor correct??], decreasing copper loss. But increasing frequency also means increased skin effect which offsets some of the gain in energy efficiency.
I'm not familiar how much litz wire or copper tape can help with reducing skin effect losses.