Thanks for the answers. I found a few different research papers about this (I'm surprised there aren't more). First, I found the paper
Realistic Animation of Fluid with Splash and Foam which (by judging from the pictures) seems to handle
splash very well, but not
oceanic whitecaps (also called just
whitecaps, or
white horses) which is foam created when the crests break because of strong wind, which is what I want to simulate. It manages this by converting water into "splash particles" on those locations where the curvature of the water exceeds a certain threshold. However, this methods requires a grid with fine resolution in order for the curvature to be able to reach that threshold, so for a coarse grid simulation like mine, this method would not work, unless the threshold was lowered drastically (which I fear would introduce new problems).
In turn, it concludes that there seems to be "few papers on handling of effects of splashes and foam with fluid", even though it mentions the paper
Rendering Natural Waters (this link is to the version published in
COMPUTER GRAPHICS forum) as one of them, which "makes crude approximations". However, this paper includes an empirical formula for the "fractional area of the wind-blown water surface that is covered by foam" f (i.e. the average area of the foam/the total area). The definition of this fractional area looks like
f = 1.59 * 10^{-5}U^{2.55}\exp[0.0861(T_w - T_a)],
where U is the wind speed, T_w is the water temperature and T_a is the air temperature. It then mentiones that "as a crude approximation to the true distribution, one can put whitecaps at positions on the surface where the amplitude of the waves is the largest". As a suggestion, the water surface would preferably first be high-pass filtered before determining at which parts of the surface "the amplitude of the waves is the largest", to prevent that some (large) regions of the surface that would happen to be a bit higher elevated than other regions would get much more whitecaps area. Or alternatively, maybe as a better approximation, one can put whitecaps at positions on the surface where the curvature of the waves is the largest. This obviously turns out to resemble the method used in the previously mentioned paper (my gosh, the method I feared would introduce new problems if it was used is back!).
Anyway, in turn, this paper mentions that the formula comes from the book
Oceanic Whitecaps: Their Role in Air-Sea Exchange Processes, edited by E. C. Monahan and G. MacNiocaill, which I haven't read, so I really can't judge of which quality this empirical formula is. Would there for example be any whitecaps area at all under a certain wind speed threshold, say at 2 or 3 m/s? I doubt there would. According to the
Beaufort scale, crests don't begin to break until there is at least a gentle breeze, which starts at about 3.4 m/s. According to this formula thogh, there would always be some whitecaps area when there is just a little amount of wind. As a remedy, one could subtract a small value f_0 = f(U=U_0, T_w-T_a=\Delta T_0) (where, tentatively, U_0 = 3.4\text{ m/s} and \Delta T_0 = 0) and create a corrected estimate f^* of the fractional area, defined as
f^* = max(f-f_0,\,0).
Of course, the parameters would have to be readjusted, since we now have a new model for how to calculate the fraction between foam area and total area.
----------
If you want to use the splash particle model anyway, there are great news for you: It looks great. For example, the following really neat video seems to use a very similar model:
https://www.youtube.com/watch?v=c3JSbhgsoGg
By looking at the next video, I managed to get another clue about a trick that is probably used to make the foam that is formed on top of the surface look better. It looks like the foam particles are attracting each other, in order to prevent them from becoming just a homogenous grey mass on the surface of the water (for example, thin lines of foam will be formed where there were previously weak broad bands). This corresponds to some inherent property of sea foam bubbles to bind to each other since it minimizes the surface energy stored in the bubble walls. Anyway, here's the video:
https://www.youtube.com/watch?v=isXNkTiiAYQ
