Refractive index depends on polarizability, which is the dipole moment produced per unit volume, per unit of applied electric field. It depends mainly on the number of molecules per unit volume as the oxygen atoms are by far the most polarizable.
I'm not familiar with the nuances of deuterium oxide of the top of my head, but it is entirely possible that the density could be higher than that of water (owing to the greater molecular mass of D2O), while having a smaller number of molecules per unit volume.
For reference, look up the Lorentz-Lorenz formula.
Claude.