One wants to be very careful of saying things like this out loud...
The only appropriate response is that attractiveness was very important in your selection process.
Unless you want to sleep with the dog.
The other day, I was joking around with a workmate about "size matters" (we were originally talking about cubicle space but of course the double entendre was obvious).
She said that, really, size doesn't matter. I said "no, no the only correct thing for you to say is that size does
matter." The confusion on her face prompted me to elaborate: "If you say size doesn't matter to you, then I know something about your husband that I don't need to know." She said she'd never thought of it that way.