And for the record, non-minimal couplings do exist and pop up when you consider effective interactions between composite charged objects and the gauge field.
An example is the interaction of the neutron with the eletromagnetic field: the neutron has zero electrical charge, but non-zero magnetic moment and therefore still couples to the electromagnetic field. This coupling is non-minimal with respect to some effective neutron-field.
A similar idea applies to zero-charged mesons (i.e. the neutral pi particle)
In the end, this non-minimal coupling arises because the neutron is made up out of three quarks who do couple minimally to the electromagnetic field.