Yes they were - but they were religious in the sense Einstein was deeply religious, and modern mathematical physicists like Penrose are religious - they believed in a God like Spinoza's God - not the personal god of the usual religions:Weren't schrodinger, Oppenheimer, Bohr etc. very deeply religious men? I think they were, but people like to forget that.
Penrose takes it to an extreme - he believes in the literal existence of a platonic like realm where mathematical truth lies, and presumably God as well. Its very seductive, and I believed in it for a time - but abandoned it when I heard a lecture by Gell-Mann: