Washers serve multiple purposes - as spacers, to distribute the load across the material, to keep nuts tight (lock washers), etc.
But the torque is primarily determined by the threads and their contact. I don't see that the washer would affect this, but for special applications you could test this - such as structural bolts.
But for ordinary things you can just follow the recommended torque tables.
I could only find a PDF with a table that contains M4, but it does including the bolt grades.
http://www.boltsupply.com/catalogue_files/catalog_data/toc_data/pdf/019.pdf
I take it you know what grade the bolts are that you are asking about? (8,8, 10.9 etc).