I'm guessing no one has answered because it isn't obvious exactly what you are looking for. I will take a swing, anyway.
From your original post it almost sounds like you want to start with calculus and work your way up to linear algebra, vector calculus, PDEs and green's functions, etc., which is a long road to travel (~3 years of math if you were taking courses!). It may sound like a copout, but for me it is most efficient to re-learn a subject by looking at the book I learned it from in the first place. Do you still have you old books? If it were me, I would find the electomagnetic theory / optics books I wanted to understand, read the appropriate sections, and when I came across math I didn't remember too well I would look it up in my old books and do whatever work required to feel comfortable with the math, then move on.
Anyway, if you want to start with the math instead, to me it sounds like you aren't looking for calculus and linear algebra books at all, but rather books on PDEs, "mathematical methods for physics", "methods of applied math", etc. I would either borrow the books, or buy used copies of old editions since they can be found crazy cheap online. For a PDE book I like "elementary applied partial differential equations" by haberman, which include Green's functions and integral transforms, but doesn't use complex analysis if I recall correctly. For vector calculus, I would be tempted to use an intermediate EM book - but if that doesn't work for you perhaps the free "mathematical tools for physics" book by Nearing (google it, you can get a free pdf from the authors website - also available as a dover edition for cheap). For vector calculus, and a whole host of other useful topics, most "advance engineering mathematics" (I like Greenberg's book, which also has a very nice treatment of linear algebra) and "mathematical methods for physics" (Arfken comes to mind, which also has TONS of material on special functions and okay treatments of Green's functions, too) may work okay.
IF you find that you need much more advanced treatments of integral transforms for applications than you find in basic treatments, I really like "functions of a complex variable" by carrier, krook and pearson, and "integral transforms and their applicaitons" by Davies. Both of these assume you know complex analysis. These are not easy books; Davies has a lot more details, since it is more specialized. If you don't know complex analysis and find that you need it, I really like te book by Saff and Snider, but the book by Ablowitz and Fokas is pretty good, too, and has the advantage that it goes into more advanced topics.
best of luck,
jason
EDIT: just remembered: "a first course in partial differential equations" by Weinberger is old-fashioned (no delta functions of notions of vector spaces, etc.) but does have nice coverage of PDEs including integral tranform techniques. Uses complex Fourier transform which often shows up in graduate level EM.