Hmm, I suppose there will be two types of books available, methodology books and context books. Personally, I would avoid the methodology books because if you are trying to learn to think critically, you shouldn't be expected to swallow a methodology without critically thinking about it.
Books that tell you "ask this, consider that" are probably not all that worthwhile. However, there is one methodology book that I have read which I thought was really good, however I won't adopt every suggestion or even most of them that it made, and that's the point. This book is "How to read a book" by MJ Adler et al.
Part of what makes that book worthwhile is that it is extremely verbose, I found myself becoming impatient and skipping over much text, to the point that I was reading only the first sentence of each paragraph until something interesting came along. This is very good and part of why I liked it.
Also I would consider books about logic, but there are logic tutorials online so you probably wouldn't need to buy a book for that. Or perhaps look for one of the informal ones by Smullyan or someone similar, if you prefer that to formal stuff. I would avoid books which specifically talk about recognising invalid arguments, or give you a list of fallacies with odd names. Or perhaps read them later, when you can already recognise flawed arguments and want to know the traditional names like red herring and non sequitir.
Then I would consider what I am calling 'context books', which would be books that stretch the mind and provide good context about which one can think critically. I'll only recommend books which I have read and helped me, rather than recommend stuff I haven't read.
My suggestions for context books are:
Enchiridion - Epictetus (online)
The second sex - de Beauvoir (selected chapters online)
Psychology of programming - Weinberg (really good book if you are technically inclined)
The moral animal - Robert Wright (about evolution, some philosophical stuff)
It's difficult to say why these books are good. I think it is because they don't try to sell you anything, you get to decide if you think what they say is rubbish or not, or in what way it applies.
Of course, you must think critically about what I have said here and decide if you agree or not. Don't let me stop you from buying whatever books you feel are suitable.