When I am investigating systems for infections, I use Process Explorer(sysinterals), autoruns(sysinterals) and hijackthis.
Here's the basic steps i follow, i'll start from the very beginning so i apologise if some of this stuff seems obvious.
1) Reboot to safe mode with networking
2) Log into an admin account
3) Run autoruns as admin
The purpose of autoruns is to examine what software is scheduled to run on your system at boot/login
4) Run process explorer as admin
The purpose of process explorer is to examine what software is currently running in active memory
5) In Autoruns
Click options>Scan Options
Check all the boxes and then agree to virustotal user agreement
Then Click options>check all 4 hide items
Then Click the refresh button (2nd from the left next to the save icon)
6) While autoruns is gathering data, run process explorer as admin
When its open click CTRL+D to show DLL's attached to processes running on your system. The DLL's will show up on the bottom pane. Please sort them by Company Name.
Then Click on the Process menu item, and then select Check Virustotal.com
Now comes the painful part.
You will need to click on each process on the top pane, verify that it's location is valid. You will search for the file in google, learn about what it does, where it resides, who made and signed the file, etc.
Eg: svchost.exe must located at c:\windows\system32\svchost.exe and must be signed by Microsoft Corporation<-- This is safe (assuming virustotal also says it's safe)
if instead you see scvhost.exe located at c:\windows\system32\scvhost.exe and not signed <-- This is malicious. The file name is incorrect, the c and v letters are flipped and there's no signature from microsoft saying this is their file.
if the file is located in c:\windows\system, it is malicious, etc.
If the file passes the initial check, then you need to look at the list of ALL DLLs attached to the process in the bottom pane. This is why we sorted by company name. All the digitally signed files (files you can trust) will be lumped together. Microsoft Corporation is okay. Everything else must be checked and verified as not malicious. This means you will have to google the filename and learn about it to find out if this is a legitimate file and can be trusted or not.
Over time, you will learn patters and will be able to figure out what's safe and what's not just by looking at the file name and location. But when you start, you will have to do this hard dirty work. No pain, no gain.
If you find something you think is malicious, right click on the parent process and then select suspend. This will stop the process from running and give you an opportunity to clean/remove it.
If you accidentally suspend a system process, you will crash windows. There are certain processes you cannot suspend. System, winint, etc are things that your computer cannot run without. There's others that i don't remember and i don't have access to process explorer right now to give you a list (i'm run Debian)
Once suspended, you will have to navigate to the file in question and then change the permissions on the file from allow to deny all. This will prevent the file from running on your system after a reboot.
The really good malware programs have several threads running that monitor each other, so if you kill one thread, the others simply restart it. This is why we suspend and change to deny permissions on all the files one at a time. Then do a hard reset and then on next boot, they cannot run and your computer is clean.
If you mess up and set deny permissions on a valid system file, you will kill windows. Unless you keep a record of your changes so you can undo them, you will end up having to wipe your computer to fix the problem. So please be careful.
When you have gone through the entire list and suspended everything that you think is malicious, go ahead and kill the threads one by one until none of the suspected malicious software is running.
7) Once you are done with Process explorer, you will switch to autoruns. You will need to go through each tab at the top, Logon, IE, Explorer, Services, etc and check the entries that were not hidden (files listed are not windows, and virustotal thinks they are suspicious)
Then just like in process explorer, you will unleash the power of google on those files to figure out what they are and what they do. if the files are safe (virustotal spits out false positives) then ignore them. If they are malicious, uncheck the entry to disable the autorun on the file and then navigate to the location of the file and change the permissions on the file to deny.
Once you are done with the entire list, close process explorer and autoruns and then click and hold th power button till the computer shuts off.
Some malware, spawn with random names and locations as part of the windows shutdown process. A hard shutdown prevents this.
8) Reboot back into safemode with networking. Then run highjackthis as administrator and redo the same thing, check each entry, verify it's okay, if it's not, uncheck it. Then reboot again.
9) Finally, boot back into normal mode and if you haven't completely destroyed windows by now, you can be reasonably assured that the system is clean. The only exception is root kits as they filter information about themselves before it reaches the Windows API, meaning you will not see them in process explorer, autoruns, highjackthis. The only way to get rid of them and be sure about it, is to wipe the system.
..←... "... and if you haven't completely destroyed windows by now ..." ... lol