How do I know that my literature search is comprehensive?

[LittleMrsMonkey]

Say I want to do molecular dynamics simulations on a crystal of a specific substance. I get on Scopus and search for it, say going back a couple of years with the results, and then I also check the references in these papers. How can I be kind of sure that this background research is adequate?

 

[mfb]

Ask an expert. Better: Ask several experts.

 

[Choppy]

I don't think you can ever be completely sure that you've covered everything - even if you're an expert in the field.

Usually what happens is that you start working with a mentor or supervisor who has enough experience in the field to make sure that you're seeing all the most important and relevant papers. That's the starting point. As a graduate student, you read - a lot. There should probably be a couple of journals that you read on a fairly regular basis. The point is that you're not reading stuff that's just related to your project, but as much as you can of the material that's at arm's length to it as well. These days you can set up alerts to be notified every few days of papers that are published containing keywords that are of interest to you.

One good thing that really helps is that once you've found a few good papers in your field, they all tend to start out with a mini review in the introduction. Read these and the references. Soon, you'll probably find that everyone in a certain area is citing a particular handful of papers. These are the important ones. You can also use services like Google Scholar to track how many citations a particular paper has. The general rule is that more citations tends to mean the paper was more important (though, not always).

 

[mfb]

One good thing that really helps is that once you've found a few good papers in your field, they all tend to start out with a mini review in the introduction. Read these and the references. Soon, you'll probably find that everyone in a certain area is citing a particular handful of papers. These are the important ones. You can also use services like Google Scholar to track how many citations a particular paper has. The general rule is that more citations tends to mean the paper was more important (though, not always).

You can also go in the opposite direction. Once you identified the key papers, check the papers that cite these papers.

 

[Dr. Courtney]

Literature searches are never really comprehensive. We almost always find more references and important papers when we sit down to write the paper than we found earlier when doing the background and initially designing the study. Often, even more references are pointed out by the peer reviewers and after publication. A few years later, after we've read all the papers that CITE our published paper, there are even more references that come to our attention in the bibliographies of later papers that cited ours.

Our goal is a high level of due diligence rather than a truly comprehensive literature search. Even though our literature searches are imperfect, we have gained an excellent reputation of being better at it than many colleagues in our fields. Some tips:

1. Google scholar is the first search engine we try.
2. We use variations on the expected search terms.
3. We circle back around at different points in the background work. Being in a different mood or different frame of mind leads to seeing the results differently.
4. Multiple collaborators repeat the process. Colleagues will find important papers I miss, and vice versa. Email the links of nuggets to the collaborators.
5. When we've found a handful of valuable references (say 5-10), we search forward and backward from there using several methods:
5a. Look over the reference lists in those papers. Read all the abstracts.
5b. Look over all the papers that later cited those papers. Read the abstracts.
5c. Find the CVs of the most important authors of those papers, and read the abstracts of the related papers they have published. Figure out how to get the full publication records of the most important authors.

5a, 5b, and 5c is probably what sets our literature searches apart from others.

And we circle back around and repeat steps 1-5 when we go from the study design (writing the proposal, etc.) to writing the introduction and background for the paper. And we do it yet again when writing the discussion section of our paper. We almost always find new pearls in the literature at each step of the process.