Junior researchers frequently wonder how many references should be included in their research papers. The common response? “As many as you need.” What exactly does that mean? While we admit there are very few hard-set rules regarding this issue, in this article, we will try to provide more concrete guidelines that will help you assess whether you have enough references in your paper.
Before we do so, let us briefly explain why references matter and whether the number of references you include can project certain perceptions about the quality of your work. There is such a thing as having too many or too few.
Why references are necessary
References show that you have carefully reviewed the relevant literature and are now contributing something novel to the academic community. You establish authority and credibility when you can critically assess other literature and distinguish your findings from previous works (if any exist). We emphasize “critically assess” in the last sentence because references are only as good as you apply them to your research. Therefore, the famous adage “quality over quantity” is the key to deciding how many references are sufficient.
Quantity can matter due to perceptions
We would be remiss if we didn’t tell you that being at either extreme (having too few or too many references) can reflect poorly on your intellectual aptitude and your study’s validity. Here’s why.
- If you don’t have enough references, particularly on a topic familiar to a wide audience, readers may think that you haven’t done enough research into existing literature. Surely someone else has thought about related topics or used similar techniques. If you’re sloppy in conducting your diligence, readers will wonder whether your paper is worth reading. What’s novel and valuable about your paper? Were you just as sloppy with conducting your study? The answers to these questions need to be evident. Additionally, readers might be concerned that you may have plagiarized by failing to properly cite information. Unless you’re John Nash, who cited only two texts in his seminal 26-page PhD thesis (one of which was to his prior work), ensure that you’ve properly researched the relevant papers and included appropriate citations!
- If you have too many references, readers may wonder if you did any original research at all. Unless you’re writing a literature review, your paper’s primary focus should be on your investigation and findings. Don’t bury your hard work under strings of citations and discussion regarding other works. Show your readers what you’ve discovered and how the new information you present fits into or departs from the academic community’s current understanding of your topic.
Additionally, let us highlight the difference between the number of references versus citations. References are the source materials; therefore, each reference should be listed only once in your references section. Citations are meant to identify the source of the information you use in your paper. You can cite a reference multiple times. Therefore, the number of citations you have is typically larger than the number of references. The opposite situation should never happen!
Key factors influencing the number of references you use
The following are some of the many factors that may influence the number of references you use:
- The number of references required for a paper will depend largely on your work’s purpose. For example, literature and systematic reviews are surveys of existing studies. Therefore, their reference lists will be more exhaustive than those of research papers whose primary focus is the current authors’ findings. Indeed, if you examine many journals’ author guidelines, you’ll note that journals have a higher maximum reference limit for review articles than original research papers.
- The length of your reference list will also depend on your researchpaper’s subject matter. For example, if you are writing about a less studied field, such as a subfield of neuroparasitology, you may discover that there aren’t many papers to cite. Similarly, newer fields will have fewer published papers that can be referenced. If you find yourself in this situation, review the references used by relevant current literature and see if you can expand your research, and thus your reference list, with valuable content from there.
- Another factor will be your institution or journal’s requirements. If you are preparing a dissertation or thesis, double-check your department’s requirements. While rare, they may have specific limits. More commonly, journals restrict the number of references due to printing constraints.
- It may happen that you don’t have access to certain literature that could have served as a reference. In such a situation, you may wish to look for an institution that may be able to provide you access to that literature for the purposes of reviewing the content.
- Given that more papers are being published than ever before in most fields, it is likely that reference lists will grow longer simply because there are more data points and discussions available to cite. Keep track of changes to the size of reference lists in publications related to your field.
- Finally, a paper’s length bears some correlation to the number of references.
The “right” number of references
Below, we provide tips on how to decide if you have enough resources. We also provide some general reminders on how to effectively use references. After all, references are meant to enhance your paper while still maintaining your research as the focal point.
Let journals be your guides
- One way to gauge how many references you should have is to survey academic journals for your article type in your field. Review their author guidelines for limits on the number of references for your article type, and make sure your reference list complies with those journal restrictions.
- Read recent articles relevant to your topic; check how many references other authors have included in their papers for the same article type as yours, and how frequently those works were cited per page.
- Keep in mind that the above methods will give you an estimate of how many references you should include but will not tell you how many citations you’ll need per page. The latter is impossible to state simply because certain sections may have no citations at all (the results section, for example).
Statistics regarding the number of references and citations
To give you a general idea, the following are some estimates from a couple of studies that examined the citation characteristics of articles published in various disciplines.
According to Milojević’s study encompassing research in the fields of astronomy, mathematics, robotics, ecology, and economics, the highest and average number of references per article page were as follows:
- Ecology: highest, ~58; average reference per page, 6;
- Math and robotics: highest, ~28; average reference per page, <1; and
- Economics: highest, ~ 32; average reference per page, >1 but <2.
The above findings were based on data compiled from the first 20 years of the author’s research. Since then some fields have increased the number of references. Thus, make sure to examine your target journal’s most recent and relevant publications for a better idea of how many references to include based on the specific type of article you plan to write.
In another study by Falagas et al. (2013), medical journals averaged 29 references for articles that were 7.88 pages long (as printed in journals).
Finally, although the sample size was small (63 journals), Gali Halevi observed the following citation trends of a broader range of disciplines.
- The average number of references per article was the highest for the social sciences, physics, and astronomy, and arts & humanities (roughly 54 references per article).
- On the other hand, health professions and earth and planetary sciences had the fewest references per article at an average of 8 and 17 references, respectively.
- Math and engineering averaged at roughly 29 references per article.
- Biochemistry, genetics and molecular and other biological sciences averaged at 51.
- Hard and natural sciences more frequently cited recent literature while social sciences and math were likely to include older sources.
Note that the Halevi study is limited in size, fails to factor in article type and does little to account for variances across different fields and journals. For example, it is possible that more review articles could have been reviewed for certain fields than others. With that said, we provide the above information to provide a rough estimate.
At the end of the day, please keep in mind the requirements of your institution or target journal and the general trends for your specific article type (by examining the most recent relevant publications).
For additional information regarding journal restrictions on the number of references, click here.
Some dos and don’ts of using references
- Don’t repeat references within a reference list.
- Don’t repeatedly cite yourself. Make sure to balance your discussion with external literature citations.
- Be careful about citing old references. The rule of thumb is to go back at most five to six years. Exceptions to this rule should be reserved for “seminal” works relevant to explaining what prompted your research. Roughly 85% of all cited works should be less than five years old.
- Be careful not to cite several references in one place without discussing the relevance of each work to your research. In other words, don’t say, “We referred to previous studies in this field (1-7)” unless you later explain how each of reference #s 1-7 apply to your discussion.
- Confirm the quality of the work you cite. Are there any ethical issues regarding the paper that would disqualify it as a good source? Do your references come from reputable sources such as respected journals rather than random blogs and website links? Remember that your analysis is only as good as the verifiable information you use to conduct your research.
- One of the main purposes of citing existing literature is to show the “knowledge gap” regarding your topic. Therefore, make sure the works you reference naturally lead readers to wonder about the research question you address in your paper. To explain further, think about your favorite fictional story. A successfully written story only reveals the background information needed for the reader to follow along in the story. You’ll rarely see an author waste time writing about how the main character stubbed his toe one day while going to work unless that event relates to an important aspect of the story. Similarly, the references you cite should support the story building you create in your research paper.
- Don’t completely ignore the paper that could disprove your hypothesis. You want to show objectivity and that you took a balanced and unbiased approach to conducting your research. Mention the potentially conflicting evidence and explain why you believe it is flawed or inapplicable to your research.
- In qualitative research papers, you may have fewer references.
- Anything you cite in your paper should be listed in the references section. Anything listed as a reference should have been quoted or paraphrased in the text. If either rule is violated, something is wrong.
- Finally, remember that a paper will typically have more citations in the Introduction and Discussion sections than in other parts.
- Stefanie Haustein. Chapter 2 of Multidimensional Journal Evaluation: Analyzing Scientific Periodicals Beyond the Impact Factor. (De Gruyter Saur, 2012).
- Gali Halevi, “Citation characteristics in the Arts & Humanities.” [Note this paper involved a limited sample size and did not factor in article type.]
- Staša Milojević, “How are academic age, productivity and collaboration related to citing behavior of researchers?”
- Falagas, Matthew E., Zarkali, Angeliki, Karageorgopoulos, Drosos E., Bardakas, Vangelis, and Mavros, Michael N., “The Impact of Article Length on the Number of Future Citations: A Bibliometric Analysis of General Medicine Journals.”
The first question I posed in the title of my blog post is one that all of my students (undergraduate and graduate) and most of my research assistants ask me: how do I know when I’ve read enough for a literature review? The answer is never clear cut, unfortunately. I am someone who loves reading, and who needs to read broadly because his own work is interdisciplinary. I’m also quite systematic in how I read, and I prefer to err on the side of having TOO MANY sources rather than be accused of not knowing the field.
I also realize not everyone has the time to be on top of the literature, particularly with teaching, service, family and caring and research obligations. I am going to answer the top six questions I get asked on breadth and depth of literature reviews, and in doing so, I am going to suggest a few shortcuts that may help narrowing a literature review search and finding the “sweet spot” where you’ve read enough that you feel confident enough to start writing your paper, chapter or thesis.
1. How many sources should I read for my literature review?
This is an absurd question that is prompted by arbitrarily setting a random number of sources as “enough”. If you read the right five sources, you’ve probably covered a full field. But if you read 40 sources that all tend to pull in different directions, you’ll still be unable to cover all the sources.
Here’s my totally non-scientific take for coursework-related materials: a final research paper should at least use 13 additional sources to those in the syllabus (one additional paper per week) for an undergraduate class, and an in-depth literature review for a graduate course should be in the realm of 26 (2 additional papers per week) to 39 (3 additional papers per week for PhD students). If somebody writes a final paper for my courses that only use the readings we did during the semester, it shows they didn’t go any further and I’ll probably penalize them.
2. Where do I get sources for my literature review if I am starting up a new topic? Well, here are a couple of strategies:
- Read literature summaries and reviews published in journals.
There’s plenty of journals now that provide reviews of the literature. Three I’m well aware of are WIRES (Wiley Interdisciplinary Reviews) and Geography Compass, as well as Progress in Human Geography. When reading Progress in Human Geography, you probably want to look for the “Progress Reports”.
- Search for “a review of the literature” or “a meta-review” type of articles (either Google Scholar or other online databases). These articles will usually provide a pretty extensive range of sources. Given their goal and scope, they’re also probably comprehensive enough. Meta-reviews tend to be more synthetic and provide a research agenda and specific research questions that need to be looked at in future research.
- Ask specialists (or look for their work) for key citations from where you can do citation tracing. For example, if I were to do a search on water ethics, I would ask Christiana Peppard, Jeremy Schmidt, Lucy Rodina for suggestions. Or I would look for articles citing them. If I were to do something on climate politics, I’d look for Kathryn Harrison, Sarah Burch, Max Boykoff, Mat Patterson.
3. “When should I stop reading and start writing?”
My answer to that question is: you should be reading AND writing. Apparently, a lot of people feel like they need to Read All The Things before they can write a literature review. That’s why I always suggest that when you process a reading (a PDF or a printed source), you should generate at the very minimum a row entry in your Excel conceptual synthesis, and a synthetic note (or a rhetorical precis). Obviously, you gain a lot more if you write a full-fledged memo, but you may want to wait to write the memo up until you’ve read a few sources. But you should ALWAYS be writing as you read. You may not assemble the full literature review, but at least you can start with an annotated bibliography.
4. How do I know when to stop reading/researching/seeking more sources?
This is again a very tough question. Having mapped a very broad survey of the literature on informal waste picking, I can assure you that I felt I could not stop even after reading 50 articles. There is just simply too much published. But one of the reasons why I encourage my students to stop when they reach conceptual saturation (e.g. when they start seeing the same themes repeated over and over again) is because I don’t think you gain too much, marginally, from reading yet another paper on the same topic but using a different case study.
For example, I recently wrote a series of memorandums on the urban commons. I had basically mapped the entire body of works on urban commons using the first 10 citations I found on Google Scholar. However, I wanted to see how much more I could go in depth on the topic. What I found was that there were many case studies, but all using the same conceptual framework. So that’s when I stopped. When I saw that basically every other paper was a variation of the same central 10 ones, but using different case studies. I added those sources to my bibliography, but I didn’t need to incorporate them to my literature review.
Another way to respond to this question is: read enough to answer your questions properly.
The two biggest questions that probably would encompass the previous ones are related to breadth and depth.
5. How far reaching should your literature review be?
Scoping a literature review, as Dr. Pat Thomson shows here, is not an easy task. It requires us to search through many months or even years of published literature. I always do Google Scholar searches at least 7 years into the past (e.g. 2010 articles would totally be welcome, as would be books published in 2010) because of the very long lag-time that exists between submission, acceptance and publication. My citation tracing process also looks at the last 7 years of scholarship of key authors.
Then you have the other associated question – what about the “seminal” (I prefer the word fundamental) articles or books?
For me, this is the most challenging component. When I know a field very well (for example, agenda-setting theory in public policy), I can easily decide which authors I will be seeking (Stuart Soroka, Michael Howlett, Baumgartner and Jones, Kingdon). If I am doing policy design, I’ll go with Helen Ingram, Ann Schneider, etc. And then based on doing a citation tracing exercise, I will go to those younger scholars who are citing these key authors. But again, this requires you to know the field already.
This is where a supervisor, a coauthor, a colleague or a trusted scholar on Twitter may be helpful with narrowing the search scope. You can ask “who are the key authors I should be reading on Topic A” or “which are the key citations I should be looking at to get a grasp of Field B“. And then use those authors to create a map of the literature.
And the last question, which just about everyone asks me:
6. Do I need to do an in-depth reading of All The Things?
This is completely a question that has arbitrary answers and a broad range of parameters to work around. It also depends on what type of literature review you are writing. If you are, for example, preparing your doctoral comprehensive examinations, you DO want to read EVERYTHING and do so IN DEPTH. You need to demonstrate that you know your field of study, broadly and deeply.
However, if you are writing a literature review, for example, of agenda-setting theory and its applications to health policy, you may want to read in depth 5-10 articles on health policy, 5-7 articles on agenda-setting theory, and then start writing from there. Again, in-depth reading is correlated with the extent and degree to which you need to demonstrate that you know a field.
My method, as most people may have noticed, is usually as follows:
- I read 5-10 citations that I find key. This reading is usually in-depth.
- I create the set of questions I want to answer. I choose 3-5 citations around each question. All the reading associated with these questions I do using the AIC method, or skimming and scribbling unless I find key ones that need to be read in depth.
- I write a memorandum for each one of the questions I’m trying to answer. In this memorandum, I assemble a mini-literature review that answers the question.
- I fuse all the memorandums into a larger document where I have mapped out how each question (and answer) relates to the overall topic.
- I read my entire literature review and restart with citation tracing until I reach concept saturation.
Hopefully this post will help those who are struggling with literature reviews, as the summer approaches! You may also want to revisit my Literature Review posts.
Posted in academia, research.
Tagged with literature review.
ByRaul Pacheco-Vega–June 17, 2017