Peer Review as Scientific Community Dialogue
By Ernesto Muñoz, PhD
The experience of being a scientific reviewer is an inherent part of our scientific education and careers. Whether it was as an Peer review in actionundergraduate student sharing criticism to a peer, as a doctoral student receiving criticisms from the dissertation panel, or as a reviewer of grant proposals for a funding agency––we all may have been in the situation of reflecting on our peers’ work and sharing those reflections as feedback. It is a process so rooted in science that any aspiring scientist should seek this experience and reflect on her or his practice as a reviewer.
Some scientists may think that the review process is plagued with shortcomings and biases. Yet, the lack of experience with the review process by some scientists perhaps raises some rumors to the level of myths (Duce et al., 2012). Thus, my goal is to provide some perspective and guidelines on the review process based on my experience as reviewer of articles and proposals, and my experience as panelist.
Peer Review: An Essential Skill for a Successful Career
By serving as a peer reviewer, scientists can have an opportunity to recognize best practices for writing a paper or proposal, develop an objective perspective on their own work in reflection to what they are reviewing, and acquire tools to become a better writer.
Keep in mind that journal editors are in need of reviewers for articles submitted to their journals, and program directors as well are in need of reviewers for proposals submitted to their programs. Scientists should, therefore, expect to be asked to serve the community as reviewers, and do so as a way of establishing a dialogue with the scientific community at large.
Invaluable resources on how to write a helpful review of a scientific article can be found in the Eos, Transactions, American Geophysical Union article by K. A. Nicholas and W. Gordon (2011) entitled “A quick guide to writing a solid peer review.” Another guide that has been helpful to me when writing my reviews has been chapter 20 of the book by David M. Schultz entitled Eloquent Science: A Practical Guide to Becoming a Better Writer, Speaker and Atmospheric Scientist.
Challenges and Goals of Proposal Reviews
The process of reviewing proposals is somewhat similar to that of reviewing articles. In addition to the scientists who wrote the proposal, referred to as Principal Investigators (PIs), proposal reviews involve essentially three groups. Program Directors (PDs) of the funding agency read the proposal and its reviews, and make recommendations based on their assessment of both. Proposals are also read by mail reviewers and/or panelists. Mail reviewers make recommendations based on their own assessment of the proposal and do not have access to other reviews of the proposal. Panel reviewers have the benefit of reading other expert reviews and discussing them with fellow panelists. PDs and panelists read many proposals and many reviews; and, a review is most helpful if it is readable and if its arguments are clearly stated and clearly parsed.
In very basic terms, the goal of a funding agency is to fund research, and the challenges are to parse out the proposals with great potential and distribute the limited funding according to the funding agency’s criteria or mission. Yet, to parse out the competitive or appropriate proposals, all proposals have to undergo the process of expert assessment, and be evaluated against a set of criteria as determined and published by the funding agency.
Imagine that you are a Program Director of a funding agency, and in a single call for proposals your program receives eighty proposals. For each of these, several reviews (let’s say five) are to be requested. Then, the PDs and panelists may have to deal with 400 reviews altogether in a single announcement of opportunity (i.e., 80 proposals and five reviews for each). Therefore, the review will be most helpful if it can ease the overwhelming task of assessing the appropriateness and competitiveness of each of the proposals.
To do this, the reviewer needs to be familiarized with the subject, be impartial and avoid conflict of interest, and limit its assessment to the evaluation criteria as defined by the funding agency. In addition, the review must be clear and readable so that the panelists and Program Directors can make sense of it as they discuss dozens of proposals in just a few days (or as they sometimes discuss each proposal and its corresponding reviews in less than 15 minutes per proposal!).
Proposal Review Tools of the Trade
- Basic information to help quickly identify the proposal
- Evaluation criteria of the funding agency
- Summary of reviewer’s understanding and evaluation of the proposal
Typically, programs of the National Science Foundation (NSF) also ask the reviewers to rate the proposals on a quality scale from “Excellent” to “Poor.” The rating should be substantiated by the comments in the evaluation.
In the first section of the review, I include basic information and the pitch of the proposal. The basic information is intended to assure me that I am uploading the corresponding electronic review for a given proposal. Given that a panelist may have to write many proposal reviews, it is important not to confuse reviews. I also include the pitch or a one-line statement of the proposed work. The pitch is intended as a reminder of the topic and the gist of the proposal amidst the hectic panel discussions. I devote considerable effort to writing the pitch, as I consider it an exercise on finding the essence of the proposal.
In the second section of the review, I address the evaluation criteria as defined by the funding agency. If the reviewer already separates the arguments by category when drafting the review (e.g., intellectual merit vs. broader impacts), the task of submitting the review is much easier given that the electronic system may require that the comments be submitted as different sections. Furthermore, separating the comments by strengths and weaknesses helps the panelists speed up the process of going over the major points of your review.
The final item is a summary of the review. As part of the summary section, I often include a paragraph of the synopsis of the proposed work followed by a paragraph of the actual summary of my evaluation. This combination of synopsis and evaluation may be helpful for panelists and PDs as they discuss the proposals. The synopsis should also help the PIs (after receiving the reviews) in assessing whether the reviewer grasped the proposal main ideas.
Toward Full Participation in Science
Peer review is an activity that gives a “voice” or representation in the science-making process and gives “life” to our participation in the scientific community. It serves as a way to legitimize a dialogue between co-authors and the community, as it transcends from an internal conversation among co-authors to a community discourse with a different cohort of scientific-literate peers.
With the guidelines on peer reviewing discussed above, it is my hope that junior scientists are encouraged to seek out peer review opportunities so that they may fully contribute to the scientific process. As a community of scientists, we must strive collectively to diversify peer review and bring fresh perspective and insight to the scientific dialogue.
About the Author
Dr. Ernesto Muñoz is an Associate Scientist in the Oceanography Section of the National Center for Atmospheric Research Earth System Laboratory (NCAR- NESL) Climate and Global Dynamics Division in Boulder, Colorado.
Photos courtesy of NIH Center for Scientific Review.
Duce, R. A., K. J. Benoit-Bird, J. Ortiz, R. A. Woodgate, P. Bontempi, M. Delaney, S. D. Gaines, S. Harper, B. Jones, and L. D. White (2012), Myths in funding ocean research at the National Science Foundation, Eos Trans. AGU, 93(51), 533.
Nicholas, K. A. and W. S. Gordon (2011), A quick guide to writing a solid peer review, Eos Trans. AGU, 92(28), 233.
Schultz, D. M. (2009), Writing a review, in Eloquent Science: A practical guide to becoming a better writer, speaker and atmospheric scientist, pp. 229-241, American Meteorological Society, Boston, MA.
Whether you’re a student presenting research & connecting with mentors or a professional giving back to the next generation, membership means taking part in something greater than yourself. With SACNAS, you're part of a community of scientists; you belong.