Key Findings from our Multi-year Study
We wanted grant reviews as excellent as our applicants.
So, we took a closer look at our process and introduced a blinding requirement to the letter of intent for Beckman Young Investigator program applicants. Over an eight-year period, we reviewed 2,291 applications. In 2020, at the midpoint of the eight-year period, the blinding requirement was introduced, omitting name, gender, gender-identifying pronouns, and institutional information from early-stage review.
Key Findings from the Study:
- Distribution of applicants invited to the full application phase shifted from “prestigious institutions” to other institutions outside the group with the new policy.
- Trending shift carried through to final program awards.
- Pre-blinding, 75 percent of BYI awards went to applicants from Top 25 institutions.
- After blinding, 45 percent of BYI awards went to applicants from Top 25 institutions.
- Reviewers reported blinding facilitated streamlined reviews and discussions, reduced workload, and decreased potential for burnout.
“Meta-Research: Blinding Reduces Institutional Prestige Bias During Initial Review of Applications for a Young Investigator Award” was authored by Anne E. Hultgren and Nicole M.F. Patras of the Arnold and Mabel Beckman Foundation in collaboration with Jenna Hicks of the Health Research Alliance. The published study is available under the persistent Crossref digital object identifier (DOI) 10.7554/eLife.92339.
Read the published study online here.
See the press release about this news on PRweb here.
Download the publication PDF:
About the Authors
Anne Hultgren, PhD
Dr. Hultgren joined the Arnold and Mabel Beckman Foundation as Executive Director and CEO in 2015. Previously, she was at the Department of Homeland Security, Science and Technology Directorate, working in chemical and biological defense technologies. She received her PhD in Physics and Astronomy from the Johns Hopkins University, and BA in Physics and Mathematics from Franklin and Marshall College.
Nicole Patras, PMP
Mrs. Patras joined the Arnold and Mabel Beckman Foundation in 2014 and serves as the Senior Program Officer for the Beckman Young Investigator Program. She also leads the Foundation’s JEDI (Justice, Equity, Diversity, and Inclusion) Committee and oversees event planning for the Foundation’s annual Beckman Symposium. She holds a BA from the University of California, Santa Cruz and is a Certified Nonprofit Professional through Nonprofit Leadership Alliance and Project Management Professional through Project Management Institute.
Jenna Hicks, PhD
Dr. Hicks joined the Health Research Alliance in 2023 as Project Lead of the Inclusive Grantmaking Initiative and was promoted to Assistant Director later that year. She came to HRA with a background in research (both biomedical and education research), program development, and evaluation. Prior to joining HRA, Dr. Hicks worked in graduate education administration at the University of Minnesota, where she developed, implemented, and evaluated professional development programming for biomedical graduate students and postdocs. Dr. Hicks received her PhD in biomedical sciences from the University of California, San Diego, and completed postdoctoral training in biology education research at the University of Minnesota.
Praise for this Study
“I feel strongly that Dr. and Mrs. Beckman would be extremely proud of this work by the Foundation. The mission they set us out to fulfill is an important one – we are tasked with supporting the most innovative and ambitious young scientists in the chemical and life sciences. Clearly, a biased review process would be a major hindrance in our work and these studies have shown a path forward to creating more equitable, inclusive, and effective proposal review methods.”
L. Andrew Lyon, PhD
Chair of the Board of Directors, Arnold and Mabel Beckman Foundation
Former Beckman Young Investigator (’00)
“I have long suspected that young investigators at top ranked universities have a better chance of obtaining grants from foundations than their counterparts at lower ranked institutions. This well documented study by Hultgren, Patras, and Hicks shows that this sort of bias likely is widespread in the science funding space. In my view, the study will be of great value to foundation directors and other officers who have or soon will have programs to support exceptional young scientists.”
Harry B. Gray, PhD
Arnold O. Beckman Professor of Chemistry, California Institute of Technology
Founding Director of the Beckman Institute, California Institute of Technology
Former Chair & SAC Member, Beckman Foundation
BYI Program Architect, Beckman Foundation
“For funders in the nonprofit sector, the results of this study conducted by the Arnold and Mabel Beckman Foundation are particularly exciting because they not only demonstrate the value of blinding in effectively reducing institutional prestige bias during the review process, but also underscore the importance of grantmaking organizations remaining steadfast in their dedication to constant program re-evaluation and process improvement. Many HRA members share Dr. Beckman’s aim of supporting young scientists “who do not yet have the clout to receive major research grants” but who demonstrate excellence, and this study enables us to advance that goal, proactively, and in a more unbiased way.”
Maryrose Franko, PhD
Executive Director, Health Research Alliance
In the News
Orange County Public Relations Society of America (OCPRSA)
2024 PROTOS Award of Excellence
Beckman Foundation's "Reducing Institutional Prestige Bias" public relations campaign surrounding this published study was recognized with an award of excellence for community relations/institutional programs in September 2024.
Blinding Reviews Reduces Bias
Featured coverage in Altum, Inc.'s May 2024 newsletter (subscription) highlighting Beckman Foundation's published study and the role of blinded reviews in equitable selection processes
Overview of attention for "Blinding reduces institutional prestige bias during initial review of applications for a young investigator award" published in eLife, March 2024
Anonymizing research funding applications could reduce ‘prestige privilege’
Study of Beckman foundation funding for early-career scientists adds to growing body of knowledge about the practice
April 18, 2024 | By Phie Jacobs
C&EN - Chemical & Engineering News
Removing institutional information from grant application materials could reduce reviewers’ bias
The Beckman Young Investigator Program saw fairer reviews after applicants anonymized their application materials
March 25, 2024 | By Krystal Vasquez
Frequently Asked Questions
Q: How did the blinding impact any conflict-of-interest processes normally used by Beckman Foundation?
A: During administrative review, we collected university information for each applicant internally, then made sure to not assign reviewers to applications where they would have a conflict of interest.
Q: For institutions or hospitals that are part of the same, larger system, how were those handled during data collection for the study?
A: Any hospital and/or research center associated with a university was counted together. Here are two examples: 1) "Harvard" included the University’s associated medical schools, and 2) "University of Wisconsin, Madison" included Morgridge Institute for Research.
Q: Were the [Beckman Young Investigator] awards’ relative advantage calculations indexed to letters of intent (LOIs) or to invited full applications?
A: In this study, the relative advantage calculations were indexed to letters of intent.
Q: What did the relative advantage look like over time?
A: There is a lot of data in the study that addresses this question. Trying to sum that up is complicated but this may help: If you look at the clusters by year of pre-and post-blinding, especially for the Top and Bottom categories, the clusters do not overlap, indicating that there wasn’t a natural trend over time to the reduced bias outcomes we saw after blinding.
Q: Was the “Top 51-96 Institutions” group blinded/unblinded difference statistically significant?
A: Chi-square tests were done to examine the relationships between the institution category (there were five categories) and whether or not the letter of intent (LOI) was invited to submit a full application, or whether the applicant ultimately received an award. The “51-96” result was not specifically tested to determine if there was a statistically significant difference from the other institution groups.
Q: If an applicant submitted a letter of intent (LOI) and inadvertently missed blinding something that was part of the requirement, were they given the option to fix the error or was the application disqualified?
A: That opportunity was not provided to applicants due to the organization’s internal bandwidth limitations. However, it is an option that another organization could explore if more bandwidth existed for accommodating updated submissions.
Q: Aside from gender, race, and institutional prestige, what other types of biases might be in proposal reviews that a funding organization should consider when evaluating their processes?
A: Another type of bias “career-path bias,” where breaks in an applicant's career or not moving to other institutions from PhD to postdoc, etc., can be seen in a very negative light without understanding the potential personal context of those decisions.
Grantmaker Resources
Webinar: Unmasking Bias in Peer Review and Grant Funding
Join us for an insightful webinar, hosted by Altum on June 25, 2024 at 1 PM EDT / 10 AM PDT, focusing on the critical issue of bias in peer review and grant funding. Registration is now open.
Bias can significantly impact the fairness and effectiveness of research funding, influencing which projects receive support. This webinar will delve into various forms of bias, such as institutional prestige and identity-based biases, and present evidence-based strategies to mitigate these issues. Three expert speakers will share their latest research findings and practical approaches to promoting equity in the review process, ensuring a more level playing field for all applicants.
Speakers:
- Jenna Hicks, Assistant Director at the Health Research Alliance
- Anne Hultgren, PhD, Executive Director, Arnold and Mabel Beckman Foundation
- Alison J. Hatt, PhD, Independent Consultant for the Advanced Light Source
Key Takeaways:
- Understand how bias affects peer review and grant funding.
- Learn about the impact of blinding on reducing institutional prestige bias.
- Explore practical strategies to minimize identity-based biases.
- Discover methods to improve the statistical integrity of peer review.
Read the transcript of this video presentation:
Candid insights is the blog of Candid.org. It is a resource for "the big picture of the social sector, backed by data and expertise."
First published on Candid insights under tips & training on June 10, the article Grant application and review practices for more equitable and inclusive funding by Kaerie Ray, Curator and Communications Officer for the Arnold and Mabel Beckman Foundation, invites grantmakers to "discover five grant application and review best practices that the Arnold and Mabel Beckman Foundation uses to make their funding more equitable and inclusive."
Media Resources
Interviews are available with:
Anne Hultgren, PhD
Executive Director, Arnold and Mabel Beckman Foundation
First Author, “Meta-Research: Blinding Reduces Institutional Prestige Bias During Initial Review of Applications for a Young Investigator Award”
Dr. Hultgren joined the Arnold and Mabel Beckman Foundation in 2015 as Executive Director and CEO. Previously, she was at the Department of Homeland Security, Science and Technology Directorate, working in chemical and biological defense technologies. She received her PhD in Physics and Astronomy from the Johns Hopkins University, and BA in Physics and Mathematics from Franklin and Marshall College.
L. Andrew Lyon, PhD
Chair of the Board of Directors, Arnold and Mabel Beckman Foundation
Former Beckman Young Investigator (’00)
Dr. Lyon joined the Board of Directors for the Arnold and Mabel Beckman Foundation in 2015, first as Chair of the Science Committee and Nominating Committee, then as Chair of the Board in 2022. He was named a Beckman Young Investigator Awardee in 2000. Dr. Lyon received his PhD in Chemistry from Northwestern University, and BA in Chemistry from Rutgers University.
For interview booking and other media inquiries, contact:
Kaerie Ray, Communications Officer
Access story-related graphics, logos, and images:
With our Dropbox folder for Media
PDF downloads available for:
Cite this article:
Anne E Hultgren, Nicole MF Patras, Jenna Hicks (2024) Meta-Research: Blinding reduces institutional prestige bias during initial review of applications for a young investigator award eLife 13:e92339.
https://doi.org/10.7554/eLife.92339
Keywords: eLife, journal, peer reviewed, publication, blinding, bias, prestige bias, grant reviews, blinded reviews, process improvements, funding, philanthropy, study, grantmaker, grantmaking, Science, C&EN, Altmetrics, interview, article, feature, findings, research, press kit, press release, author, LOI, letter of intent, BYI, young investigator, award, Anne Hultgren, Nicole Patras, Jenna Hicks, PROTOS, award, public relations, Kaerie Ray
