Is Blinded Review Enough? How Gendered Outcomes Arise Even Under Anonymous Evaluation

Innovative research by women is underfunded because of gender differences in writing style.

Introduction

In the scientific community, research grants can be subject to gender bias when they are awarded on a less than scientific basis. To level the playing field, the use of blinded reviews has become increasingly popular – meaning that reviewers evaluate submitted grant proposals without knowing the identity (gender, race, etc.) of the applicants. Innovative fields particularly suffer from greater gender disparities than other sectors. Considering how diverse innovators have much to offer, it is important to explore whether gender barriers still exist after implementing blinded reviews.

To address this issue, the authors analyzed the blinded review process for grant proposals to the Gates Foundation’s Global Challenges: Exploration program from 2008-2017. Over the program’s first nineteen rounds, a total of 635 proposals received grants of $100,000 each. To award these grants, reviewers evaluated an average of 100 proposals per round, rating their favorite proposal “Gold” and up to five other proposals “Silver.” The authors measured the innovative outcomes of these grants through a combination of proposal-based metrics, article publications, National Institute of Health grant awards, and text-based analysis of Medical Subject Heading (MeSH) terms. By evaluating the gender barriers that may still exist post-blind review, the authors explored the mechanisms that can lead innovative organizations to become more gender diverse, and the potential benefits of an increase in gender diversity.

Findings

In evaluating research proposals, the focus on writing style resulted in a gender bias.

  • Despite implementing a blind review process, female applicants were approximately 15% less likely to receive a silver rating and 20% less likely to receive a gold rating from reviewers compared to male applicants.
  • Reviewers mainly based their decisions on the writing style of proposals, favoring the use of broad words (for example, “control” or “detection”).
  • Female applicants tended to use narrower, more topic-specific phrasing - word choice associated with low reviewer scores (for example, “contraceptive” or “brain”). After controlling for the impact of word choice, the gender-based score disparity effect size dropped by over 50%.
  • However, applicants with broad phrasing in their proposals tended to underperform after receiving a Gates Foundation grant, with fewer publications and National Institute of Health grants.
  • Gates Foundation grants had a larger impact on women’s careers in innovation compared to men.
    • Without funding, women performed worse than their male counterparts in terms of articles, new journals, coauthors, and MeSH terms derived from applicant publications.
    • With funding, women performed on par with their male counterparts. In terms of National Institute of Health grant funding, female Gates Foundation grantees even outstripped their male counterparts.

In short, even when proposed innovations were actually high quality, women received lower scores because of differences communication and writing styles, as they tended to use excessively narrow words in their proposals. Compared to male applicants, the failure to achieve funding posed more of a barrier to future innovation by female applicants. Based on these findings, innovation-driven organizations have room to improve their application procedures in selection of high-quality projects and allocation of resources to underrepresented innovators.

Methodology

Using a sample of 6,794 proposals submitted by 5,058 applicants (34% female) to the Gates Foundation’s Global Challenges: Exploration program between 2008 and 2017, the authors analyzed two major components of the interplay between diversity and innovation. First, the authors examined the determinants of diversity in innovative organizations by examining the role of gender in explaining reviewer evaluations of innovative proposals. Second, the authors constructed a difference-in-difference estimator to explore the interaction between funding and applicant gender, in order to identify the differential impact of funding across applicants. In these analyses, the authors focused on a homogenous sample of US-based life science researchers, and identified the causal impact of gender on both reviewer scores and subsequent outcomes.

Related GAP Studies