There’s an App for That

Review of Online Tools to Combat Gendered Citation Bias

Joanna Blair
Wilfrid Laurier University Library
Wilfrid Laurier University
Waterloo, ON, Canada
joblair@wlu.ca

Abstract

Keywords: Citation bias, Gender bias, Women in science

Recommended Citation:

Blair, J. (2025). Review of online tools to combat gendered citation bias. Issues in Science and Technology Librarianship, 111. https://doi.org/10.29173/istl2895

Introduction

Many bibliometric analyses have pointed to the existence of gender imbalances in citation lists as part of a larger problem of citation bias in scientific disciplines (Caplar et al., 2017, Dworkin et al., 2020; Fulvio et al., 2021; Teich et al., 2022). Penders (2018) argued that the act of citation is not a neutral act because citations lend credibility to ideas and adds visibility to both authors and journals, making citations a type of academic currency. Gendered citation bias results in women scholars being under-cited relative to the number of papers in their field (Zurn et al., 2022) and this phenomenon can affect a researcher’s career trajectory when impact metrics are used to assess candidacy for employment or promotion (Ray et al., 2022). Many journals and publishers (e.g., American Psychological Association, Biomedical Engineering Society, Royal Society of Chemistry) are working actively to improve the equity, diversity, and inclusion (EDI) support in their publications through working groups, new guidelines for editors, and the introduction of “citation diversity statements” (Kwon, 2022, p. 569). New digital tools have also been developed to help researchers examine the gender balance of the authors in citation lists and work toward improving the gender balance (Heidt, 2023).

Gender imbalance in citation has been documented in several scientific disciplines. West et al. (2013) examined a JSTOR dataset with 1.8 million research articles and showed gender inequities in citation for many disciplines including ecology, molecular and cellular biology, and mathematics. Research in astronomy shows that women are cited approximately 10% less than would be expected for the field (Caplar et al., 2017). A study focused on physics literature showed that women authored papers are cited 11% less than expected (Teich et al., 2022). Another study in the field of academic medicine showed that when women are both the primary and senior author, they receive approximately half as many citations as articles authored by men (Chatterjee & Werner, 2021). Neuroscience is a discipline that has been particularly active in studying the existence of gender bias in the field and shows discrepancies of up to 13.9% (Dworkin et al., 2020).

The literature suggests that gender disparity has several possible causes. Causation has been linked to the act of self-citation and the occurrence of male researchers preferentially citing male researchers (Dworkin et al., 2020; Teich et al., 2022). Another possible cause is the “first mover advantage,” whereby the first researcher to write about a phenomenon is cited preferentially downstream and in many STEM fields there is still a disproportionate number of male senior researchers (Kong et al., 2022). There is also evidence that longer reference lists have better gender balance, so limits on the length of reference lists may contribute to the disparity (Teich et al., 2022). Gendered citation bias is largely an invisible problem and limited awareness of the issue is another suggested contributor to the problem (Dworkin et al., 2020), one that science librarians and scholarly communications librarians are well-positioned to improve.

The digital tools reviewed below are designed to help authors examine their citation lists and potentially reduce gender disparities in citation. Knowledge of these tools along with their function, strengths, and limitations can help science librarians discuss citation justice and citation bias with faculty and students by communicating strategies toward rectifying the existing imbalance. All the tools reviewed below were listed as “diversity assessment tools” to help authors combat citation bias in the Nature article titled, “Racial Inequalities in Journals Highlighted in Giant Study” (Heidt, 2023). The examination below is a detailed look into the strengths and functionality of the citation tools and how they can help researchers.

Tools

GCBI-alyzer

The GCBI-alyzer (https://psych.wisc.edu/gcbialyzer/), currently hosted by the University of Wisconsin–Madison Department of Psychology, is a free tool that calculates the gender ratio of a bibliography and Gender Citation Balance Indices (GCBIs) based on the gender of the first and last authors of each reference. Examining the gender of the first and last author is the common method in bibliometrics literature for examining gender biases (Chatterjee & Werner, 2021; Dworkin et al., 2020; Teich et al., 2022). The GCBI-alyzer was developed by the Journal of Cognitive Neuroscience (JoCN) to ascertain if gender disparity in citations exists in the field of cognitive neuroscience (Fulvio et al., 2021) and for authors to use the tool to examine their citation lists.

To use the GCBI-alyzer, the user submits their citation list, complete with Crossref DOIs, into the input text box. Crossref DOIs are of use in this situation because Crossref makes their metadata openly available and, when the DOI is registered, the registering organization is asked to supply each author's first name and ORCID (Feeney, 2021). The tool analyzes the first and last author of each reference on the citation list and assigns the authors to a male or female category if the algorithm determines that the name is more than 70% likely to be a specific gender.

The assignment of gender is based on the algorithm used by Dworkin et al. (2020) where first names are derived from Crossref metadata and then assigned probabilistic gender using GenderAPI (https://www.genderapi.io). GenderAPI includes over 4,000,000 names from 240 countries including non-Latin scripts (GenderAPI.io, 2024).

The output of GCBI-alyzer tells the user how many DOIs were successfully categorized and how many failed. According to the GCBI-alyzer documentation, failure is usually due to inaccurate DOIs or articles published before 2009 where first names were not standard in the Crossref metadata.

The analysis is delivered in two parts. The first part of the output provides the gender ratio of the first and last authors of the references according to four categories where M=Man and W=Woman. The four categories are:

• M/M
• M/W
• W/M
• W/W

The second part of the output is the GCBI “Range,” which is compared to the JoCN base rate for references. The Gender Citation Balance is represented by GCBI range where a score of less than zero (<0) means that the gender category is under-cited and a score greater than zero (>0) means the gender category is over-cited.

I tested this tool by submitting a bibliography from a paper I wrote about information literacy assessments. The bibliography had 18 references with DOIs and 15 of the DOIs were successfully analyzed. Not surprising for a reference list from the library discipline that is heavily dominated by women - the citations were heavily skewed to women authors (see Figure 1).

The interface for GCBI-alyzer is minimalistic and offers only basic text boxes for submissions and outputs. The clarity of the interface could be improved by adding small design elements, such as making the “Submit” feature into a button rather than the current appearance of a single word beneath the text box (see Figure 2).

The big limitation of this tool is that the benchmark used for the “Range” analysis is the standard citation gender balance for JoCN. This analysis is most useful for references lists in the field of neuroscience and disciplines that have known gender balance metrics that are like JoCN’s balance of 40.7% M/M, 11.5% M/W, 32% W/M, and 15.9% W/W. The GCBI-alyzer could be greatly improved if users were able to reset the benchmark for gender balance metrics based on the norms for their specific discipline. One year after the launch of this tool, the editors of JoCN reported seeing a modest improvement in the gender balance of citation lists published in the journal (Postle & Fulvio, 2021).

There are limited tools available that perform similar services to GCBI-alyzer. There is code available through Github that produces a similar analysis and output (Zhou et al., 2022) if the user has appropriate technical and coding knowledge. Another analyzer of citation lists called Gender Balance Assessment Tool (GBAT) (https://jlsumner.shinyapps.io/syllabustool/) analyzes a list and returns a gender ratio of the authors in the list; however, the bibliography must include the author’s first names (Sumner, 2018).

Citation Transparency (v0.0.8) Browser Extension

Citation Transparency (Citation Transparency Tools, 2023) is a free browser plug-in for Chrome that assigns a probabilistic gender to the first and last author of the citations in the results list from Google Scholar and PubMed. Because the gender assignment is obvious in the record itself, this is a potentially powerful tool for researchers to quickly identify the diversity of authors in their results list and recognize the gender imbalance of citation lists. The description on the extension download page is brief:

In many fields, men and women are not cited at the level that one would expect given their prevalence in the field. One way we can help mitigate this bias is by making the perceived genders of first and last authors of papers more transparent in the search for references. Towards this goal, this extension adds probabilistic gender information to paper searches on Google Scholar and PubMed (Citation Transparency Tools, 2023, Overview section).

The Chrome extension download page has very little detail about the tool, functionality, or developers. However, the project’s GitHub page (https://github.com/jastiso/citation_bias_ext) contains a great deal more information about the underlying code and developer.

This tool follows the protocol Dworkin et al. (2020) developed for assessing gender by looking at the gender for the first and last author of the paper. Like GCBI-alyzer, this tool uses GenderAPI to assess and assign probabilistic gender to the authors. Greater transparency on the extension download page would be helpful for assessment purposes and would provide increased confidence in the output.

When conducting a search in PubMed or Google Scholar, the plug-in adds the probabilistic gender information beneath the brief record where possible (see Figure 3). It may take up to a full minute for the gender assignment results to appear. The tool does not produce results for every article, likely due to success linking to the article and availability of the author’s complete first name in the source article metadata. The extension can assign gender to non-English names, due to the aforementioned breadth of the underlying GenderAPI data for 4,000,000 names from 240 countries (https://www.genderapi.io).

One key issue was noted with this tool: users can expect as few as 20% of search results to have probabilistic gender assigned. In one case, as shown in Figure 3, the first and last authors were listed incorrectly. Another issue was that the plug-in did not function consistently on either PubMed or Google Scholar and required frequent cookie and cache clearing to load results.

The citation transparency plug-in does not yet seem to be in a state where it would be a great help for improving the gender disparity in citation lists as the results are too few and inconsistent. The premise, however, does have a great deal of potential for helping researchers because revealing genders during the literature search allows researchers to strive toward inclusivity early in the literature review.

Gage

Gage (https://gage.500womenscientists.org/) is a directory of women and gender-diverse people in science, technology, engineering, and mathematics (STEM) that was created by 500 Women Scientists, an organization devoted to inclusion in the scientific community. The Gage landing page states that the directory is “designed for journalists, educators and policymakers to identify gender-diverse STEMM professionals.” In 2020, over 14,000 entries were listed from 146 different countries (McCullagh et al., 2021).

This tool allows you to search by keyword and find women scientists in different fields along with their contact information. You can also search by discipline, race/ethnicity, or country and additional filters include gender identity and spoken language. While the scope of the directory is international, scientists are largely based in North America and Europe.

The records include the scientist's name, position, field of study, biography, spoken languages, affiliated institution, and contact information. Some records contain a link to the scientist’s website, but there is not a required field for an ORCID or Google Scholar link. Of a sample of 20 records examined, only five records included a link to the researcher’s website and two records had a link to ORCID or Google Scholar. Lack of consistent connection to the scientists’ output is a hurdle for people using this tool to identify specific research authored by women and gender diverse scientists. Nonetheless, it is a handy tool for identifying women scientists in a specific field and in many cases the records reveal the intersectional identities of the scientist through self-disclosure of race and languages spoken.

There are several other directories that link researchers to women scientists. Many of them are discipline-specific including Microscopy Gender Equality Database (https://www.rms.org.uk/community/equity-microscopy-resource.html), Women in Microbiology (https://docs.google.com/spreadsheets/d/1xw6W2noEUrpc5AFbLgkBGwR1KHIBnay2M9tT_bHaxqk), Women in Polar Science (https://womeninpolarscience.org/100polarwomen), among others. These tools are helpful for identifying people in a specific area of study but are not granular enough to contribute directly to citation lists.

Conclusion

It is an exceedingly positive sign that tools to combat gender disparity in citation exist. The development of these tools shows awareness and action toward improving gender inequities in science. It is worth noting that one of the limitations of these tools is that they work on the assumption of a gender binary. Many publishers are working to improve diversity and gender disparity in their disciplines (e.g., American Psychological Association, 2025; Royal Society of Chemistry, 2025) so these tools may continue to develop or be integrated into search tools.

The problem of gendered citation bias in the sciences is acknowledged as an invisible problem (Chatterjee & Werner, 2021) and science librarians are well-positioned to increase awareness of the issue through information literacy classes and faculty meetings. Science librarians involved in scholarly communications can also advocate for longer and more inclusive reference lists and educate academics, particularly graduate students, about the merits of citation diversity statements (Teich et al., 2022).

Reliable tools that can provide a gender breakdown of a citation list is an excellent premise for improving inequities. Of the three tools examined, the GCBI-alyzer offered the most complete set of results, and it is promising that the editors of JoCN noted an improvement in gender balanced citations after the tool was launched. The Citation Transparency plug-in browser extension and Gage directory may not directly improve citation lists in their current iterations but are useful tools in helping researchers reflect on their citation lists and work toward change. It will be interesting to see how this field develops in the future and whether there might be any resistance to the perceived value of these tools.

References

American Psychological Association (2025). Equity, diversity, and inclusion toolkit for journal editors (2nd ed.). https://www.apa.org/pubs/authors/equity-diversity-inclusion-toolkit-journal-editors.pdf

Caplar, N., Tacchella, S. & Birrer, S. (2017). Quantitative evaluation of gender bias in astronomical publications from citation counts. Nature Astronomy, 1(6), Article 0141. https://doi.org/10.1038/s41550-017-0141

Chatterjee, P. & Werner, R. M. (2021). Gender disparity in citations in high-impact journal articles. JAMA Network Open, 4(7), Article e2114509. https://doi.org/10.1001/jamanetworkopen.2021.14509

Citation Transparency Tools (2023). Citation Transparency (v.0.0.8) [Computer software]. Chrome Web Store. https://chromewebstore.google.com/detail/cepnbdbhabaljgecaddglhhcgajphbcf

Dworkin, J. D., Linn, K. A., Teich, E. G., Zurn, P., Shinohara, R. T., & Bassett, D. S. (2020). The extent and drivers of gender imbalance in neuroscience reference lists. Nature Neuroscience, 23(8), 918–926. https://doi.org/10.1038/s41593-020-0658-y

Feeney, P. (2021, October 22). Metadata principles and practices. Crossref. https://www.crossref.org/documentation/principles-practices/

Fulvio, J. M., Akinnola, I., & Postle, B. R. (2021). Gender (im)balance in citation practices in cognitive neuroscience. Journal of Cognitive Neuroscience, 33(1), 3–7. https://doi.org/10.1162/jocn_a_01643

GenderAPI.io (2024). GenderAPI. https://www.genderapi.io/

Heidt, A. (2023, April 28). Racial inequalities in journals highlighted in giant study. Nature. https://doi.org/10.1038/d41586-023-01457-4

Kong, H., Martin-Gutierrez, S., & Karimi, F. (2022). Influence of the first-mover advantage on the gender disparities in physics citations. Communications Physics, 5(1), Article 243. https://doi.org/10.1038/s42005-022-00997-x

Kwon, D. (2022). The rise of citational justice. Nature, 603, 568–571. https://doi.org/10.1038/d41586-022-00793-1

McCullagh, E., Nowak, K. & Zelikova, J. (2021, January 13). Gage: Quashing the Matilda effect. 500 Women Scientists. https://500womenscientists.medium.com/gage-quashing-the-matilda-effect-5d917f55a0e1

Penders, B. (2018). Ten simple rules for responsible referencing. PLOS Computational Biology, 14(4), Article e1006036. https://doi.org/10.1371/journal.pcbi.1006036

Postle, B. R. & Fulvio, J. M. (2021). One-year update from the Editor-in-Chief. Journal of Cognitive Neuroscience, 34(1), 1–3. https://doi.org/10.1162/jocn_e_01799

Ray, K. S., Zurn, P., Dworkin, J. D., Bassett, D. S., & Resnik, D. B. (2022). Citation bias, diversity, and ethics. Accountability in Research, 31(2), 158–172. https://doi.org/10.1080/08989621.2022.2111257

Royal Society of Chemistry (2025). Inclusion and diversity resources. Royal Society of Chemistry. Retrieved March 21, 2025, from https://www.rsc.org/policy-evidence-campaigns/inclusion-diversity/resources/

Sumner, J. L. (2018). The Gender Balance Assessment Tool (GBAT): A web-based tool for estimating gender balance in syllabi and bibliographies. PS: Political Science & Politics, 51(2), 396–400. https://doi.org/10.1017/S1049096517002074

Teich, E. G., Kim, J. Z., Lynn, C. W., Simon, S. C., Klishin, A. A., Szymula, K. P., Srivastava, P., Bassett, L. C., Zurn, P., Dworkin, J. D., & Bassett, D. S. (2022). Citation inequity and gendered citation practices in contemporary physics. Nature Physics, 18, 1161–1170. https://doi.org/10.1038/s41567-022-01770-1

West, J. D., Jacquet, J., King, M. M., Correll, S. J., & Bergstrom, C. T. (2013). The role of gender in scholarly authorship. PLoS ONE, 8(7), Article e66212. https://doi.org/10.1371/journal.pone.0066212

Zhou, D., Stiso, J., Bertolero, M., Cornblath, E., Teich, E., Sizemore Blevins, A., Murphy, K., Oudyk, K., Michael, C., Urai, A., Matelsky, J., Virtualmario, Camp C., Alacantra Castillo, R., Saxe, R., Dworkin, J., & Bassett, D. (2022). Diversity statement and code notebook: dalejn/cleanBib. (Version 1.1.3) [Computer code]. Zenodo. https://doi.org/10.5281/zenodo.3672109

Zurn, P., Teich, E. G., Simon, S. C., Kim, J. Z., & Bassett, D. S. (2022). Supporting academic equity in physics through citation diversity. Communications Physics, 5, Article 240. https://doi.org/10.1038/s42005-022-00999-9

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.