NASA Science Mission Directorate Dual-Anonymous Peer Review Virtual Town Hall

Hello, my name is Daniel Evans and I’m a Program Scientist in NASA’s Science Mission Directorate. This is a recording of a virtual community Town Hall that we held on March 3rd 2020 to discuss NASA’s Science Mission Directorate’s implementation of dual-anonymous peer review. On March 3rd, I was joined by two SMD leaders: Dr. Thomas Zurbuchen, the Associate Administrator for NASA’s Science Mission Directorate and Dr. Michael New, the Deputy Associate Administrator for research.

00:40 - During the town hall, we asked anybody to submit any questions using the following link and later in this presentation I will actually go back and review some of the most frequently submitted questions. Let me start by saying that NASA’s Science Mission Directorate is strongly committed to ensuring that the review of proposals is performed in an equitable and fair manner that reduces the impacts of any unconscious biases. To this end, and motivated by an extremely successful study conducted for the Hubble Space Telescope, the Science Mission Directorate is conducting a pilot program in ROSES-2020 to evaluate proposals using dual-anonymous peer review. Now, under the system not only are proposes unaware of the identities of the members of the review panel but those reviewers themselves do not have explicit knowledge of the identities of the proposing team during the scientific evaluation of the proposal. This webinar covers four main subjects: number one, we’ll begin by introducing exactly what is dual-anonymous peer review before moving on to discuss which programs are converting to dual-anonymous peer review.

01:55 - Then, we’ll be discussing how exactly the proposing community can go about making their proposals compliant with this new method ,and then finally discuss how proposals will be reviewed under the auspices of dual-anonymous peer review. Let’s start off with the motivation: what exactly is dual-anonymous peer review? SMD is strongly committed to developing an excellent work force both within NASA and also within our extramural community. We have started a whole host of measures to develop the workforce and one important technique that we’re doing to do just that is dual-anonymous peer review. I want to pay credit to the pioneering work performed by the Hubble Space Telescope team in implementing and standing up dual-anonymous peer review, and indeed it’s from Hubble that I’m showing the following plot. This is data from the Hubble Space Telescope for each of its annual call for proposals over the past dozen or so years, and you can see the inferred male PI success rate exceeds slightly the inferred female PI success rate year after year after year.

It’s like flipping 03:14 - a coin and coming up with the heads twelve times in a row, and so that motivated the Hubble team to switch to dual-anonymous peer review in Cycle 26. In cycle 27 you can see there on the right the implementation of dual- anonymous peer review first resulted in a slightly higher success rate in inferred female PI. In Cycle 27 that trend was actually flipped but their data from their most recent mid-cycle review shows that parity exists between genders: in other words there’s a forty three percent success rate for inferred male PIs and a forty three percent success rate for inferred female PIs. But gender is course is one of the many many parameters of diversity that we wish to think about, and it may merely represent the tip of the iceberg. So what else are we missing? To motivate that further this is a plot of the success rate by institution type for ROSES programs in the NASA Science Mission Directorate pilot: you can see that government contractors, federal government, and research one universities all have in excess of a twenty percent success rate for their proposals, but take a look at research two universities or minority serving institutions: the success rate declines dramatically.

But I 04:38 - wanted to point out the one of the main motivations for moving to dual- anonymous peer review is to the level the playing field for everyone. I should point out that dual-anonymous peer review is not completely a double-blind process, however, and in NASA’s implementation of dual-anonymous peer review we require that our proposing community submit two documents: the first one is an anonymized proposal; the second one is a not-anonymized expertise and resources document. And here that latter document allows the proposing team to demonstrate why exactly it has the qualifications, experience and resources that will make the execution of the proposed investigation successful. I’m showing now some feedback from the Hubble panelists I just wanted to note Rachel Somerville’s remarks - she was the chair of the Cycle 27 Hubble time allocation committee. Dr. Somerville noticed that several time allocation committee members themselves noted that they felt the discussions at both the panel level and at the time allocation committee level seemed more collegial less emotionally charged than previous TACs, perhaps because either positive or negative feelings about the people involved in the proposal were largely removed.

Let’s 05:55 - move on which programs are converting to dual anonymous period in NASA’s ROSES- 2020 pilot. We have selected four programs to be performed under dual-anonymous peer review, one from each SMD division. We have the Astrophysics Data Analysis Program, otherwise known as ADAP. We have Earth Science US Principal Investigator. We have Habitable Worlds and then finally we have the Heliophysics Guest Investigator-Open program.

In addition to 06:29 - that, all astrophysics Guest Observer Guest Investigator programs are permanently converting to dual- anonymous peer review, and we held a separate Town Hall for that on February 27th. So, how does the proposing community go about making their proposals compliant with dual-anonymous peer review. I wanted to say that NASA has provided very detailed guidance to assist the proposing community with this major change in the submission of proposals. The first one is that every single program element which will be performed under the auspices of dual-anonymous has very specific program-level guidance. In addition the NSPIRES page of each program element contains a document which we term the Guidelines for Anonymous Proposals, which describes in detail the specific requirements for anonymous proposals.

Number two: SMD 07:21 - has a whole host of resources including a quick-start tutorial as well as frequently asked questions, both of which may be found at the URL provided. The five main points to remember when submitting anonymized proposals are these. Number one is to exclude the names and affiliations of the proposing team including in figures and in references to personal websites. Number two: this is an important change - please to not claim ownership of past work so don’t say “my previously funded work” or “our analysis shown”. Instead we need you to write, including references in the passive third person, “prior analysis”. We need you to use square brackets.

Number four: 08:04 - of course, please do describe the work proposed so absolutely you must say “we propose to do the following” or “we will measure the effects of”. Finally you’ll need to include a separate not anonymized “Expertise and Resources” document, and we’ll be providing more details for that later on. A very common question is how do you reference unpublished work or how do you reference proprietary results? Our advice is as follows: we understand that it may be occasionally important to cite exclusive access datasets or non-public software or unpublished data or findings that have been presented elsewhere in public but are not yet citable. Each of these may reveal or strongly imply the investigators on a proposal. So in these instances we recommend that proposers use language such as “obtained in private communication” or “from private consultation” when referring to such a potentially identifying work.

09:00 - Recall that the goal of dual-anonymous is to create a shift in the tenor of discussion, not to make it absolutely impossible to guess who the team members are. Another common question is in proposals where a team member has institutional accesses to unique facilities such an observatory or a laboratory, both of which would be required to complete the proposed work. In italics here I’m sharing an example in the anonymized section of proposal. You can say that the team has access to telescope time on the Keck Observatory, which will enable the spectroscopic follow-up. So that doesn’t really give clues to the identities of the proposing team but we recommend that you validate any claim in the not anonymous expertise and resources document. Here’s an example of anonymization.

The 09:49 - paragraph above has not been anonymized; the paragraph below has been anonymized. Thanks to the Hubble team for allowing us to include this. In yellow, you can change “in Rogers et al. 2014 we concluded”, to “prior work square brackets 12 concluded”. In green “If our model from Roger et al. 2014 is correct”, you would simply change to “If the model from [12] is correct”. And “our first epoch” you can change to “a first epoch” to again not signify ownership. A very common question of course is” how is the capability of the team to execute the proposed investigation taken into account? And here this is where we provide details for what we termed the “Expertise and resources - Not Anonymized” document. This document will get distributed to panelists for a subset of proposals - typically the top third according to the distribution of the assigned grades, projected selection rates and so forth. It contains many elements - a key summary is as follows. You need to include a list of team members together with their roles; brief descriptions of the scientific and technical expertise the each team member brings; a discussion of the contribution that each team member will make to the proposed investigation; a discussion of the specific resources that are required; a summary of work effort, to include a not anonymized table of work effort; bio sketches if required; statements of current and pending if required; letters of resource support again if required by the solicitation.

The 11:27 - guidelines for anonymous proposals document includes a worked example. Here are some other requirements which I won’t go into in detail here, but I just wanted to point out that the guidelines for anonymous proposals document spells them out in significant detail. How will your proposals be reviewed? The overall flow of the review is as follows: we’ll start off with an anonymized scientific review. This is where the review panel discusses those anonymized proposals. All assessments get completed, all grades get finalized, all panel summaries are written.

And then, number 12:03 - two: the “Expertise and Resources” document will be distributed to panelists for a subset of proposals, typically the top third. The panelists go forth and assess the team resources capability to execute the proposed investigation. What do we tell our review panelists? We tell our review panelists to consider proposals solely on the scientific merit of what’s proposed. We ask them to not spend any time attempting to identify the PI team, and even if our reviewers think they know the identities of the team we instruct them to discuss the science and not the people. We have NASA appointed Levelers in each panel room to ensure that the discussion does not veer off.

We 12:46 - also tell our reviewers to keep in mind that language can be very important and we ask them to use appropriately neutral pronouns, so “what they propose” or “the team has evaluated” when they write their panel summaries. As the discussion unfolds we have NASA-appointed Levelers in each room in addition to panel support staff. The role of the Leveler is to ensure that the panel discussions focus on the scientific merit and unlike the chairs they’re not listening for issues pertaining to the science, but they’re actually primarily focused on the discussion itself. Now, if the discussion does veer into comments about the proposing team, or their past work or their identities, we make sure that our Levelers simply refocus that discussion. But if that discussion gets out of hand, which is very rarely, Levelers have the authority to discuss to stop the discussion.

Let’s move on to the 13:40 - discussion of the “Expertise and Resources - Not Anonymized” document. So again, the scientific evaluation of all the proposals is completed, then we distribute this document to panelists for a subset of proposals - typically the top third according to the distribution of assigned grades and projected selection rates. We ask our panelists to assess the team capability to execute the proposed investigation using a three point scale: uniquely qualified, qualified, or not qualified. We provide details of what each of those criteria and ratings means. We set the expectation that the vast majority of the proposals, say 98% or so, will fall into the qualified category.

Here are 14:28 - some answers to the questions that were submitted in advance of the webinar. The first one is this: if the identity of the proposing teams and institutions is shrouded in secrecy then how are the proposing teams and institutions to discuss their track record, their ongoing work, their complementary endeavors, the institutional assets and so forth? Our answer to that is as follows: the anonymous proposals have absolutely no prohibition on discussing these aspects, but we do ask that they be discussed without attribution to a particular investigator or group. In situations like this, we simply recommend writing “previous work” instead of “our previous work” or using words such as “obtained in private communication”. Remember, as always, proposals should be able to make their case through their description of their proposed program of observations and analysis and of course there is the opportunity for them to demonstrate and validate the claims using the “Expertise and Resources - Not Anonymized” document. Again, remember that the goal of dual- anonymous is not to make it completely impossible to guess the identities of the investigators, but to shift the focus of discussion away from the individuals and towards the proposed science.

Another 15:51 - question that was highly upvoted is this: why not just solve the demonstrable problems in selection practices at the Space Telescope Science Institute rather than change the systems across all of NASA’s Science Mission Directorate? Firstly I’d say we would actually dispute there any problems in the selection practices at Space Telescope: we actually think they run an extremely competent, fair and thorough review, but we would point out again that a key goal of dual-anonymous is to level playing field for everyone. Once more, remember that this is a pilot study for just four programs out of over 100 SMD ROSES elements. Another question: while it’s not possible for the proposing teams not to show any information in the proposals that might reveal their identities, such as the context and motivation of the proposed research, unique methodologies, and cited references, why keep the reviewers guessing, leading to undesirable consequences? Furthermore, the track records of the proposers should be part of the merits of the proposals. Once more, we respond by saying that it’s entirely appropriate for the context and motivations be addressed in the anonymized proposal, but again you must not attribute these to a particular investigator or group, or claim ownership of that work. Again, the track records of the proposing team will be addressed in the “Expertise and Resources - Not Anonymized” document.

Three very similar 17:21 - questions were asked, and they all circle around the concept of demonstrating whether there are sufficient institutional resources or access to custom-built facilities and laboratories, and indeed how in general does the proposing team’s ability to accomplish the research get evaluated? I hope this webinar has served as a reminder that what we’re really trying to do here is to separate out the anonymized science proposal from any claims or validation, which subsequently get discussed in the “Expertise and Resources - Not Anonymized” document. So I hope that this webinar has provided a little bit of confidence to the proposing community that the qualifications, teams, and experience of the PI and track record are not entirely eliminated, but they are evaluated rather in a separate step after the scientific evaluation of the proposal is completed. Another question was: what do you expect the unintended consequences of this action to be and does this really serve the meritocracy? Well the experience with the Hubble Space Telescope indicates that there are very few unintended consequences, but NASA is proactively taking steps to ensure the following: the SMD programs in this pilot lend themselves to the dual-anonymous peer review. In fact, they were very carefully chosen. We’re also ensuring the proposers have sufficient information and guidance to adequately anonymize their proposals. We’re making sure that review panels are sufficiently briefed about dual-anonymous peer review.

19:00 - We’re actually trying to make sure that the duration of each panel is not significantly increased. Once more, we’re trying to ensure that conflicts of interest are identified ahead of time and not during the review if at all possible. Then finally, we’re trying to make sure that high risk high-impact proposals are not disproportionately affected - we have a new SMD blue-ribbon panel to ensure just that. Some final remarks: NASA understands that dual- anonymous peer review represents a major shift in the evaluation of proposals, and we understand that there may be occasional slips in writing of anonymized proposals. However, NASA does reserve the right to return without review any proposals that are particularly egregious in terms of the identification of opposing team.

We 19:48 - further acknowledge that some proposed work may be so specialized that, despite good-faith attempts to anonymize the proposal, the identities of the PI and the team members are readily discernible. As long as those guidelines are followed, NASA will not return these proposals without review. Finally we want to make sure and we encourage all of our PIs and proposing community to plan adequately and feel free to contact your Program Officer or to email this address. Thank you very much for listening. Bye. .