Hicks – Open science can’t solve the replication crisis – DS² 2021

00:14 - It’s this weird kind of out of body disconnected like all right, yes, everything should be good. Now I believe everybody’s pulled across into the session. So without further ado, the last talk of the of the meeting, PS stick around, we will do a little tiny, I promise not too long. I’m not we’re not going to bore you. But we’ll do it. We’ll do a very quick closing remarks after this, there won’t be a break for that with a swap as quick as the machines will will let us pull over into into one little little final zoom.

Wonderful final last session for that. But the last proper talk of the meeting from a dear old friend of mine, Dan Hicks, who is going to talk to us yet again, looping back Today’s been a sort of articulated reticulated looping kind of kind of day to to the question of open science to talk about open to to to argue for it. So the title is the thesis that’s so nice, open science can’t solve the replication crisis. So please take it away.

01:15 - Yeah, well, so thank you, everyone, for sticking around for the last talk. Thanks to Charles and Luca, is that how it’s pronounced. I’m gonna guess it’s Luca for organizing this conference, I’m I’m on the west coast of the US. So because of the the time difference, and other responsibilities I I’ve only been able, I’ve really only been able to watch the talks on YouTube. Um, but super interesting, really looking forward to seeing the other open science stuff this morning that I had to miss because I was teaching literally now.

But yeah, okay, getting into my talk. So just as an overview, I’m going to start by giving some background on the replication crisis for anyone in the audience who might not already be familiar with that. Then I’m going to introduce the distinction between reproducibility and replicability and talk about some of the problems that have emerged in the discourse surrounding the replication crisis because of a failure to distinguish those two things.

02:24 - Then I’m going to talk about open science. I’m going to give you some background on Open Science talk about how open science is supposedly going to address the replication crisis, and then give my argument that, in fact, open science can’t solve the replication crisis. So first, the replication crisis is an ongoing epistemic crisis unfolding primarily in social and behavioral sciences as sort of a tagline the crisis is that, um, experiments especially some high profile experiments, such as priming studies, fail to replicate much more frequently than one might have expected.

The replication crisis really came to the attention of sort of the general educated science minded public around 2015 when outlets such as a 538 published pieces with titles like science isn’t broken, and the Atlantic published a piece with a title, how reliable our psychology studies um, however, the clouds of this epistemic crisis have been gathering for some time. Already in the 1960s. The psychologist Paul Mele has raised a lot of the concerns that we see today in the replication crisis.

In the 1980s per Gigerenzer, idag. GERD Gigerenzer, the psychologist was was raising some concerns about the misinterpretation and misuse of P values and statistical hypothesis tests that also appear today. In 2005, Johnny needies, an epidemiologist published a paper I was going to double check the title. It’s something like why most research why most published research is false, which attracted an enormous amount of attention.

04:17 - But really, the current crisis precipitated around 2011 especially in psychology with a couple of major events that happened in that field. So first, Daryl bem, who at the time was a very highly respected psychologist, published a paper purporting to find evidence of pre cognition using methods that at the time were regarded as perfectly acceptable. That same year, Another prominent at the time, prominent psychologist, Diederik stapel, was fired from both of his university positions for fabricating data.

These these these two events, really caused psychologists to look at to take a close look at the methods they were using the established standards in their fields. And within the year, the first large, large scale replication studies in psychology had launched by 2015, one of these large scale studies called the Open Science collaboration, published the figure you can see up at the top of this slide, which there are a few different ways to interpret this slide.

The most common interpretation was something like only about a third of psychology studies can successfully be replicated. I think that reading of the figure is a little bit strong. But certainly it attracted a lot of concern and attention.

05:48 - This slide is just a quick and dirty bibliometric analysis to give a sense of the disciplinary scope of the crisis. So you can see up at the top, the crisis really is has as its a center or Epicenter psychology, which has about twice as much discussion of a replication crisis as the next field that seems to be the most concerned about it, which is business and economics. But you can see there’s a lot of spill over into other social and behavioral sciences, primarily quantitative fields.

So you can see government law, which probably refers to political science, sociology, a bit further down geography, communication, education research, there is some partial overlap with some fields of biology. So you can see up towards the top biochemistry, molecular biology with some discussion, and then down at the bottom cell biology just barely making the cut off for this slide of more than 25 hits in the Web of Science search. In molecular biology and cell biology, though a lot of the concerns about ear replicability relate to things like being able to properly identify cell lines or reagents, which don’t quite aren’t really the issues in social and behavioral sciences.

The crisis has prompted a enormous and very vigorous literature, which is proposed numerous diagnoses, or I’m later going to call them bad habits that are thought to contribute to the crisis. I’m not going to go through all of these, I will highlight a pee hacking down in the analysis and interpretation session, because that may be a technical term that some folks haven’t encountered before. Pee hacking is a habit or an analysis practice in which researchers try various statistical analyses repeatedly with the same set of data until they’re able to achieve an analysis that is conventionally statistically significant.

So a p value below the conventional point oh, five threshold. And then they only report that final analysis that was statistically significant. And the problem here from the perspective of statistical hypothesis testing is that this approach completely validates invalidates what it is a p value is supposed to report.

08:10 - Um, I’m not going to really make claims here about which of these factors might be actually or not contributing to the crisis, which factors might be more important than others. The replication crisis is a social phenomenon. And like most social phenomena, probably, it’s over determined with a lot of these different factors not only contributing to the problem, but also interacting with each other, to make really complex causal relationships here.

08:44 - So with that background on the replication crisis, now, I want to draw out a distinction between reproducibility and replicability. And in this talk, the definition I’m going to use for those terms come from a consensus report published by the US National Academies of Science, Engineering and medicine, or the NA s in 2019. Early on in this report, the authors note that this distinction is really fundamental for understanding what the nature of the problem might be, and what might be done about it.

They defined a reproducibility in the blocks here are quotations as obtaining consistent results using the same input data, computational steps, methods and code and conditions of analysis. So reproducibility is really a very computational concept. The idea here is that if I’m attempting to reproduce some of your research, I have the exact same data that you use the exact same code that you use to analyze it. And in addition, I’m expecting to get if the reproducibility attempt is successful is the exact same numerical outputs from running that script on your same data.

So it’s a computational concept, whereas replicability on the other hand, as they put it is Obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. So whereas reproducibility is a purely computational concept, replicability is really about running another study and looking to see if we get similar results. A key difference here that the consensus report authors emphasize is that reproducibility involves the original code and data, whereas replicability involves new data collection.

And this difference also corresponds to the criteria for success of these two notions. So, again, reproducibility is looking for as the authors of the consensus report put it full bitwise reproduction of the original values with the exact same numerical values, whereas replicability is about qualitatively similar findings from qualitatively similar studies. And, oh Yana feast in a few recent papers has looked very closely at replicability disputes, and examined the ways in which this qualitatively similar notions open the space for possible reasonable disagreement about assessments of similarity assessments of whether one study is even a replication attempt of another.

I want to emphasize that reproducibility and replicability are logically independent deceit or Rada for computational or digital science. On the one hand, we can have a study where errors in the data analysis inappropriately exclude a range of data resulting in biased estimates, so this would not be considered replicable. Another study that gathered similar but different data and used a similar but corrected analysis strategy would reach very different findings.

This was a case in macro economics in the study that was very influential in sort of arguing for austerity politics after the Great Recession. This study, however, can still be perfectly reproducible, we can have the original data and the flawed data analysis and get exactly the same numbers from it, but it wouldn’t be considered replicable. Similarly, or conversely, we might have a study that is replicable, but not reproducible. For example, suppose the original data have been lost, there’s no way to reproduce the exact same numbers that were produced from the analysis because we don’t have the original data to plug into that analysis.

But if similar studies that collect new data from the same population tend to produce similar findings, then that original study would still be considered replicable.

12:44 - I emphasize this distinction because there’s been a lot of terminological chaos in discussions surrounding the replication crisis about these two notions. So for example, I have a quotation here from letterings are in 2018, which is a white paper published by two economists that’s been very influential in arguing for really stringent open data requirements in science used to make regulations in the United States. And they say, access to the data necessary to replicate scientific studies is essential because the results of so many peer reviewed scientific publications have been proven to be impossible to reproduce.

And they cite a bunch of studies to try to support this statement. However, the studies they cite, look at either replicability or reproducibility. But they don’t say any studies that look at both of these things together and establish the kind of relationship that they’re asserting between replication and reproduction. And in addition, that opening few, that sort of opening clause access to the data necessary to replicate scientific studies, that’s incoherent because by definition, data from the original study is never necessary for a replication attempt.

A replication attempt is always by definition about going out and gathering new data. And finally, in this section, it’s worth emphasizing that the replication crisis is really a replication crisis. It was really precipitated by concerns about replicability, not reproducibility. So in the pre cognition study by Daryl bem in Diedrich staples, fraudulent work, the concern wasn’t, oh, can we get their data and produce the same numbers that they reported in their paper? The concern was, if someone else ran the same study, would they get similar results? This is a replication concern.

In addition, the the the sort of large scale studies that have gotten the most attention and seem to indicate that there really is a replication problem, these have been replication studies, rather than reproducibility studies. This means that evidence of a replication problem isn’t necessarily evidence of a reproducibility problem and vice versa. And it also means that things that promote reproducibility don’t necessarily promote replicability.

And I think Open Science clearly reproduce, clearly promotes reproducibility. But it’s not obvious that it promotes replicability just because it promotes reproducibility. Okay, so now let’s talk about open science. Open Science, and specifically, a movement for open science has been a very popular response to the replication crisis. Proponents of open science often adopt an attitude that kind of facetiously, I call open science to the rescue.

One very prominent proponent of open science has been the psychologist Brian nozick. And I have some quotations here on the bottom of the slide with various papers, where he’s been a co author. In one relatively early piece, he called Open Science, the ultimate solution to the replication crisis. Elsewhere, he’s argued that reproducibility increases confidence in results, and that transparency is superior to trust. So you’re making assertions that I think about the connection between the reproducibility promoted by open science and replicability.

The Open Science movement has been rather successful, especially in psychology in getting some of its recommendations adopted by journals, for example. So up at the top, this is a screencap, of an article that was recently published in I forget which, but one of the major psychology journals, I’ve highlighted some badges sort of towards the lower right of the screencap. These badges indicate that this study has implemented various Open Science policies.

So authors are being recognized and thereby explicitly rewarded for adopting Open Science practices. There are a lot of different things that people mean when they talk about open science, I really recommend this survey study by 11, at all from 2016, which looked at was researchers in the UK about the various not just policies that they mean by open science, but also the values they associate with open science. I just want to note very quickly how I’m going to define open science for the purposes of my argument.

Specifically, when I talk about open science, I’m going to refer to open data and open code that is practices where researchers published the data used in their study, and the code they use to analyze that data, where researchers publish those things in publicly available repositories. This might be GitHub, this might be OSF the Open Science Framework, which was developed by nozick and other advocates of open science. And I want to be explicit that today, I’m not talking about pre registration, or the specific form of pre registration and called registered reports, or open access publishing.

These things are often included when people talk about open science or open science practices. During the q&a, I can explain why I want to exclude them from the argument I’m making today. So now we have the question. Now we turn to the question of whether open science will help with replication problems. My claim is that open data and code will help with replication problems only insofar as three criteria are satisfied. First of all those bad habits those purported causes, or factors contributing to the replication crisis back from part one, those habits need to leave traces in the data or code so that by inspecting open data or code, we discover that these things might have played a role.

Second, these bad habits shouldn’t already leave traces in papers or other traditional academic publications, because otherwise, we can just look at the paper, we don’t need to go to the data or code. And third, open data or code need to help prevent or mitigate the bad habits. It’s somewhat helpful if they help us detect that there’s a problem but what we really want are new habits or practices that will help prevent the problems in the first place.

19:29 - So what I’ve done on this slide is I simply went through the list of factors, quote unquote, bad habits from section one. And I’ve I evaluated them according to these three criteria. What you should notice immediately on the bottom half of the table is that for a lot of these things, I don’t think open science will help at all. So take for example, imprecise theory and underspecified phenomenon. One concern in the replication crisis is that often social media All scientists are kind of vague about the relationship between the theory they’re trying to develop the phenomena they’re trying to characterize and their data, this vagueness isn’t going to leave any sort of trace in the data or the code, it’s really a matter of how things like the data and the experimental design are interpreted.

And so I don’t see how open science can help with these kinds of issues in any way whatsoever. Jump up to the top for one issue where I think open science will certainly help and that is detecting and dealing with software bugs, or that could be either code that the researchers have written themselves or packages or other dependencies that they’re using in their analysis. If I publish my data and my code and you access it, attempt to reproduce it, and you get an error message, or you discover that I’m using a package that’s considered to have flaws and shouldn’t be used anymore, then yes, open data and code have definitely helped you detect that problem, and can help mitigate that by for example, suggesting changes to the code through a system such as GitHub issues.

I think open science has a much more situational value when it comes to issues like p hacking, or data mismanagement, and just for the sake of time focusing on p hacking. So if I am a researcher working on my analysis, and I try a dozen different variant analyses, and include them all in my code, and then publish my code, and you go and look at my code, you can see I tried a dozen different variant analyses, I only reported the last one, you can see that I’ve engaged in pee hacking.

So that’s a case where open sides could be helpful in directly detecting pee hacking. However, suppose rather than dutifully recording every single variant analysis, I try, I try an analysis, I write a few lines of code, it doesn’t work out in the sense of not giving me statistical significance. So I raced those lines of code and write a new variant analysis, and I keep repeating this process until I have something that’s statistically significant.

Well, then what you will see when you go and look at my code that’s finally published is just the one analysis that I reported in the study, there won’t be any sign of pee hacking. So in this case, open science doesn’t help detect pee hacking. Um, in terms of prevention, or mitigation, so suppose I’m already planning when I’m writing my code to publish the code that’s used in the final paper. Okay. So I know I’m going to be publishing open code.

And I know other researchers are going to potentially see this, I don’t want them to think that I’m p hacking, and so I’m not going to be hack, Okay, with that train of thought, then open code has helped prevent p hacking. On the other hand, my thought process might be okay, I’m going to publish open code, I don’t want people to think I’m pee hacking.

23:04 - So I’ll pee hack anyways. And then I’ll very carefully erase all of those other analyses that quote unquote, didn’t work.

23:12 - Well, now, open data hasn’t done anything to prevent p hacking.

23:17 - Okay, so the ability of open science to prevent or mitigate and detect things like pee hacking, I think is much more situational depends a lot on how researchers actually respond to open science. I think you can tell a similar story about data mismanagement. But I’m already a little over my target time. So I will come back to that in q&a if people are interested. I will note about fraud I think, well, open science can probably helped detect fraud in the present day.

We’re really committed fraudster who wants to fabricate data, I think can adopt simulation techniques used by statisticians to evaluate methods as is to create a simulated data that would completely past current data forensic techniques would look totally legitimate, but in fact, it’s completely fabricated. So I don’t think open science will really help with fraud either.

24:16 - Now, I do think open science can be good and valuable, but not because it will solve the replication crisis. In my own empirical work. I use open science practices. I teach data science methods courses in my graduate program, where I encourage my students to learn and use open science techniques.

24:34 - I think open science is good and valuable, but because of the way it’s valuable for teaching and learning by understanding how a researcher conducted an interesting study, I think open science can be valuable because of the way it can be used to extend prior research with reanalysis or data aggregation.

24:52 - And under the right circumstances, I think open science can support participatory research and research in the global south. I think open science can To be good and valuable for all of these reasons, but I don’t think it will solve the replication crisis. Thank you.

25:10 - Fantastic. Okay. Um, again, I’m going to keep doing my usual cuz this has been working so well, I’ll take advantage of the of the tape play to get my own first question in this is like not what you would never do if you were the chair in real life.

25:23 - But for some reason it makes sense here. So this is related to the I assume to the traces not being being fundable in the in the papers point I’m, I want to I want to pick up the open access publication question you might have expected that I did, I might have picked that up, because it’s one of my things.

25:42 - So So yeah, what do you how does that change the landscape? Do you think I mean, I mean, obviously, there’s, there’s questions about, there’s separate ethical questions about financing and democratization of access and all that kind of good stuff. But as far as far as it being an arm of the Open Science movement. How do you how do you read that? Um, okay, I just inhaled some tea as he started asking her question. So I want to make sure I understood the question correctly.

Um, so were you asking, like, why did I draw this line here? Why did I exclude Open Science here? or open access? Yes, in short, yes.

26:27 - Yeah, I’m good. There are a couple of reasons here. One is, so on the one hand, I think open access publishing is included because it sort of exemplifies a lot of the values that proponents of open science point to in other contexts sort of notions of transparency, giving sort of everyone access to knowledge and sort of this mortoni income communism sort of sense. Okay. Um, so I think open access publishing gets bundled in because it exemplifies a lot of the same values.

However, I haven’t seen anyone sort of argue, for example, that open access publishing will somehow do something about the replication crisis. Specifically, this is a longer answer, that I will try to avoid giving because it can turn into a rant, I got specifically interested in this over some proposed policies in the US federal government, I kind of gestured before at a strong Open Science requirement. That was, in some sense, implemented for about five minutes at the US EPA and the Department of Interior that really would have caused a lot of problems developing protective regulation.

That was sort of how I got interested, a big part of why I got interested in developing the argument of this talk was responding to that initiative. That initiative was focused entirely on open data and open code with nothing in there about open access publishing. So there was sort of that arm of why I was less interested in open access publishing was because it wasn’t sort of the problem I was trying to address in here.

28:23 - That makes that makes good sense. A question coming in from from Stefan, his plugin, who writes a gecko, good. So eating our own dog food? What’s the value of reproducibility? Not replicability in in digital science studies, and what’s the value of open science in the kind of stuff that we do? Yeah, good. Um, so I actually, I had a slide that I said, that that I decided to cut for the sake of time, a pair of slides, actually, one talking about the value of replicability and the other talking about the value of reproducibility.

Um, what I say on the reproducibility slide is that I think reproducibility is good, invaluable in much the same way that internal logical consistency is valuable in scientific theories, okay. Um, if a piece of computational science isn’t reproducible, specifically, in the sense where, you know, I get your data and code and try to rerun it, and I get a bunch of errors, or I get different values from what you reported in the study. Like we have reason to think something has gone pretty seriously wrong.

It’s hard to tell what just from that, but like, that is at least a yellow flag. That your numbers don’t seem to match up with the numbers in the paper. I think that’s a reason to be concerned. So much like logical consistency and the theory like if the theory is logically inconsistent, we’ve got some reason to be concerned about the quality of the theory. That said once that criterion is satisfied, reproducibility doesn’t do much to establish the credibility of a piece of research, right? So you might actually go back to this case right here.

So this macro economics example, completely reproducible, actually the way so Herndon, Ash and Paulin were the folks who got the Excel spreadsheet from the original authors and figured out they were dropping a bunch of countries from their analysis.

30:34 - And when that was fixed, the analysis actually supported exactly the opposite conclusion from the original authors. Okay, um, this was entirely reproducible, right, they could get exactly the same numbers, because they had the Excel spreadsheet bundling together the data and the analysis code.

30:52 - But that didn’t establish the credibility of the study that actually helped people figure out why the study was not credible at all. Um, and so I think the same thing about other cases of reproducible research, it’s like, it’s pretty important to have, but once you have it, it’s not going to take you very far. Um, so jumping back here now. So in terms of promoting open science, in sort of digital hpscs, what I would give are the arguments I give on this slide.

I think it’s incredibly valuable for teaching and learning to be able to look to see how another researcher conducted their analysis and kind of poke through it and work through it step by step. This is actually how I learned a lot of the data science I learned was reproducing published studies that had open data and code. And it’s also something I use in my own data science courses, I have my students look at a study and try to reproduce it.

And we learn a lot from doing that. I’m similarly making the same points here about this can allow us to extend our research if we can build on each other’s data sets. And this can also make our research more accessible to community partners if we’re doing community based research, or researchers in the global south.

32:17 - Fantastic. Next question. Let’s see just double checking. Yes.

32:21 - Next question comes in from from Sarah one who writes, Sue says, Thanks, Dan. I’ve wondered about the social forces in play in thinking of of open science as the solution to the replication crisis. So for example, is it less directly confrontational to say, Please be transparent about what you’re up to in your studies than it is to say, Please stop doing pee hacking, designing studies badly behaved other kinds of bad behavior in science? Yeah, yeah.

Um, I was dm dming, with Sarah yesterday about this.

32:52 - So I know some of the broader context that I said in our DMS that I wasn’t going to talk about the particular thing she was asking about, and I’m going to stick by that I’m not going to comment on a very recent open science, bad thing. That that’s been unfolding over like the past 24 hours. So I won’t come on comment on that. Um, in terms of, so I think part of what Sarah’s asking there is sort of carrot versus stick. Um, and so commenting on that, yeah, I think in general, carrots are preferable to sticks in this context.

Um, you know, what I’m giving on these arguments here are really carrots right there, here, all of these additional good things that your research can do if you do it, using open data and open code. I also think, although I also think that for example, software bugs, um, using open science practices and being sort of really conscientious about trying to learn how to do it, well, can lead you to become, in effect, a much better software engineer, which is something that you know, even in sort of the the fields of science, where open science and computational methods are widely used.

So for example, psychology and psychology and cognitive science on the students generally aren’t explicitly taught software engineering principles that will make their code like more reliable, and more reproducible and potentially more replicable. In terms of sticks, and the social dynamics of it, um, I think strong open access requirements are a mistake, just because there are I did, I didn’t have a slide here. I didn’t have time to put it together, because this week has been a mess.

Um, there there are downsides to open science. Um, there are a lot of concerns about, for example, publishing open data with sort of anonymized patient medical information, because a lot of that can be de de de identified or re identified. There are concerns in the context of ecology about publishing locations of endangered species, because that tells poachers where to go. There are a lot of concerns in archaeology and certain areas of genomics about publishing data on indigenous communities, because for example, then, you know, outsiders can know where their sacred sites are, and trample over them, which is, you know, really not good.

I think there are enough sort of provisos on Open Science, that it’s better to teach scientists how to be thoughtful and in other ways and virtuous about their use of open science, rather than sort of more, let’s say, deontological approach that there are these rules that you must obey these, you must follow these practices.

36:07 - And we’re going to wrap your knuckles if you don’t do that.

36:13 - Very nice. By the way, I just want to say cuz I’m violating the upvotes. But I want to note that that are the belfry that that question is so cool. I’m holding that for the last question. I actually want that to be the last question, because that’s awesome.

36:27 - Oh, no, no, no. Good way, you’re gonna like it, you’re gonna like it, you’re gonna like, Okay, next, next one, then coming in from from Rose trappists. Who’s Who? So as you know, maybe I missed this. But don’t you at least think that the publishing data makes fraud more difficult, like now you can’t just copy paste and mix and match values in an Excel spreadsheet? Like the like, actually a lot of the real real scientific fraudsters Have you ever been doing? Yeah, so I would agree that, um, open data makes detecting fraud as it’s currently done easier, and thereby makes it harder to get away with fraud under current circumstances.

However, I think this is a very short term thing. Um, the methods used to create what’s sometimes called fake data or simulated data. These methods are widely used by statisticians and methodologists, to basically look to see, you know, do your analytical methods do what you think they do. This is something that’s status that in undergraduate statistics courses are increasingly being taught, because they’re actually really, really useful.

But to get students to understand like, how actual data gets translated, and like what things like statistical tests are actually doing. So these methods are already being widely adopted. And they’re very easy it takes I mean, depending on the complexity of your data, it can take just a few lines of code to write a simulation to produce data that I think would really pass any current forensic methods for detecting fabricated data. And so I think over the medium term, much less the long term, I don’t think Open Data can really help with fraud.

I think a committed fraudster can find, you know, can take a couple of days to learn how to use these methods, and then simulate perfectly legitimate looking data.

38:35 - Sure, sure. Next question coming in from Max writes, would you share some thoughts on the role of theory in replicability and expand on what you what you already briefly mentioned, it seems as though your point is that open science is sort of overtly focused on methodology. Can we say that in order to get better replicability besides open science, we’re missing? We’re missing good theory? Yes, yeah. Um, okay. I don’t want to just say like, I really think sort of the study conceptualization level is probably the level where is probably the most important level for a lot of these replication problems.

I kind of think that but I’m enough of an outsider to psychology and psychological practices that I don’t want to like, really state like really assert that very, very strongly. I will note, there is a lot of literature happening in psychology, other social and behavioral sciences, philosophy of psychology, on these sorts of issues about the way psychologists use theory about what they mean by theory and also like sort of down the down the ladder at sort of the level of phenomena characterization, like our our psychologists actually measuring the things that they think they’re measuring.

When they measure them, again, oh, Yana feasts. her recent work on this is really fantastic, really, really like fine grained, careful case studies, of how these things play out in practice, like how psychologists try to characterize phenomenon, and how sort of a failure to recognize disagreement over phenomena characterization has led to replication controversies in some particular cases. So I really recommend, I’m really it was her work that made me think like, Oh, you know, maybe it’s not sort of so much on sort of the statistics, data analysis, end of things, maybe there’s like something quite a bit deeper here.

And I’m in I’m not in a philosophy department, because we don’t have a philosophy department here at UC Merced said, I’m in a cognitive and Information Sciences Department. And so my colleagues are cognitive scientists, including a few psychologists and this is something they talk about quite a bit, as well.

41:16 - cool to see that it’s getting, it’s getting uptaken with the practitioners. That’s that that’s good stuff. Um, next question comes from Eugenio Petrovich, who asks. So sometimes we have to work with proprietary data, full text of publications records from Web of Science or Scopus, they just can’t be open, because their proprietary are protected by copyright and their entire fields like scientometrics that are that are affected by this kind of issue.

What do you what do you think about this? What are people? What are people saying about this? Yeah, that is something I have run into repeatedly. Um, I started doing sort of bibliometric studies 2015 ish.

41:54 - Um, and I’ve published several papers, excuse me, that you Scopus data, and what I have to say is, I can show you the code here it is on GitHub, I can’t give you any of the data I use, because Elsevier says that that belongs to Elsevier. I know there have been initiatives. I want to say it’s the it’s the umbrella organization that does do I’s. Cross ref. Cross ref.

42:29 - Yeah, I think it’s cross ref. Um, I think they have an initiative to try to include, at least like the reference lists, making those as open as possible. Um, but that requires working with publishers. And so whereas like, plus is totally cool with it. Like Elsevier and Springer have been really hesitant to make the references open. Um, I think it’s also beer and Springer, I might be misremembering exactly which companies so I don’t want to necessarily give people yet another reason to dislike Elsevier.

Um, yeah. Oh, and I guess I should also note, for a couple of years, I had a postdoc that was funded by Elsevier. Just full disclosure there that’s in my past now, but just so people know that. Um, yeah, so there, I think there have been initiatives to try to get at least, you know, like, the reference lists or the citations openly available, but it requires working with publishers who have been really hesitant to do that, um, J store has done a lot of really cool work, of course, on making the text of the journals that they archive available for text mining and bibliometrics, which is really, really cool.

I have a couple projects that for various reasons have been installed.

43:54 - Text mining j store data, and that’s great. If you’re working with journals. Jason Yeah. So I share the frustration there. Um, and I don’t unfortunately see, like any immediate solutions on the horizon.

44:14 - Yeah, that seems that seems that seems right to me. And I agree.

44:19 - I agree wholeheartedly. I have two more questions here. One that I think let me let me let me lean on you to answer a little briefly and then and then one that I think will take a little more time. But one from Nicola Batali who says a great talk and the answer. Would you say that the replicability crisis is more about technical problems, like statistical power and significance, or more about collections of social and epistemic norms.

44:46 - Okay, I’m going to do a very characteristic thing for me, and it’s both. If you know my work, you know that I love sort of arguing that like, these very technical problems are always So social problems and matters of social norms. And I think just to pick out, you know, pee hacking, I think illustrates this really well. This is both, in some sense, a technical problem, because it’s, um, you know, it’s sort of it technically violates the assumptions of statistical hypothesis testing.

And so your P values become completely meaningless. On the other hand, it’s also a sort of social problem of norms, because like one of the big things people realized after Darryl bems publication is that it as I understand it, right, I haven’t looked at it too closely. But like, the way he got his results was pee hacking. And so there was this realization that’s like, Oh, we have this practice that is perfectly normal in our field, like, we kind of teach our students to do this.

And it completely violates, like, all of the statistics that we teach them at the same time. So I think it’s both a technical issue as well as a social issue a matter of the norms in the field.

46:04 - Fair enough. Yeah. Like that. And then lastly, so I love this as a list as a final as a as a final question. So from lalibela, folks who writes so we can make our data open so that everyone can see it, we can situate ourselves, in terms of indicating our personal biases. We’re aware of power dynamics, we’re respectful of indigenous and community ownership of data, we’re as transparent as we can possibly be with our methodology, our funding, etc.

What else do we do as researchers to build the kind of ethical data that we seem to be aiming for? Oh, a nice, tiny, nice, small question for you.

46:45 - Yeah, and I love this, because this really brings together our whole day, right, this is that this has been a this has been the sort of sort of underlying theme, I think of a lot of the talks today. So I wanted to I wanted to get your thoughts.

46:57 - Okay. Um, I’m going to bring in an answer from the major area of my work, which is in science and values. So if you if you know, my work as a philosopher of science, you probably know me for the work I’ve done on science and values, arguing for this kind of Aristotelian approach to thinking about values in science. Um, something I’ve really emphasized in this work is building relationships. Because a lot of the science I focus on is like, Environmental Public Health, it’s very directly tied into public policy.

That’s the kind of science I’m most interested in. So what I emphasize in the context of science and values is strengthening relationships between scientists, policymakers and the public. Where I give as sort of an exemplar of doing this. Well, certain instances of the environmental justice movement, were members of communities, often impoverished communities of color situated right next to sources of pollution, suffering, all kinds of health effects, as a result of that are doing their own science, partnering with scientists in order to sort of get the kind of institutionalized credibility that they need in order to get regulators to take them seriously.

Where I want science to go is sort of in that direction. So if you ask me, you know, you step back from that question and ask like, what do I mean by good science, I mean, science that is used to make people’s lives better, rather than worse. And I think the way this really needs to be done is sort of building connections to other communities, especially, for example, environmental justice communities that like our most desperate have the biggest need for the support of scientists to pursue goals of justice and human well being.

So that’s why, you know, we come back to this slide. That’s why I think actually, like the last argument on here are the ways that open science can under the right circumstances support this kind of participatory research and also research in the global south. To me, this is actually the most compelling argument for open science, because under the right circumstances, it really can move science in a really powerful way towards promoting justice and promoting human wellbeing.

The citations I have here, do a really nice job of sort of looking at how open science in practice doesn’t necessarily do this super well.

49:56 - And like things that might change to do this Better.

50:02 - Fantastic with that, I think we will leave it there. Again, as I mentioned, we’ll be back in in as quick as the platform will let us be back with a with a closing with a closing message.

50:13 - So so so thanks again, Dan. Much appreciated. We will be back in just a few moments. Thank you.