CHAOSS Value Working Group 6-17-21

00:07 - Vinod Ahuja: So we are recording welcome to the class value working group meeting.

00:14 - Vinod Ahuja: 17. Vinod Ahuja: Yes, so.

00:18 - Vinod Ahuja: Stephen is here that you make on research and application they propose that true.

00:29 - Stephen Jacobs: Okay Oregon Kevin. Georg Link: Oh.

00:48 - Vinod Ahuja: Can you share what this is about.

00:51 - Stephen Jacobs: yeah so i’m. Stephen Jacobs: The national academies.

00:57 - Stephen Jacobs: Are kind of big guns. Stephen Jacobs: In.

01:03 - Stephen Jacobs: academia used to means old TV.

01:05 - commercials. Stephen Jacobs: For. Stephen Jacobs: NGOs Merrill Lynch one of the other financial groups right for their cocktail parties or their their dinners out or whatever and.

01:19 - Stephen Jacobs: And there’s lots of parking and bustle in the background and one guy leans over to somebody else’s.

01:24 - Stephen Jacobs: By broker is Merrill Lynch and Merrill Lynch says in the whole room goes quiet everybody it’s going like this that’s kind of like the national academies in academia is like they speak up hmm.

01:39 - Matt Germonprez (he/him): That is no reference, by the way.

01:41 - Stephen Jacobs: Yes, I know I know, but it still works right.

01:45 - Stephen Jacobs: I refrained from adding a YouTube link to the old commercial so you try not to take us too far down the road.

01:57 - Stephen Jacobs: We know there’s specific call outs.

02:01 - came down. Stephen Jacobs: To to the stuff that we’re trying to figure out with with academic value and you define the ruling.

02:11 - one. Stephen Jacobs: So this is. Stephen Jacobs: The the invitation letter is for.

02:23 - Stephen Jacobs: University Presidents in Provence to.

02:27 - Stephen Jacobs: designate someone to go to the meeting to create the beginning of a word of colleges and universities to advance open scholarship practices.

02:42 - Stephen Jacobs: And so, as part of that. Stephen Jacobs: In the letter itself.

02:46 - there’s. Stephen Jacobs: Here, this bullet review and update university open scholarship policies to harmonize with blah blah blah.

02:57 - Stephen Jacobs: So there’s some kind of. Stephen Jacobs: relationship there and services and training toward fair principles of research, but especially track progress or open scholarship well as sharing the new scholarship standards and policies set by research sponsors are satisfied.

03:16 - Stephen Jacobs: that’s just the body of the letter and then.

03:21 - Stephen Jacobs: This two page document. Stephen Jacobs: That there’s a link to in the middle of it, the guide to supporting open scholarship for President and provost.

03:32 - Stephen Jacobs: has again to call out to policy work about reviewing how tenure and promotion or valuing diverse types of research privates metrics and incentivize the open dissemination of articles data and other research outputs and value in collaborative research.

03:53 - Matt Germonprez (he/him): So there, this is interesting because, looking at the two page letter that PDF.

04:00 - Matt Germonprez (he/him): One of the things that.

04:03 - You see if I can share my screen. Matt Germonprez (he/him): So down in here services and training.

04:13 - yeah. Matt Germonprez (he/him): A lot of this a lot of these are like here are the things that you should do they don’t really give much guidance as to how to do them.

04:25 - Matt Germonprez (he/him): And metrics could perhaps start drying some of that out.

04:30 - Matt Germonprez (he/him): We are already talking with the fair for RS folks so on, do you know, Dan cats it went away.

04:40 - Stephen Jacobs: I know that name. Matt Germonprez (he/him): Okay, and then Michelle Barker who is she’s down in Australia.

04:47 - Stephen Jacobs: So and Mike in our current full stack person that’s been devoted to this me.

04:56 - Stephen Jacobs: are making some real progress in.

05:00 - Stephen Jacobs: bouncing around between osf and personable and from war and that kind of stuff they have some specific technical questions Mike was going to see if you could try and drop in today but.

05:14 - Stephen Jacobs: He might also reach out to Sean or somebody just asked who they should talk to address some things the API.

05:22 - Stephen Jacobs: doesn’t expose that they like it to.

05:25 - Matt Germonprez (he/him): Some of this stuff is like probably not software a bowl.

05:31 - Stephen Jacobs: yeah oh yeah. Matt Germonprez (he/him): You know so like, even if I look above like policies academic hiring tenure and review and promotion like this whole this whole thing.

05:43 - Matt Germonprez (he/him): I mean we’ve been down this road, before right like in terms of an rpg process.

05:49 - Stephen Jacobs: yeah well, I think. Stephen Jacobs: there’s a lot more activity around the state.

05:58 - Stephen Jacobs: That I put that I think was a Harvard law thing on.

06:04 - Stephen Jacobs: on how the software is scholarship that I put that in here at any point.

06:09 - Matt Germonprez (he/him): not sure I wasn’t weeks ago so maybe if you did.

06:16 - Matt Germonprez (he/him): find it now it’s I mean Stephen are you thinking like how the work we’re doing here could supplement this document because I would see.

06:29 - Matt Germonprez (he/him): kind of the list of things that are in that two pager as being potential high level candidates for.

06:36 - Matt Germonprez (he/him): kind of further definition.

06:43 - Stephen Jacobs: um I think it’s the universe, is going to be looking to things to do this.

06:52 - Matt Germonprez (he/him): Right so like. Matt Germonprez (he/him): If I just picked like a highest level thing like financing open scholarship right.

07:05 - Matt Germonprez (he/him): Like what would be.

07:08 - Matt Germonprez (he/him): The ways that we could.

07:12 - Matt Germonprez (he/him): Are you thinking, what are the ways that chaos could help kind of define what.

07:18 - Matt Germonprez (he/him): metrics would be around how we determine financial support for like where to look, you know what I mean like do you have a grant program for for providing open access, do you have a.

07:30 - Matt Germonprez (he/him): what’s that what’s that thing we have here at you know, a word or even on the open digital commons thing you know what i’m talking about.

07:40 - Stephen Jacobs: Some of it will be that kind of guy I mean I think all of that are pieces right think it’s also i’m.

07:48 - Stephen Jacobs: Looking at this merger between the kinds of things, the.

07:52 - Stephen Jacobs: Chaos does and. Stephen Jacobs: The OSS open science format.

08:01 - Stephen Jacobs: We open science platform that’s that are proven science it’s like what what’s the Van what could they do independently or together to kind of support some of this stuff.

08:18 - Stephen Jacobs: here’s the. Stephen Jacobs: Software a scholarship thing from MIT.

08:28 - Stephen Jacobs: Just from last year. Stephen Jacobs: provides the first systematic account and justification of software applications as work of stock.

08:48 - Stephen Jacobs: And then the other thing I found recently is.

08:57 - Stephen Jacobs: This. Stephen Jacobs: So you know, the question is always, how do we get beyond just the raw metrics of downloads right.

09:15 - Stephen Jacobs: So what we do, we know that.

09:21 - Stephen Jacobs: Ls insights is is cracking.

09:28 - Stephen Jacobs: your attendance at their conference right for your presentations on your stuff at their conferences and what conferences you’ve attended or going to try to make that a model for other things.

09:46 - Stephen Jacobs: I think the real. Stephen Jacobs: The real impact of the nascent stuff is.

09:57 - Stephen Jacobs: Around open science, there have been journal articles and research studies and says no broadcasters think this is good idea, but most of them don’t actually follow the practices right.

10:14 - Stephen Jacobs: Now they don’t follow the practices, because they’re not trained it’s more work, there is no support for it right i’m going to do all this work to be open and yet all the university cares about as the journal article anyway, so I went bothering.

10:28 - Stephen Jacobs: Right, so what this nascent piece does is, I think, turn the heat up on that to make universe, the universe, is trying to better value the open work that faculty are doing so they’ll do more of it.

10:44 - Matt Germonprez (he/him): OK, so the the path that i’m kind of seeing is the chaos is a project is focused on trying to improve transparency on Community health right and value is one of those.

11:03 - Matt Germonprez (he/him): And so the intersection as faculty members are producing software artifacts like, how do we ascribe.

11:15 - Matt Germonprez (he/him): Academic value in that software work, and so what would be the metrics that we would need to develop to help people administrators for people that aren’t the accurate in the academic themselves.

11:31 - Stephen Jacobs: Right it ends up it ends up being the academic themselves.

11:33 - Stephen Jacobs: Because, as as time moves as way the the academic cast defendant.

11:41 - Stephen Jacobs: Did we won’t it isn’t just the admins defining it it’s the faculty members here’s a you have to pitch right, this is why this is important, this here’s here’s my Community around my research right.

11:53 - Matt Germonprez (he/him): Right and so, if cass can help that pitch by saying here are here are actually published metrics.

12:00 - Matt Germonprez (he/him): yeah he’s so like the way that the the MIT article, it was MIT right that you shared like that helps the pitch right you’re like listen MIT is arguing.

12:13 - Matt Germonprez (he/him): care about this and.

12:15 - Matt Germonprez (he/him): The most of that.

12:16 - Stephen Jacobs: Even that health policy thing helps.

12:18 - Matt Germonprez (he/him): yeah right so and then, if we can publish metrics it really says here are the here are the things that kind of make my case and here are the things that we can actually look like.

12:29 - Stephen Jacobs: He but but part of it is also trying to understand how the non software, people are different.

12:36 - Matt Germonprez (he/him): So I have them the non stop for people, or what.

12:39 - Stephen Jacobs: What are different. Stephen Jacobs: Right, so my.

12:44 - Stephen Jacobs: My my summer students staffer me.

12:48 - Stephen Jacobs: is looking at USF and she. Stephen Jacobs: says that she was trying to set up grow more impersonal again we’re trying to get more and more of those two platforms to talk to each other, share data.

13:00 - Stephen Jacobs: And she said sorry, they said.

13:06 - Stephen Jacobs: there’s no forks on any of these repositories.

13:10 - Stephen Jacobs: You want me to follow ford’s want me to make a dashboard item for books, I said yes, because.

13:17 - Stephen Jacobs: First of all, you’re only looking at the alrighty people you’re not looking at everybody.

13:22 - Stephen Jacobs: secondarily there’s value in here is this open science platform, or you can share a lot of stuff one of the things you can share insert software, but there’s also these other artifacts and your research.

13:38 - Stephen Jacobs: But it turns out. Stephen Jacobs: numbers of osf are members of the Center for open science aren’t actually forking each other’s work.

13:48 - Stephen Jacobs: webinar. Stephen Jacobs: Right is it.

13:53 - Stephen Jacobs: Is it replicates so you just download it and copy it, you know really for off of it is it that you don’t know.

14:01 - Stephen Jacobs: That you can port and make your own thing and upload a variation is a, you know as a part of the practice now pro practices that ignorant why right.

14:12 - Stephen Jacobs: The ios APP platform will pull in stuff as well chaos my form from your github stuff it’s software wise nobody 4k what’s what’s the.

14:27 - Stephen Jacobs: what’s the science behind one metric of a fork and how do you track right, how do we find a way to say, well, yes, people are actually.

14:35 - Stephen Jacobs: replicating my work, but it does the way to show big that show up is through aside journal article CITES, there are also these other ways in which the fact that people are using my stuff shows up and what are their.

14:49 - Matt Germonprez (he/him): So listening to talk I jotted a note down to like maybe from a cost perspective when something like downloads or forks or like number of closed issues like kind of these repository II things.

15:06 - Matt Germonprez (he/him): right too granular of a metric and I dropped a totally random metric in there that’s called like repository software activity, it could be measured in a variety of different ways, like.

15:20 - Matt Germonprez (he/him): to your point like forks, as you know, forks may or may not be a good measure, depending kind of on the context downloads may or may not be a good measure kind of again, depending on the context.

15:32 - Right. Matt Germonprez (he/him): Issues right issues may may or may not be a good measure, because somebody may be using bugs zilla.

15:40 - Matt Germonprez (he/him): You have zero in a repository management perspective, like just on github so you know, maybe from an academic perspective, just to start like just to kind of like break the ice on this it’s something like.

15:56 - Matt Germonprez (he/him): repository software activity.

15:59 - Matt Germonprez (he/him): And another metric is something like you know software citations.

16:09 - Matt Germonprez (he/him): You know, like. Matt Germonprez (he/him): Our people.

16:15 - Matt Germonprez (he/him): it’s like i’m thinking of like the James house and stuff site, as you know, like are people actually citing.

16:22 - Matt Germonprez (he/him): My software in the research that they’re doing, and maybe try to like think maybe we abstract a little bit at least a start.

16:32 - Stephen Jacobs: yeah and here’s something that literally just appeared as I was.

16:40 - Stephen Jacobs: was listening to you um there’s this group on Meta data for reprints.

16:48 - Stephen Jacobs: That may be another piece of.

16:51 - Stephen Jacobs: You know I don’t know what.

16:54 - Stephen Jacobs: How much of this stuff already does with crossover is but.

16:57 - Matt Germonprez (he/him): I think pre prints are really important too, because so.

17:03 - Matt Germonprez (he/him): So there’s the published journal article which from from concept to publication, as we know, can take five five years and so.

17:16 - Matt Germonprez (he/him): Pre prints really became pretty critical during coven because in the medical community waiting for a paper to be published in a journal as having scientific impact, it was just too long, like the cycle was just too long.

17:33 - Matt Germonprez (he/him): And so pre print serve as a way to get that scientific information out.

17:39 - Matt Germonprez (he/him): To a Community quite a bit faster it’s just more timely than then oftentimes journal articles are.

17:47 - Matt Germonprez (he/him): And so tracking so I don’t know if site as tracks software insight as as a piece of software that tracks software citations in articles and I don’t know if they track it in pre prints but that’s I think the printer a good point.

18:04 - Matt Germonprez (he/him): Here, that my son is dying in the background, I think.

18:07 - Yes. Matt Germonprez (he/him): mean other other ways that people can think of that.

18:34 - Matt Germonprez (he/him): You know, software.

18:41 - Matt Germonprez (he/him): Pro software falling under the fair principles.

18:45 - Matt Germonprez (he/him): So fair is a there’s fair for research software which is fine double accessible interoperable and.

18:54 - Matt Germonprez (he/him): reusable. Matt Germonprez (he/him): yeah yeah and so some measure by which.

19:02 - Matt Germonprez (he/him): A piece of software is fine double some measure by which.

19:07 - Matt Germonprez (he/him): Or, as determined as findable or can be determined as findable some measure by which it’s considered interoperable and so on and so forth.

19:17 - Matt Germonprez (he/him): So I mean maybe i’m thinking.

19:21 - Matt Germonprez (he/him): These what I have here, something like repository software activity could be a measure and it’s not like the super not a definitive measure at all, it might be helpful at all righty but it might not be helpful here at you, oh no you don’t I mean like that.

19:35 - Stephen Jacobs: yeah it’s. Stephen Jacobs: A placeholder for what might be 10 things are my.

19:40 - Stephen Jacobs: turn out to be the right. Stephen Jacobs: neighbor it’s too early to tell.

19:43 - Matt Germonprez (he/him): yep and then software citations can be another one as found in journals and pre Prince, if you could get that data that would be fantastic like my piece of software has been cited 100,000 times, and then the software itself.

20:00 - Matt Germonprez (he/him): As being it’s kind of adhering to fair principles, maybe this is a good place to start.

20:07 - Georg Link: Did you do we have a place where we can jot down these ideas, how the reference bolts could lead to some metrics you.

20:17 - Matt Germonprez (he/him): Like just didn’t like ideas from specifically to fair I do.

20:23 - Matt Germonprez (he/him): I can go track those down.

20:25 - Vinod Ahuja: It doesn’t sure. Georg Link: yeah well, I was thinking we were just talking about how find the bowl on the can operationalize that with metrics.

20:36 - Vinod Ahuja: If I if i’m thinking like find double as a separate category for developing that PICs accessible as a separate category for developing that thinks this is not kept trying to take you.

20:52 - Matt Germonprez (he/him): So highest level could simply be we have three metrics that help like i’m just trying to get the again break the ice on metrics that would help in this domain and the three metrics would be those listed there repository software activity.

21:10 - Matt Germonprez (he/him): Software citations and adherence to fair principles like it if down the road, it made sense to make a new metric that is simply the ability, then great if it makes sense to make a new metric that’s just about interoperability that’s also great, but some.

21:32 - Matt Germonprez (he/him): The software authors are attentive to.

21:36 - Matt Germonprez (he/him): To the fair principles.

21:42 - Matt Germonprez (he/him): And so Eric I will put track that stuff down.

21:45 - Stephen Jacobs: So how and how much of it is, is the software authors and how much of it is working with people like github and get lab to build that into what they do.

21:55 - Matt Germonprez (he/him): yeah sure I mean the sourcing of that could come at it from a variety of different places totally okay.

22:03 - Georg Link: i’m adding adding the thoughts here to how we can.

22:09 - Georg Link: You know, if we take this as metric how we would go about collecting that can be done for find the ball.

22:18 - Georg Link: I heard you say cited, in other words, I was also saying blog post published maybe there are some key words.

22:28 - Georg Link: That make it short and Google search not specific to the name of the project, but if it’s isn’t a space solve specific problems someone types in how do I solve this that that’s operations that’s also find it in the D.

22:47 - Matt Germonprez (he/him): Agreement, thank you.

23:58 - CHAOSS Community: pause the recording. Stephen Jacobs: Are you suggesting it’s not fun to watch the scrolling screen.

24:06 - CHAOSS Community: I mean, it would be helpful, I guess it’s the one watching, but quite.

24:22 - Vinod Ahuja: That you cannot tell you you’re muted.

24:24 - Matt Germonprez (he/him): Oh, I just said Hello watchers of the YouTube video you can.

24:30 - Matt Germonprez (he/him): see me browse the Web so i’m sorry if I was talking so gay are these are your points, this is to your point here, maybe some some guides to what can constitute the particular components around FA ir.

24:51 - Matt Germonprez (he/him): i’m not saying we read all these, but I think.

24:55 - Matt Germonprez (he/him): You can get the general idea.

25:01 - Matt Germonprez (he/him): You know what I mean So these are some of the bars that that we could use.

25:05 - Matt Germonprez (he/him): In that highest level fair like we could actually just point people.

25:10 - Stephen Jacobs: yeah. Matt Germonprez (he/him): To this page and say here.

25:14 - Matt Germonprez (he/him): You should be attending to some of these things.

25:18 - Matt Germonprez (he/him): We just had this talk in the Di meeting when we were talking about demographics, like we don’t need to reinvent a lot of things that people have already published and so.

25:29 - Okay. Vinod Ahuja: Oh, maybe the idea would be like be sure to start with one metric and start working on that and see how it evolves and brings.

25:42 - Vinod Ahuja: At that time. Matt Germonprez (he/him): yeah I mean I would I would be happy to start on the fair metric Okay, just because, not just because, but also because i’m working with folks in the fair project, and I think we could get some feedback from them as well, just as to how we present such a metric.

26:07 - Vinod Ahuja: Of maybe add them to the crib sheet and start a new document on the air but.

26:14 - Matt Germonprez (he/him): that’d be great air I was shown on the screen.

26:19 - Georg Link: The table. Georg Link: yeah yeah I open that just a side.

26:23 - Matt Germonprez (he/him): Okay, does that does that sort of what you’re asking for like just in guidance as to what FA ir could be.

26:31 - Georg Link: I think that is that is helpful and then.

26:36 - Georg Link: In looking through the table they’re more aspirational like this is what the outcome should be.

26:45 - Georg Link: And then, what I was looking for is how to be give guidance on how do you answer this so if reusable this software is Richard described that the morality of accurate involvement attributes, how do you actually demonstrate that your software meets this criteria.

27:04 - Matt Germonprez (he/him): gotcha okay. Georg Link: Or if we for for for final for software is registering deck in surgery searchable resource.

27:22 - Georg Link: that’s where it’s It shows up when you do search it’s part of the well known software which, like github aren’t getting.

27:33 - Matt Germonprez (he/him): I think that’s fair, I think, not to be fair, fair but i’m like summer considerably more applied like what to your point and then others like I saw it right, I didn’t see one inaccessible like adheres to Community standards like.

27:49 - Matt Germonprez (he/him): Whatever that might mean.

27:53 - Georg Link: An open source become, we can then say well part of it is you have a code of conduct you’ll have a good rate near good contributing dark.

28:03 - Georg Link: There are some Community standards.

28:06 - Matt Germonprez (he/him): Okay that’s good.

28:08 - Matt Germonprez (he/him): So i’m what i’ll propose is i’ll make a highest level just singular fairer metric.

28:18 - Vinod Ahuja: broken down, and we can do that I have created a template for the peer mentoring.

28:27 - So. Vinod Ahuja: You can go ahead and start adding all right.

28:33 - Matt Germonprez (he/him): i’ll put that as an action item for me OK, I can two weeks.

29:20 - Matt Germonprez (he/him): So should we update the update the spreadsheet.

29:24 - Vinod Ahuja: I have added the same to the spec sheet to.

29:29 - Vinod Ahuja: snap to pass the link okay. Vinod Ahuja: So, in the meantime, we have one metric which we compete a little bit maybe we take a look at it now, or what everyone said yes.

29:53 - Vinod Ahuja: That is open source project in fact previously RPT metric.

30:05 - Matt Germonprez (he/him): What are you asking us to.

30:07 - Vinod Ahuja: So. Stephen Jacobs: metric.

30:11 - Vinod Ahuja: PT metric that we we developed it’s like almost maybe it’s a percentile we just take a look and finalize it.

30:36 - Matt Germonprez (he/him): Now I think can pause the recording was a bit. .