OA21: Breakout (3) Building a modern discussion tool backed by scholarship and usability research

Jun 28, 2021 20:34 · 4464 words · 21 minute read

free to send me a a direct chat and i’m glad to help out in any way that i can so with that it is it is 12 12 and let’s get started uh hannah’s saying that when she leaves and tries to join it just gives her the headphones listening option well maybe we can change it once we get her in there and josh no we can’t hear you oh yeah that’s me she might need to try with a different browser hannah maybe maybe you and i can work on this offline while the presentation gets started okay with that note i am going to bring in the presentation does everyone see a um slide almost no loading for me at least there we go okay great well hello and welcome to our presentation i’m heather valle and i’m here with heather hans and hannah rogers we hope we’re from duke university’s learning innovation team and we’re going to discuss the development of a new asynchronous communications tool as the use of online education tools increased to meet the needs of learners during the recent pandemic some new pain points became apparent to address an unexpected hole in our teaching and technology portfolio duke university decided to create a new online discussion platform to facilitate compelling 21st century learning conversations by leveraging its commitment to learning innovation dedication to open source open source software deep engagement with the duke community expertise in user informed design and strong partnership with the with longsight a trusted education technology vendor duke has reached an advanced prototype stage of development on this tool with the goal of piloting it for this coming fall semester here is our full team from duke learning innovation duke creative and user experience and longsight but as you’ll see the range of people providing input is much broader the recent transition to online education has demonstrated how valuable online conversation is to our learning experiences unfortunately our faculty are not enamored with sakai’s forums tool so projects like piazza have become increasingly important even before the pandemic their use increased dramatically during last year’s higher education pivot to online education when piazza changed its business model in 2020 we at duke took the opportunity to evaluate some competitive projects like ed discussions and campus wire at the same time we began the development of an entirely new discussion tool within sakai our goal is to replace the current discussion forums with a tool designed to better meet our community’s needs and facilitate meaningful compelling learning conversations in order to transform our current discussion tool we brought together a wide variety of stakeholders students instructors teaching assistants lms administrators user experience designers and vendors we delved deeply into pedagogical research and leveraged modern web development technologies and we asked our primary users instructors and students to provide input and feedback at every stage of the process this holistic approach to course discussions aims to make them strongly aligned with both user needs and pedagogical best practices and see yes now hannah will dig a bit deeper into the research side of things thanks heather i’m hannah rogers so on the slide that you’ll see in just a second you can see some but not all of the research categories we explored relating to online discussion so by exploring topics as diverse as gamification and social media we sought to gain a holistic understanding of effective online discussions we centered pedagogical research from the beginning of this process initially collecting reviewing and summarizing about 70 articles from there we crafted 42 online discussion design recommendations directly tied these findings about directly type defines about 50 articles at the heart of every recommendation is the understanding that user experience plays a key part in not just using a tool but in learning itself we are in the process of connecting these research recommendations with other forms of data to see where the needs of instructors graduate students and undergraduate students align with the findings from teaching and learning scholarship we are currently working with a group of six instructors from across schools and disciplines at duke who are providing sustained consultation on the process as we began the process of designing the new discussion tool we collected user stories from faculty once we had our recommendations from pedagogical research we presented these recommendations to duke instructors to see what overlapped with their needs and what questions we had not yet asked additionally we have provided faculty with our findings from the undergraduate focus groups that heather will talk more about in a moment and ask for their reflections the faculty themselves pushed for us to consider how to build values directly into the tool spurring our team to integrate these questions into our design and research methodologies we are coding each user story based on the values underlying each action and pictured on this slide you can get a peek of our workflow in air table after we code these user stories we will be running a values exercise with faculty and now i’ll turn it over to heather who’ll speak more about our user research process thanks hannah to complement the work we are doing with our faculty group we conducted several focus groups with undergrad and graduate students in addition to conducting user testing of prototypes with students and faculty after completing initial interviews with two student interns to gain some valuable background information we recruited undergrad students for three one hour focus groups and the students were compensated for their time we kept these groups relatively small at three to seven participants and we intentionally recruited students from a variety of disciplines in total we spoke to 18 undergrads from duke and duke kunshan university about their experiences with sakai forums piazza and online discussion in general the focus groups were held via zoom and included several formats we broke up the group into smaller groups based on their experience with sakai forums and or piazza and asked them questions about their use of those respective tools then we encouraged them to use jamboard to identify what they liked and disliked about each tool here you can see one group’s feedback on sakai forums in jamboard we also had larger group discussions about the tools the online discussion experience in general and related topics including privacy and gamification uh the focus group methodology was designed by bendy fagg a lead information architect ux designer here at duke who is part of our project team we are currently recruiting for two rounds of focus groups with graduate students ideally from a variety of disciplines again we will use the same methods as the undergrad focus groups but some of the questions will be different to account for graduate students experiences as tas and instructors of record again the grad students will be compensated for their time all of the information from our focus group research is being collected grouped by theme and synthesized in reports that we then compared to the other data collected for the project we will share some of the most common themes across our research a bit later in the presentation on another note another major component of user research in this project was of course usability testing of two prototypes by faculty students and community members during the design phase since we already had a weekly meeting with our faculty group we sent them into individual breakout rooms to test the first prototype of the tool then brought them back into the main room to share their feedback with us also we are doing usability testing with undergrad and grad students currently in 20 to 30 minute one-on-one slots in which we ask them to complete a series of tasks and then ask some follow-up questions about the experience in addition we will be doing usability testing with community members very soon uh more information on that at the end if you’re interested and we plan to test uh one more prototype with these user groups but hopefully with different participants later in the summer now i’ll turn it back to hannah to talk a bit about some of our findings from all this research so in designing this tool we ask ourselves where does the pedagogical research student and instructor needs overlap so i’ll just take you through three quick examples the need for asking questions and holding more extensive discussions was represented by both instructor user stories and student focus groups students in particular spoke about how current tools like sakai and piazza work at cross purposes so on this slide you can see a current version of sakai forums where there is not a specific question and answer option and if you look at the next slide you will see pictured a sample screenshot of what we’re currently calling forums next the tool when users create a new topic they can choose between asking a question and engaging in q a and starting a discussion another big uh thing that was important to both students and instructors was keeping posts organized through tagging on this slide you can see some example tags from ed discussion and piazza sakai forums does not currently support tagging posts so on the next slide you can see a screenshot of our initial tagging layout in the forms next tool instructors can also edit tags for the course to make sure the organizational flow fits the needs of the course and in order to build on improving organization for the forum’s next project we sought to improve upon the long list of forums and topics that sakai forums currently displays as you can see in this screenshot sakai forum’s vertical list of texts can make it difficult to easily find topics important to the user so in our final screenshot of the forms next you can see a new left-hand organizational column that displays alongside the topic you are currently viewing instructors can pin important topics to the top of this column so that members of the course can easily find them individual users can also add topics to something we’re currently calling my list so that they can follow along with discussions important to them and while we have a solid research basis for these changes we still have some open questions about how we might design this tool as we continue further along this plot process so what questions remain based on our research firstly we value student privacy and instructors have emphasized this as have students but what should we show instructors to help them measure their pedagogy another topic in the consideration is gamification gamification can increase student motivation if integrated properly into a course but how should we approach this especially with sakai’s overall goals and now i’ll turn it back over to heather who will talk us through some final next steps thanks hannah so our next steps in addition to the graduate student focus groups and the values exercise with faculty are uh pulling in experts from duke to evaluate the accessibility of the tool as well as maybe help us test that a bit in the usability testing uh prototyping the next iteration of the interface uh we’ll leave that to adrian and jen and others we are planning to do two prototypes in total and do additional usability testing on the second prototype and our plan is to launch the tool in very early fall so in august so with all that being said uh thank you all for joining us we hope this presentation gave you an overview of the holistic work we are doing to develop a new discussion tool for sakai which as you can see is currently in progress now we’d be happy to answer any questions you have about this presentation josh are there any questions that we missed as we were talking i was i was muted twice no there there were no questions in the chat so this this is really good a couple of people are typing in the shared notes so this is uh there’s a whole lot of activity there one person notes that uh the comments tool is a great way of finding balance between anonymous posting and analytics for the faculty members and uh wonders is such a principle possible in forums next i wonder could that person say more about what they mean about the comments tool or how they would suggest using it if you can identify yourself in the public chat i’ll be glad to uh unlock you so that you can unmute yourself and explain or not if you decided to you know um while we wait for that let’s uh oh actually heather go ahead and we can move to the next question while the first question mutes well i’m not sure i’m the person to talk about sk how this is going to scale but and as far as question about the tool being made open source i believe our current plan is to contribute it back to the sakai community so tonko was the first questioner tonko from the uh the hotel school in in the hague um tonka you’re unlocked you can turn on your microphone and uh um and explain further if you want to do that tonko looks like he’s on with headphones only so while he uh works out that audio issue uh matt writes the interface looks great any thoughts on performance and reaching new scales i.

e mega courses i think it’s the tagging that we’re looking at is going to be somewhat helpful as far as um organizing things appropriately so you can tag things for specific groups or questions which should help narrow stuff down for larger courses i believe we’re also going to be looking whether it’s going to be mvp one or perhaps the later iteration of this um tying this into the uh groups function which means that people would be able to subset things specifically for smaller groups which is how instructors at duke at least tend to approach uh the large classes that we have to make them more manageable yeah 100 percent i was going to say you know for q a it might not it might work just fine but yeah i think you’d want to also implement um you know groups in a fashion probably using tags uh if not also you know actually defining user groups for more discussion based work and i believe that quite a few of the faculty that are consulting on this project and some of the grad students have worked in larger classes so there there are considerations that we’re taking even though i i think that it’s fair to say that we haven’t looked into them as depth in this initial prototype um yet um but you know there are things like um how like if someone has like 90-something comments on a post how will that look and how will that be functional that we’re thinking about i see also adrian is responding on about performance uh in the notes saying i’m sweating performance on this all the time believe me so lots of paging only pulling data to the front end when clicked into ignite caching that would probably make sense to someone who’s not me for that part adrian i just unlocked you do you want to unmute and elaborate for those who don’t know adrian is the person doing the heroic coding required to make this entire thing work i’ve not really got anything more to say about that i mean yeah so yeah i mean i’m thinking about performance a lot with it i mean it’s it’s obviously a big worry we know how how hard it is to scale forums especially when things are all loaded into one screen so i’m thinking in terms of uh infinite scroll with proper database page in the back rather than loading all the data up front and yeah only loading data only putting data to the front when you actually click on something right so everything comes down sparse first right so you get a sparse a sparse kind of topic a sparse kind of post whatever you know the terminology may change for these things but you know you get the idea people click it then we pull it so additional questions uh from dave eveland are stats to be relegated to the stats tool instead of embedded in forums next well we’re currently in an ongoing conversation with um our advisory groups uh instructors and so on about what stats they’re going to need and what stats you’re going to want and i think that may affect how they get pulled down uh they’ve been looking at how stats are handled in ed discussion for example and you can get a lot that’s very specific to the online discussion for that um there and um i don’t believe we have have we made a decision on how that’s going to play within our tool i don’t believe we have um we know that there are going to be stats that are going to have to be pulled out of here but we’re still deciding where does the need for information versus student privacy where is that line going to be for what it’s worth we the intent is not to leverage well the intent is to leverage sakai’s capabilities and not build uh shadow versions of those capabilities into forums next so one of the things that’s been of interest to us is making sure that uh we don’t replicate what current forums does which is to maintain its own permissions the realms tool has to handle that you know so i think that that’s the overarching principle but how that gets applied in the case by case basis is being worked out as we go i see a question about questions versus discussions that’s something that is um handled in piazza and many of our instructors have said that they really like having that distinction questions are um q a i have this problem or the instructor is presenting this problem and the follow-up responses are attempting to answer that specific thing an instructor can go in and say this answer is correct as opposed to a discussion which is much more wide-ranging and people can debate topics back and forth and um it’s i’d say it’s far less structured although of course both of them are going to have to have structure because they are within an interface but um it was something that was brought to our attention as being a thing that a different project does well that they want to bring into sakai yeah and i think it’s a good point that students that sorry hannah students at first might uh not be sure about the difference so i think that’s definitely something um you know we’ll make sure to look at for the usability testing and i don’t know what you were planning to say uh yeah so i i guess i just generally speaking about usability testing in terms of instructors some of the team at learning innovations had a look at and we’ve had some student tests i i think that that is a correct concern that people might not be like and just in general in general with what we’re calling any of the functionalities of this tool we need to think carefully about how we name things and also how like we integrate explanations to make things intuitive within the tool um and which is why i think usability testing is so important and we’ve had some conversations amongst team members about how to do that so it is definitely something we’re thinking about there’s also a question about tracking discussions for student participation requirements i believe we are planning on bringing the ability to grade things into the tool i believe in the first version of it that’s launching this fall because instructors have said that in some cases possibly most cases if there isn’t a grade attached students won’t do it so they feel that is a necessary part of a tool that’s going if it’s going to be useful in their classes that’s also come through in the research right hannah and and also in the student focus well pedagogical research um in terms of teaching learning articles and the student focus groups in fact students said i know this is controversial but i would like forums to be used more consistently and or graded to give me the motivation to do it it was a breath of honesty yeah they were very honest it was very helpful so an additional question has come into the shared notes which says sorry if you already covered this but will forms next enter sakai 22 as a contrib tool or already core so that it can replace discussions um so i can i can say a little bit and adrian may want to say more so the plan currently is to have the first version of forums next enter sakai 22 as a core tool but not as a replacement for current forums at this moment you know so it would be a little bit presumptuous and a little bit risky i think to replace forums quite that quickly but the the hope is over time that will happen in the same way that uh you know dashboard for example might over time replace overview and there have been other examples of that kind of thing so uh adrian or earl if you’re here do you guys want to say more about that no i think you’ve i think you’ve captured that yeah i mean there’s so many um i mean just this conversation now i mean uh it makes it really apparent that people have particular requirements around the threaded discussion tool right so so we’re kind of i suppose we can’t we’re kind of focusing primarily on the question and answer functionality in the in the kind of first you know the first the first version but i’m i’m i’m hoping we are going to have a threaded discussion functionality in the release for 22 but like like josh says it’ll be probably a lot a big reach to uh to a student we can just say right you can all just kind of shut off discussions now you know well that’s the ambition of course you know yeah we’re hoping that um as it becomes part of sakai and more people use it we can get feedback on how it does and doesn’t suit their needs and that can affect future iterations so it is 12 37 eastern that means we’re about three minutes from the end of the session um i i wonder wilma would you like to say a few words about the community input opportunities that are going to be forthcoming in the next few weeks um sure i i put a link in the chat um it’s in there a couple different ways there’s a bitly and then there’s just the link to my calendly um for appointments and right now the appointments are scheduled the week of um june 21st through 25th so there’s um a number of time slots throughout that week and um we really want to encourage faculty end users to sign up so if you want to distribute that link to faculty at your institutions that would be great and if they happen to be heavy discussions users that would be even better um so the idea is just to get some community feedback on the the tool and to be able to get kind of a first look at the prototype for some of the folks outside of duke to provide some feedback on how it would work for the wider community we may open up additional usability testing opportunities later um so if all the spots fill up and we decide to run another set um you know we’ll let you know but right now it’s just the one week for um for the testing maybe we only have such a problem that people are banging down our doors to help provide usability test input yeah so there’s a follow-on question in the shared notes that says does the usability user have to be familiar with sakai i think this this relates to the notion of getting community faculty involved not necessarily i mean if they use any sort of discussion tool whether because i mean some of the folks that have been involved in the testing now have used things like piazza or ed discussions and so having relevant experience with other similar types of tools would also be beneficial i think and can i just add that when we’re doing usability testing with graduate students and undergrads we aren’t looking for them to be familiar with anything because i think part of what we’re hoping is that if you’re not familiar with these tools because you might understand questions if you’ve seen piazza but you might also have no idea if you’ve not seen piazza like getting that viewpoint is also really important you know i think we had someone who basically uses the email to communicate and that’s it and that was a really valuable viewpoint i learned so much from that test all right it is 12 39 and will be 12 40 momentarily and with the the lightning talks coming up at 12 45 um i think we should let that be the last question i do note that uh matt clare has a note about uh an idea for a transition from forums next uh from forums classic to forums next so thanks matt for putting that in the shared notes we’ll definitely take a look at that all right it is 12 40.

thanks to all of you for for being here thanks to the the duke folks for this great presentation and a project about which i’m personally really pretty excited so thanks gang we’ll see you all in other sessions in the conference next up are the lightning talks at 12 45 so those should be pretty interesting there are six or seven of them there they cover a wide range of projects in a wide range of topics so definitely come check it out all right thanks all take care see you soon thanks bye everyone bye.