Setting up, Troubleshooting, and Innovating on the Delivery of Online Instruction
Setting up, Troubleshooting, and Innovating on the Delivery of Online Instruction:
A Case Study of an LMS Q&A forum
( Accepted to Learning@Scale 2021 ACM )
User Research/Human Centered Interaction Project
Ramesh, K., Vermette, L., and Chilana, P. . (2021) Setting up, Troubleshooting, and Innovating on the Delivery of Online Instruction: A Case Study of an LMS Q&A Forum. Proceedings of the ACM Learning at Scale Conference (L@S’21).
Full Conference Talk for L@S 2021
Motivation
In 2020, I could see the impact that transitioning to online and remote teaching had on my own education. This drove me to find ways to support instructors and students with these new challenges, by applying UX methodology to understand current user needs and address ways to solve them.
Overview
What is this project?
This is a collaborative UX research project aimed to understand how instructors have been most impacted by the transition to online classroom course delivery with regards to their learning management system. Conducted this project with supervising researchers and PhD student at Simon Fraser University.
How did I approach the research?
We used grounded-theory qualitative analysis on a segment of LMS-help forums during the early months of the covid-19 pandemic, we recently synthesized our findings in a submission paper for the CSCW Conference, (Computer-Supported Cooperative Work and Social Computing).
What were the deliverables?
From there, I developed design recommendations, and delivered clear guidelines for adjusting a LMS help-tool after workshopping and iterating over my ideas.
What did I learn?
I learned how to conduct a qualitative analysis under unique pandemic constraints. This experience provided me with an insight on current (2020) best practices for online education tools. And how to approach qualitative research goals that aim to support users as they interact with complex systems, like learning management systems.
We wanted to understand the challenges that users of Canvas experienced transitioning to full remote online-teaching in 2020, by looking at the content of posts in the Canvas Community help forums.
Research Goal
There is a lot more online activity on LMS forums compared with the same time in the previous year
Canvas is one of the most widely-used learning management system in Canada and the US. Used in K-12 and up to university, with over 30 million users. The Canvas Community Help Forum saw an uptick in activity as users of the LMS and instructors adapted their Canvas courses or adopted the Canvas LMS, for online teaching when covid-19 lockdown began around March 2020.
Key research questions and areas:
1. What learning management problems are educators struggling with?
2. What issues do they face in seeking help with these problems?
3. Observe and understand some of unique challenges of online and remote teaching for instructors and their LMS?
Methodology
Sample help-forum posts from period when instructors transitioned to online. Conduct grounded-theory qualitative analysis on Canvas user forum posts, as they seek help and advice with Canvas tasks and learning to adapt Canvas for their online-teaching needs. Explore findings further with secondary research, usability studies, interviews etc.
Phase 1: Explore forum for qualitative content and develop schema for analyzing Canvas forum posts
Phase 2: Apply grounded theory qualitative analysis on sample of forum posts, iterating as needed
Phase 3: Develop key insights and further research opportunities based on results
Phase 4: Support the delivery of design recommendations with usability research and further inquiry
Why study learning management systems?
Learning management systems are complex software
Wide range of features -> steep learning curve
While many instructors learn from their colleagues, sometimes it can be easier to seek help online. But the tradeoff, they may have to wait a while, and no guarantee of a response
We want to know how instructors go about seeking help and what they seek help for?
Phase 1: Open Coding
Using grounded-theory approach we developed a scheme for understanding the content of the forum posts on Canvas Help Forum, and what kinds of issues instructors were discussing etc.
How we took a grounded-approach to understanding the experiences of users?
We collected 80 Posts from recent months (March-June 2020) and explored posts with questions and content related to the UI and other issues.
We observed common themes emerge in the content:
Troubleshooting
Setting up a course
Student-facing UI issues
Seeking guidance on best practices from other instructors
We developed a preliminary coding scheme using the themes observed in the content, and iterated over this scheme as needed, in order to ensure it could represent a majority of the types of posts in the Canvas Community Forum.
After iterating, we decided to use two coding schemes to understand and explain the content of the forum posts.
1. It made sense to first separate posts by the type of question that they author was asking ranging from simple “how-to” questions to questions about “where” to find a feature.
2. Then once we segmented posts by the question type, we could dig deeper into the content of the questions and patterns, themes that emerged among these posts.
What do we know about our users, the instructor’s, who use learning management systems?
Often have to turn to on-the-job learning
But are routinely strapped for time, making this hard (e.g., Vermette ‘19)
Recent events and pandemic has forced unprecedented numbers of educators to quickly adapt to online teaching
Some are learning to teach with an LMS for the first time
Many have some LMS experience, but need to learn many new features (e.g., Vermette ‘20)