Research Publication for CHI 2022
Uncovering Instructors’ Diverse Practices and Perceptions:
A Field Deployment of a Customization-Sharing Platform that Supports Course Management
( Accepted to Learning@Scale 2021 ACM )
Field Study Project with Domain Experts
Vermette, L., Ramesh, K., Chilana, P and McGrenere, J. (2022) Uncovering Instructors’ Diverse Practices and Perceptions: A Field Deployment of a Customization-Sharing Platform that Supports Course Management. Proceedings of the ACM CHI Conference (CHI’22).
Under construction - check out paper and deck soon~
Motivation
In 2021, I supported a field deployment study of an in-context customization sharing tool, for the learning management system, Canvas. I executed a multi-week field study, where instructors were given access to the tool, Customizer, developed by Laton Vermette (PhD Candidate).
Overview
What is this project?
This is a collaborative research field study aimed to understand how instructors use customization sharing and in-context support tools, like Customizer to more effectively build their online classroom course delivery. Conducted this project with supervising researchers, Dr. Parmit Chilana and Laton Vermette PhD student, at Simon Fraser University and additional collaborators, Joanna McGrenere of University of British Columbia.
How did we approach the research?
We developed a field study deployment methodology to allow for semi-structured use of the in-context tool, Customizer as instructors planned their courses and customized their LMS for teaching the following term.
What were the deliverables?
Initially, deliverables included the initial authoring interface design, and authoring usability study— to help populate the tool. After the field study was complete, design recommendations and research contributions included: an understanding of how customization sharing can fit into current instructor workflows.
What did I learn?
I learned how multi-stage complex field deployment research requires precise planning and execution. I also learned about how gathering qualitative insights on a complex system, requires framework driven qualitative analysis rather than just a grounded theory approach. Additionally there were unique learnings in how to conduct remote and online field study research, under unique pandemic constraints. This experience provided me with an insight on current (2021) best practices for online education tools. And how to approach qualitative research goals that aim to support users as they interact with complex systems, like learning management systems.
We wanted to understand the challenges that users of Canvas experienced transitioning to full remote online-teaching in 2020, by looking at the content of posts in the Canvas Community help forums.
Research Goal
There is a lot more online activity on LMS forums compared with the same time in the previous year
Canvas is one of the most widely-used learning management system in Canada and the US. Used in K-12 and up to university, with over 30 million users. The Canvas Community Help Forum saw an uptick in activity as users of the LMS and instructors adapted their Canvas courses or adopted the Canvas LMS, for online teaching when covid-19 lockdown began around March 2020.
Key research questions and areas:
1. What learning management problems are educators struggling with?
2. What issues do they face in seeking help with these problems?
3. Observe and understand some of unique challenges of online and remote teaching for instructors and their LMS?
Methodology
Sample help-forum posts from period when instructors transitioned to online. Conduct grounded-theory qualitative analysis on Canvas user forum posts, as they seek help and advice with Canvas tasks and learning to adapt Canvas for their online-teaching needs. Explore findings further with secondary research, usability studies, interviews etc.
Phase 1: Explore forum for qualitative content and develop schema for analyzing Canvas forum posts
Phase 2: Apply grounded theory qualitative analysis on sample of forum posts, iterating as needed
Phase 3: Develop key insights and further research opportunities based on results
Phase 4: Support the delivery of design recommendations with usability research and further inquiry
Why study learning management systems?
Learning management systems are complex software
Wide range of features -> steep learning curve
While many instructors learn from their colleagues, sometimes it can be easier to seek help online. But the tradeoff, they may have to wait a while, and no guarantee of a response
We want to know how instructors go about seeking help and what they seek help for?
Phase 1: Open Coding
Using grounded-theory approach we developed a scheme for understanding the content of the forum posts on Canvas Help Forum, and what kinds of issues instructors were discussing etc.
How we took a grounded-approach to understanding the experiences of users?
We collected 80 Posts from recent months (March-June 2020) and explored posts with questions and content related to the UI and other issues.
We observed common themes emerge in the content:
Troubleshooting
Setting up a course
Student-facing UI issues
Seeking guidance on best practices from other instructors
We developed a preliminary coding scheme using the themes observed in the content, and iterated over this scheme as needed, in order to ensure it could represent a majority of the types of posts in the Canvas Community Forum.
After iterating, we decided to use two coding schemes to understand and explain the content of the forum posts.
1. It made sense to first separate posts by the type of question that they author was asking ranging from simple “how-to” questions to questions about “where” to find a feature.
2. Then once we segmented posts by the question type, we could dig deeper into the content of the questions and patterns, themes that emerged among these posts.
What do we know about our users, the instructor’s, who use learning management systems?
Often have to turn to on-the-job learning
But are routinely strapped for time, making this hard (e.g., Vermette ‘19)
Recent events and pandemic has forced unprecedented numbers of educators to quickly adapt to online teaching
Some are learning to teach with an LMS for the first time
Many have some LMS experience, but need to learn many new features (e.g., Vermette ‘20)