AI Feedback Assistant: Student
Getting Started
I was a member of an artificial intelligence / machine learning (AI/ML) research and development team that had a number of design challenges with early-stage products that had Natural Language Processing (NLP) AI components. Utilizing my background in design, statistics, psychometrics, and psychology, I helped solve a number of design and data challenges related to designing and training an AI assistant that provides feedback to college students on their writing. The student proof of concept discussed in this case study pairs with an instructor proof of concept discussed in another case study.
The project included a number of steps:
-
Generative research, including interviews and cognitive walkthroughs with students at a large university
-
Design thinking sessions with stakeholders and initial proof of concept designs
-
Concept testing with instructors and students
-
Iterating designs
Client
Pearson & a Large University
Duration
3 months
Role on Project
AI research & design
Skills Demonstrated
Quant & qual research
Product strategy
Concept testing
Proof of concept design
User flows
Interaction design
The Team
Learning Design
Four learning researchers and designers
External University
A large university with approximately 4 core project team members from their innovation center.
AI Team
-
2 data scientists, ML modeling
-
1 VP AI products & solutions (background: cognitive psychology & CS
-
(me!) data scientist, AI research & design
Generative Research
Student Interviews & Cognitive Walkthroughs
Twelve college students participated in an hour interview and walkthrough of how they receive, perceive, and implement writing feedback on assignments.
Research Objectives:
-
Understand if/when students access feedback on their writing
-
Understand what elements of the writing feedback they attend to
-
Understand how students process the writing feedback
-
Understand the range of actions students take in response to feedback
-
Understand any frustrations students experience in the writing feedback process
Synthesis of Findings
I created three journey maps that outline three different types of students in regards to how they approach writing feedback and how they perform as a student. An overall, summary journey map was also created to give a high-level view of the writing feedback process, as well as outline user goals, attitudes, behaviors, wants and needs, and frustrations and pain points.
View Student User stories
Ideating & Concept Testing
Stakeholder Design Thinking Workshop
About 15 internal and external stakeholders took part in a design thinking workshop. Participants were shown research results and were given the following problem to solve: Students access writing feedback in a number of locations and frequently don't read or implement feedback, how can we deliver writing feedback to students so that they have an easier time finding, reading, and applying the feedback? The goal is to deliver writing feedback in a way that facilitates continual improvement of writing skills.
The workshop resulted in a number of different concepts. The concepts were voted on and I mocked-up two, including my own concept, to be tested with instructors and students.
Initial Concepts
The first concept, start-to-finish support, was my concept. It addressed the following learner needs: 1) actionable and focused feedback that tells the student where to focus their revisions, 2) clear expectations at the beginning of the writing process, 3) speedy turnaround of feedback and request for help via email.
The second concept, personal feedback tour, addressed the learner need of receiving a positive emotional impact from the writing feedback.
Assignment details all in one location
Editing paper with instructor's feedback populated
AI assistant provided feedback
Feedback tour with an avatar walking you through your feedback step-by-step
Concept Testing
Concept testing was conducted in separate sessions with 2 instructors and 4 students. Participants were shown two student concepts and asked for their feedback on aspects of the concepts.
The two concepts shown in Miro during testing.
Concept Testing Insights
-
Students and instructors thought an AI assistant that gives substantive feedback while the students are completing the assignment is very helpful.
-
The instructor's inline feedback is a big plus.
-
Assignment details (i.e., prompt, rubric, instructor policies, announcements, resources) is very helpful.
-
The ability to reply to the instructor's feedback is very helpful for students; currently, in order to ask a clarifying question, students have to send an email where it is hard to clarify the context.
-
Instructors were less enthusiastic about the students' ability to reply to a feedback comment. They worried it would increase their workload and that students would only use it to try and improve their grade.
-
Students preferred to see all of their feedback at once, as opposed to a feedback tour which walked them step-by-step through their feedback.
-
Students were neutral on the idea of an avatar walking them through their feedback in the feedback tour.
-
Students wanted the ability to save drafts of their papers.
-
Students wanted all feedback boxes open (i.e., no clicking to view the feedback)
-
Students wanted an expandable rubric that contains the overall feedback comments for each rubric element.
-
Overall, students really loved the idea of the start-to-finish feedback; they particularly liked the ability to edit the paper will their instructor's inline feedback and liked the AI assistant that could give them instant feedback on the spot, rather than waiting possibly days for their instructor to email them back.
Final Proof of Concept User Flow
User Flow
Results from the concept testing were used to iterate on a user flow for the project's final proof of concept. Click any of the images to view the full user flow in Miro.
Assignment details all in one location
Editing paper with instructor's feedback populated
Viewing rubric-level feedback
AI assistant provided feedback
Viewing instructor feedback, with the ability to respond to feedback for clarification
Feedback tour with an avatar walking you through your feedback
Feedback tour continued
