The Learner Activity View for Advisors (LAVA) Overview
This document describes the Learner Activity View for Advisors/Canvas Flags (LAVA), which is a new resource accessible via the Advising Gateway.
During the Spring 2023 term a learning analytics pilot is being conducted with the intent of exploring providing academic advisors with access to and awareness of trends across the course array of a student in two broad areas: performance and participation.
The “Learner Activity View for Advisors” (LAVA) resource will be made available to the pilot academic advisors via the Advising Gateway. Advisors will not see detailed visualizations, but rather high-level trend flags indicating patterns across a student’s enrolled course array. Communications about the pilot to the pilot student cohort and to the instructors whose courses they are enrolled in for Spring 2023 were sent out in late-January.
Some of the major goals of the 2023 LAVA pilot include:
- Evaluating this new approach to learning analytics for advisors (providing awareness of broad trends ACROSS a student’s enrolled course array)
- Understanding the utility and efficacy of this approach and of the LAVA resource
- Working with academic advisors to develop/determine what intervention(s)/actions(s) could look like based on insights provided by access to this new information
- Understand the impact(s) of access to learner activity data on advisors (bandwidth, impact of seeing flags, ability to engage/intervene, etc.)
- Assessing whether this insight into learner activity is helpful to advisors in their work
- Assessing whether this insight into learner activity proves helpful to students (getting ahead of potential bad outcomes, improving outcomes, etc.)
Notes:
- This project is part of UW-Madison's broader efforts around learning analytics. More information about learning analytics at UW-Madison is available from the Vice Provost for Teaching and Learning and the Learning Analytics Center of Excellence.
- This document describes a learning analytics approach to help support student success. Students are unique individuals and are more complex than their data - be sure to consider them holistically.
- Please review the UW-Madison Learning Analytics Guiding Principles and the Office of Undergraduate Advising Guiding Principles before using LAVA.
- For more information about the use of data, review the Teaching and Learning Data Transparency Statement.
What is the Learner Activity View for Advisors/Canvas Flags (LAVA)?
The Learner Activity View for Advisors/Canvas Flags is a feature that displays performance and participation data to authorized users through a new Canvas Flags tab in the Advising Gateway. This tab provides information about how a student is participating and performing in online academic activities across their array of enrolled courses, and relative to their peers within each of those courses.
By incorporating these data directly into the Advising Gateway, it offers authorized users a view of student academic activities during the active term, within the normal workflow and familiar Advising Gateway interface.
Features:
- This is a learning analytics approach, using course activity and gradebook data from Canvas (and other integrated tools) and accessed within the Advising Gateway. More details about the data, and when performance and participation flags will appear are provided below.
- Data appear in a new Canvas Flags tab within the Advising Gateway. If low performance or participation trends are present across multiple courses (relative to their peers within those specific courses), a flag will appear in that tab on the student profile page.
- The flags may help an advisor start a conversation with a student, if there’s a potentially concerning trend or pattern of behavior across multiple courses.
About the Flags
Two types of flags may appear:
- Participation - course activity
- Performance - grades
To view flag data and details, click on the tab to show the accordion menus that are labeled with the flag names.
If flags are present, there will be a label telling you which flag has been raised. In the example below, the Performance flag has been raised.
In the example below, there are two flags raised - both Participation and Performance flags have been raised.
Flagged Courses
When the flag menus are expanded, you can see the list of courses the student is in, and which specific courses contribute to the active flag.
- Course lists will appear under both the Participation and Performance sections, since there may be a flag raised in one area but not in the other, for the same course. (For example, a student’s performance is not yet flagged, however they have not participated in Canvas for the last five days, compared to their peers, for several courses.)
- Courses that meet the criteria in the flag details will appear in the Flagged Course column. All other courses will appear in the No flag or no data column.
Two or More Courses - Looking at Trends
LAVA/Canvas Flags is designed to show trends or patterns of behavior across a student's array of enrolled courses. A student generally must have two or more courses that meet the criteria in the flag details for the flag to be “raised”.
Caveats:
- If a student is only enrolled in one course, a flag will be raised based on that one course.
- If a student is enrolled in two courses, a flag will be raised if the threshold is met in only one course.
Flag Details
The flag details box describes the course data that is used to determine if courses have flags.
Important reminders to keep in mind:
- Instructors teach using many different strategies and pedagogical approaches. How a course is set up, and how much an instructor uses Canvas and other linked tools for teaching activities will impact the data that can trigger these flags.
- The more an instructor uses Canvas and other linked tools, the more useful Canvas Flags may be.
- Don’t assume anything based on just one data point; remember to validate and ask questions. This is a starting point to help with a conversation.
Participation Flag Details (Course Activity)
Student activity in the course is compared with other students enrolled in that course. Participation data uses activity in Canvas, along with several other tools if they are integrated within the Canvas course (more details below).
- Participation is measured as any activity in an online tool in the course.
- Wherever possible, students are compared to the other students enrolled in all sections of a course.
What is Participation in a Course?
Canvas participation is based on page views in a course, along with student activities. Student activities may include:
- Accessing a course page
- Announcement - posting a new comment
- Assignments - submitting an assignment
- Commenting - posting a new comment on an announcement or discussion
- Collaborations - viewing and/or editing a collaborative document
- Conferences - joining a web conference
- Discussions - posting a new comment
- External tools that are linked within the course (see more info, below)
- Pages - creating a page
- Quiz - starting a quiz, submitting a quiz
More About External Tools
Other tools may be linked within a Canvas course; for example Kaltura MediaSpace, Piazza, TopHat, Pressbooks, Engage eText or Zoom are other Learn@UW tools that instructors use.
- When a student opens an external tool from within the Canvas course, that participation will be counted.
- Activity occurring within external tools is NOT generally captured.
Activity in other online tools that are not connected to Canvas - or are not launched from Canvas - will NOT be included as participation for flag calculations. For example, a course that uses a standalone WordPress website for course activities won’t show participation data.
A participation flag will be raised if:
- The student is 5 days behind the median of their peers in aggregate participation. In other words, they are well behind their peers in accessing the course activities.
- There must be at least two courses that the student hasn’t logged into for five days. (If a student is only enrolled in one course, a flag may still be raised if they’ve not interacted with that one course.)
How the Participation Flag is Calculated
- LAVA calculates the median most recent activity time for all students in the course.
- If the students’ Last Activity DateTime is 5 or more days before the median, the flag may be raised (if this occurs across two or more courses, when the student is enrolled in more than one course).
- This flag will be most useful for courses where many students are frequently logging in (for example, they all take a quiz synchronously in a live class).
- If the student does not have a last activity date (has never interacted with the course), and the course start date is set, the students’ last activity date defaults to the course start date.
- If the course start date is not set, the default start date is the start of the term.
Additional Details
- Peers for the activity quartiles are based on the course the student is enrolled in Canvas. This means if sections have been split across multiple courses (for example, the discussion sections have been moved to their own course, separate from the lectures) they may not be compared with all their peers in the course.
Performance Flag Details (Grades)
Performance data uses the Canvas Gradebook. It uses the current calculated grade for a course.
- The more assessment and grading activity that is handled in Canvas, the more useful this flag will be.
- Grades are included based on the grade posting policy set in Canvas.
- Grades can be included from external tools, if they are integrated within the Canvas course (e.g. Top Hat, Atomic Assessments, ScanTron, Kaltura Quizzes.)
- Grades can be manually entered or imported into the Gradebook.
What is Performance in a Course?
- Performance is measured as any and all grades entered in the Canvas Gradebook.
- Wherever possible, students are compared to all sections of a course
A performance flag will be raised if:
- A student is a full letter grade (8 percentage points) behind the bottom quartile of their peers.
- Students are scoring in the bottom quartile - their grade performance is in the lowest 25th percentile for the course.
- There must be at least two courses that meet the performance threshold (If a student is only enrolled in one course, a flag may still be raised.)
How the Performance Flag is Calculated
- LAVA calculates the bottom quartile grade (the lowest 25th percentile) for a course.
- It then subtracts a single grade (e.g. from A to B) which is 8 percentage points.
- If the students’ grade is less than the above calculation, a flag may be raised (if this occurs across two or more courses, when the student is enrolled in more than one course)
Additional Details
- If the Gradebook is not turned on, nobody gets flags (for example, a face-to-face course that doesn’t really use Canvas).
- Peers for grades are based on the course and enrollments as originally defined in the Student Information System (SIS). If different sections are later split apart (e.g. the discussion sections have been moved to their own course, separate from the lectures) and grades are given in different sections, we will get duplicate grades for the same SIS course. In this case, the maximum score from the two sections will be used, as there is no way programmatically to determine how an instructor will later combine these grades before final grade submission. We default to the maximum score to reduce the risk of raising a flag unnecessarily.
- Instructional choice about grading (and activity for that matter) is a personal choice of each instructor; consistent grading practices should not be assumed.
Where can I find more information about learning analytics?
For additional information on using learning analytics at UW-Madison, check out:
- The learning analytics guide, What are the pedagogical uses of learning analytics?
- The Learning Analytics at UW-Madison page on the the Vice Provost for Teaching and Learning website.
- The The Learning Analytics Center of Excellence information about projects, resources and events.
- The Learner Engagement Analytics Dashboard (LEAD) for instructors.
- Learning Analytics Guiding Principles.