Controlled study shows VideoNote improves exam grades 9.5%

Summary

In the Fall 2008 semester, ECE 2100: Circuits participated in a study with a new online learning tool called VideoNote.  The course of 77 students was randomly divided into two groups.  Students in Group A were given access to the VideoNote service through www.videonote.com for 9 lectures covering operational amplifiers.  Group B was given access to VideoNote for 12 lectures covering transfer functions and Laplace transforms.

The final exam consisted of 6 questions, each graded out of 15 points.  Questions 3 and 4 pertained to material in Group A’s VideoNote recordings.  Questions 5 and 6 pertained to material in Group B’s VideoNote recordings. 

If VideoNote improves grades, one would expect that Group A would score higher on Q3 and Q4 and Group B would score higher on Q5 and Q6.  This is in fact what was observed.  Note that since we randomly divided the groups at the start, there is no self-selection bias present here.  We were not simply identifying the diligent students that studied more and received higher grades but rather isolating access to VideoNote as a factor and its influence on grades and (hopefully) comprehension.

Graphical Summary

Full Version:

Overview of VideoNote

VideoNote is a web 2.0 service that captures lectures with high quality audio, video and linked notes to be placed online for students to view.  VideoNote captures the same feed as someone in the lecture would experience.  VideoNote links all the notes to the video and make them searchable so users can instantly jump to any part of a lecture that contains the topic that they need to review.  VideoNote also collects additional data to determine which topics students are having difficulty with so users can search for the most relevant topics.

Participants and Protocol

For ECE 2100: Circuits in the Fall 2008 semester, taught by Professor Belina, we divided ECE 2100 into two randomly selected groups.  We received an exemption from the IRB at Cornell and stripped the names from the grades and only retained ID numbers corresponding to group and usage.  Group A, which consisted of 38 students, received access to VideoNote for the first section of the study, which consisted of 9 lectures primarily revolving around the topic of Operational Amplifiers.  Group B, consisting of 39 students, received access to VideoNote for the second section, consisting of 12 lectures revolving primarily around the topics of Transfer functions and Laplace transforms.

Access to VideoNote was provided through the website www.videonote.com and students needed to log in with their Cornell NetID in order to access the lectures.  It is assumed that students would not share this login, as it also accesses sensitive material such as grades and bursar billing.  It is possible that some students not in the same group shared access by looking at the same computer screen, but this is unlikely to be a significant effect. 

Both groups had over 50% of the students use VideoNote at least once.  About 25% of each group used it heavily (>5 times); group A had only about 15% of students not use it at all and group B had about 40% of students not use it at all.

Method of Analysis

In order to analyze the effect on grades by using VideoNote, we analyzed the grades from the final exam, which is broken out by question.  The exam had 6 questions, the last 4 of which were relevant to our study.

Q3 and Q4 were questions with material on Operational Amplifiers: Group A had VideoNote

Q5 and Q6 were questions about Transfer Functions and Laplace Transforms: Group B had VideoNote

Each question is graded out of 15 total points.  These are the mean grades for each group for each question:

 Group 2 happened to have better grades overall – something that happens when you randomly divide a population into two groups.  We take this difference of 0.71 points and add it to all of the group 1 grades to make the means equivalent – eliminating the random effect of the group selection.

Hypothesis

Looking at the difference in scores between the two groups shows the effect of having access to VideoNote on the grade for each exam question.  Q3 and Q4 were questions where Group 1 had access to VideoNote and Q5 and Q6 were questions where Group 2 had access to VideoNote.  If VideoNote improves grades then we should see Group 1 score higher on Q3 and Q4 and Group 2 score higher on Q5 and Q6.

Stated formulaically:

H0: VideoNote has no effect on exam grades: We should see no difference between Group A and Group B’s performance for each question.

H1: VideoNote improves exam grades: We should see Group A score relatively higher on Q3 and Q4 and Group B score relatively higher on Q5 and Q6.

We see that for all 4 of the exam questions, having access to VideoNote improves that group’s grades by between 8.2% and 11.9% - on average a 9.5% improvement in exam grades.  This means that on average, a student that would score 70% without VideoNote would get a 77% with having that as an option.  Further, these results are statistically significant at the 5% significance level.  The p-value (meaning the probability that these results were observed simply by chance) is between 0.017 and 0.086 for each question – the probability that we would observe all of these by chance is extremely small. 

Conclusions – eliminating the self selection bias

As the only changing variable between the groups is whether they have access to VideoNote or not, we can be relatively certain that this is the cause of the improved grades.  The purpose of this study was to eliminate the natural self-selection bias that occurs with an optional learning tool.  The students who spend the time to use VideoNote might be more motivated students already in which case one might merely be observing that the better students use VideoNote, as opposed to VideoNote actually making students better.  We can make a stronger statement here due to the fact that we randomly selected the two groups at the beginning.  Of course, this conclusion is specific to this course and style of teaching, whether this applies to all types of courses is yet to be determined.  Further study is warranted to determine how the use of VideoNote can improve comprehension in other courses.