There was an exchange on Twitter back in April, between two edu-Tweeters. One edu-Tweeter tweeted a study on how the use of iPads in the classroom lowered geometry proficiency scores and increased off-task behaviour. Another edu-Tweeter replied with a study showing the use of iPads increasing achievement and engagement, pointing out how easy it is for us to cherry-pick our evidence.
This is common on edu-Twitter. People (myself included) retweet studies where we like the results, without opening them up to take a look inside at the methods, analysis, and limitations. And we usually can’t; as mere teachers, we’re stuck on the wrong side of paywalls.
I, however, was enrolled in a university course, and took the opportunity to mass-download journal articles, these two studies on iPads included. While hitting the “Download PDF” button repeatedly, I noticed lots of debates and back-and-forth in the literature; someone publishes an article, someone else responds, the first person responds to the response, and so on. It’s actually pretty cool, when these things can be discussions and dialogues, and not just declarations.
So, I’ve decided to do a blog series on education debates, borrowing the hashtag from the weekly debates run on Twitter for my title. Arguments from academic literature will be put head to head, to see which one is stronger. I’ve judged middle school debates, so I am totally qualified to evaluate the merits of these arguments.
The rules of engagement are as follows:
- I only consider the evidence and arguments put forth in these articles. This is important when judging middle school debates, to resist the temptation to “round up” or fill in the gaps for the students. There may be other articles that add more weight to one side of the debate or the other, but I am not conducting a literature review or meta-analysis here.
- I must declare a winner. No ties allowed. Even if both sides have obvious flaws. Yes, the reality is that there is not a winner, its complicated, and it depends, but that’s no fun.
- I set my personal opinion aside. Which is hard, because I’m forming all sorts of opinions about education, but I’m judging a debate, not justifying my own ideas.
This week’s debate → Resolved: That iPads hinder classroom learning.
Perry, D. R., & Steck, A. K. (2015). Increasing Student Engagement, Self-Efficacy, and Meta-Cognitive Self-Regulation in the High School Geometry Classroom: Do iPads Help? Computers in the Schools, 32(2), 122-143. doi:10.1080/07380569.2015.1036650
This study was conducted in two secondary geometry classes, over the course of a semester. One class received iPads, and the other did not. The impact of the technology was measured in three ways: geometry proficiency scores, taken at the beginning, middle, and end of the semester; classroom observations of off-task behaviour; and self-reports of self-efficacy and metacognitive reflection on a Likert scale.
There is a glaring flaw in the design of the study: the two teachers use very different methodologies:
The teacher of the iPad (experimental) group was proficient with its use and implemented a student-centered pedagogy of inquiry and discovery to facilitate student engagement, discovery, and understanding of mathematical concepts. Prior to demonstrating the proper method of solving geometry problems, this teacher engaged students in completing problems using iPad applications…The teacher of the non-iPad (control) group used a teacher-centered pedagogy that frequently employed direct instruction and focused on demonstrating geometry concepts and problem-solving approaches prior to students working individually on problems. (130)
To me, that is a more important variable than the use of iPads in the classroom. Any differences in outcome between the two groups cannot be confidently attributed to the use of the devices. On the other hand, edu-tech advocates often assert that the use of technology should transform classroom pedagogy. If the pedagogy between the control and experimental groups were too similar, it would probably be said that the use of technology wasn’t done right. A better study design would have been two teacher-centered classrooms and two student-centered classrooms, one of each trialling iPads. However, because this was a “convenience” sample (130), these were likely just the teachers willing to participate in the study.
A second problem with the study is the lesson observations data. One lesson was observed of the non-iPad classroom, but two lessons for the iPad classroom. I do not understand why they would not do two for both, other than ‘convenience’. This creates problems, as the data between the two observations is quite different. On-task verbal behaviour increases from 38% of observations to 84%, while on-task non-verbal behaviour falls from 94% to 79%. There is no reason to believe the non-iPad group would not have similar variation between lessons, so I don’t think we can extrapolate the influence of iPads on behaviour from that data.
Finally, there is the data on geometry proficiency. The non-iPad group ended the year with higher scores than the iPad group. However, the scores for both groups fell, between the test at the beginning of the semester and the one in the middle. Other than speculating about math anxiety (139), this drop in achievement is not discussed.
Tay, H. Y. (2016). Longitudinal study on impact of iPad use on teaching and learning. Cogent Education, 3(1). doi:10.1080/2331186x.2015.1127308
This study took place over the course of three years. In the first year, two grade levels (Sec 2 and Sec 3) trialed iPads in a select number of classes. In the second and third year, iPads were used in all classes in Sec 2 and Sec 3. Thus, in the first year of the study, the data is compared to the non-iPad control classes. In the second and third year, the comparisons are made between those who were part of the first year of the study and those who are new to the iPads, between grade levels, and between the different years of the study to see change over time. Because of this design, the data I care about the most is from the first year of the study.
The study collected three forms of data: achievement data from year-end exams, lesson observations to see the activities teacher use, and surveys and interviews to measure engagement. We will look at each of these in turn.
In the first year of the study, the Sec 2 iPad users achieved statistically significant higher results in the end-of-year exams than the non-iPad students, particularly low ability and high ability. Students. The researcher speculates that this is because lower-ability students could use the iPads to review, while the higher-ability students could expand on what they learned in class.
The lesson observations were used to see how iPads impacted pedagogy. For the first year, iPad classrooms were compared to classes taught by the same teachers (without iPads), or teachers who matched them in skill and subject. Peculiarly, despite the loss of the control group in the second and third years, the researcher kept up the lesson observations, but instead compared when the iPad was used to when it wasn’t. I don’t know what the researcher intended to learn from this. We won’t learn how iPads change pedagogy. Instead, we’ll see, once the pedagogy has been changed (if at all), which activities a teacher will use with iPads.
The lesson observation data was also categorized into teacher talk and group discussion, which seemed to carry the assumption that teacher talk was bad. There was no consideration as to the quality of the teacher talk or the quality of the group discussion, or the role the iPads played in either.
But, none of this matters. In the first year of the study, in the only year where we can actually see how iPads impacted pedagogy, there was no statistically significant difference between the iPad and the non-iPad classes in terms of lesson activities. So, unlike the affirmative study, the pedagogy between the two groups seems to be the same, which reduces confounding variables.
Finally, we get to the survey and interview data, the former completed by both teachers and students. This was to measure the impact on engagement and was, oddly, only done with the iPad groups. The only comparisons they could hope to do, then, is between year groups and over time. This is in addition to the problems associated with perception data and, with the survey, self-selection bias (as not everyone answered the survey). It would not have been difficult to include both iPad and non-iPad groups in the survey. For example, the statement “My learning extends outside of class time with the use of the iPad” (4) could have been changed to “I frequently find ways to extend my learning outside the classroom.”
One of the interview questions also seemed to be a bit leading: “What were the things made possible with the iPad over the last three years?” (5). There was no question on how iPads may have hindered learning.
That being said, I appreciate the mixed methods approach of the study. Qualitative data can help illuminate quantitative data, and provide the ‘why’ behind the patterns we see. In the interviews and the comments in the surveys, students said they used the devices to review while waiting for the teacher to arrive and to search on the internet things they don’t understand.
Which brings me to my final point. The school in this study is composed of middle and upper class students who are in the top 10% of their age group for academic ability (measured by the national exams). As far as I can tell, this study also took place in Singapore, a country whose high valuation of education helped it develop and prosper. These students may have had the motivations and ability necessary to make the best use of the iPads.
And the winner is…
It is good that both studies looked at achievement data, and not just self-reports of engagement and self-efficacy; after all, we care most about the impact of edu-tech on learning. As well, both studies included control groups for comparison.
The affirmative study had a huge confounding variable (teaching methodology), which may have held a larger influence over the achievement results than the use of iPads. As well, the negative study was held over a larger time period; one year of their study had controls, as opposed to the one semester length of the affirmative study. This gives teachers and students more time to become adjusted to the new technology.
We should take into consideration the characteristics of the student population for the negative study when looking at its results. However, the affirmative study had far too many issues to be said to have any clear results at all.
I award the win to the negative side.