A Schmidt Futures-backed effort lets teachers test their intuition about what works best for their students
From The Hechinger Report
By Jill Barshay
June 14, 2021
During the pandemic, Krystal Clifton, a veteran fourth-grade teacher in Kankakee, Illinois, 60 miles south of Chicago, ran a study in her classroom to see how well her students were learning, both remotely and in-person. Over five months, from October 2020 to February 2021, Clifton carefully tracked the performance of her 25 students in three forms of instruction: traditional in-person teaching; live, interactive lessons over Zoom (synchronous instruction); and self-paced instruction, where students could watch pre-recorded videos and complete online worksheets in their own time (asynchronous instruction).
“I think we’ve all been wondering about learning since we started going online in the pandemic,” said Clifton. “I also thought this could really change the way we teach, after the pandemic, if we find asynchronous teaching is really effective. With teacher shortages, this might make a big difference.”
Clifton was among 30 teachers around the country participating in a new effort, funded by the former CEO of Google, to encourage more teachers to conduct high-quality classroom research, from miniature randomized controlled trials to A/B tests, a type of split experiment often used by online retailers. In commercial A/B tests, website visitors are shown different versions of an ad at the same time to see which is most effective in driving sales. Applied to education, teachers can use A/B tests to see which approach is most effective in driving learning. In this experimental program, a bilingual teacher compared two ways of teaching vocabulary, one bilingual and one explaining words in English only. A fifth grade teacher broke the class into groups and tested different ways of teaching note taking.
These new teacher-led experiments are an attempt to address long-simmering tensions between researchers and teachers. Researchers often complain that teachers don’t use their insights and discoveries in the classroom. Teachers complain that researchers aren’t solving their problems or answering their questions about how to best teach. The gap leaves students without much evidence-based instruction in their classrooms.
Many researchers and teachers have tried to bridge this gap. For example, research practice partnerships pair up groups of researchers and large school systems to devise a research agenda together. These partnerships have generally been better at helping school administrators understand their problems than solving them. Other efforts have encouraged teachers to become researchers in their own classrooms by trying new approaches and documenting what happens. But these “action research” studies are often poorly designed without comparison or control groups and their conclusions — my idea worked! — are colored by teacher subjectivity.
This latest attempt to improve teacher-led research was inspired by Bill Hinkley, a blueberry farmer in Maine who is also a middle school math teacher. He was using a website, ASSISTments, to assign daily homework but was skeptical about the site’s digital tools for calculations. Working with Neil Heffernan, a professor at Worcester Polytechnic Institute who invented the homework site, Hinkley divided his students into two similar groups with equal grade distributions. Both were assigned the same problems for homework. One group completed the problems as usual while the other group was shown an embedded video reminding them to do their calculations using pencil and paper. The students who were prompted to use pencil and paper scored 13 points higher on homework than those who used the online tool as usual and they had better end-of-course results.
Ulrich Boser, founder of the Learning Agency, a consulting company, happened to be doing work with ASSISTments and was intrigued. He wondered if more teachers might be able to test their intuition with well-designed experiments.
“Technology is driving down the cost and ease with which we can do these experiments,” he said, explaining that online learning platforms make it a lot easier for one teacher to assign different kinds of instruction to different students in the same classroom.
His nonprofit arm, the Learning Agency Lab, launched the Teacher Run Experiment Network this 2020-21 school year. Schmidt Futures, a philanthropic initiative founded by former Google CEO Eric Schmidt and his wife, Wendy, is funding it, including a $2,000 stipend for each teacher.
“It’s not necessarily for every teacher,” said Boser. “But for a certain teacher, this is a really valuable way to hone their craft, and then add to the research literature. It’s exciting to see research ideas coming from the ground up and elevate teachers’ voices.”
The teachers received training over Zoom from research scientists on basic statistics and how to design a proper experiment. Coming up with a good hypothesis that can be tested was a challenge. In small Zoom groups, the teachers discussed and vetted each other’s experiments before launching.
Clifton, the fourth grade teacher, tracked daily and weekly “exit tickets” for her students, a short assessment to check if her students comprehended the lesson of the day. In her classroom, most of the students alternated between in-person and remote learning, using interactive video software such as Zoom and Nearpod. The exit tickets scores for both formats were surprisingly similar. But the exit ticket scores for students who learned independently in their own time were “much” lower. When her hybrid students had to switch to asynchronous independent learning because of pandemic quarantines and childcare arrangements, their exit ticket scores also dipped.
“On the assessments, a lot of times, they were getting zero percent, 10 percent, 20 percent. Like, they literally had no idea what they were doing,” said Clifton. “They went through every single lesson. They did the work and answered the lesson questions correctly. I think they figured out the system of getting through that lesson. But they were sort of passive and not getting the concepts.”
Afterward, Clifton told her superintendent that she recommended against asynchronous instruction in the fall of 2021 for families who want to continue learning remotely.
The giant drawback of these mini-experiments is their tiny size. Scientific standards require lots of students in different types of schools in various communities in order to draw reliable conclusions. Many statisticians argue that an experiment needs a minimum of 30 or 50 students to arrive at results that are statistically significant. Anything smaller could be a random fluke, a roll of the dice. One shouldn’t put much stock into any of the conclusions from any of these single classroom studies, which tend to involve fewer than 30 students, no matter how well designed they are.
Indeed, Clifton was teaching a gifted fourth-grade class with an advanced curriculum. It’s entirely possible that a different group of students wouldn’t have fared as well in remote learning as her high-achieving students did.
“It’s not a crazy idea to have teachers involved in research but it will be of little value if it doesn’t go farther than one teacher in isolation,” said Adam Gamoran, president of the William T. Grant Foundation, which funds and studies research-practice partnerships in education.
In medical science, many clinics are recruited to participate in a drug trial. Some patients get the drug and other get a placebo. “I don’t see that as any different from recruiting a lot of schools and having teachers deliver synchronous lectures to some students and asynchronous teaching to other students,” said Gamoran. “It’s not a crazy idea to do a bunch of small experiments and pool the results from them and try to get a reliable answer to a question.”
As a next step, the Learning Agency Lab hopes that teachers can pair up together and run an experiment in two classrooms simultaneously.
Outside the United States, teachers in the United Kingdom have been experimenting with conducting randomized control trials for several years. Education Development Trust, a U.K. nonprofit, sponsored a cluster of these before the pandemic, and published a paper about them in May 2020. Most focused on how teachers can help children better memorize information .
One hurdle here will be the idea of “experimenting” on children, which can sound ominous to many parents. During the training sessions for this program, teachers were told about when they need to obtain informed consent from parents. Advocates will need to find a way to communicate the importance of classroom research to the public.
I like the way Gamoran makes an argument for turning classrooms into laboratories. “If we already know the answer, if A is better than B, it’s unethical to do an experiment,” he said. “And we should all do A. But if we don’t know the answer, A might be better or B might be way better, then it’s actually unethical to just plunge ahead with one or the other just because we feel like it.”
The ethically correct approach, he said, is to “experiment and figure out what the more effective one is.”
Photo: Your Teen Magazine
Read this and other stories at The Hechinger Report