A math program that offers computer-based, bespoke lessons for students could, with teacher support, help them learn more math, according to a study released last week.

The report, from the Center for Education Policy Research at Harvard University, found encouraging, but mixed, results for a California-based charter school network and a Maryland school district that used this math program, made by DreamBox Learning. The research suggested that students who spent more time using the program made more progress on grade-level math tests than other students did.

It’s a fundamentally different learning experience than most math programs

DreamBox uses technology to create customized, computer-based lessons for students that are neither too easy nor too hard. It collects and analyzes data for teachers. It provides students with an engaging format to improve their skills and steadily make progress. And the program provides instruction for teachers to help them learn how to use it effectively in the classroom.

“It’s a fundamentally different learning experience than most math programs,” said Tim Hudson, DreamBox Learning’s vice president of learning.

More than 3,000 students in grades 3 to 5 were included in the study, which gauged results at the Howard County Public School System, a traditional school system in Maryland, and Rocketship Education, a California-based charter school network known for its intensive use of education technology. Students who completed at least some of the math program performed, on average, between 1.5 and 3 percentile points better on standardized assessments. Higher levels of achievement correlated with how much of the computer-based program students successfully completed.

There was significant variation in how schools used the program. Many teachers were using the program less than the recommended amount of time. And teachers, rather than students, seemed to be deciding how children were using the math program, according to the research.

Students used DreamBox an average of 35 minutes per week in Howard County schools and 44 minutes per week at Rocketship. The recommended level of use is 60 to 90 minutes per week. And there were other important differences in how the math program was used in the classroom. In Howard County, many students using the program were behind grade level, which suggests teachers were assigning it most often to struggling students. At Rocketship, however, various levels of students were using the program, and there was no expectation that students use the software outside of the typical school day.

The researchers could not say for sure that the computer program was the reason students were doing better. The study did, however, attempt to control for other factors, such as the quality of teaching the students were receiving or individual student motivation.

It’s difficult and costly to research education technology and figure out what’s working – and what’s not.  This has been the case with education research, in general, but the problem takes on new urgency with technology. Slow studies can be practically useless; the technology examined could be obsolete by the time the study is complete.

The Harvard study of DreamBox presented a streamlined method for evaluating the program, according to the researchers. For instance, the Harvard report used student data from computers to figure out how the program was being used in schools. This is a change from other ways of studying classrooms, such as visiting schools to observe practices or surveying teachers.

“Our goal was to develop a streamlined, low-cost evaluation model that could be replicated easily,” the authors wrote.

Cross-posted at The Hechinger Report.

Nichole Dobo

Nichole Dobo writes about blended learning. Most of her 10-year career as a reporter has focused on education. She has also covered stories about government, courts, business and religion. She was a staff writer at The News Journal in Wilmington, Del., The York Daily Record/Sunday News in York, Pa., The Times-Tribune in Scranton, Pa. and The Citizens' Voice in Wilkes-Barre, Pa. and has been published in The Atlantic's online edition. She won first prize and best of show for education writing in 2011 from the Maryland-Delaware-D.C. Press Association. She earned a B.A. in journalism at the Pennsylvania State University.