Evaluating EdTech: A Strategy for Selecting Digital Tools

Bewildering. Confusing. Overwhelming. Those are some of the milder terms I’ve heard educators use to describe the landscape for educational technologies. The good news: If you have a learning requirement in your school district, there’s probably a mobile app that promises to solve that for you. The bad news: There are likely to be hundreds, if not thousands, to choose from. On Apple’s App Store alone, more than 80,000 learning apps are available.

Navigating through EdTech options can feel like a tightrope walk for school administrators hoping to balance the desire for innovation with the need to make fiscally sound choices. Fortunately, it does not need to be a binary choice. Over many conversations I’ve had in recent months with digitally savvy educational leaders, I’ve learned that it’s possible to do both.

I’d like to share with you what I’ve learned.

Where to Start?

Most educators I’ve spoken with agree that defining your need is a good way to begin.

“Let’s say you want kids to learn more math,” says Steven Hodas, former executive director of the New York City Department of Education’s Office of Innovation and now practitioner in residence at the Center on Reinventing Public Education at the University of Washington. “An administrator might see the problem as kids doing poorly in math. But a teacher might say that the problem is they’re unable to give students sufficient individual attention. Both point to kids learning more math, but they define the nature of their objectives differently.”

Hodas suggests administrators engage people who work with the problem on a daily basis to help refine the need before jumping to possible solutions. That also includes parents, school staff, classroom aides and even the community at large. Once the need is better defined, it becomes easier to narrow down the universe of digital tools to the ones that address your need.

Next, Barton Dassinger, principal of the Cesár Chavez Multicultural Academic Center in Chicago, likes to proceed with a “research question” — turning the need into a testable hypothesis. This forces people to think of ways to quantify their objectives. “An example of a research question is: ‘Do students using product X show higher gains in Y than similar students not using the product?’” where Y is the metric used to gauge progress, whether that’s behavioral referrals, attendance, college admissions or test scores, Dassinger explains.

“Before you make a purchase, you can work with the company to provide the data that will help you answer your research question,” adds Dassinger, who recommends pressing vendors for weekly data reports. “That way, you can evaluate their claims on the metrics that matter to you.”

Improving Discovery

With tens of thousands of choices, finding the right tool can seem like searching for a needle in a haystack. Many educators have devised their own systems for filtering in the most relevant tools while remaining open to changing possibilities.

The Omaha Public Schools in Nebraska developed an app approval process that allows any educator in the school district to submit recommendations for digital products throughout the year. Since the process launched last fall, Omaha’s teachers have submitted more than 200 apps. Most receive a decision within five to seven days. If the app is rejected, the teacher receives feedback on why the product didn’t pass muster.

“Before this system, it was a manual process that would take months” to process a recommendation, says Rob Dickson, Omaha’s executive director for information management services.

In Chicago, Dassinger prefers to interact directly with teachers within the classroom environment and have informal conversations to surface new products, keeping a list of prospects on a spreadsheet.

Shared Characteristics

These examples suggest there’s room for a wide range of approaches. But the most effective systems tend to have several characteristics in common — they are ongoing, transparent and adaptable.

• Ongoing. Technological advancements occur nonstop, and teachers’ needs don’t always coincide with adoption cycles. The best systems continually evaluate what’s new in the market.
• Transparent. Danny Wagner, manager of Common Sense Education’s STEM Education Content and a former science teacher in Boone County, Ky., believes teachers need feedback so they can better understand why their submissions did or didn’t make the grade. This is critical. Feedback shows teachers their voices are valued, and it will help them stay involved, providing a fresh flow of ideas.
• Adaptable. Dickson calls his system in Omaha “a workflow for change” because he can update it with new curriculum requirements and learning standards. “If we create a sound system for evaluating apps, it shouldn’t matter what new technologies are thrown at us,” he says. “We can still do what’s best for our kids.”

If your district’s discovery process involves issuing a request for proposals, Hodas recommends focusing on the learning goal, not the solution. “If your RFP contains a list of specifications, people try to check the boxes and tell you what you want to hear. That’s not likely to be productive,” he advises. “You need to reach out in a way that gives people the freedom to give you creative solutions.”

Filtering Options

Let’s assume your discovery phase leads to a flood of options. An efficient way to narrow the field, Hodas says, is to ask teachers to watch a five-minute demo of each product (most tech products have video demos posted on their websites). After viewing, ask the teachers to answer this question: Would you use it in your classroom? If the answer is no (or mostly no), toss it out of consideration.

Once you have a manageable list of candidates, you can put them through a more rigorous evaluation. In Omaha, reviewers use a wide range of criteria, from compliance with the federal Children’s Online Privacy Protection Act and best practices to pedagogy and alignment to P21’s Framework for 21st Century Learning.

In the Charlotte-Mecklenburg School District in North Carolina, Valerie Truesdale, the chief technology officer, says she first determines whether the products are interactive and adaptive. “Students have to be interacting with the software and not just be given digitized worksheets,” she says. “And the software has to adjust its pace to individual progress so every student can have a personalized learning experience.” Products then are evaluated using North Carolina’s Standard Course of Study.

At Common Sense Education, where we’ve reviewed more than 2,500 educational technologies, our rubric includes engagement (how compelling the product is), pedagogy (how well it supports deep learning) and support (how accessible the experience is to students and teachers).

Recently, we added a fourth pillar: privacy. Over the past two years, we’ve worked with more than 100 school districts to create two free resources for schools: (1) a program that evaluates educational software products on their safety, security and privacy policies, along with how well they comply with government regulations; and (2) an open source toolkit for districts that want to do their own evaluations. By the end of the school year, our director of privacy review, Bill Fitzgerald, plans to have completed evaluations for 300 educational technology products on our site https://privacy.commonsense.org/.

Start Modestly

Finally, once you’ve selected a technology to try out, start small and iterate.

“Think of the first implementation as a pilot,” Hodas advises. “Don’t think of pilots as free software to try out. Think of the vendor as someone who is collaborating with you to create a product that will result in the changes you both want to see.”

This can be a cultural leap for school districts used to being given fully developed materials from established publishers. With technology companies, particularly the younger and more innovative startups, products are in constant evolution, and their developers are continually adding features, fixing bugs and improving the overall experience. While it can be frustrating to deal with a moving target, schools can take advantage of that flux by jumping in to influence which features get added and how the product is improved.

At the end of the day, I am reminded of some sage advice for maintaining perspective, provided by Truesdale, a veteran school leader in the Charlotte-Mecklenburg Schools. “The key is to remember that these are all digital tools,” she says. “Just like building a house, you have many tools in your toolbox. What tools we use depends on the job we want to accomplish. But there’s no substitute for quality teaching.”