The 'Drill Down' Process

A strategy for asking the right questions and applying the data in useful ways by Philip A. Streifer

Ithought I had prepared well, having thoroughly reviewed the school district report card published annually by our state education agency. I figured I could anticipate any question my school board might ask about our performance.

All was going well until one board member, who had been a reading specialist prior to retirement, picked up on an interesting combination of facts from among the hundreds of statistics compiled by the state about our schools and students. He observed a disparity between our two elementary schools. One appeared to be spending more time in language arts instruction than the other, and it also performed better on the language arts portion of the state's mastery test.

There it was, right in front of our noses, but we hadn't picked up on it. When asked to comment, all I could say was "We'll get back to you on it."

It was a couple of months more before I reported my findings to the board. The path I followed in figuring out what had occurred was time consuming and not at all apparent. In fact, the "answer" was quite surprising as it turned out that the school spending less time in language arts (and that had attained poorer test results) was actually "overperforming" given the entry language arts level of the school's students. Had I run with just the surface information displayed on the state report, I would have unfairly accused a school and its faculty of poor performance. The "drill-down" process that I engaged in, however, revealed a very different result.

Valuable Comparisons

The drill-down process starts with a global question or issue, which then is broken down into its component parts for analysis. Once the analysis is completed, all the data are considered from one's "helicopter view" to make a reasoned decision on next steps.

In the case of the state report card, the global question was "Is there a relationship between time allocated to learning and performance?" But the real question posed here by the school board was "Should we force the lower-performing school to add language arts instructional time?"

A good example of how the drill-down process works is one that many school district leaders have experienced in making a major purchase such as a home. On the surface the issue seems straightforward. We want to buy a home, which leads us to ask a series of questions: Where do I want or need to live? What is the cost of homes in one locality versus another? How much can I afford? What are interest rates and are they stable? How much are taxes? How much are the condominium fees?

As these factors are narrowed down, I can begin looking at various properties to make comparisons. Then, once I have selected one or two very good prospects, I can consider making an offer to buy. But how much should I bid given market conditions and potentially rising interest rates? If I bid too low, dragging the purchase process out, I risk losing that amount in rising interest rates over the term of the mortgage. Or I risk losing the home altogether, having then to settle for a higher-priced or same-priced home or a less desirable alternative.

Thus, I need to conduct a cost-benefit risk analysis to continue the process. A final decision or determination requires what I call a helicopter view—an overview and consideration of all the facts that can be gathered and analyzed in a reasonable period of time.

Statistical Differences

In the case of the language arts performance at the two schools, I started by simply verifying all of the information. Next I wanted to know if the schools were somehow using different instructional techniques. I didn't think so, but it was worth checking. Then I reviewed the experience of the staff—perhaps one school had a young, less-experienced staff. That proved to be a dead end as the staff was well balanced. Then I decided to disaggregate the data by student mobility rates, gender and special education enrollment. No luck there either.

A next logical step was to see if the mastery test scores between schools, as reported by the state, were statistically different. The score reported by the state was the percentage of students attaining mastery. But I wanted to know if there was a practical difference in scores between the schools.

To find out I entered each student's score into a spreadsheet and ran a T-Test—a powerful statistical technique to determine if group scores are significantly different. The analysis showed no statistical difference between schools. This was surprising. I wondered how these two schools, one with more students attaining mastery than the other, could have no statistical difference between their scores?

The answer, it turns out, is critical and lies in the nature of the measures reported by the state. Percent mastery is a simple "frequency," based on a cut-off score, while the T-Test is a more powerful statistic designed to help one better understand the nature of and the differences in the data.

While discussing all of this at a superintendent's cabinet meeting, one principal remarked that the school with lower test results seemed to have students who had poorer reading skills and suggested that we check their verbal ability. We had a verbal ability score on these students, but I was cautious about using it for important decisions because of the inherent lack of validity. But given the previous analyses and the fact we had run out of other options, we decided to look at the verbal ability score averages more carefully. We found that students from the lower-performing school on the mastery test also had significantly lower verbal ability scores.

This trend told me we were onto something. The two schools' mastery test scores were not significantly different (statistically speaking), yet one school had lower verbal ability scores and fewer students reaching mastery level. What did this all mean?

We realized that it was possible for a school to be successful while not having as many students reach an arbitrary cut-off score—the state mastery standard. The school with lower mastery test scores had actually done quite well moving as many students to mastery as they did, all the while working with students of lower verbal ability levels who had spent less time on language arts instruction.

The drill-down process resulted in a significant change from the original assumption or hypothesis. Had we acted on the initial results reported by the state on its annual report card we would have made a serious error. However, by following a logical sequence of questioning we were able to come to a deeper understanding of the issue.

Our response to the board's question could not have been predicted. We showed how the school with lower mastery test results was actually doing very well. What might have begun as criticism turned out to be a tribute and a recommendation to keep up the good work.

Logical Exploration

This example demonstrates that there are two frameworks for guiding the drill-down process: the variety of questions posed and the level of data analysis used.

The questions fall into three categories having to do with (1) disaggregation of data across one or more factors; (2) longitudinal analyses over time; and (3) a category I simply refer to as "exploring." The story of the two elementary schools fits the latter category because it is a discovery process without any clear, predetermined direction to follow.

Logic is the most important attribute of exploration. Buying a home also is an example of an exploration inquiry. The case discussed in a moment demonstrates the disaggregation and longitudinal drill-down processes.

Statistical power is the second guide for the drill-down process. In the case of the two schools, we started with basic statistics such as frequencies—the simple straightforward display of aggregate information (percent of students reaching mastery and the amount of time allocated to language arts instruction). Then we reached a much deeper understanding of the issue when we looked at the data more deeply using more powerful statistics.

But we can't continue the drill-down process indefinitely as there is a cost-benefit consideration here. Ending the drill-down process and analysis process is a judgment call informed by our helicopter view and the importance of the issue. Just as we cannot visit every home on the market (or in most cases we can't), the same is true for educational queries.

Available Data

A colleague recently wanted to know the impact of instructional time spent in a special reading program on achievement at the elementary level. The first step was breaking the problem down into its component parts. Thus, the drill-down questions were these: What was the growth in reading achievement from fall to spring (longitudinal)? What was the impact of time spent in the special reading program to that growth (disaggregation)? What is the impact by school (disaggregation)? Could we identify teachers who were particularly successful to use as models (disaggregation)?

Fortunately in this case, all of the data was in a "data warehouse"—a large database that can hold data for analysis from disparate sources and multiple years. This allows us to do these analyses rather quickly—in several hours instead of the weeks it had taken me in the previous case.

The first drill-down question about the growth in reading achievement from fall to spring was easily addressed by comparing the fall state mastery test results with the locally administered spring reading assessment results (all using the same measure). Looking at simple averages, there was an overall 6.8 point gain from fall to spring. The district considered that significant.

To test this a little further, remembering my experience with the two schools, I decided to run a T-Test and found the difference between the fall and spring scores to be statistically significant. Now I was satisfied.

The next question was a lot harder to address, even though all the data was in the data warehouse. Identifying the impact of class time spent in the special program on spring scores was not going to be easy. To do this I decided to use a more powerful statistical technique that allows one to handicap or modify one variable based on another.

In this case I would handicap the fall to spring score based on the amount of time a student spent in class weekly. (Fortunately the district had been collecting these data, expressed as the number of hours in special reading instruction each week.) Once the statistics program does the handicapping, it then determines if there is a significant difference between fall and spring scores.

I found a positive impact of the amount of class time spent in the special reading program on the fall-to-spring reading scores. For data junkies, this was a cool finding.

Drilling down further I wanted to know if these results were significant by school and teacher. Indeed they were, but with an even more interesting finding that only a drill-down process could uncover. All of the school and teacher fall-to-spring averages showed that students had grown by at least the level one would consider significant. That was great news for the district. The final question was whether we could identify any teachers with extraordinary gain rates to use as models. Indeed we did.

Time-Saving Process

From a helicopter view it was important to address the fundamental cost-benefit question of whether the special reading program works. While no educational research can be absolutely certain in its conclusions, the district's conclusion was the program was worth keeping.

However, this is time consuming, complex and hard work. And it is often specious to derive meaning from simple scores as was the initial case with the two elementary schools. My colleague who wanted to know if the special reading program was working also suspected that some teachers were not performing well, which turned out not to be true. In both cases, findings were determined after significant drill-down into the data.

The work to gather all the data for the two elementary schools was done by hand and took weeks to complete. The work to assess the reading program was performed in hours because all the needed data resided in a data warehouse. But data warehouses are still rare and most of this work needs to be done by hand.

To address this challenge and ensure a successful school improvement process in a data-driven environment, school districts might consider the following suggestions:

* Develop data work teams for each school or district with staff members who have expertise in various areas: curriculum, testing, database manipulation and basic research.

* Give these teams adequate time to do their work. Districts might consider compensation as well. To adequately support school improvement work, every member of the school staff needs to be involved in some way.

* Teams should focus on trends in the data over time, not one data point. Schools and schooling are just too complex for single scores to drive solutions. Armed with diagnostic information gained through drill-down analyses, teams should rely on their judgment and expertise in planning program changes.

* Once the team believes it has identified the root cause of a problem (if we are focusing in on problems rather than gains), check the research on what works to solve the problem to avoid reinventing the wheel.

* Build a culture that supports the review and use of data for decision-making by including it regularly in the school improvement process. Change will be incremental—the shift to standards-based learning will have increasingly greater impact on the work of schooling. It is easier to build collaborative relationships before your school system is faced with tough issues. Doing data-driven decision-making only when a serious problem exists can result in an anti-data culture. Using these techniques regularly to support teacher and administrator decision-making will make for a much more positive and successful approach.

Phil Streifer is an associate professor of educational leadership at the University of Connecticut, 249 Glenbrook Road, U-93, Storrs, Conn. 06269-2093. E-mail: streifer@erols.com