Science, Education, and Science Education

Every dataset has a story. We usually look only at the data and ignore the story. For example, according to my original findings, and as approved by my committee of esteemed researchers in education and science, I could make this statement:

Pre-service elementary teachers showed a statistically significant gain in their learning about the moon and teaching elementary students about the moon by inquiry.

And this supporting statement:

The study shows that pre-service teachers average gain scores from pre-test to post-test increased by 7 points on a 21-item test.

If this were taken as the only finding from my dissertation, these pre-service teachers obviously demonstrated significant learning. All is well.

But wait. (See below for the tl;dr version.)

At this point, the story needs a disclaimer.  I did not teach this class.  The instructor of record is no longer teaching at this university. The course has been taught in a very different, and, it is hoped, more successful manner for a few years now.

The preservice teacher participants’ meager reflections confirmed their apparently considerable leaning. Likert scale survey questions showed much agreement that the course had significantly deepened their learning.

Another look, from a perspective otherwise easily missed, is this finding:

While there was a significant increase in constructed knowledge among the experimental group, the mean posttest score of 11.17 out of 21 possible is only 53%.

And then there’s this:

At the individual student level, these data are of even more interest….. The experimental group, however, had 15 out of 24 pre-service teachers (or 63%) who ended the semester with more misconceptions than they had at the beginning.

Misconceptions were measured on the main instrument, a pretest/posttest. The pretest/posttest had an “I don’t know” option. The number of “I don’t know” responses was much higher on the pretest than on the posttest. The same questions on which teachers responded with “I don’t know” on the pretest were correlated with incorrect responses on the posttest. Overwhelmingly, the participants were willing to admit they didn’t know something on the pretest. On the posttest, they did not want to admit they didn’t know, they made up an answer hoping to earn some credit, or they genuinely thought they DID know and gave an incorrect response.

One of the most important things I learned while writing this dissertation was the story behind the data.

This is human subject research, folks, with many uncontrolled variables and many stories not easily measured with numbers. Using data from standardized student tests alone is very, very dangerous. I’ll stand unwaveringly on that position until this whole NCLB/RTTT/evaluation madness goes to the grave.