The One About Data, Numbers and Truth

Much has been made by the corporate reform community about how poorly American students do on the Program for International Student Assessment (PISA) tests. According to the Organisation for Economic Co‑operation and Development, which administers the exams, "the United States performed below average in mathematics in 2012 and is ranked 27th. Performance in reading and science are both close to the OECD average. The United States ranks 17 in reading...and 20 in science." Now, if these were "apples to apples" comparisons, we would be well advised to be concerned. But they aren't; and the reformers know it, don't care, and are more than willing to use this misleading data to advance their agenda.


What the reformers don't share--because it would not support their agenda of failing schools, bad teachers, disengaged students, uninvested parents, and greedy unions--is that more American students take these exams than do students in other countries, and that US students from schools with low rates of poverty do extremely well--better than students in most other countries.


(An important caveat here: I don’t think that test scores are a terribly meaningful way to assess what students know and can do, but it’s the data we are used to seeing. The idea that we need standardized test results to tell us what our children know is one of the biggest lies told by the reformers. Teachers know a LOT about our students: we know their strengths and weaknesses, we know about their siblings and families, we know when they are in trouble, and we know what they are thinking and feeling. Only persons who have never taught would believe that we need standardized test scores to let us know how our students are doing. The truth is that these test scores are among the LEAST meaningful things we know about our students, and no amount of hype and reformer drama is going to change that.)


The issue here is that the PISA test scores are usually not disaggregated--that is, the scores are not parsed out by poverty rates, but instead are all lumped together. Many in the corporate reform community are fond of reporting these international scores without sharing the fact that US poverty rates are nearly 30%, while countries like Finland are at 5% or less.


There is a strong correlation (not causation) between SES and test scores (and zip code and test scores, for that matter)--which is not to say that wealthy kids are smarter than poor kids. They aren't. Its just that kids with more financial resources have more opportunities and advantages than their less -advantaged peers. 


It's important to be clear when reporting these stats: US schools are not "failing," and in fact, when we provide our kids and schools with adequate resources they "compete globally" just fine (although that is not our goal as teachers). It's also important to remember that less than 7% of the differences in student learning are attributable to in-school factors, such as teacher quality--with more than 90% of the difference being a function of out-of-school factors, like test prep tutors, private music lessons and the resources to purchase instruments, after school sports, and access to travel, concerts, books, and movies. The reformers like to say that these things don't matter, but teachers know that they do--and we want nothing more than for all of our students to have access to schools with rich and vibrant music and art programs, athletics,  school nurses, librarians, psychologists and counselors, and all of the other advantages their peers in the suburbs enjoy and from which they benefit.


Here's the Truth...


So, the next time that someone tells you that American students are not "competing globally," and that our "failing schools" are a threat to our national defense, our economy, and our very future as a nation, please share these disaggregated stats from the most recent PISA tests. Because when we look at US students from schools with free or reduced lunch rates lower than 10%, here’s how American kids do—for comparison’s sake, Finland’s rankings are also included:


Science literacy


U.S. schools with less than 10% free/reduced – score=556 [1st in the world]


Finland – ranked 4th in the world


Reading literacy


U.S. schools with less than 10% free/reduced – score=559 [1st in the world]


Finland – ranked 5th in the world


Mathematics literacy


U.S. schools with less than 10% free/reduced – score=540 [5th in the world]


FInland – ranked 11th in the world


Then, do what my students would do, and…<drops the mic>

Write a comment

Comments: 7
  • #1

    H (Sunday, 14 June 2015 02:12)

    I agree, another interesting data set would be investigating: is the growth of students in each country. This would need the same kids sitting the test at least twice. A good education is all about impact.

  • #2

    Sarah (Sunday, 14 June 2015 03:47)

    Can you please let me know where you got this data.
    'It's also important to remember that less than 7% of the differences in student learning are attributable to in-school factors, such as teacher quality--with more than 90% of the difference being a function of out-of-school factors'

    We are getting John Hattie's work pushed fown our throat which say 30% student outcomes are attributed to teacher quality, but he ignores SES in his studies. I would like to see a study that does include this.

    Ta

  • #3

    Mitch (Sunday, 14 June 2015 08:40)

    Sarah,

    According to a recent statement
    issued by the American Statistical Association (ASA):
    "VAMs typically measure correlation, not causation:
    Effects—positive or negative—attributed to a teacher may
    actually be caused by other factors that are not captured in
    the model. . . . VAMs should be viewed within the context
    of quality improvement, which distinguishes aspects of
    quality that can be attributed to the system from those that
    can be attributed to individual teachers, teacher preparation
    programs, or schools. Most VAM studies find that teachers
    account for about 1% to 14% of the variability in test
    scores, and that the majority of opportunities for quality
    improvement are found in the system-level conditions.
    Ranking teachers by their VAM scores can have unintended
    consequences that reduce quality.”"

    American Statistical Association. 2014. ASA statement on using valueadded
    models for educational assessment. Accessed May 13, 2014, at
    http://www.amstat.org/policy/pdfs/ASA_VAM_Statement.pdf.

    see: https://www.academia.edu/12968982/The_Inchworm_and_the_Nightingale_On_the_Mis_use_of_Data_in_Music_Teacher_Evaluation

  • #4

    diseño gráfico mallorca (Wednesday, 07 June 2017 07:01)

    Thanks for every other informative site. The place else may just I get that kind of information written in such an ideal means? I have a venture that I’m just now operating on, and I have been on the look out for such information.

  • #5

    Sinus Infection (Tuesday, 22 August 2017 17:12)

    It turned out to be Very useful to me and I am certain to every one of the observers here!

  • #6

    EssayPedia10off (Monday, 02 October 2017 06:08)

    This is my first time i visit here. I found so many interesting stuff in your blog especially its discussion. From the tons of comments on your articles, I guess I am not the only one having all the enjoyment here! keep up the good work

  • #7

    www.55printing.com/blogs/brochure-design-tips/ (Monday, 02 October 2017 09:15)

    Moreover, this will not break your bank. If you do not have to conform to a specific postal regulation, you will be able to save money on the production cost. You can forget about the regular envelop and make a postal billboard which will generate business.