The "Brave New World" of Teacher Evaluation: Be Afraid...


Just when you think the masters of the universe who run the corporate reform movement can't be even more clueless, you catch wind of the newest effort to turn education into a profit-generating endeavor--this time, a hare-brained scheme to evaluate student teachers, by...and I swear I'm not making this up...observing them as they pretend to teach...wait for it...a fake "classroom" of avatars, in a virtual reality environment.




Don't believe it? Neither did I until I took a look at the new "National Observational Teaching Exam," or NOTE, brought to you by the Educational Testing Service (ETS)--the same company that brings us the Graduate Record Exam (GRE), the SAT, the PRAXIS tests, the AP Exams, and the National Assessment of Educational Progress (NAEP).

NOTE works like this: a teacher candidate (student teacher) "teaches" a sample lesson for 6-7 minutes using a "virtual class" populated by 5-6 "avatar students," and receives feedback from an artificial intelligence program. The NOTE system is a combination of video production and "advanced technology" developed by the US Military, according to Mark Atkinson, the CEO of the tech firm Mursion, Inc., which is responsible for the creation of the avatars and virtual reality environment used in NOTE.

ETS describes NOTE as a program that promises to "measure a teaching candidate’s readiness to teach in ways that are representative of real-life teaching experiences." The operative word here is "representative," as the teacher candidates are not assessed by observing their work in an actual classroom, with real live children...

"Because a teacher's interaction with students is an integral part of certain high-leverage teaching practices, ETS and TeachingWorks are designing and prototyping virtual classrooms with interactive avatar students. In partnership with Mursion™, which provides a mixed-reality teaching environment with simulated students, the avatars are produced by trained, calibrated human "interactors" using standardized protocols. The use of virtual classrooms not only supports greater standardization of instructional contexts and settings for candidates, but also eliminates disruption to classroom activities, curriculum and student learning that occurs in schools."


"Simulated students."


"Trained, calibrated human 'interactors.'"


"Greater standardization of instructional contexts and settings."


These words should be chilling to anyone who has ever set foot in a classroom, has ever taught a child, or who believes that the purpose of education is the development of relationships between learners and teachers--and among the learners themselves.


Teacher candidates can't develop relationships with "simulated students."


Teacher candidates should not be evaluated by "trained, calibrated human 'interactors'"--to be honest, I'm not a fan of the word "training" in most educational settings, and I don't even want to know how humans are "calibrated."


And "greater standardization of instructional contexts" isn't something we should consider a goal worth pursuing--standardization serves only to diminish and devalue the richness and diversity of the teaching and learning process.

To be fair, the educational principles behind NOTE are not without merit. The intellectual basis upon which the exam is based comes from TeachingWorksan initiative housed at the University of Michigan under the leadership of Dean Deborah Loewenberg Ball of the UM College of Education. Dr. Ball's work in identifying the particular kinds of knowledge and skills needed to teach specific subjects (for example, see: math knowledge for teaching) has been recognized as groundbreaking, and holds great promise.

Could one make the argument that virtual reality simulators, such as those used in the training of pilots, might be useful in certain methods classes, or as a supplemental or enrichment experiences for prospective or practicing teachers as ways to practice their teaching "moves" under low-stakes, relatively stress-free conditions? Of course.

However, what is profoundly troubling here is that there seems to be no understanding, or even consideration, of the impact of the use of this work in creating a high-stakes evaluation tool for prospective teachers. It's one thing for teachers to lose their jobs for their students' test scores, or for being denied certification as a result of not passing a high-stakes assessment such as edTPA. But we need to ask ourselves some serious questions as a profession when student teachers are being judged on the basis of their ability to teach a "class" of avatars, created by a high-tech firm--and being charged for the privilege.

The seemingly small step from "methods class supplement" to "high-stakes assessment tool" is a devastating miscalculation, and betrays a disappointing lack of judgment on the parts of all involved. And just so there can be no mistake about labeling the NOTE as a high-stakes tool, Mr. Atkinson, the CEO of Mursion, Inc., can be heard saying the following in this video: "We are working with ETS around the new NOTE assessment that they are rolling out as the Tier 1 licensing alternative to the edTPA."

It is also worth mentioning that the funders for this initiative include the New Schools Venture Fund, Gates, Broad, Walton, Pearson, Bloomberg; aka, the usual suspects when it comes to corporate reform projects.

Perhaps the most troubling thing I encountered while looking into NOTE came while watching a video of the CEO of Mursion explaining the advantages of learning how to teach with computer simulated students, rather than real, live children. As he described a hypothetical teacher candidate making an error in explaining a math problem (why is it always math with the reformers?), Mr. Atkinson waxed poetic about the benefits of simulation in providing feedback to students: "You're not missing a kid who may be turned off to mathematics, you're missing an avatar that doesn't matter."

And therein lies the rub. There's nothing about teaching that doesn't matter. And learning how to teach with students, real or simulated, who "don't matter" will only lead to the further deprofessionalization of teaching, not to the Brave New World promised by Mursion and ETS.

Teaching is a human activity, not a video game.

Becoming a teacher is a developmental process that is guided and nurtured by experienced colleagues and mentors, not measured by "trained, calibrated human 'interactions.'

And learning how to teach requires caring--not the elimination of it.

Write a comment

Comments: 14
  • #1

    Christine Langhoff (Sunday, 08 November 2015 03:03)

    My 24 year old son has been reading this post over my shoulder. He is incredulous that this could even be proposed. His take is that you need a human being for teaching and when I pointed out that he was echoing you, Dr. Robinson, he said "You don't need a PhD to know this, just simple common sense."

    Which is sadly lacking among the billionaires and their reformy friends.

  • #2

    Mitchell Robinson (Sunday, 08 November 2015 09:44)

    Your son sounds pretty smart, Christine...;)

  • #3

    Peter Ellertsen (Sunday, 08 November 2015 10:03)

    The arrogance and stupidity here is breathtaking! My teaching was formed by Carl Rogers' theory that teaching and learning occur in human relationships not unlike the therapeutic relationships he discusses in clinical practice, and it immediately made me more effective in the classroom. You just can't learn to have human relationships with something that isn't a human being.

  • #4

    Chris (Sunday, 08 November 2015 12:06)

    Why don't administrators just evaluate our own digital teacher avatars?

  • #5

    Cathy (Sunday, 08 November 2015 13:29)

    ETS doesn't do SAT anymore - Pearson now does SAT. I have a lot of respect for ETS, especially compared to other testing companies. There has to be more to this than what the blog implies. If this was Pearson, I'd totally believe this insanity.

  • #6

    Mitchell Robinson (Sunday, 08 November 2015 17:37)

    The SAT has always been administered by ETS. The Stanford Achievement Test is a Pearson product.

  • #7

    John (Monday, 09 November 2015 10:24)

    Don't knock it until you've tried it. I can see how, on the surface, this seems like another "big brother" is watching AND EVALUATING in a completely artificial environment. But do entertain the notion that you might be a bit misinformed about the system.

    I've been teaching for over 30 years and coaching teachers now for 6 years. A couple of years ago I had the pleasure of seeing this system in action. Looking back at the pictures on your blog I'm amazed that I am remembering the "personalities" of the students represented as avatars on the screen. At that time I had the opportunity to not only witness an education student "perform" in front of the virtual classroom, but also to see thoughtful, caring reflection with a REAL human being about the experience. The student teacher became emotional in her reflections about what she felt went right and wrong during her stint in front of the "class." Her professor masterfully walked through the experience with her student and gently guided her through the process of coming up with her own successful strategies should these situations occur in her own class. It might have been the most beneficial methods experience I've had the pleasure to witness.

    But the real kicker was when I was asked to stand in front of the screen and try to teach a lesson to these students. It was uncanny how my emotional response to the students was extremely similar to the experiences I have in my own class. Some students were over-eager while others were clearly slackers. One was quite shy but, when prodded a bit, showed incredible talent--just like in a real classroom.

    My colleagues and I have been talking about this experience on occasion for the past two years. We still refer to some of the "students" by name. In fact just last week I was observing a small class of students with special needs when I found myself saying to myself, "Oh my goodness, I'm actually IN the virtual classroom we experienced a while ago.

    Of course a virtual classroom can never replace what it is like to actually stand among a group of wonderful kids and all of the emotions and energy they bring to a classroom. But practicing in front of a low-stakes situation like this could be a wonderful reflective experience for EVERY teacher and not just the ones in training. My confidence for the production of the system itself actually makes me think the assessment might be just as good.

    As an evaluator of student teachers, I've had a few who could have really used some avatar training before they stood in front of my students--the ones I deeply cared about and hurt for when I saw them having to suffer the experience of an untrained, uncaring college student who was just jumping through the hoops to get his degree. The human aspect of my evaluation probably resulted in a better-than-deserved recommendation due to the fact I cared about him as well. Perhaps a disconnected observer who really knew what she was doing might be able to make an objective judgement of what she has seen even in this virtual environment better than my emotionally attached observations.

    Sorry for the long response but I just wanted to shed some light on the other side of the equation. Just knowing how to "program" such a situation demands incredible insight into kids themselves.

    It is always easy to rage against "the machine" but I think you should give it the benefit of the doubt until you see it in action.

  • #8

    Dr. J Patrick Tiedemann (Thursday, 12 November 2015 18:24)

    One key consideration that I don't see being raised is the underlying assumption that ETS is more qualified to assess a student teacher's teaching ability than the professors who teach, observe, and mentor those student teachers.

    The attempt to move assessment of teaching proficiency from professors who observe and guide multiple teaching experiences in authentic contexts with real students to ANY external entity not remotely connected to the real teaching and learning process must be received with great skepticism.

  • #9

    kids (Friday, 21 April 2017 08:36)

    Thank you very much for the sharing! COOL..

  • #10

    meditation (Saturday, 29 April 2017 02:19)

    Thanks for a wonderful share. Your article has proved your hard work and experience you have got in this field. Brilliant .i love it reading.

  • #11

    West Michigan vinyl and graphics (Tuesday, 06 June 2017 06:40)

    This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!.

  • #12

    frosty the snowman- miraclepianist (Wednesday, 09 August 2017 04:14)

    Thanks for your information, it was really very helpfull..

  • #13

    Claim (Thursday, 05 October 2017 06:46)

    Great Information sharing .. I am very happy to read this article .. thanks for giving us go through info.Fantastic nice. I appreciate this post.

  • #14

    Lateacher (Thursday, 19 April 2018 17:14)

    Hope you continue to research this, and in the same spirit we teach students to research—consider counterclaims that a reasonable person (as opposed to a caricature of evil) might offer, be open to the fact that it may not be as cut and dry as you initially thought.

    Lacking that, your interpretations may feel satisfying, but they will be little more than consipracy theory