How do teachers measure up?

The is a big push to improve teacher accountability and rating systems, and with good reason.  Many teachers get almost no feedback, rated either "satisfactory" or "unsatisfactory."  Feedback is often not continuous and does not feed continuous improvement: the feedback may occur once a year, it may be sporadic, and it may be reduced in frequency as one becomes a more experienced teacher.  There is often a very limited amount of "data" that goes into evaluations, and there may often be a limited amount of evaluators feeding into the assessment.

Michelle Rhee's The New Teacher Project proposed a new framework for teacher evaluations (pdf) that is largely based upon her very pro-charter, pro-standardized testing agenda.  It has a lot of great points, but it also is based upon some pretty egregious flaws.

The framework is based upon the idea that every teacher should be excellent, that several years of excellent teaching can bridge the gap between poor and wealthy students' performance, and that teachers need to be evaluated in a more rigorous way that focuses most heavily on the improvements made by their students.  On its face, this sounds very reasonable.

The  framework opens by citing a variety of studies that show how important great teachers are, which is certainly true.  Two of the studies she cites, however, paint an incomplete picture:

1. A 2006 study y Gordon, Kane and Staiger found that "having a top-quartile teacher rather than a bottom-quartile teacher four years in a row could be enough to close the black-white test score gap."
2. A 2002 study by Rivkin, Hanushek and Kain found that "having a high-quality teacher throughout elementary school can substantially offset or even eliminate the disadvantage of low socio-economic background."

These two studies, which are perfectly valid on their face, leave out some important underlying assumptions:

1.  Teacher quality is static - Northwestern University's Helen Ladd (pdf) evaluated teachers (2008) in the highest and lowest quintiles according to student assessments and found that most highly-effective teachers one year were not highly effective the next and many ineffective teachers were no longer ineffective the next.
2. An effective teacher is effective for all groups of students s/he teaches - If you are given a class that is significantly less prepared than the last one you taught, your performance may not improve and even a great teacher might fail to produce the desired gains.
3. Student gains and failures can be universally attached to the performance of a given teacher - Economist Jesse Rothstein surveyed data for 99,000 5th graders in NC and performed a statistical test asking "What effect do fifth grade teachers have on their students' 4th grade performance?"  Obviously, the effect should be 0, as they had yet to teach the kids.  Nevertheless, he tested three different "value added" measures and in all cases found that fifth grade teachers had an enormous impact on their students' test scores before they had even taught them for a day. There is obviously a flaw with the use of such value-added tests if such preposterous results are statistically significant. 
4. That the results of a good teacher can be added as the two studies suggest - Diane Ravitch notes in her The Death and Life of the Great American Education System that "nowhere was there a real-life demonstration in which a district had identified the top quintile of teacher, assigned low-performing students to their classes, and improved the test scores of low-performing students so dramatically in three, four, or five years that the black-white test score gap closed." 
5. Bridging the socio-economic gap is sufficient and means that our students are receiving a quality education:  plenty of white students from higher income backgrounds are doing terribly.  Bridging the achievement gap between rich and poor is a start, but certainly not the "destination" if a quality, world-class system is our goal. 

Now, the framework then continues with six key characteristic of a "good evaluation system":

1. All teachers should be evaluated at least annually;
2. Evaluations should be based on clear standards that prioritize student learning;
3. Multiple sources of data should be considered, especially those measuring student's academic growth;
4. Multiple rating levels to better differentiate teacher effectiveness;
5. Rating encourages regular, ongoing, and  constructive feedback; and,
6. Evaluation outcomes must have teeth, that is they should feed into teacher employment, bonuses, and pay.

I agree in general with the first five, though I see some constraints to the sixth characteristic.

All teachers should be evaluated annually

I think the more feedback a teacher can get from different evaluators during different types of lessons over the course of a year, the more useful a tool can be.  This seems like a great basis for an improved evaluation system that all teachers can use to improve.

Base evaluation on clear standards, emphasize student learning
Any evaluation, to be fair, should be based upon very clear standards with limited room for interpretation.  I agree also that they should be based on student learning, but I would urge caution in operationalizing the concept.  I think having impartial master teachers and principles observing or conducting a pop quiz to see if lesson plans are having an effect on a student's learning would get at this a lot better than using standardized tests.  Further, it would give teachers the freedom to teach a diversity of lessons that cover materials that are of extreme import but not necessarily on a standardized test.  The document does identify some opportunities like having a master teacher come in and note how many kids raise their hand or seem to "get" the material presented, though it does express a lot of support for the use of standardized tests.

Multiple sources of data should be used, focused on student growth measures
Diverse data--both in type and person evaluating--is critical to getting a more balanced assessment of a teacher's performance.  The focus on standardized tests is problematic as student performance on tests can vary and these tests may not test material that is all that worthwhile to know (or they may not test many subjects).  There is a further issue: a successful goal, according to the Harvard Business Review, is one that is concrete, that you can identify clearly when you have fulfilled it, and that is not dependent upon others.  Setting a goal for teachers that is dependent upon someone else (their students) is somewhat unfair.  Worse yet, these tests are not designed to test teacher performance.  They do not have the external validity to be misappropriated in this manner.  Create a standardized test for teachers, as that would at least have the validity necessary to make them an appropriate measure.  Additionally, if a test is administered mid-year, are the gains (or lack thereof) attributable to the current teacher or the previous ones? This is not clear. As I noted, I would prefer multiple observations and student and parent feedback.

Multiple Rating levels and on-going feedback
This is indeed preferable because it does improve upon evaluations to make them a tool for teacher encouragement and feedback rather than a narrow filter to remove only the worst teachers.  Further, if the ratings are meaningful and accompanied by concrete feedback, it gives teachers the actual tools to look at how they might improve and for the school to perhaps pair up that teacher with resources to help on their weaknesses. The more regular, the better.

Tie teacher ratings to their pay and employment 
All of the studies that I noted earlier should make us very cautious about this.  If teachers drop in and out of the highly-effective category (and the ineffective category) between years, then you need to be cautious about wantonly firing or punishing someone for doing poorly one year or rewarding someone who anomalously does brilliantly one year.  I think a more appropriate sixth metric would be to use student data and teacher performance data to try to determine what kinds of students a teacher teaches most effectively for future class assignments to try to set up student and teacher alike for success.  I recognize that this may not be realistic or might be logistically quite challenging, but it might be interesting to see what limits there are to this idea in practice.


In the end, my preference is for a rating system that looks a little different:

1. Monthly evaluations by an independent master teacher (15%)
2. Quarterly evaluations by school administrator/principals with experience in the classroom (15%)
3. Semester evaluations by external education evaluation experts (15%)
4. Round robin evaluations in which teachers evaluate their peers (10%)
5. Amount and quality of efforts made by the teacher to improve on areas identified in previous observations (15%)
6. Evaluations of student portfolios that look at growth on the subjects taught (10%)
7. Use of interviews to get randomized student and parent opinions of teacher (10%)
8. Performance on testing that can be attributed to that specific teacher, is included in that school's curriculum for the year, and that is value-added in nature (10%)

The weightings are my personal preference, but I think they reflect the importance of regular, directed feedback being used by a teacher for continuous improvement.  Ultimately consistently bad teachers should be counseled out, but those teachers who have a decent record of doing well should be retained and efforts should be made to match them with students with whom they will succeed (insomuch as is possible while still giving the most difficult to teach students access to quality teaching).

Speak out!

How would you evaluate teachers?

What are some interesting evaluation methodologies or criteria that you have seen/experienced?