How do teachers measure up?

The is a big push to improve teacher accountability and rating systems, and with good reason.  Many teachers get almost no feedback, rated either "satisfactory" or "unsatisfactory."  Feedback is often not continuous and does not feed continuous improvement: the feedback may occur once a year, it may be sporadic, and it may be reduced in frequency as one becomes a more experienced teacher.  There is often a very limited amount of "data" that goes into evaluations, and there may often be a limited amount of evaluators feeding into the assessment.

Michelle Rhee's The New Teacher Project proposed a new framework for teacher evaluations (pdf) that is largely based upon her very pro-charter, pro-standardized testing agenda.  It has a lot of great points, but it also is based upon some pretty egregious flaws.

The framework is based upon the idea that every teacher should be excellent, that several years of excellent teaching can bridge the gap between poor and wealthy students' performance, and that teachers need to be evaluated in a more rigorous way that focuses most heavily on the improvements made by their students.  On its face, this sounds very reasonable.

The  framework opens by citing a variety of studies that show how important great teachers are, which is certainly true.  Two of the studies she cites, however, paint an incomplete picture:

1. A 2006 study y Gordon, Kane and Staiger found that "having a top-quartile teacher rather than a bottom-quartile teacher four years in a row could be enough to close the black-white test score gap."
2. A 2002 study by Rivkin, Hanushek and Kain found that "having a high-quality teacher throughout elementary school can substantially offset or even eliminate the disadvantage of low socio-economic background."

These two studies, which are perfectly valid on their face, leave out some important underlying assumptions:

1.  Teacher quality is static - Northwestern University's Helen Ladd (pdf) evaluated teachers (2008) in the highest and lowest quintiles according to student assessments and found that most highly-effective teachers one year were not highly effective the next and many ineffective teachers were no longer ineffective the next.
2. An effective teacher is effective for all groups of students s/he teaches - If you are given a class that is significantly less prepared than the last one you taught, your performance may not improve and even a great teacher might fail to produce the desired gains.
3. Student gains and failures can be universally attached to the performance of a given teacher - Economist Jesse Rothstein surveyed data for 99,000 5th graders in NC and performed a statistical test asking "What effect do fifth grade teachers have on their students' 4th grade performance?"  Obviously, the effect should be 0, as they had yet to teach the kids.  Nevertheless, he tested three different "value added" measures and in all cases found that fifth grade teachers had an enormous impact on their students' test scores before they had even taught them for a day. There is obviously a flaw with the use of such value-added tests if such preposterous results are statistically significant. 
4. That the results of a good teacher can be added as the two studies suggest - Diane Ravitch notes in her The Death and Life of the Great American Education System that "nowhere was there a real-life demonstration in which a district had identified the top quintile of teacher, assigned low-performing students to their classes, and improved the test scores of low-performing students so dramatically in three, four, or five years that the black-white test score gap closed." 
5. Bridging the socio-economic gap is sufficient and means that our students are receiving a quality education:  plenty of white students from higher income backgrounds are doing terribly.  Bridging the achievement gap between rich and poor is a start, but certainly not the "destination" if a quality, world-class system is our goal. 

Now, the framework then continues with six key characteristic of a "good evaluation system":

1. All teachers should be evaluated at least annually;
2. Evaluations should be based on clear standards that prioritize student learning;
3. Multiple sources of data should be considered, especially those measuring student's academic growth;
4. Multiple rating levels to better differentiate teacher effectiveness;
5. Rating encourages regular, ongoing, and  constructive feedback; and,
6. Evaluation outcomes must have teeth, that is they should feed into teacher employment, bonuses, and pay.

I agree in general with the first five, though I see some constraints to the sixth characteristic.

All teachers should be evaluated annually

I think the more feedback a teacher can get from different evaluators during different types of lessons over the course of a year, the more useful a tool can be.  This seems like a great basis for an improved evaluation system that all teachers can use to improve.

Base evaluation on clear standards, emphasize student learning
Any evaluation, to be fair, should be based upon very clear standards with limited room for interpretation.  I agree also that they should be based on student learning, but I would urge caution in operationalizing the concept.  I think having impartial master teachers and principles observing or conducting a pop quiz to see if lesson plans are having an effect on a student's learning would get at this a lot better than using standardized tests.  Further, it would give teachers the freedom to teach a diversity of lessons that cover materials that are of extreme import but not necessarily on a standardized test.  The document does identify some opportunities like having a master teacher come in and note how many kids raise their hand or seem to "get" the material presented, though it does express a lot of support for the use of standardized tests.

Multiple sources of data should be used, focused on student growth measures
Diverse data--both in type and person evaluating--is critical to getting a more balanced assessment of a teacher's performance.  The focus on standardized tests is problematic as student performance on tests can vary and these tests may not test material that is all that worthwhile to know (or they may not test many subjects).  There is a further issue: a successful goal, according to the Harvard Business Review, is one that is concrete, that you can identify clearly when you have fulfilled it, and that is not dependent upon others.  Setting a goal for teachers that is dependent upon someone else (their students) is somewhat unfair.  Worse yet, these tests are not designed to test teacher performance.  They do not have the external validity to be misappropriated in this manner.  Create a standardized test for teachers, as that would at least have the validity necessary to make them an appropriate measure.  Additionally, if a test is administered mid-year, are the gains (or lack thereof) attributable to the current teacher or the previous ones? This is not clear. As I noted, I would prefer multiple observations and student and parent feedback.

Multiple Rating levels and on-going feedback
This is indeed preferable because it does improve upon evaluations to make them a tool for teacher encouragement and feedback rather than a narrow filter to remove only the worst teachers.  Further, if the ratings are meaningful and accompanied by concrete feedback, it gives teachers the actual tools to look at how they might improve and for the school to perhaps pair up that teacher with resources to help on their weaknesses. The more regular, the better.

Tie teacher ratings to their pay and employment 
All of the studies that I noted earlier should make us very cautious about this.  If teachers drop in and out of the highly-effective category (and the ineffective category) between years, then you need to be cautious about wantonly firing or punishing someone for doing poorly one year or rewarding someone who anomalously does brilliantly one year.  I think a more appropriate sixth metric would be to use student data and teacher performance data to try to determine what kinds of students a teacher teaches most effectively for future class assignments to try to set up student and teacher alike for success.  I recognize that this may not be realistic or might be logistically quite challenging, but it might be interesting to see what limits there are to this idea in practice.


In the end, my preference is for a rating system that looks a little different:

1. Monthly evaluations by an independent master teacher (15%)
2. Quarterly evaluations by school administrator/principals with experience in the classroom (15%)
3. Semester evaluations by external education evaluation experts (15%)
4. Round robin evaluations in which teachers evaluate their peers (10%)
5. Amount and quality of efforts made by the teacher to improve on areas identified in previous observations (15%)
6. Evaluations of student portfolios that look at growth on the subjects taught (10%)
7. Use of interviews to get randomized student and parent opinions of teacher (10%)
8. Performance on testing that can be attributed to that specific teacher, is included in that school's curriculum for the year, and that is value-added in nature (10%)

The weightings are my personal preference, but I think they reflect the importance of regular, directed feedback being used by a teacher for continuous improvement.  Ultimately consistently bad teachers should be counseled out, but those teachers who have a decent record of doing well should be retained and efforts should be made to match them with students with whom they will succeed (insomuch as is possible while still giving the most difficult to teach students access to quality teaching).

Speak out!

How would you evaluate teachers?

What are some interesting evaluation methodologies or criteria that you have seen/experienced?  

The Hands-On Classroom

My mom was a waitress when I was growing up.  I watched as her hard work, dedication, and skills were undervalued by customers, employers, and society at large.  When I picked up (does it count as picking up on a Kindle?) a copy of the book The Mind at Work by Mike Rose, I felt like it vindicated my mom and people who work as hard as she did with no recognition.  The book takes the reader through a host of blue collar professions (including waiting tables) and identifies all of the different skills and intelligences that these workers use on a daily basis.  It also showed some of the dangers of making the classroom too cerebral, as students lose the ability to execute a project fully, to work in groups, to engage with their environment on a concrete level, and to respect all people around them that contribute to the economy and society.  It also showed how our devaluing of the mental prowess behind some "menial" professions causes us to develop vocational training that is just as mind-numbingly bad as the fact-based education that "more successful" peers receive, stripping it even of the practical value it is supposed to have.

As I have argued before, there is a strong reason to have more practical, hands-on elements to all students' education.  For me, this includes a variety of things from languages to the arts to computing to vocational education.  As I noted, they improve test scores and higher order thinking (problem solving, creativity, etc.) and strengthen teamwork and community service.  In fact, my lack of a shop class left me restless for work to do with my hands, and I am really excited to begin wood-working classes this year to fill this gap in my education.  That said, I am excited by two different models that attempt to integrate the vocational with the "intellectual" and at the same time pull student learning outside of the school and into the community.

The first model (video) comes from Emily Pilloton, a designer who relocated to Bertie County, NC, with her husband and was inspired to add new skills and new life to her new community by teaching her craft.  Initially, Ms. Pilloton saw the community as a place ripe for her practice, lacking architects and designers.  She redesigned spaces within the school (like the computer lab) to make them more dynamic spaces in which the students interacted with one another rather than mindlessly receiving a brain dump.  Then she decided to share her talents by creating a unique, hands-on design course in which students learned how to identify a need in their community, practice developing the idea, design a prototype to see if it was viable, and then build something for their community. They were hired using private funds to carry out their projects.  The community became the classroom, kids were learning practical skills and earning money, and community members could see the concrete results of what the class was doing.  Their first project is reclaiming an abandoned property to create a farmers market.  The next two projects are a bus station and a care facility for the elderly.  It is nothing short of inspiring, and kids that are not as engaged by classroom learning have really taken to the program.

The second model (pdf) is called the Studio School Trust (html), a form of charters in Great Britain that takes the same concept (video) but makes it the entire educational experience for kids age 14 to 19.  Studio schools recognized that so many children don't learn well in an artificial setting and noticed a widening gap between the labor market and education system.   The model thus makes a school that operates in concert with local business, teaching children using projects.  The schools get kids knee-deep in their learning and teach them CREATE skills (Communication, Relating to people, Enterprise, Applied learning, Thinking, and Emotional intelligence).  The schools are committed to making children ready to work without sacrificing their ability to go on to university, should they so desire (something Ms. Pilloton also highlighted about skills in her classes). 

Many people may be quite surprised to see me advocating for a class and school system that are effectively charters, given my previous critique of charter schools.  I obviously argued very passionately that I think that applying charters on a systemic level is largely unproven to bear fruit and has the effect of segregating kids so that the worst performers and most challenged students are ghettoized.  I further find the whole concept uselessly duplicative: why create a parallel, supposedly-better school system when you could just fix the one that already exists?  That said, my problem largely comes from how charter advocates have forgotten why charters exist: charters were supposed to be monitored laboratories that generated unique models that could be applied in the public school system (not to create a private system that would destroy public and high-performing private education like Catholic schools).  I think these models have it right because they identify a specific set of the population that doesn't do well in traditional schools and offer a program that is tailored to their needs without giving up on their success.  Further, success in these models is well-rounded in a way that most vocational schools are not designed to be. 

To update my previous opinion: charters are not the problem, policy-makers that have mistaken the bottle for the Robitussin are the problem.  It's not the new bottle that makes you better, it is the different type of medicine that suits your particular cold that does.  Charters have enormous potential, and I LOVE how they bring the community and big tranches of private money into educating kids who really need our support.  That said, they are not systems, and they are not an alternative to reforming the schools that already exist.  They should, however, inform how we educate a diverse child population with its diverse educational needs.  I think these particular models highlight that the death of shop classes and truly vocational education (not remedial "you are going to fail at life" education) have pioneered something that can be scaled for some kids and that might improve our whole system by adding some hands-on components to more traditional curricula. 

As Marge Piercy says in her poem "To Be of Use," "The pitcher cries for water to carry
and a person for work that is real."  Our children aren't pitchers to be filled endlessly with facts, isn't it time we gave them work that is real?

It Ain't Easy Being Upwardly Mobile and Poor

As you may recall, I grew up poor and ended up going to college.  Surely, that’s success, right?  My investment in education and motivation behind this blog stem from the belief that by writing about these issues we can craft a better system that ensures that more people go from poor to more.  I have to say, though, that I read a rather disappointing article in Jezebel that highlights a Cornell study.  The study shows  that poor people who go to college are less likely to marry than rich people who go to college or poor people who don't get advanced degrees.  Especially guys (oh boy...is it stuffy in here or is that just my college degree laughing at me?).  This post will be less study-based and a bit more of a personal reflection on some of the largely unconsidered barriers to poor people succeeding.  It is no small task providing the ideal system that promotes equitable advancement through education, but even if you did that, it is no guarantee that people will (or perhaps should) jump to take advantage of it. 

There are a lot of impediments to a poor person that make you really give pause as you move up the educational ladder.  One of the big ones is that a lot of the information you learn has no relation to your background.  As my European History teacher said, “this class is the story of DWG” (dead white guys).  One of my favorite poems "Prière d'un petit enfant nègre" (Prayer of a little black boy) from Haiti provides the thoughts of a young Haitian boy who laments going to the French schools that talk about history from places he will never know and that take him from walking under the mangroves with his dad.  If you are not wealthy and white, it is really hard to connect.  It’s not that you can’t relate to or learn from people that are unlike you, but when everyone you learn about is remarkably similar to each other but completely different from you…well, it sends a signal.  It says that history doesn’t remember your kind.  School reinforcing your smallness while you are trying to escape it is not the greatest encouragement that you are going to go far.

Even if you, as I and many others have, get past your historical absence from textbooks, you then have to deal with a feeling of extreme guilt.  Every time you mess up, it is wasting all the sacrifice of your family.  When I went to Stanford, I was suddenly in an upper-middle class environment that I wasn’t paying for.  Even as I lived relatively frugally, I’d get to go on ridiculous things like ski trips and wine tasting that I wasn’t paying for.  It is hard to not feel guilty when you are skiing in Tahoe (and trying not to mow down defter six year olds on the slopes) while your single mom is heading to work at the crack of dawn to make ends meet.  Guilt is a powerful impediment to enjoying your success and to even viewing it as success.

There is also the intense separation of points of views that creates big tension between your past and future relationships.  As you get more educated, people start to resent you for being condescending or “thinking you are better.”  It is funny because they encourage you to get an education, but once you try to encourage them, it sounds like meddling or judgment.  To have friends or relatives think you are some out-of-touch, paternalistic know-it-all is a strong disincentive.  In minority communities, this can manifest itself as “being too white.” Meanwhile, you start to resent them for their stubbornness, for refusing to acknowledge that studies have shown that “x” will make their lives so much better.  In this, you and the person who once encouraged you both start to question the wisdom behind upward mobility.

The separation is not just the tension of your differing educational levels, but also of your changing interests.  Typically, you are educated outside of your community if you are an upwardly-mobile poor person (my high school was 12 miles from home in Chicago, my grammar school about 6).  My education got me excited about languages and foreign countries, and conscientious travel is not all that big where I come from.  I really care about buying local.  My mom and I have constant spats about it because she’s very right when she says, “It’s nice that you can afford to buy at small places, but I need to go to Target.”  Likewise, I’m right when I say, “Yes, but places like Walmart are shown to kill jobs in Chicago and you are hurting people just like you.”  It’s an awful catch-22, and it’s an impediment.  What the hell do you talk about at Thanksgiving that won’t provoke a fight?  I thought that was the job of a drunk uncle, anyway.

It is also really hard when people you knew can’t understand what you are doing with your life.  You are an “international something or other” major who wrote a very long paper on “something to do with Arabs.” “Yes, I did write my honors thesis in International Relations on Iraqi refugees.”  “Right, and what is this diplomat thing again? I know you’ve told me, but I can’t remember.”  Then try explaining to people “I work on promoting the labor provisions of free trade agreements.” (blank stares abound).  The thing is that people suspect you are successful, but all the people you really want to impress can’t understand well enough to be fully proud because they get what you’ve accomplished.  It's even worse because they expect you to be knowledgeable about EVERYTHING, and you suddenly get questions like "The doctor says I should get a biopsy, but I don't know...what do you think?"   Try responding, "Ask me about the Democratic Peace Theory, not medicine."

What’s worse is that in the end, you are changed but you don’t fit into your new life so well either.  Your middle class friends talk uninformedly about how to help the poor—your people, what YOU yourself once were—or what poor people need.  People laugh at you when you say that you were on welfare as a kid and were attacked by gangs on the way to school because “what? You went to Stanford, give me a break!”  Worst of all is that gnawing feeling that you’ve changed.  You know you can’t go back, but you can’t fully buy into your new class.  You will buy expensive coffee and think with disgust, “damn, when did I become such a bougie tool?” And when you don’t fit in, that means it is harder to find someone to relate to where you are coming from or where you are heading to, and that is why the Jezebel article notes the upwardly-mobile poor have such a hard time marrying.  

So, to be poor and then successful, you need to overcome leaving your neighborhood, inviting a widening gap between you and everyone you knew, learning new things that you can’t use to help those who once encouraged you, feeling perpetually out of place, and constantly questioning yourself.  Put that way, it doesn’t feel like success.  Plus, if it is success, then why do those who encouraged you have such a hard time with so much of it?   If you are just changing to fit society’s model of success, are you not just a pawn like all the other poor people who feel they lack any control in their government or society?  

These questions don’t get raised in policy circles because poor people rarely get that entry and because these are frankly hard emotions to just put out there.  Class makes people very uncomfortable, so they also don’t often want to hear these kinds of things.  Worse yet, because you escaped and so many others did not, you may not be the best voice for those other poor…maybe you too are different?  

So the next time you offer a voucher to a parent who doesn’t take it, the next time you hear about high drop out rates in charter schools with great results, kids who reject scholarships to stay close to home or not go to college…maybe you’ll understand why.  Maybe you will also get why the battle to improve our education system is not just logistical, political, and systemic…it’s also psychological.