THE BOOK--Playing The Percentages In Baseball

To properly do regression, don’t we need to know the year to year variance for each student? Someone might score 60 the first time around, but we won’t know until they’ve taken multiple tests how much natural variance there is, and thus how much regression is needed. As best I recall, there was very little variance on my own standardized test scores, like 95 +/- 4.

Right, we need to do a year-to-year correlation.

Excellent.  This is a topic that interests me tremendously, especially since there are (literally) hundreds of millions of dollars set to flow to school systems nationwide base in large part on how states and localities implement teacher evaluation methods.  Needless to say, lots of what is being sold by consultants and information management companies isn’t worth nearly the price sought, but too many decision makers (principals and superintendents) don’t have the understanding of statistical methods required to evaluate any of the claims.

Sanders is a statistician by trade, and employs a mixed model. Here’s a couple of descriptions of the system Sanders helped develop (the second one also addresses some of the weaknesses): http://www.sas.com/govedu/edu/mixed_model.pdf and http://web.missouri.edu/~podgurskym/Econ_4345/syl_articles/ballou_sanders_value_added_JEBS.pdf .

That said, I personally question how any sort of measurement system can filter out all the noise of a student’s environment. Certainly, everyone wants good teachers, but the quality of my child’s teacher has far less to do with her success in school than the totality of the rest of her life.

Of course, I prefer a value-added approach over a “what percentage of your students meets a minimum requirement” approach. Value-added rewards teachers who challenge ALL their students. Because my daughter’s last school’s emphasis was strictly on making sure all kids met the minimum standards, smart kids like her and her friends were basically ignored because the teachers knew they’d pass the tests.

I think we do need to have a discussion about how to spend our limited resources on student achievement. Not to demean manual labor, but should we spend a disproportionate amount of our education budget on future ditchdiggers or on future doctors?

Tango - I would think that any tendency to regress to the mean would be overwhelmed by selective sampling issues that would tend to push the scores in the other direction.  If you are above average as a six year old on a standardized test the most likely reason is that you come from a home with economic and sociological advantages that are going to continue as you get older.  And the older you get, the more you are going be able to make use of those advantages to learn on your own.  This should increase the variation from the mean for both low advantaged and high advantaged children even without teacher intervention.

A second consideration is how much the testing is measuring “true talent”, i.e. innate intelligence, and how much it is measuring acquired knowledge.  Innate intelligence should not change much from year to year and should be little affected by teachers.  Acquired knowledge is the area where teachers and teaching methods can make a difference.  Designing standardized tests that actually measure what you are trying to measure without bias is an extremely difficult business.

Peter, even though innate intelligence might not change much (and you’re right about that… in my psych testing classes, the rule of thumb we were given was that IQ is pretty stable after one’s 6th birthday), there’s still multiple-choice questions on these exams.  So if you have a bunch of true talent 95th percentile kids, some are going to get a few guesses right and some will get a few guesses wrong, so you’ll end up seeing them score 93 to 97.

I like your point about the socioeconomic advantages allowing the spread in talent to increase as time goes on.  Though, I will counter it by saying, there is also typically some social pressure to regress to the mean as well.

There is enormous regression to the mean in these metrics, and in many cases these value-added models (VAMs) do not correct for it.  First, the teachers regress:  their value-added scores in one year are only somewhat predictive of future year’s scores.  There is some signal there, but it takes several years of data to have any idea what that teacher’s “true talent” is, and even then there’s a lot of noise.

But even more troubling—and a factor which has received almost no attention—is the KIDS’ reversion to the mean.  According to multiple studies, they lose 50-75% of the “gain” within one year, and almost all of it after two years. This means that teachers who have a lot of kids who posted big gains the prior year (by accident or design) will tend to have very poor value-added scores—and very few of these models factor in students’ prior-year gain (only their prior year score). But the student regression raises a much bigger question:  what teacher talent, if any, is actually being measured, if it fades out within two years?  This strongly suggests these high value-added teachers are just doing a better job of test prep, rather than truly improving students’ skills and/or knowledge base.

Interesting stuff - I haven’t thought about this in years.  The College Board claims that there’s basically zero variance in your performance on its standardized tests, so no regression to the mean is required.

Basically, the College Board in that case is claiming that if Felix throws 20 FB today and he throws 20 FB in five days, that his FB speed will remain unchanged.

I believe that for FB speed, once you hit a certain “n”.

I believe that for test scores, once you hit a certain “n”. 

But, there’s always regression.  For FB speed, maybe you need to add 1 pitches of league average speed, so that there’s only 5% regression on FB speed.

For test scores?  Well, that’s the question.  It would depend on the number of questions on the table.  The longer the test, the less regression needed.

A few days ago I got in an argument with my department chair over statistical evaluation.  Our English department collects one essay from every student each semester, the essay being up to the instructor’s discretion.  The chair presented these data to us: in fall 2008, 34% of all freshmen were deemed proficient in composition, and in the spring, just 32%.  In fall 2009, 32%, and in the spring 2010, 28%.  She used this as a way of saying that the program was failing.  Aside from the data’s obvious SSS, she hadn’t controlled for anything - e.g. the average SAT score had declined from 2009 to 2010 - and she hadn’t even considered regression. 

When I objected, she informed us that we were collecting data ONLY to try and convince the administration that the English department’s spring program worked.  Bureaucratic details unimportant.  We need to show that scores improve from the fall to the spring, and since they’re dropping - which is to be expected, due to the spring’s increased difficulty - the administration won’t be impressed. One solution, I suggested, was to engineer the results.  In the fall, do not tell the instructors what assignment to collect for the department.  The rubric for “scoring” these essays favors research and argument; and if your data has a lot of narrative, creative essays, it will naturally score poorly.  You cannot get a 4 out of 4 on the “thesis” component if the short story you’ve collected was not even supposed to HAVE a thesis.

Then in the spring, have the instructors collect a standard research-based assignment, a cookie-cutter argumentative research paper.  Now the scores should skyrocket, and you have your “proof” that the spring program works, and hopefully your money.

A deeper question is if the standardized tests are even measuring the right stuff.  If you have a flawed measurement technique then all of this will just reinforce that and amplify the problem.

@7/9: These tests are *designed* to have incredibly high “reliability”, i.e. high self correlations, correlations to other tests, and to be as much of a measure of ‘true talent’ as possible. Unlike baseball, which serves the purpose of being a game with some indeterminacy as to outcome, the purpose of these tests is to measure.
I don’t know which tests in particular are being talked about, but a test like the CELF has corrected correlations of .9, and as high as .98 for some subtests.

So, as per 8, little-to-no regression required and certainly nothing that would compare to all of these other effects in terms of effect size.

Peter/5 wrote: “And the older you get, the more you are going be able to make use of those advantages to learn on your own.  This should increase the variation from the mean for both low advantaged and high advantaged children even without teacher intervention.”

Indeed. Being somewhat familiar with issues in education, and being married to a PhD candidate who studies education, I can confidently say that this is true - the strong (and the middle and upper class) students and the weak (and the working and lower class) students tend to move further from the mean as they age, and the number of people occupying the space in the middle shrinks.

Another confounding issue via marginalrevolution: http://www.marginalrevolution.com/marginalrevolution/2010/06/does-professor-quality-matter.html

Students taught by inexperienced/less qualified teachers tend to perform better on tests that year, but relatively worse than those taught by experienced/more qualified teachers in later years in more advanced classes. The (unsubstantiated) explanation that jumps to mind is that inexperienced teachers teach to the test while experienced teachers provide a deeper understanding.

When I read this article I thought of this blog and regression to the mean.

In my personal experience (obviously ultra SSS), the College Board’s standardized scores do appear to have a fair amount of variation.  After taking the SAT the first time I felt I underperformed my (subjective assessment of my) own true talent, and indeed when I retook it (without really extra studying in between) I scored 180 points higher.  There *is* random variation involved in multiple choice tests.

I’m completely speculating at this point, but it seems to me that, for a fixed student, the types of questions on tests can be divided into questions a student can solve and those the student cannot (at the time).  People like to assume that those the student can solve are answered correctly and vice versa.  I speculate that more studying would lead to a tighter distribution around true talent.  I imagine students who studied less would miss a larger portion of “solvable” questions and thus have a larger amount of variability.  Of course I’m completely making this up based upon my sample size of 1 person, but it makes sense to me.  Repetition probably doesn’t help the students learn much, but it probably does make it less likely that they will miss a solvable question.

Averages of percentiles would not necessarily regress to the mean in the good old predictable way.

Let’s pretend that there’s a normal distribution of scores and that the best model predicts a student’s score to regress 50% to the mean.  In a certain class let’s say you have 3 students who are in the 98

(post getting cut off)

In a certain class let’s say you have 3 students who are in the 98, 26 and 26 percentiles.  The class “average” is 50% so you might not expect any regression to the mean but the student’s scores (in terms of stdev’s from the mean) are +2.05, -0.64 and -0.64 and we should therefore project (regressing 50% to the mean) +1.03, -.32 and -.32 giving us percentiles of 85, 37 and 37 for a projected class average of 53 percentile.  The class average percentile regressed away from the mean in this case.

Anyway, more to the point, I worry a little bit about what teacher evaluation based on WOWY would do to incentives. 

I teach one of two sections of AP Chemistry at my school.  The other teacher and I are both competitive in a friendly way and, of course, we compute our class averages after we get the AP scores and see how we did.  While I suspect that she’s a better teacher than I am (she’s been doing it for much longer and has better insight into each individual kid’s stumbling blocks) the difference isn’t big enough to emerge through the noise in the data over two or three years and over that time frame teaching prowess is likely to change. 

Perhaps more importantly, while we’re competitive, it’s not to the point where we don’t help each other (and each other’s students) out whenever we can with lessons and labs and homework.  We also aren’t nearly sick enough to manipulate who is in which class (though this would be easy to do) or coerce students into spending gobs of time perfecting chemistry minutia to the detriment of their other pursuits.  What would happen if our pay or even our jobs depended on these scores though?  How would it change how I view the students and my fellow teachers?  Frankly, just knowing that I’ll see the scores as will my fellow teachers and department head provides all the *extra* incentive I need (you know, beyond wanting to educate children).  And, btw, this is in the class (of those I teach) where the curriculum is the most standardized and is least likely to be perverted by a drive for higher test scores.  I’m also pretty fortunate that the AP in Chemistry is a thoughtful test as they go. 

I do, however, like the idea of teachers getting feedback on how their students are progressing in a variety of areas relative to the students of other teachers.  Feedback is always good and the more specific the better.  Also, posting these scores where all other teachers in the school can see them might provide extra incentive while not being over the top.

Oh, I should add, I don’t think something like WOWY is needed for a school to know who the crappy teachers are.  Everyone already knows who the crappy teachers are.  I think the idea is just that it would be easier to fire crappy teachers when you have objective evidence to back up what you already know.

You know, that’s a good point.  If in baseball everyone has 600 PA, then the wOBA rankings will remain the same, regressed or not.  So, percentile rankings should remain unchanged.

Greatest thread ever. Reminds me that any science in which the term “science” is preceded by an adjective is not a science.

Dick/19:
Wow, that was a real douche-y thing to say (sentence 2). Your douchiness might be excused if you weren’t also wrong.
First, why don’t you pick up the magazine _Science_, published by the American Academy for the Advancement of Science, and peruse the articles; the first one listed in the current issue is on Sociology. Or look at Proceeding of the National Academy of Science (articles on archaeology and meditation this week). Second, people that practice what I imagine you’re calling just “science” often refer to what they work on as “physical (adjective) science” and “biological (adjective) science”. Third, things like brain research are often published in places like Psychological (adj) Science.
Finally, you’re besmirching the work of the people on this site, which most closely resembles the work of social (adj) scientists and not, say, chemists (e.g., baseball analysis generally involves posthoc controls, unlike planned experiments).
I’m curious what you even mean by “real science”.
On second though, I don’t think I am.

to be fair to #19, Dick Feynman *would* probably say something like that.

A related saying is that putting “social” in front of anything reverses its meaning.  “Social justice” is unjust.  “Social security” is insecure.  “Social science” is unscientific.  And so on.

The snarky sayings are the best, because, even though they’re not 100% true, they really piss off their targets. 

As a former teacher and current principal, I’ll add 2 others:

[1] How seroius the student takes the test, and [2] how hard they try on it.

If non-educators could observe some of the students taking the test, they would not value the results as highly.

I’m all for accountability and assessment. I just prefer a growth model.

Our district is lower income (50-55% low income), but the big thing is the 20-30% mobility. It’s no exaggeration to say that our test scores are mostly affected by who moves in or out.

-----------------------

I agree with another who said that many administrators do not understand statistics. Looking at our district’s reading data, I was asking question like “How many students are represented by 1%” (as we were looking at how reading scores in one area went up 2%, and we were just discussing 2nd grade), and like “Who are these students?” .... again, our scores could go up/down 3-10% and it could be due to simple mobility (students moving in/out) and not reflect on the instruction at all.

The biggest thing I always preachis to make sure that your data is telling you what you want to discover, not just telling you “something”. Confidence levels in education data is simply ignored. %’s either went up or down. That’s it.

*Shrugs*

I think we have a done a good job hear of critiquing the scientific method, and illustrating how damn hard it is to measure these types of things.

That said, I would like to suggest that instead of WOWY, a better solution from Tango’s tool bag is “Wisdom of the Crowds”.

Instead of the experts looking down and trying to determine the winners and losers, instead have the parents look up and decide where the kid will go. I don’t care why they make their decision, just create an environment where schools have to compete to gain the business of the customer, not the favor of the bureaucrat.

I realize this is not the miracle solution to everything, but I do believe it is a major part.

1. Parents and teachers must convince the children that learning is essential. Work hard now and learn, work hard later as an adult and get ahead. I’ve seen Korean families who drill into their children that the only way to beat the system is be smarter and work harder than everyone else. It helps if it’s fun, but that’s not essential. We try to find jobs we love, but it’s still not always fun.

2. Teachers and schools need an economic incentive, like the rest of us, to be the best we can be. Many days I get bored at work, but it’s the fear of the wrath of my boss if he discovers my lack of production that pushes me forward. I don;t want to get yelled at. I want a raise. I don’t want to be laid off (half my department was last year, and thy were the worst half in productivity). Part of this is what J. Cross said above, it has to be easier to fire bad teachers. It’s addition by subtraction, but also that the fear of losing one’s job may motivate some to get their act together.

3. And a good portion of that economic incentive comes from the opportunity of parents to decide which are the best schools. Instead of the folks in the state capitol or Washington deciding which schools get funded to which extent, give that power to the parents. I know there are practical limits to where parents can send their children, and situations vary geographically, but where I grew up I lived three miles from my public high school. Two other public high schools and a Catholic school were closer, but without paying property taxes and then tuition on top of that, I would be excluded from attending even the other public schools. Today (living less than a mile away) we do send our grandson to a Christian school a fifteen minute drive away.

Brian/24: I think that’s more of the “it takes a village” approach than “wisdom of the crowds”.

I more or less agree with Brian.  I certainly like the idea of teachers and schools negotiating salaries individually.  The collective bargaining between cities and unions can’t really be optimal for either teachers or schools.  I also, at least in theory, like the idea of parents being able to move their kid to the school of their choice although I think presents several logistical problems that I can’t see offhand how to solve.

Certainly it should be easier to fire lousy teachers, however, one of the draws of teaching is the job security.  Teachers are cautious people, I think.  There’s a reason they didn’t go into investment banking or whatever else.  Those big paychecks come with a great deal of uncertainty.

The one thing I disagree with is that fun isn’t essential and that we should get kids to buy in based on long-term gain.  Kids are generally short-term thinkers.  You can’t blame them for this or scare them out of it, it’s just part of being a kid.  They do enjoy learning new things, however, to an extent that I think is vastly underestimated.  We should insist that it be fun and try, where possible, to make it immediately relevant. 

There doesn’t seem like there’s much of a push to do the things that we know would make education better.  You don’t hear a lot of noise about making the school year several weeks longer even though this would almost certainly be better.  And no one really thinks that high school age students learn best starting at 8 in the morning and yet this doesn’t change.  I think there’s good reason to believe that exercise and healthy lunches matter but we don’t hear much about that either.

In general, I think a lot of the problems stem from the fact that the concern exhibited for schooling is mainly just political lip service. We like to think education is a priority, but it is so obviously not given the decisions we make as a country. J Cross is wrong in a sense - it’s not kids that are unable to factor in long term consequences - it’s Americans in general. Education is hard and the arc of its ROI is long.

Compare prison expenditure changes the last 20 years to school expenditures. Witness the drastic defunding of the best public higher education in the world - the University of California - so that people can keep their property taxes a little bit lower. Compare changes in technology, productivity, and research in banking and marketing the last 30 years to schooling. It goes on and on; short term gains, long term losses.

It’s cliched and doesn’t acknowledge the complexities of the issue, but the famous bumper sticker is still basically right:
“It will be a great day when our schools get all the money they need and the air force has to hold a bake sale to buy a bomber.”

By the way, a solution is to give teachers equity in future earnings of their students. There are a million problems to be ironed out, but witness the geniuses and brain power that would flock to the field if 1% of your earning went to pay for what made you what you are.

mettle/28: In that case, what impetus would there be for teachers to teach anything in the arts or humanities, where even the most high-profile jobs tend to be relatively low-paying? It’d be far better (easier?) for the teacher to try and create a bunch of accountants.

@28,

Where I live, we already have teachers abandoning the higher needs areas for the towns filled with yuppies. Your idea would only make that worse.

As for those talking about introducing competition to give teachers economic incentives, have you looked at teacher salaries lately? People definitely aren’t doing it for the money.

I agree with mcsnide: there are already many disincentives to teaching.  Teachers are at the mercy of parents, who will support their kids no matter what, and they are at the mercy of the kids, who know they can get their parents to sue their teachers (or make life miserable for them).

Teachers SHOULD be considered an extended guardian, and therefore partner, of parents.  That’s not the way it works.

I have no doubt that if we look at all professions, the % of people leaving the teaching industry within 5 years of starting it will be far above the average for other professions.

In order to improve teachers, you need to increase the pool of potential teachers.  Imagine the incentive to play baseball is not there, so you have Pujols being a tight end, and Longoria being a DB, and Glavine being a defenseman.  And then you say “we have to make the baseball standards tougher for baseball players to make sure the best players make it to the pros”.  Well, that’s all fine and well, but if Pujols, Longoria, and Glavine are not part of the pool to begin with, you won’t be going anywhere fast.

I find this whole discussion (and I live in So Cal and take the LA Times which started all this, at least recently) a take off on “The Emperor Has No Clothes”.  We spend so much time on teachers as if one can be a magician relative to another.  We hear about a teacher who has had some seemingly remarkable success (but it’s likely a small sample problem).  I teach kids.  Here’s a likely list of the things that cause a kid to do well (or do poorly):
1.  The student’s aptitude (I’ve spent an entire hour teaching one kid how to solve 3x -4 = 8.  At the end of an hour they can do it.  Tomorrow I have to start over.  And the next day, etc.) If the student can’t retain, he won’t do well.  I’m guessing aptitude represents at least 60% of the problem.
2.  The student’s attitude.  Those who are willing to work hard and to think for themselves can improve if they can retain some of the work.  I think this represents about 20%.
3.  Parents.  Parents who consistently work with their kids can help them--maybe 10% out of the 100.
4.  Teachers.  Well, we’re already at 90%, so what’s left for teachers and all the money that we spend for useless things like computerized white boards????  Hmmmm
I understand that we all like to think that our kids would do better if they had better teachers, and maybe to some degree it’s true, but it’s small potatoes in the big picture.
So we spend countless hours and dollars trying to evaluate teachers (which as noted above is incredibly difficult to do) instead of working with the kids and providing them with real skills that they and society can use.

28-30
Yes, there are lots of details to be ironed out. For example, one simple way to address #30 is to calc a child’s expected earnings based on current SES, race, zip code, etc and teachers could get a bonus when kids exceed that. And yes, something needs to be done about humanities. There are lots of specific problems with this idea in it’s simplest form but if enough people think through it, I think it could all be ironed out to meet our goals.

I stand by the basic idea, though, which is that we find innovation and talent when it’s rewarded. Right now, only people with a powerful personal pull toward teaching do it (and some even do not) because of the economic disincentive. That’s not currently drawing enough talent into the field, unlike something like banking.

So, I believe the basic idea I’m presenting solves what I think are the 3 basic problems: funding, incentivizing success and reminding people where they came from and what made them what they are.