Friday, May 22, 2009

SAT Coaching Boosts Scores -- Barely

The SAT has a fascinating history as a test. Started as more of an "aptitude" assessment designed to find highly intelligent children who hadn't had top-notch academic preparation, it later turned into an "achievement" test designed to cover high school coursework. It was "re-centered" in the mid-1990s to push declining average scores back toward 1000 on a 1600 point scale. Then, four years ago, the College Board added a writing section, putting the potential point total at 2400.

In all its versions, though, here is one thing we do know about the SAT. It is one of the few academic measures that means something nationally. High schools vary vastly in their rigor; it is quite possible for two children at two different high schools to take math classes with the same names, both get As, but have one score 200 points higher on the math section of the SAT. In that sense, it is harder to manipulate.

So it's no surprise that a cottage industry of folks who don't like the fact that the SAT can expose bad schooling has sprung up to claim that the SAT can be rigged, too. The chief criticism is that scores are meaningless because they can be coached. Rich parents can afford to pay thousands of dollars for Kaplan courses or tutoring, and hence buy their children admission to top colleges. The commercial coaching companies give ammunition to this idea by advertising massive score increases. According to an article in Wednesday's Wall Street Journal, Elite Educational Institute says its average score improvement is 240 points. Ivy West guarantees a score increase of 200 points. Revolution Prep promises 200 points, Summit Educational Group says their proven increases are 180-400 points, and Sylvan pledges 160 points.

I've always been skeptical of these claims because if it were so easy and straightforward to raise your score by 400 points just by paying a fee, why don't you see more 2400s than you do? Why doesn't every kid who's been coached get a 2400? It's not that I don't believe coaching helps -- I paid for voice lessons for 2 years, for instance, and saw an improvement in my singing -- but I believe that any increase is coming, largely, from the fact that the kid is learning material that's on the SAT and that she didn't know before, or becomes familiar with the test. You could learn the material in school, too, but for a variety of reasons, that doesn't always happen as it should. There are also a variety of old SATs available from the College Board or even the library, so one can gain test familiarity without paying a cent.

Now, a new report from the National Association for College Admission Counseling released this week says I'm on to something. The average gains from commercial coaching were more in the neighborhood of 30 points on the SAT (the old version, which was out of 1600 points), not the 100 or 200 points claimed by many coaching companies. NACAC accuses some colleges of fueling the industry by setting cut-off scores that give 30 points a significance they shouldn't have, but as William Fitzsimmons, Harvard's admissions dean, told the Wall Street Journal, "It breaks my heart to see families who can't afford it spending money they desperately need on test prep when no evidence would indicate that this is money well-spent."

So what's behind the massive score increases claimed by the commercial coaching companies? Apparently a high school senior named Jonah Varon from San Francisco came up with a theory. As the Wall Street Journal noted, he took a mock SAT from a company called Revolution Prep, and scored a 2060. Then, without doing any of the company's coaching, he took the real SAT a few weeks later, and scored a perfect 2400 -- more than 300 points higher. He was suspicious. So he had several of his classmates do the same thing. Almost all of them scored hundreds of points higher on the real SAT than the mock SAT, with no coaching whatsoever. He wrote an article for his school paper claiming that the mock SATs were either harder or scored more harshly than the real SAT in order to inflate gains and make it look like the coaching created a miracle. It all makes perfect sense.


Anonymous said...

I've always wondered how they could make such claims. I'm certain they don't refund your money if it doesn't work. My son took the SAT only once in the fall of his junior year. He scored 2140 which was solid enough for anywhere he wanted to go and good enough to get him a full-ride scholarship. Spending thousands of dollars wouldn't have made a big difference I'm sure.

Crimson Wife said...

I think that test prep can result in legitimate gains of a non-trivial amount. I went up 100 points between the first and second time I took the PSAT, and went up another 100 points by the time I took the SAT. And that was starting from a fairly high level (mid 1200's).

Kevin said...

@Crimson Wife: repeating the test does usually result in some gains. It isn't clear that test prep has much additional effect, based on the data collected by the College Board.

atxteacher said...

I love the resourcesfulness of gifted kids. Great job, Jonah!