This blog is still alive, just in semi-hibernation.
When I want to write something longer than a tweet about something other than math or sci-fi, here is where I'll write it.
Wednesday, September 30, 2009
Wednesday Math, Vol. 90: Are z-scores obsolete?
Statistics is a calculation intensive field and it has been for well over one hundred years. Some of the calculations are nightmarishly difficult to do, especially the integral calculus stuff. For that reason, one universally used method found in statistics books, both modern and historical, is the look-up table. Instead of asking students to find the area under the normal curve when it is cut by a vertical line at some pre-determined position defined by a z-score, the student only needs to find that z-score, round it to the nearest hundredth and look up the value on a table, so long as the z-score lies between -3.5 and +3.5. This is not a major inconvenience, as problems that produce z-scores higher or lower than those boundaries are very rare indeed.
The z-score is a middle step needed to find information. The idea is that you have data taken from a set where the average and the standard deviation is known. For example, let's say you get an 82 on a 100 point test. Is that good or bad? Some grading systems would say that score is a B, or possibly a B-. The idea of "grading on the curve" is to see if 82 is lower or higher than the average.
Example #1: If the average is 74 and the standard deviation is 5, then the z-score for 82 is (82-74)/5 = 8/5 = 1.6. This says that a score of 82 is 1.6 standard deviations above average. If this were a normally distributed set, that would correspond to a score that is better than about 95% of all scores. Unless the teacher using the "grading on the curve" method is super strict, this is probably worth an A.
Example #2: If the average is 84 and the standard deviation is 4, then the z-score for 82 is (82-84)/4 = -2/4 = -0.5. This says that a score of 82 is 0.5 standard deviations below average. If this were a normally distributed set, that would correspond to a score that is better than about 31% of all scores. This time, the 82 isn't worth a B or an A, probably more like a C or even a D.
Normal distribution as a concept is not dead. Nassem Nicholas Taleb, author of The Black Swan, hates the normal distribution, but his is a mind without nuance. Normal distribution should definitely not be used in all cases, but in its correct venue it is invaluable. What could be going away is the z-score.
The z-score is a middle step. We need the raw score, the average and the standard deviation to find out the percentile. Going the opposite direction, if we have a percentile, the average and the standard deviation and we want to know the raw score that corresponds to it, there are methods for that as well.
If you use Excel, the formula to use to find a percentage given a raw score is =normdist(score, average, std. dev., 1). The "1" at the end is for a Boolean variable dealing with whether you want the information cumulative or not. No middle step of calculating the z-score is involved, the computer does it all for you. Likewise, if you want to find the raw score that corresponds to a given percentile, the formula is =norminv(percent, average, std. dev.). The most popular high end Texas Instrument calculators like the TI-83 and TI-84 can do these calculations as well, without ever resorting to the student finding the z-score. Besides being easier, the answers are more precise, because any table look-up system is prone to rounding errors.
This presents a conundrum. The z-score is still an important idea, it's just not a necessary tool anymore to get from one set of information to a desired new piece of information. A lot of math teachers over the past few decades railed against calculators, that the students were incapable of doing anything by hand. The teachers have a point, but they may be fighting a battle that is far behind the times. Calculators themselves may be getting out of date now that computers are so ubiquitous. It's a little like someone arguing that we should return to vinyl records and stop using compact discs when a growing number of young people look on CDs as an out of date technology, preferring the easier MP3 standard to move their music from a computer to a player or back.
To the outside world, math may seem like an unchanging edifice, handed down to us by great minds now long dead. But math at the forefront, seen by only a handful of people around the world, is most definitely still changing. More to the point of this post, math education, which effects nearly everyone at some time in their lives, is in a constant state of flux, and some advances which are clearly progress still come with a cost.
Yay, Flags Of Many Lands™! Yay, New Caledonia!
Like French Guiana and Martinique, New Caledonia is still a outpost of French colonialism, but at least the New Caledonians get their own flag.
Before this, the only thing I knew about New Caledonia was that it was an important destination for the getting of cool stuff on the TV show McHale's Navy.
Bienvenue, mes amis!