Tuesday, June 26, 2007

LIES, DAMNED LIES, AND STATISTICS (Part 2)

All about the abuse of anecdotal math to falsify the truth and truthify falsehood.

This is the second part of my "presentation" on the topic of "Lies, ..."

Click here for the first part: http://tvpclub.blogspot.com/2007/06/lies-damned-lies-and-statistics-part-1.html


In this part, we will explore "truth and consequences" and "playing the percentages."

EXAMPLE #1: What did I say?

This year is 2007. If I told you I was born in the year 2000, how old am I?




Well, 2007 - 2000 = 7, so, I guess I am seven years old. Right?

All right, not necessarily. This is the month of June 2007, so, if a person was born before June 2000, he or she would be 7, but, if after June, he or she would only be 6. So the answer is six or seven years old. Right?

>>>>>>>>>>>>>>>>>>>

Sorry, nope! No matter what I told you, I am the age that I am. That happens to be 68 years old. I am sixty-eight years old!

LESSON: No matter what someone may say, that does not change the truth of the matter.

EXAMPLE #2: Let us move on to playing the percentages.

From now on in this posting, let us assume (for the sake of this discussion) anything in purple bold italic type font is a literal truth.

Poupon University has five academic departments. Four out of the five have more male than female professors. Thus, 80% of the departments at Poupon U. are male-dominated.

The percentages of Male/Female are as follows:

  • Engineering: 90% male / 10% female.
  • Physics: 80% male / 20% female.
  • Philosophy: 65% male / 35% female.
  • Foreign Languages: 60% male / 40% female.
  • Humanities: 25% male / 75% female.

If you average the percentages of the five departments, you get 65% male / 35% female.

That would seem absolute proof that old PU is discriminating against female professors! Right?



Not necessarily! Here are the numbers for each department, and the sum of the numbers for Poupon U. as a whole:



Note that there are EXACTLY 200 female professors and 200 male professors at old PU! The genders are EXACTLY 50/50!

What happened to the discrimination? (The same thing that happens to your fist when you shake hands!)

THE LESSON: Beware of percentages, particularly when they are averaged.


EXAMPLE #3: Percentage Increase and Percentage Decrease

In 1980, gas in the US cost about $1.50 per gallon and now it is up to about $3.00 per gallon, which is a 100% increase.

If gas prices should drop back from $3.00 to $1.50 per gallon (I'm not predicting that, just suggesting it for the purposes of illustration), that would be a 50% decrease.

What is going on? When prices go up by $1.50 we get twice the percentage increase as when they go down by the exact same amount!


If we look at the historical record, US gas prices peaked in 1980 when they were about $1.50 per gallon. Considering inflation from 1980 to 2007, that is about $3.00 per gallon in constant dollars! If you calculate the gas price as the number of minutes the average US worker must devote to earn the price of a gallon of gas, the current price is less than the historical peak in the 1980's!

The same percentage increase and percentage decrease confusion holds for unemployment, crime rates and all things that are bad.

As an example, in the Orlando Sentinel for June 26th 2007, there is a report on the increase in Florida gun crimes, based on data for 2005 and 2006. Gun Murders in Florida went from 521 in 2005 to 740 in 2006, an increase of 42.0%

What if they happen to decrease in 2007 to the same number as in 2005. That is, if they went down from 740 in 2006 to 521 in 2007? That would be a decrease of only 29.6%, 12% less of a decrease than the increase reported above.

If the number of Gun Murders in Florida happened to be 521 in odd years and 740 in even years for a decade, and you averaged the percentages, that would show an average increase of 6.2% per year while, in truth, the rate was unchanged for the decade!

THE LESSON: Beware of percentages, particularly when percentage increase and percentage decrease are compared.


Please comment on this material. (Stay tuned: I plan to post yet another part of this "presentation" in a week or so.)

Ira Glickstein

11 comments:

joel said...

Ira said:
THE LESSON: Beware of percentages, particularly when percentage increase and percentage decrease are compared.


Ira is exactly right concerning percentages. I've tried to formulate some rules concerning this matter, but I can't go beyond "beware." The problem is that mathematics is a set of tools for most of us. How can you formulate a general rule which tells you not to use a screw drive to pound nails, a hammer to turn screws, etc. except to say "Beware, do only what is appropriate." Application is everything.
Let's take the averaging of percentages for example. Ira pointed out that strange or non-intuitive results occur when one adds percentage increases or decreases. Actually, it depends on what you have in mind. Suppose that you intend to use the stock market to make your own annuity. If a stock started at $100 and it went up by 100% and down by 50% every other year, we could say there was no increase in the stock price at the end of the ten year period, but the average annual stock appreciation would be 25% or 250% over the total period. Certainly this is counter intuitive since there's been no net increase in the stock price. But, there is an situation in which this is meaningful. Suppose your market investment is for income purposes. Each year you intend to withdraw the earnings, so that at the beginning of the next year you still have your original investment in the market. When there's less than the fixed principle, you add back into the account. In this case, your average annual income will be 25%. Over ten years you would have received 250% of the original investment, just as predicted by the average of the annual percentage fluctuations.
My point is the same as Ira's. Beware. A solution technique is correct or incorrect, useful or useless, only in the context of a well-stated problem.

Ira Glickstein said...

More fun with statistics about percentages!

On this evening's ABCNEWS TV program they reported, with great alarm, from June 1-15, canceled flights were up 91% (compared to last year at this time I suppose, they did not say!).

I'm sure these data are accurate, but WHAT do they MEAN in PRACTICAL terms to ME? What are my chances of having my flight cancelled or excessively late if I fly tomorrow?

I checked the Internet and found similar alarm but no details that mean anything.

For example, if cancellations last year were 1 in 1000 flights, a 91% increase would bring them up to a bit less than 2 per thousand. I certainly don't want to be caught by a cancellation, but the difference between 0.0010 and 0.0019 is not that alarming.

Ira Glickstein

joel said...

Ira's example of a kind of statistical distortion is often used by the press to make a story more interesting and grab attention. An example I recall from a front page AP story went something like this. Both the headline and the first paragraph gave the impression that African-Americans were not receiving equal medical treatment, resulting in more deaths due to cardiac arrest. A study was quoted which said that whites survived cardiac arrest in the streets twice as often as blacks. Down near the end of the story one finds out that whites survive 6% of the time while blacks survive 3% of the time. In other words, your chances of dying after a sudden heart failure is 94% if you're white and 96% if you're black. It would seem that the real story is that if your heart stops, you're as good as dead whatever the color of your skin. The small difference between black and white can be explained by many factors other than discrimination. With respect -Joel

wwp said...

two comments: 1. How does a nice looking 5 year old, grow up be a Saddam lool-a-like ? 2. Since in the non-physical world, everything is happening simultaneously, Ira would be partially right, he would be 5 and 68 at the same time.

joel said...

Ira said that no one had commented on his example of the misuse of statistics at Pugugly University. Before we can determine if Ira is right or not about this being an abuse , we need to define the problem and then see whether or not the conclusion drawn from the statistics is correct. If the thesis is that the university is discriminating against women, then Ira is exactly right. The raw data shows no such thing. Averaging the percentages over the departments is nonsense given that the departments are different sizes. More properly, the percentages need to be weighted with the relative department size. This would turn out to be the answer that Ira got, that the total percentage of males and females were the same. However, there is a caveat here.

It is highly unlikely that there is an inter-departmental effort to balance the number of male and female hires. Logically, one needs to consider one department at a time to see whether some are guilty of discriminating against men and others against women. Engineering might favor men while humanities favors women. The imbalance is an indication of something gone askew. It's pretty clear that the important question, with respect to civil rights, is whether or not roughly equally qualified candidates have equal probabilities of being selected whether male or female. If there's any statistic that might come close to answering this question it's the ratio of the number of females selected to the number of females applicants, over let's say five years, as compared to that same ratio for males for the same period. We also have to note another caveat.


There are those who would hold PU responsible for the lack of female applicants under the doctrine of affirmative action. The figures could be interpreted to say that PU did not do a good enough job in reaching out to females in graduate school or college or high school or in the cradle thus causing a lack of applicants. This is a natural consequence of a belief that men and women are fundamentally the same and therefore the number of applicants of each gender should be the same. Unfortunately, that is the attitude of many civil rights watchdogs.

Stewart A Denenberg said...

Joel said at the end of a comment about using gender as a basis for hiring:

"This is a natural consequence of a belief that men and women are fundamentally the same and therefore the number of applicants of each gender should be the same. Unfortunately, that is the attitude of many civil rights watchdogs."

My L-brain says that this is not quite fair. I think a better way to say is that there is no reason to assume that a man can do a better job than a woman in most occupations and there certainly should not be a bias in men's favor just because they're male.

On the other hand my C-brain reminds me that my former college seemed to have the policy that all other things being equal a woman must be hired over a man rationalizing this action by saying that this would act to rebalance the imbalance --- in other words righting a wrong with a wrong.

Stu

Ira Glickstein said...

Joel and Stu:

Thanks for your thoughtful and erudite Comments!

Joel's Comment gets exactly to the point about the departments being of different sizes. (Reminds me of the farmer who has black horses and white horses. He kept statistics that showed the black horses ate ten times as much as the white horses. The mystery was solved when someone pointed out he had about ten times as many black horses as white horses.)

Joel goes on to suggest equal opportunity types might compare the percentage of male/female hires with the percentage of male/female applicants, but then asks if PU is reachingout as much to the potential female applicant pool. If someone wants to "prove" discrimination, they can do so if the numbers are anything but equal.

There are certain fields that attract more men than women and vice-versa. Some of that is certainly cultural, but I think much of it is genetic. Certain jobs require lots of travel and overtime and, as long as wives are more likely to spend more time with young families than husbands, those jobs will be sought mostly by men. Some of that is cultural but most is genetic!

The best way to stop discrimination on the basis of race or gender is to cease discriminating on the basis of race or gender! Perhaps it should be illegal to keep records of the race or gender of individuals (except for medical purposes where that information might be necessary).

Ira Glickstein

Stewart A Denenberg said...

More on "sadistics" :

Flatlander stops in middle of nowhere and asks farmer, how many cars go by each day. "Don't know, different numbers on different days." replies the farmer. "Well," says traveler, "What would you say the average is?".
"Mister," the farmer replies, "The average varies".

While most folks know what an average is I would guess that most, including college students, don't know how to compute a weighted average. I've had to show many students how to compute their "GPA" or Grade point average which is a classic example of a weighted average. Usually I point out that getting an A grade in a 4 credit class should count more towards the GPA than an A grade in a 1 credit class.

Delving a bit deeper I think many folks have trouble with concept of ratio itself. This test was given to Junior Physics majors and only 50% got it correct:

Mr Short measured in buttons is three buttons high and measured in paper clips is four paper clips high. Mr Tall is six paper clips high; how tall is he in buttons?

I think the problem might be that division is more abstract than subtraction. And people being lazy will tend to choose the easier path.

For example,if the problem were:
There is a 3 hour time difference between NYC and LA where it's earlier in LA. If it's 3PM in NYC, what time is it in LA?

Obviously this is a simple subtraction problem but if we follow up with:

If 3 cans of paint cost 4 dollars,
how much will 4 cans of paint cost?

Children who are weak on division and therefore ratio will say 5 dollars.

If you think ratio is easy here's a problem I made up and always have difficulty solving (mostly dependent on how much sleep I had on the prior night):

Which would you rather have: a pound of dimes or a pound of quarters? Do not specify an answer, instead describe the process you would use to solve this problem.
Hint: this is a sort of inverse ratio problem...

Sadistically,

Stu

Ira Glickstein said...

Stu: THANKS for your sadistically humorous Comment!

If *any* college student needs help understanding weighting of GPA based on the course credits he or she should have their GPA reduced by 1.0! (If fact, since Stu is a Computer Science Prof., any CS student who needs help in this area should be transferred to the English Department and the transfer would raise the standards in both departments :^)

A paper clip is 3/4 as high as a button. So, 6 paper clips equals 6 x 3/4 = 4.5 buttons high. Right? And half the Junior PHYSICS MAJORS got it wrong? Oh my!

Each can of paint costs 4/3 dollars. So 4 cans cost 4 x 4/3 = 16/3 = 5 1/3 dollars.

On the dimes and quarters. I'd get a balance and put five dimes (=50 cents) on one side and two quarters (=50 cents) on the other. If the quarters were heavier, I'd choose a pound of dimes. If they were equal, it doesn't matter what I choose. If the dimes were heavier, I'd choose a pound of quarters.

I haven't done the experiment, but I think two quarters weigh more than five dimes. So. I'd end up choosing the pound of dimes.

Ira Glickstein

Stewart A Denenberg said...

Good work Ira; basically if a quarter weighs more than 2.5 dimes you'd want the pound of dimes because you'd get more 25 cent units in one pound. But this involves inverse ratio thinking and even though I invented the problem, some days I really have problems with it...

So my question is: is it reasonable to expect the average college graduate to answer this question correctly?

Stu

Stu

Ira Glickstein said...

Stu:

I just checked and, according to http://www.factmonster.com/ipka/A0774847.html the standard weight of a US dime is 2.268 g and a quarter is 5.670 g. If you do the math, 50 cents in either denomination weighs 11.34 g, so it does not matter which you choose.

They are both made of the same alloy, Cupro-Nickel clad (8.33% nickel [Ni], the rest copper [Cu]), so, even if you were going to melt them down it does not matter which you choose.

On the other hand, if you were going to use them to mark the position of your ball in golf, you'd have 2 1/2 times as many markers if you chose the dimes over the quarters.

The half dollar is the same alloy in the same proportions as the dime and quarter.

The cent, nickel, and golden dollar are each different alloys. Since the golden dollar is only 8.1 g I'd pick a pound of dollars if I could chose any denomination.

Ira