Bayes Theorem has practical applications. Use it to make real world decisions. [Updated 8 Sep 2013]
Bayes Theorem is not just an obscure artifact of the statistics of probability handed down to us from centuries ago. You can use it now to make decisions that will affect your financial well-being.
A relatively simple Excel-based tool helps you choose the right course of action in the face of uncertain probabilities and inexact test results. It is available for FREE.
[Presented to the Science-Technology Club, The Villages, FL, 9 September 2013. For a copy of my PowerPoint slide show, send an email to ira
What can the Bayesian Inference Advisor Tool do for You?
I know you are probably not in the oil business (and neither am I), but the best way to understand an abstract concept is to go through a practical example. The example below has to do with oil exploration and drilling, but the Bayesian Inference Advisor can handle any problem where you know:
1) the probability of success if you take action without doing further testing,
2) the cost of further testing and the probability the test results will be reliable,
3) the cost of taking some action,
4) the financial benefit to you if the action is successful, and
5) the compensation you expect for taking the risk.
You can download the Bayesian Inference Advisor at:
https://sites.google.com/site/bigira/stuff-ira-knows/BayesianInferenceAdvisorV3.xls?attredirects=0&d=1
The Bayesian Inference Advisor will compute the most likely financial implications of your actions. What if you proceed without further testing? What if you get a positive test result and proceed? What if you get a negative test result and proceed? Several other examples in very different domains are included later in this posting, but for now, put on your hard hat and let us imagine we are in the oil business!
Example #1- Should we drill here, drill now?
Situation
Twenty oil wells have been drilled in a particular area and only three of them have yielded commercially-viable quantities of petroleum. The others were “dry holes”. So the chances of a newly-drilled hole striking oil are about 15%. You could do just go ahead and drill without further testing, but that would be relatively expensive and you could turn up a dry hole. Seismic testing is less expensive than actually drilling, but even if you get a positive result, that doesn’t guarantee oil will be found. The probability of getting a positive seismic test result in this area is 25%. The tests results have a probability of 80% of being positive if there is indeed oil at the site. You input the three probabilities (15%, 25%, and 80%) into the Bayesian Inference Advisor INPUT panel as indicated above.
The Bayesian Inference Advisor uses Bayes Theorem to calculate that there is an inverse probability of 48% that you will be successful finding oil given a positive seismic test. OK, now that Bayes has spoken, what should you do?
To answer that question you first have to input four pieces of financial data. According to experienced oilmen in the area, it will cost $1M to do a seismic test and an additional $20M if you decide to drill. if you do strike oil, you could get $70M based on current prices, but you need a 15% return on investment (ROI) to account for the risk and the possibility petroleum prices could go down further. ROI is: { Gain from Investment – Cost of Investment } / {Cost of Investment}
The figure above shows the numbers you would input into the Bayesian Inference Advisor for the given situation, and the output you would get. The Bayesian Inference Advisor is pretty sure what you should do in this case:
Test first. If Test is Positive, do the Procedure. Proceed with the seismic testing and, if you get a positive result, go ahead and drill and get rich. If you get a negative test result you are advised to abandon the plan to drill at that point. Do not throw good money after the “bad” $1M you spent on the seismic testing. Move to another spot and continue seismic testing and hope for better test results.
But what if you and your investors happened to be more greedy (or cautions) and demanded at least a 30% ROI?
The figure above shows the numbers you would input into the Bayesian Inference Advisor for the given situation, and the output you would get. The Bayesian Inference Advisor is pretty sure what you should do in this case:
Test first. If Test is Positive, do the Procedure. Proceed with the seismic testing and, if you get a positive result, go ahead and drill and get rich. If you get a negative test result you are advised to abandon the plan to drill at that point. Do not throw good money after the “bad” $1M you spent on the seismic testing. Move to another spot and continue seismic testing and hope for better test results.
But what if you and your investors happened to be more greedy (or cautions) and demanded at least a 30% ROI?
The figure above shows the numbers you would input into the Bayesian Inference Advisor for the given situation, and the output you would get. The Bayesian Inference Advisor is pretty sure what you should do in this case:
Hopeless venture. (Can you reduce expected ROI?) Forget about it unless you and your investors will accept a lower ROI. Given these inputs, if you absolutely need 30% ROI, it is HOPELESS. However, if you and your investors are willing to reduce expected ROI to 20%, you should TEST FIRST, then GIVEN A POSITIVE TEST RESULT, DRILL.
On the other hand, what if you are in a different oil patch where the PRIOR probability of striking oil or gas is much higher, say 80%?
The figure above shows the numbers you would input into the Bayesian Inference Advisor for the given situation, and the output you would get. The Bayesian Inference Advisor is pretty sure what you should do in this case:
No need to Test. Go ahead with Procedure. Here, with such a high expectation of hitting oil or gas, you don't even have to test.
NOTE: These are just made-up values to illustrate the process and may or may not represent the actual financial situation in any real-world situation. If you use this tool to make money, please send me my share. If you lose money, you are on you own :^)
Choices What to do?
1) Go ahead and drill without seismic testing?
2) Do the seismic testing and, if you get a positive result, go ahead and drill?
3) Forget about it – all these numbers hurt my head!
The Bayesian Inference Advisor crunches the numbers you entered and gives you the results and recommendations on what to do!
According to analysis using Bayes Law, in the first situation above, if you drill for oil or gas without doing the seismic testing, you are most likely to invest $133M, for a net loss of $73M and an ROI of -55%. OUCH! Not a good result!
If you go ahead and invest the $1M for the seismic testing of several locations and eventually get a positive result and then drill, you are most likely to invest $50M for each successful oil well, for a net gain of $10M and an ROI of +20%. WOW! That would be great!
What happens when oil prices go up or down and other variables change? Given different circumstances, the Bayesian Inference Advisor is capable of recommending that you proceed to drill without testing or that you abandon plans to get into the oil business in that area entirely.
Background – What is Bayes Theorem? If you really want to know all the intimate statistical probabilistic stuff, just Google “Bayes Theorem” or “Bayesian” and you will get lots of links replete with mathematical symbols in all their glory. (Don’t worry if you can’t really understand all those symbols -perhaps your daughter could play them on her flute :^)
Here is All You Really Need to Know about Bayes The Rev. Thomas Bayes was a Presbyterian minister and mathematician who lived in the 1700′s. His great contribution to mathematics was the concept of “inverse probability”. He came up with it at a time when only “forward probabilty” was generally known.
Forward probability has to do with making predictions based on previous knowledge. Say you know the following about Poupon University:
1) There are a total of 3000 students,
2) 2340 are Liberal Arts majors,
3) 60 are Math majors, and
4) 600 are Engineering majors.
If you go to Commons and pick a student at random, what is the probability you’ll pick a Liberal Arts major? A Math major? An Engineer? (Assuming, of course, that Engineers are as likely as, say English majors, to take a break from their studies and to go to the Commons :^)
That is easy using forward probability:
1) Liberal Arts: 2340/3000 = 0.4 or a 78% chance of picking a Liberal Arts major.
2) Math: 60/3000 = 0.02 or a 2% chance of picking a Math major.
3) Engineering: 600/3000 = 0.2 or a 20% chance you’ll pick an Engineer.
What Bayes figured out, inverse probability, is harder. (An English major once told me that “backward poets rhyme inverse” but that is a different matter :^)
Hey, here’s a student wearing a Bayes Theorem T-Shirt! Based on the facts given above, what is the probability he or she is a Liberal Arts major? A Math major? an Engineer? The formula on his shirt can help you do the inverse probability to figure that out.
See the top figure in this posting, a T-Shirt with Bayes Theorem in Mathematical Symbols
But first you need some additional facts:
a) Based on sales by the college store, about 300 students out of the 3000 on campus, or 10%, own a Bayes T-shirt.
b) 100% of the Math majors own them (of course :^).
c) 30% of the engineeers.
d) Only 2.6% of the Liberal Arts majors.
e) You are in luck, today is April 7, the day Rev. Thomas Bayes died 1761, so everybody who owns a Bayes T-Shirt will be wearing one today!
If you pick a student who is wearing a Bayes T-Shirt, at random, what is the inverse probability that he or she is a Liberal Arts major? A Math major? An Engineer? Why don’t you guess right now – please write your answers down so you will be properly amazed when you find out how wrong (or right) you were!
Bayes Theorem is shown in mathematical symbols on the T-Shirt at the top of this posting. It translates to mathematical English as follows:
Probability of A given B equals Probability of B given A multiplied by the Probability of A and divided by the Probability of B In case you did not understand that translation, here it is in plain English using the Bayes T-shirt and Engineering students as an example:
The Probability a Student is an Engineer given that he or she is wearing a Bayes T-Shirt, is equal to the Probability of wearing a Bayes T-Shirt given a student is an Engineer multiplied by the Probability a Randomly-selected student is an Engineer, all divided by the Probability a Randomly-selected student is wearing a Bayes T-Shirt. Let’s run the numbers:
- P(Wearing Bayes T-Shirt given Engineer)=30% [from a few paragraphs above]
- P(Engineer) = 20% [from the forward probability stuff]
- P(Wearing Bayes T-Shirt) = 10% [from a few paragraphs above]
- P(Engineer given Wearing Bayes T-Shirt) = P(Wearing Bayes T-Shirt given Engineer) x P(Engineer) / P(Wearing Bayes T-Shirt) = 30 x 20 / 10 = 60%
So, using the “test” that a student is wearing a Bayes T-Shirt increases your chances of picking an Engineer by a factor of three. WOW, the power of Bayes Theorem is impressive.
If you run the numbers for Math majors you get 100x 2 / 10 = 20%. There is a 20% chance he or she is a Math major. WOW! A surprising result because only 2% of the students on campus are Math majors! So using the “test” that a student is wearing a Bayes T-Shirt increases your chances of picking a Math major by a factor of ten. WOW, the power of Bayes Theorem is impressive.
If you run the numbers for Liberal Arts majors you get: 1x 40 / 10 = 4%. There is only a 4% chance he or she is a Liberal Arts major, surprising because they make up 40% of the student body. Bayes predicts you are ten times less likely to pick a Liberal Arts major using the Bayes T-Shirt test, a negative result. (You could use that test to avoid Liberal Arts majors -or they could use the same test to avoid Engineers and Math majors:^)
So, in summary, if a student happens to be wearing a Bayes T-Shirt on this campus today, there is an 80% chance he or she is in Math or Engineering even though only 22% of the students are in those two majors.
Bayesian Controversy
According to http://psychology.wikia.com/wiki/Bayesian_probability there is considerable controversy between “Bayesians” and “Frequentists” as to the true meaning and interpretation of “probability”. (IMHO, the controversy is like the legendary spat between the "Hatfields" and the "McCoys" - as a practical matter it does not mean anything TO ME. There are situations where standard "Frequentist" probability is most applicable and situations where "Bayesian" probability is most applicable. Both are valid in their own ways.)
“In the philosophy of mathematics Bayesianism is the tenet that the mathematical theory of probability is applicable to the degree to which a person believes a proposition. Bayesians also hold that Bayes’ theorem can be used as the basis for a rule for updating beliefs in the light of new information —such updating is known as Bayesian inference . In this sense, Bayesianism is an application of the probability calculus and a probability interpretation of the term probable, or —as it is usually put —an interpretation of probability.”
“A quite different interpretation of the term probable has been developed by frequentists . In this interpretation, what are probable are not propositions entertained by believers, but events considered as members of collectives to which the tools of statistical analysis can be applied.”
An interesting example would be if ten coin tosses resulted in seven heads and three tails. A frequentist would say the probability is 70/30 heads unless and until further tosses proved otherwise. A Bayesian would consider the situation from a larger prospective. Was the coin provided by a trusted person or some stranger in a bar? Is there reason to believe the coin is fair or loaded? Based on that consideration, one Bayesian might assign a probability of 50/50, since ten tosses with a 70/30 result is statistically possible with a fair coin. Another Bayesian might conclude, from the situation, that the coin is probably loaded and assign a 70/30 probability. Yet another might split the difference and assign 60/40, …
Targeted Marketing Example
Guess what, the Bayesian Inference Advisor loves targeted marketing problems!
Say we are marketing something fairly expensive, with a correspondingly high profit margin. Suppose it would appeal only to a highly specialized audience. For example, something that only mathematically and technologically-oriented college students would buy. Indeed, what if ownership of a Bayes T-Shirt turned out to be an excellent “test indicator” to qualify a prospect? If we could get hold of a list of students who bought those shirts -or just go on campus and approach anyone wearing one- we could restrict our sales pitches to them.
If the Bayesian Inverse probability was high enough for this select group, we could afford to lure them to our sales pitch with the promise of a gift or a free meal.
Bad Reputation of “Free Bait” Not Always Justified Of course you are familiar with this type of targeted marketing approach. Offer a free trial subscription to a magazine or a 30-day free trial for a software program or computer game. Give a lower monthly price for cell phone service or cable-TV for a year and jack it up after the customer is hooked. Offer a free vacation to a time share property, etc. The key to the success of these plans is to qualify the targets and then offer them the “free bait”. You better qualify them well because the “free bait” can be costly to you if all you attract are people who have no intention of buying and are just looking for a free meal or gift!
Although some instances of targeted marketing have given the genre a bad reputation, there is nothing wrong, in principle, with offering something of value to qualified people to get them to listen to your sales pitch. For example, when I interviewed seniors on a college campus for possible jobs as engineers at the company where I worked, I was allowed to invite up to 25% of the top candidates for an expense-paid trip to our facility – if they met certain GPA and other qualifications.
The key is to make sure that the qualifying “test” is effective enough to justify the “free bait”.
Marketing Case - Test the Person, Offer Free Bait if Qualified
This is a situation where the “Prior” Probability is very low – only about five out of one-thousand people would want our product and be able to afford it. That is only 0.5%.
Assume we have developed some type of “qualifying test” that identifies about ten out of one-thousand people as targets. That is only1%. (This “test” might be a prospect list we could buy from a market data company that tracks interests and buying habits and disposable income of specific types of people. Alternatively, we might send agents to yacht clubs or professional societies or whatever type of affinity group that tends to have people who qualify for our product. Our agents would glad-hand people and identify likely prospects. Another possibility would be to have our current customers recommend their associates with similar interests and income.)
Notice that these two probabilities 0.5 and 1% are very much lower than the probabilities we used for the oil exploration cases. Targeted marketing makes sense only when the targets are a very select population and our product has a high markup. (If the targets were a higher percentage of the general population, we would use the mass media such as the internet, TV, and print publications.)
The third probability we need to input is conditional probability, the probability that someone who would buy our product would pass our qualifying test. We could survey current customers using the “test” we have devised and determine what percentage of them would pass the “test”. In this case, let us assume 75% of our current customers would pass the test.
We input these three probabilities and the Bayesian AI Advisor calculates the inverse probability that we will be successful in selling our product to a person who passed the test. In this case that comes out to be 37.5%.
In this case, with such a small percentage of the population as our audience, it would be foolish to invite everybody for the “free bait” – we would lose big! According to the Bayesian Inference Advisor, an untargeted approach would cost us about $8000 for each person who buys, a net loss of $7250 and a ROI of -90%. We expected results like that which is why we intend to do targeted marketing.
The Bayesian Inference Advisor is pretty sure what you should do in this case:
Hopeless venture. (Can you reduce expected ROI?) Forget about it unless you and your investors will accept a lower ROI. Given these inputs, if you absolutely need 20% ROI, it is HOPELESS. However, if you and your investors are willing to reduce expected ROI to 17%, you should Test first. If Test is Positive, do the Procedure. Proceed with the qualification testing and, if you get a positive result, go ahead offer them the free vacation or gift or meal and get rich.
Medical Testing - Highly Accurate Tests but Ultra-Low “Probability of Success”
Medical testing illustrates another aspect of the value of analysis based on Bayesian inverse probability. Generally, medical tests, such as tests for early detection of serious diseases or use of illegal drugs in the workplace are very good (with 95% to 99.99%accuracy) but the targets are rare in the population. Of course it is good that only a very small percentage of people have these serious diseases or use illegal drugs in the workplace, but, as you will see based on Bayesian analysis, that leads to a fairly high level of false positives.
Screening Tests for Serious but Rare Diseases
Assume one in ten-thousand people in a given population is in the initial stages of some serious disease. Early testing could detect the disease in time to take preventive action. P(S) = 0.01% expresses that one in a ten-thousand probability.
Assume further that there is a test that can detect the disease in its early stages with a 99.99% probability. That is, if a person has the disease and takes the test, there is a 99.99% probability the test will come back positive. However, the probability of a positive test is 0.1%, which means that, while 99.9% of the population will get a negative result, 0.1% will get a positive, and nine out of ten of the positives will be false positives.
The computed Bayes inverse probability is 10% which means that only one in ten people who get a positive test result will actually have the disease. There will be about nine false positives for every ten-thousand people tested. All ten people will have to be called back in for more intensive and intrusive (and expensive) re-testing using other techniques to assure they do not have the disease. For every person who has the disease, nineteen will be alarmed and inconvenienced unnecessarily.
If we assume the cost of the initial testing is about $10 per person, which is quite reasonable for any kind of medical test, and that the cost of follow-up testing will be about $500 for each person identified as possibly having the disease, the cost of finding that one person in ten-thousand will be over $100,000 dollars! If we have to test one-million people, we will find about one-hundred with the disease and unnecessarily alarm and inconvenience nearly two-thousand, and spend over $2M! If we have to test three-hundred million people, it will cost over $3B!
Testing for Illegal Drugs in the Workplace
The Bayesian AI Advisor could be used to determine the cost of doing screening tests in the workplace to identify employees who may be using hard drugs. They are putting themselves and their co-workers in jeopardy through their actions and risking liability actions against their employer if a co-worker or a member of the public is affected by their actions.
However, even if the tests are highly accurate and correctly catch 99% of the employees who have used illegal drugs recently, they are likely to have false positives for even more innocent employees. This will necessitate follow-up testing that may be more intrusive and alarming, as well as very expensive. The Bayesian AI Advisor will compute the likely costs of such a screening program. Management will have to determine if the elimination of various types of drugs among employees is worth the overall expense and disruption.
For example, it may well be worth the expense and disruption of innocent employee’s lives to do complete screening for airline pilots, bus drivers, employees in high-security jobs where drug use may expose them to blackmail, and others whose actions could cost lives and expose employers to gigantic liability claims. On the other hand, employees in less critical jobs might be screened only if evidence comes up that they are behaving strangely at work, are in financial stress, are having domestic troubles, etc.
Error Testing Feature of Bayesian Inference Advisor
What if we enter probabilities that don’t make sense? Well, the Bayesian AI Advisor has some error detection capabilities, as illustrated by the red boxes in the rightmost panel.
Notice the far right column in the above examples, where four red "OK" notices indicate error checking has found no problems. If there was an error, a red "Error" notice would appear on the affected line. For example, a red “Error” would denote that P(+) < P(S) x P(+|S), which is not possible, or that the combination of P(S), P(+), and P(+|S) causes P(S|+) > 0.999.
The Bayesian Inference Advisor also checks that the Gross Benefit if Successful is greater than the Cost of Procedure and that the Cost of Procedure is greater than the Cost of Test. A little red “OK” indicates that those values are not in error.
Sensitivity Analysis – How Robust Are Your Results?
Why Do Sensitivity Analysis?
The results and recommendations of the Bayesian AI Advisor are, of course, dependent upon the probabilities and cost factors you input. There are two reasons to do sensitivity analysis and the Bayesian AI Advisor provides graphical aids for both:
YOU ARE UNCERTAIN ABOUT YOUR INPUT PARAMETERS
The inputs are “just estimates”. What if they are off by 10%? 20%? – or more? The Bayesian AI Advisor graphically indicates what would happen to your ROI if the true value was lower than you estimated, down to half what you input. It also shows what the ROI would be if any of your input variables were double what you input. (If you can’t estimate a variable closer than a factor of two, you are really not ready to do anything that will put real money at risk!)
YOU MISSED YOUR ROI GOAL
If the Bayesian AI Advisor indicates you missed your ROI goal you may be able to make changes to achieve it. The most obvious change is to reduce your desired ROI. For example, if you expected to achieve 30% and the Bayesian AI Advisor indicates you will only meet 20%, perhaps you should settle for that.
If reducing minimum acceptable ROI is a deal-breaker for your investors, then you need to consider more difficult changes. These include:
1) Improving the qualification test to tighten the limits,
2) Changing the drilling location or market sector to an area where you are more likely to strike oil and/or find customers,
3) Modifying your product so it has more appeal,
4) Raising the price to gain more benefit from each sale,
5) Reducing the price to increase volume of sales,
6) Reducing the cost of the qualification tests, etc.
The Bayesian AI Advisor graphically indicates which of these changes are most likely to have the best pay-off for you.
1) Improving the qualification test to tighten the limits,
2) Changing the drilling location or market sector to an area where you are more likely to strike oil and/or find customers,
3) Modifying your product so it has more appeal,
4) Raising the price to gain more benefit from each sale,
5) Reducing the price to increase volume of sales,
6) Reducing the cost of the qualification tests, etc.
The Bayesian AI Advisor graphically indicates which of these changes are most likely to have the best pay-off for you.
Example of Sensitivity Analysis – Oil Exploration Case #2
FINANCIAL FACTOR SENSITIVITY
The figure shows your financial factors (see Oil Exploration Case #2 subsection above): Your minimum acceptable ROI is 30% (indicated by the horizontal dashed line) . The most likely achieved ROI is indicated as the point where the other curves cross. It is about 20%.
Here is how to use the sensitivity analysis graph:
Benefit if Successful: The blue line marked with diamonds represents changes to the Benefit if Successful. If you can increase that by 10%, indicated as where it crosses the ROI line, you will be at 30% ROI. That means waiting for oil prices to go up or reducing your refining or transport costs, etc. In the marketing case, it means raising your prices, but that may be self-defeating if sales decline as a result, or reducing your prices to gain volume, but that too may be self-defeating if profits decrease as a result of a smaller margin per item sold.
Procedure Cost: Alternatively, according to the green line marked with triangles, if you reduce your Procedure Cost by about 15%, you meet the ROI. That means reducing the cost of drilling that well. In the targeted marketing case, perhaps you can just serve coffee and donuts rather than a full meal.
Test Cost: The third possibility is reducing your Test Cost, as indicated by the pink line with squares, but that would require reducing the Test Cost by over 20%. Perhaps you can get the seismic test company to lower their price? Perhaps the market research company will sell you that list of hot prospects for less money?
PROBABILITY FACTOR SENSITIVITY
The figure shows your probability factors: Again, your minimum acceptable ROI is 30% (indicated by the horizontal dashed line) and the achieved ROI, indicated as the point where the other curves cross) is about 20%.
Here is how to use the sensitivity analysis graph:
Probability of Success: The black line marked with starry squares represents changes to P(S), Probability of Success. If you could improve that by about 10% you would meet your ROI goal. P(S) is determined by where you intend to drill or the market segment you are addressing, etc., so you would have to move to a different area or market segment to improve that value.
Probability of Positive Test: Alternatively, according to the blue line marked with circles, if you reduce your Probability of Positive Test by about 10%, you meet the ROI. That would require making the qualification test tighter, so it discriminates the chances of success more sharply and has fewer false positives.
Probability of Positive Test given Success: The third possibility is increasing your Probability of Positive Test given Success by about 10%. This requires tighter classification of your previous customers in the targeted marketing case or of the geological assumptions in an oil exploration case.
Please Let Me Know How this Tool Has Helped You
I’d appreciate it if you would Comment on this (former Google Knol) posting and let me and others know your experiences with the Bayesian AI Advisor, ideas for improvement, etc. You may also reach me by email at Ira@techie.com. advTHANKSance!
Ira Glickstein
No comments:
Post a Comment