I wanted to shore up my understanding of Statistical Confidence Levels ( “shore up” is a joke, there is no shore, no beach, no nothing) so I contacted my brother in law, Paul Vogt, who is a genius at this stuff, and asked him to explain…
“So say you’ve got two sets of data from two sites, and say that over 20 days you get an average of 35 hits on one and 58 hits on the other. The average difference is 23. Is that difference statistically significant, meaning how likely are you to get an average difference that big by chance alone? You get the answer using a t-test. The answer is always a p-value. P stands for probability. You want the p value to be small because it is the probability that your result is just a coincidence. So if the p val is .05 there is only a 5% chance that the difference is just a fluke.”
“Confidence is something else. You might get something like the following: 95% Confidence Interval CI = 12, 35. You can be 95% confidence that the true value is between 12 and 35. Your margin of error is plus or minus 11. If you are willing to lower your confidence to 90% you can reduce your margin of error. Conversely, if you want to be 100% confident you can say, for example: I’m 100% confident that the true value is somewhere between zero and infinity. In brief, the lower the confidence the narrower the margins of error, the higher the confidence the wider.”
“Doing all this by had is not only slow, it’s prone to error. There is probably some freeware you can use to do it once you get the data.”
To learn more about this and other interesting quanta please visit him at http://vogtsresearchmethods.blogspot.com/