I wanted to shore up my understanding of Statistical Confidence Levels ( "shore up" is a joke, there is no shore, no beach, no nothing) so I contacted my brother in law, Paul Vogt, who is a genius at this stuff, and asked him to explain...
"So say you've got two sets of data from two sites, and say that over 20 days you get an average of 35 hits on one and 58 hits on the other. The average difference is 23. Is that difference statistically significant, meaning how likely are you to get an average difference that big by chance alone? You get the answer using a t-test. The answer is always a p-value. P stands for probability. You want the p value to be small because it is the probability that your result is just a coincidence. So if the p val is .05 there is only a 5% chance that the difference is just a fluke."
"Confidence is something else. You might get something like the following: 95% Confidence Interval CI = 12, 35. You can be 95% confidence that the true value is between 12 and 35. Your margin of error is plus or minus 11. If you are willing to lower your confidence to 90% you can reduce your margin of error. Conversely, if you want to be 100% confident you can say, for example: I'm 100% confident that the true value is somewhere between zero and infinity. In brief, the lower the confidence the narrower the margins of error, the higher the confidence the wider."
"Doing all this by had is not only slow, it's prone to error. There is probably some freeware you can use to do it once you get the data."
To learn more about this and other interesting quanta please visit him at http://vogtsresearchmethods.blogspot.com/
Edward worked at Lawrence Berkeley Laboratory in the hills of Berkeley Ca in the 70's, where the current internet backbone was being developed. He consulted scientists and programmed in fortran, cobol, pascal, C, and Oracle and Informix 4gl's. He worked corporate in the city as a freelancer for 12 years. Higher Source Web Sites started developing web sites in 2001, while in the real estate business. He started blogging and developing sites for his colleagues and his company, then other Real Estate companies. His background in business programming, database administration, & software development made staying within the soft technologies more appealing than other choices. He lives in Santa Rosa Ca with his wife Pam and runs a beagle rescue with Pam, teaches chess, and loves music and cinema.