13. A level of confidence, as defined in Table 3, can be used to characterize uncertainty that is based on expert judgment as to the correctness of a model, an analysis or a statement. The last two terms in this scale should be reserved for areas of major concern that need to be considered from a risk or opportunity perspective, and the reason for their use should be carefully explained.

Table 3. Quantitatively calibrated levels of confidence.

Terminology                     Degree of confidence in being correct
Very High confidence      At least 9 out of 10 chance of being correct
High confidence               About 8 out of 10 chance
Medium confidence          About 5 out of 10 chance
Low confidence                About 2 out of 10 chance
Very low confidence         Less than 1 out of 10 chance

14. Likelihood, as defined in Table 4, refers to a probabilistic assessment of some well defined outcome having occurred or occurring in the future. The categories defined in this table should be considered as having ‘fuzzy’ boundaries. Use other probability ranges where more appropriate but do not then use the terminology in table 4. Likelihood may be based on quantitative analysis or an elicitation of expert views. The central range of this scale should not be used to express a lack of knowledge – see paragraph 12 and Table 2 for that situation.

Table 4.                Likelihood Scale.

Terminology                Likelihood of the occurrence/ outcome

Virtually certain            > 99% probability of occurrence
Very likely                    > 90% probability
Likely                            > 66% probability
About as likely as not        33 to 66% probability
Unlikely                        < 33% probability
Very unlikely                < 10% probability
Exceptionally unlikely    < 1% probability

quote:

ORIGINAL: FirmhandKY

Questions.

1.  If, according to their analysis, they have a "high level of confidence" (8 out of 10 chance) of something being "likely" (>66% probability) ... what is the statistical probability of that event happening?

2.  In the "likely" category .... what is the statistical difference between a 66% chance of something happening, and an 89% chance of something happening?

3.  Without a detailed breakdown of the statistical background, and the math that was used to arrive at their "level of confidences" and their "Likelihood Scale", how confident are you in their conclusions?

Any input and thoughts would be appreciated.

FirmKY

quote:

ORIGINAL: Wildfleurs

quote:

ORIGINAL: FirmhandKY

Questions.

1.  If, according to their analysis, they have a "high level of confidence" (8 out of 10 chance) of something being "likely" (>66% probability) ... what is the statistical probability of that event happening?

2.  In the "likely" category .... what is the statistical difference between a 66% chance of something happening, and an 89% chance of something happening?

3.  Without a detailed breakdown of the statistical background, and the math that was used to arrive at their "level of confidences" and their "Likelihood Scale", how confident are you in their conclusions?

Any input and thoughts would be appreciated.

FirmKY

I have a statistical background, and its 3:30 in the morning so take my comments for what they are:

1) The thing that is throwing me with their parameters of confidence is that there literally is a statistical test that you can do for confidence, which is distinct from a self-defined determination of confidence that they seem to be (setting) arbitrarily .

2) It looks like they are using Likert scales to set their parameters, which is why you'll never be able to fully tease out the difference between a 89% and 66% chance of something happening.  Thats just the bad that happens with doing groupings like that, which is inevitable for risk analysis (that I've done a bit of and seems to be largely a function of plugging in numbers in spreadsheets).

3) Not very confident - the devil is always in the details, especially details as general as what you provided would never be enough for me.

C~

Confidence levels are typically given alongside statistics resulting from sampling.

In a statement: we are 90% confident that between 35% and 45% of voters favor Candidate A, 90% is our confidence level and 35%-45% is our confidence interval.

It is very tempting to misunderstand this statement in the following way.

[math deleted]

This conclusion does not follow from the laws of probability because θ is not a "random variable"; i.e., no probability distribution has been assigned to it. Confidence intervals are generally a frequentist method, i.e., employed by those who interpret "90% probability" as "occurring in 90% of all cases". Suppose, for example, that θ is the mass of the planet Neptune, and the randomness in our measurement error means that 90% of the time our statement that the mass is between this number and that number will be correct. The mass is not what is random. Therefore, given that we have measured it to be 82 units, we cannot say that in 90% of all cases, the mass is between 82 − 1.645 and 82 + 1.645. There are no such cases; there is, after all, only one planet Neptune.

But if probabilities are construed as degrees of belief rather than as relative frequencies of occurrence of random events, i.e., if we are Bayesians rather than frequentists, can we then say we are 90% sure that the mass is between 82 − 1.645 and 82 + 1.645? Many answers to this question have been proposed, and are philosophically controversial. The answer will not be a mathematical theorem, but a philosophical tenet. Less controversial are Bayesian credible intervals, in which one starts with a prior probability distribution of θ, and finds a posterior probability distribution, which is the conditional probability distribution of θ given the data.

For users of frequentist methods, the explanation of a confidence interval can amount to something like: "The confidence interval represents values for the population parameter for which the difference between the parameter and the observed estimate is not statistically significant at the 10% level". Critics of frequentist methods suggest that this hides the real and, to the critics, incomprehensible frequentist interpretation which might be expressed as: "If the population parameter in fact lies within the confidence interval, then the probability that the estimator either will be the estimate actually observed, or will be closer to the parameter, is less than or equal to 90%". Users of Bayesian methods, if they produced a confidence interval, might by contrast say "My degree of belief that the parameter is in fact in the confidence interval is 90%". Disagreements about these issues are not disagreements about solutions to mathematical problems. Rather they are disagreements about the ways in which mathematics is to be applied.

The Bayesian interpretation of probability allows probabilities to be assigned to all propositions (or, in some formulations, to the events signified by those propositions) in any reference class of events, independent of whether the events can be interpreted as having a relative frequency in repeated trials. Controversially, Bayesian probability assignments are relative to a (possibly hypothetical) rational subject: it is therefore not inconsistent to be able to assign different probabilities to the same proposition by Bayesian methods based on the same observed data.

Differences arise either because different models of the data generating process are used or because of different prior probability assignments to model parameters. Differences due to model differences are possible in both Frequentist and Bayesian analyses, but differences due to prior probabilities assignments are distinctive to Bayesian analyses. Such probabilities are sometimes called 'personal probabilities', although there may be no particular individual to whom they belong. Frequentists argue that this makes Bayesian analyses subjective, in the negative sense of 'not determined from the data'. Bayesians typically reply that differences due to alternative models of the data generating process are equally subjective, and that such model choices are also (ideally) chosen prior to analysis of the data, by the analyst. Since the analyst is also a natural person to assign prior probabilities, the two subjective inputs can be seen as deriving from the same source.

Although there is no reason why different interpretations (senses) of a word cannot be used in different contexts, there is a history of antagonism between Bayesians and frequentists, with the latter often rejecting the Bayesian interpretation as ill-grounded. The groups have also disagreed about which of the two senses reflects what is commonly meant by the term 'probable'. More importantly, the groups have agreed that Bayesian and Frequentist analyses answer genuinely different questions, but disagreed about which class of question it is more important to answer in scientific and engineering contexts.

Confidence Level is the likelihood - expressed as a percentage - that the results of a test are real and repeatable, and not just random. The idea is based on the concept of the "normal distribution curve," which shows that variation in almost any data (such as the heights of all fourth-graders, or the amount of rainfall in January) tends to be clustered around an average value, with relatively few individual measurements at the extremes.

So if your confidence level is, say, 92%, that means, according to probability theory, there's a 92% chance that you'd see similar results in a repeat of the test. (It does not mean you'd receive the same number of gifts, or that the difference between the packages would be the same. It only means that the package that received more gifts in the first test would be likely to receive more gifts in the second as well - unless, of course, some significant other factors have changed.)

A confidence level of 50% would mean the difference is truly random, with only a 50-50 chance that you'd see the same results in a repeat of the test. Even at 75% the odds are not good - there's a one in four chance that your results are meaningless. Some statisticians consider 90% to be the minimum confidence level for statistically significant results, and that's reportedly the standard used in many election polls. Others insist on a minimum of 95% to be considered significant. And in medical research, for obvious reasons, there's a strong preference for even higher levels of confidence.

It's important to remember that we're talking about probabilities, and there's no magic number that guarantees your results will be repeatable. While it's always best to have a confidence level of 95% or higher, you shouldn't ignore results in the 80% to 90% range. Those results may indicate trends and provide clues about how to improve your mailings; at the very least, they're worth re-testing, preferably in larger quantities. (In any test, a larger sample size will generally give more reliable results.)

Finally, a few words of caution: These confidence levels are only valid when you're comparing test panels that can be thought of as a single event. Don't try to adapt them to a situation that changes over time (such as your total number of active donors), or use it to compare appeals that mailed at different times (there are far too many uncontrolled variables in that case).