Applying Bayes' Theorem in Deduction

Events

Simple form

For events A and B, provided that P(B) ≠ 0,

$P(A|B)={\frac {P(B|A)P(A)}{P(B)}}$.

In many applications, for instance in Bayesian inference, the event B is fixed in the discussion, and we wish to consider the impact of its having been observed on our belief in various possible events A. In such a situation the denominator of the last expression, the probability of the given evidence B, is fixed; what we want to vary is A. Bayes' theorem then shows that the posterior probabilities are proportional to the numerator, so the last equation becomes:

$P(A|B)\propto P(A)\cdot P(B|A)$.

In words, the posterior is proportional to the prior times the likelihood.

If events A₁, A₂, ..., are mutually exclusive and exhaustive, i.e., one of them is certain to occur but no two can occur together, we can determine the proportionality constant by using the fact that their probabilities must add up to one. For instance, for a given event A, the event A itself and its complement ¬A are exclusive and exhaustive. Denoting the constant of proportionality by c we have

$P(A|B)=c\cdot P(A)\cdot P(B|A){\text{ and }}P(\neg A|B)=c\cdot P(\neg A)\cdot P(B|\neg A)$.

Adding these two formulas we deduce that

$1=c\cdot (P(B|A)\cdot P(A)+P(B|\neg A)\cdot P(\neg A))$,

or

$c={\frac {1}{P(B|A)\cdot P(A)+P(B|\neg A)\cdot P(\neg A)}}={\frac {1}{P(B)}}$.

Alternative form

Another form of Bayes' theorem for two competing statements or hypotheses is:

$P(A|B)={\frac {P(B|A)P(A)}{P(B|A)P(A)+P(B|\neg A)P(\neg A)}}$.

For an epistemological interpretation:

For proposition A and evidence or background B,

• $(A)$ is the prior probability, the initial degree of belief in A.
• $P(\neg A)$ is the corresponding initial degree of belief in not-A, that A is false, where $P(\neg A)=1-P(A)$
• $P(B|A)$ is the conditional probability or likelihood, the degree of belief in B given that proposition A is true.
• $P(B|\neg A)$ is the conditional probability or likelihood, the degree of belief in B given that proposition A is false.
• $P(A|B)$ is the posterior probability, the probability of A after taking into account B.

Extended form

Often, for some partition {A_j} of the sample space, the event space is given in terms of P(A_j) and P(B | A_j). It is then useful to compute P(B) using the law of total probability:

$P(B)=\sum _{j}P(B\cap A_{j})$,

Or (using the multiplication rule for conditional probability),

$P(B)={\sum _{j}P(B|A_{j})P(A_{j})}$,

$\Rightarrow P(A_{i}|B)={\frac {P(B|A_{i})P(A_{i})}{\sum \limits _{j}P(B|A_{j})P(A_{j})}}\cdot$

In the special case where A is a binary variable:

$P(A|B)={\frac {P(B|A)P(A)}{P(B|A)P(A)+P(B|\neg A)P(\neg A)}}\cdot$

Random variables

Figure 5: Bayes' theorem applied to an event space generated by continuous random variables X and Y with known probability distributions. There exists an instance of Bayes' theorem for each point in the domain. In practice, these instances might be parametrized by writing the specified probability densities as a function of x and y.

Consider a sample space Ω generated by two random variables X and Y with known probability distributions. In principle, Bayes' theorem applies to the events A = {X = x} and B = {Y = y}.

$P(X{=}x|Y{=}y)={\frac {P(Y{=}y|X{=}x)P(X{=}x)}{P(Y{=}y)}}$

However, terms become 0 at points where either variable has finite probability density. To remain useful, Bayes' theorem can be formulated in terms of the relevant densities (see Derivation).

Simple form

If X is continuous and Y is discrete,

$f_{X|Y{=}y}(x)={\frac {P(Y{=}y|X{=}x)f_{X}(x)}{P(Y{=}y)}}$

where each $f$ is a density function.

If X is discrete and Y is continuous,

$P(X{=}x|Y{=}y)={\frac {f_{Y|X{=}x}(y)P(X{=}x)}{f_{Y}(y)}}$.

If both X and Y are continuous,

$f_{X|Y{=}y}(x)={\frac {f_{Y|X{=}x}(y)f_{X}(x)}{f_{Y}(y)}}$.

Extended form

Figure 6: A way to conceptualize event spaces generated by continuous random variables X and Y

A continuous event space is often conceptualized in terms of the numerator terms. It is then useful to eliminate the denominator using the law of total probability. For f_Y(y), this becomes an integral:

$f_{Y}(y)=\int _{-\infty }^{\infty }f_{Y|X=\xi }(y)f_{X}(\xi )\,d\xi$.

Bayes' rule in odds form

Bayes' theorem in odds form is:

$O(A_{1}:A_{2}\vert B)=O(A_{1}:A_{2})\cdot \Lambda (A_{1}:A_{2}\vert B)$

where

$\Lambda (A_{1}:A_{2}\vert B)={\frac {P(B\vert A_{1})}{P(B\vert A_{2})}}$

is called the Bayes factor or likelihood ratio. The odds between two events is simply the ratio of the probabilities of the two events. Thus

$O(A_{1}:A_{2})={\frac {P(A_{1})}{P(A_{2})}}$,

$O(A_{1}:A_{2}\vert B)={\frac {P(A_{1}\vert B)}{P(A_{2}\vert B)}}$,

Thus, the rule says that the posterior odds are the prior odds times the Bayes factor, or in other words, the posterior is proportional to the prior times the likelihood.

In the special case that $A_{1}=A$ and $A_{2}=\neg A$, one writes $O(A)=O(A:\neg A)=P(A)/(1-P(A))$, and uses a similar abbreviation for the Bayes factor and for the conditional odds. The odds on $A$ is by definition the odds for and against $A$. Bayes' rule can then be written in the abbreviated form

$O(A\vert B)=O(A)\cdot \Lambda (A\vert B)$,

or, in words, the posterior odds on $A$ equals the prior odds on $A$ times the likelihood ratio for $A$ given information $B$. In short, posterior odds equals prior odds times likelihood ratio.

For example, if a medical test has a sensitivity of 90% and a specificity of 91%, then the positive Bayes factor is $\Lambda _{+}=P({\text{True Positive}})/P({\text{False Positive}})=90\%/(100\%-91\%)=10$. Now, if the prevalence of this disease is 9.09%, and if we take that as the prior probability, then the prior odds is about 1:10. So after receiving a positive test result, the posterior odds of actually having the disease becomes 1:1, which means that the posterior probability of having the disease is 50%. If a second test is performed in serial testing, and that also turns out to be positive, then the posterior odds of actually having the disease becomes 10:1, which means a posterior probability of about 90.91%. The negative Bayes factor can be calculated to be 91%/(100%-90%)=9.1, so if the second test turns out to be negative, then the posterior odds of actually having the disease is 1:9.1, which means a posterior probability of about 9.9%.

The example above can also be understood with more solid numbers: Assume the patient taking the test is from a group of 1000 people, where 91 of them actually have the disease (prevalence of 9.1%). If all these 1000 people take the medical test, 82 of those with the disease will get a true positive result (sensitivity of 90.1%), 9 of those with the disease will get a false negative result (false negative rate of 9.9%), 827 of those without the disease will get a true negative result (specificity of 91.0%), and 82 of those without the disease will get a false positive result (false positive rate of 9.0%). Before taking any test, the patient's odds for having the disease is 91:909. After receiving a positive result, the patient's odds for having the disease is

${\frac {91}{909}}\times {\frac {90.1\%}{9.0\%}}={\frac {91\times 90.1\%}{909\times 9.0\%}}=1:1$

which is consistent with the fact that there are 82 true positives and 82 false positives in the group of 1000 people.

Propositional logic

Using $P(\neg B\vert A)=1-P(B\vert A)$ twice, one may use Bayes' theorem to also express $P(\neg B\vert \neg A)$ in terms of $P(A\vert B)$ and without negations:

$P(\neg B\vert \neg A)=1-\left(1-P(A\vert B)\right){\frac {P(B)}{P(\neg A)}}$,

when $P(\neg A)=1-P(A)\neq 0$. From this we can read off the inference

$P(A\vert B)=1\implies P(\neg B\vert \neg A)=1$.

In words: If certainly $B$ implies $A$, we infer that certainly $\neg A$ implies $\neg B$. Where $P(B)\neq 0$, the two implications being certain are equivalent statements. In the probability formulas, the conditional probability $P(A\vert B)$ generalizes the logical implication $B\implies A$, where now beyond assigning true or false, we assign probability values to statements. The assertion of $B\implies A$ is captured by certainty of the conditional, the assertion of $P(A\vert B)=1$. Relating the directions of implication, Bayes' theorem represents a generalization of the contraposition law, which in classical propositional logic can be expressed as:

$(B\implies A)\iff (\neg A\implies \neg B)$.

In this relation between implications, the positions of $A$ resp. $B$ get flipped.

The corresponding formula in terms of probability calculus is Bayes' theorem, which in its expanded form involving the prior probability/base rate $a$ of only $A$, is expressed as:

$P(A\vert B)=P(B\vert A){\frac {a(A)}{P(B\vert A)\,a(A)+P(B\vert \neg A)\,a(\neg A)}}$.

Subjective logic

Bayes' theorem represents a special case of deriving inverted conditional opinions in subjective logic expressed as:

$(\omega _{A{\tilde {|}}B}^{S},\omega _{A{\tilde {|}}\lnot B}^{S})=(\omega _{B\vert A}^{S},\omega _{B\vert \lnot A}^{S}){\widetilde {\phi }}a_{A}$,

where ${\widetilde {\phi }}$ denotes the operator for inverting conditional opinions. The argument $(\omega _{B\vert A}^{S},\omega _{B\vert \lnot A}^{S})$ denotes a pair of binomial conditional opinions given by source $S$, and the argument \(a_{A}} denotes the prior probability (aka. the base rate) of $A$. The pair of derivative inverted conditional opinions is denoted $(\omega _{A{\tilde {|}}B}^{S},\omega _{A{\tilde {|}}\lnot B}^{S})$. The conditional opinion $\omega _{A\vert B}^{S}$ generalizes the probabilistic conditional $P(A\vert B)$, i.e. in addition to assigning a probability the source $S$ can assign any subjective opinion to the conditional statement $(A\vert B)$. A binomial subjective opinion $\omega _{A}^{S}$ is the belief in the truth of statement $A$ with degrees of epistemic uncertainty, as expressed by source $S$. Every subjective opinion has a corresponding projected probability $P(\omega _{A}^{S})$. The application of Bayes' theorem to projected probabilities of opinions is a homomorphism, meaning that Bayes' theorem can be expressed in terms of projected probabilities of opinions:

$P(\omega _{A{\tilde {|}}B}^{S})={\frac {P(\omega _{B\vert A}^{S})a(A)}{P(\omega _{B\vert A}^{S})a(A)+P(\omega _{B\vert \lnot A}^{S})a(\lnot A)}}$.

Hence, the subjective Bayes' theorem represents a generalization of Bayes' theorem.

Bayes theorem for 3 events

A version of Bayes' theorem for 3 events results from the addition of a third event $C$, with $P(C)>0$, on which all probabilities are conditioned:

$P(A\vert B\cap C)={\frac {P(B\vert A\cap C)\,P(A\vert C)}{P(B\vert C)}}$

Derivation

Using the chain rule

$P(A\cap B\cap C)=P(A\vert B\cap C)\,P(B\vert C)\,P(C)$

And, on the other hand

$P(A\cap B\cap C)=P(B\cap A\cap C)=P(B\vert A\cap C)\,P(A\vert C)\,P(C)$

The desired result is obtained by identifying both expressions and solving for $P(A\vert B\cap C)$.