CS205: Building with Artificial Intelligence

Bayes' theorem is stated mathematically as the following equation:

$P(A\vert B)={\frac {P(B\vert A)P(A)}{P(B)}}$

where $A$ and $B$ are events and $P(B)\neq 0$.

• $P(A\vert B)$ is a conditional probability: the probability of event $A$ occurring given that $B$ is true. It is also called the posterior probability of $A$ given $B$.
• $P(B\vert A)$ is also a conditional probability: the probability of event $B$ occurring given that $A$ is true. It can also be interpreted as the likelihood of $A$ given a fixed $B$ because $P(B\vert A)=L(A\vert B)$.
• $P(A)$ and $P(B)$ are the probabilities of observing $A$ and $B$ respectively without any given conditions; they are known as the prior probability and marginal probability.

Proof

Visual proof of Bayes' theorem

For events

Bayes' theorem may be derived from the definition of conditional probability:

$P(A\vert B)={\frac {P(A\cap B)}{P(B)}},{\text{ if }}P(B)\neq 0$,

where $P(A\cap B)$ is the probability of both A and B being true. Similarly,

$P(B\vert A)={\frac {P(A\cap B)}{P(A)}},{\text{ if }}P(A)\neq 0$.

Solving for $P(A\cap B)$ and substituting into the above expression for $P(A\vert B)$ yields Bayes' theorem:

$P(A\vert B)={\frac {P(B\vert A)P(A)}{P(B)}},{\text{ if }}P(B)\neq 0$.

For continuous random variables

For two continuous random variables X and Y, Bayes' theorem may be analogously derived from the definition of conditional density:

$f_{X\vert Y=y}(x)={\frac {f_{X,Y}(x,y)}{f_{Y}(y)}}$

$f_{Y\vert X=x}(y)={\frac {f_{X,Y}(x,y)}{f_{X}(x)}}$

Therefore,

$f_{X\vert Y=y}(x)={\frac {f_{Y\vert X=x}(y)f_{X}(x)}{f_{Y}(y)}}$.

General case

Let $P_{Y}^{x}$ be the conditional distribution of $Y$ given $X=x$ and let $P_{X}$ be the distribution of $X$. The joint distribution is then $P_{X,Y}(dx,dy)=P_{Y}^{x}(dy)P_{X}(dx)$. The conditional distribution $P_{X}^{y}$ of $X$ given $Y=y$ is then determined by

$P_{X}^{y}(A)=E(1_{A}(X)|Y=y)$

Existence and uniqueness of the needed conditional expectation is a consequence of the Radon–Nikodym theorem. This was formulated by Kolmogorov in his famous book from 1933. Kolmogorov underlines the importance of conditional probability by writing "I wish to call attention to ... and especially the theory of conditional probabilities and conditional expectations ..." in the Preface. The Bayes theorem determines the posterior distribution from the prior distribution. Bayes' theorem can be generalized to include improper prior distributions such as the uniform distribution on the real line. Modern Markov chain Monte Carlo methods have boosted the importance of Bayes' theorem including cases with improper priors.