Probability


 * Probability**

Statistics Review

Pros Naval Computer Vision & Machine Intelligence Group Department of Computer Science University of the Philippines Pros Naval (University of the Philippines) Statistics Review 1 / 28

Outline 1 Sample Spaces and Events Pros Naval (University of the Philippines) Statistics Review 2 / 28

Sample Spaces and Events Sample Spaces and Events Sample Space
 * set of possible outcomes of an experiment

 points ! 2
 * sample outcomes or elements

 subsets of
 * events

Example Toss a coin twice.

= fHH; HT; TH; TTg Event that first toss is heads is A = fHH; HTg: Pros Naval (University of the Philippines) Statistics Review 3 / 28

Sample Spaces and Events Sample Spaces and Events Example Let ! be outcome of a measurement of some physical quantity (e.g. temperature)  =R= (􀀀1;1) = R not accurate. Temperature has a lower bound but no harm when sample space is larger than needed.  event A that measurement is larger than 10 but less than or equal to 23 is A = (10; 23]: Example Toss a coin forever.

=f!= (!1; !2; :::; !i; :::) : !i 2 fH; TggEvent E that first head appears on the the third toss: E = f! = (!1; !2; :::; !i; :::) : !1 = T; !2 = T; !3 = H; !i 2 fH; Tg; i > 3g Pros Naval (University of the Philippines) Statistics Review 4 / 28

Sample Spaces and Events Sample Spaces and Events Complement of Event A Ac = f! 2
 * ! =2 Ag

 complement of A: "not A"  complement of = ; Union of Events A and B A [ B = f! 2
 * ! 2 A or ! 2 B or ! 2 bothg

 Think of A [ B as "A or B".  If A1; A2; ::: is a sequence of sets then 1[ i=1 Ai = f! 2
 * ! 2 Ai for a least one ig

Pros Naval (University of the Philippines) Statistics Review 5 / 28

Sample Spaces and Events Sample Spaces and Events Intersection of Events A and B A \ B = f! 2
 * ! 2 A and ! 2 Bg

 Think of A \ B as "A and B".  A \ B is also written as AB or (A; B)  If A1; A2; ::: is a sequence of sets then 1\ i=1 Ai = f! 2
 * ! 2 Ai for all ig

Set Difference A 􀀀 B = f! : ! 2 A; ! =2 Bg  If every element of A is also contained in B, we write A  B.  If A is a finite set, then jAj denotes the number of elements in A. Pros Naval (University of the Philippines) Statistics Review 6 / 28

Sample Spaces and Events Sample Spaces and Events Disjoint or Mutually Exclusive Sets A1; A2; ::: are disjoint or mutually exclusive if Ai \ Aj = ; whenever i 6= j: Partition A partition of is a sequence of disjoint sets A1; A2; ::: such that 1[ i=1 Ai = Indicator Function Given an event A, the indicator function of A is IA(!)  I(! 2 A) =  1 if ! 2 A 0 if ! =2 A Pros Naval (University of the Philippines) Statistics Review 7 / 28

Sample Spaces and Events Sample Spaces and Events Monotone Increasing/Decreasing A sequence of sets A1; A2; ::: is monotone increasing if A1  A2  ::: and we define lim n!1 An = 1[ i=1 Ai: A sequence of sets A1; A2; ::: is monotone decreasing if A1  A2  ::: and we define lim n!1 An = 1\ i=1 Ai: In both cases, we write An ! A. Pros Naval (University of the Philippines) Statistics Review 8 / 28

Sample Spaces and Events Probability We assign a real number P(A) to every event A called the probability of A.  P: probability distribution Definition (Probability) A function P that assigns a real number P(A) to each event A is a probability distribution if it satisfies the following axioms: Axiom 1: P(A)  0 for every A Axiom 2: P = 1 Axiom 3: If A1; A2; ::: are disjoint then P

1[ i=1 Ai ! = 1X i=1 P(Ai): Pros Naval (University of the Philippines) Statistics Review 9 / 28

Sample Spaces and Events Probability Interpretations of P(A) Frequentist: proportion of times that A is true in repetitions (e.g., probability of heads in coin toss = 1/2 when repeated an infinite number of times)  an idealization (similar to the concept of line in geometry) Degree of Belief: P(A) measures an observer’s strength of belief that A is true. Difference in interpretation will not matter much until we deal with statistical inference. For both interpretations, the three axioms must hold. Pros Naval (University of the Philippines) Statistics Review 10 / 28

Sample Spaces and Events Probability Some Properties of P derivable from the Axioms  P = 0  A  B =) P(A)  P(B)  0  P(A)  1  P(Ac) = 1 􀀀 P(A)  A \ B = ; =) P(A [ B) = P(A) + P(B) Pros Naval (University of the Philippines) Statistics Review 11 / 28

Sample Spaces and Events Probability Lemma For any events A and B P(A [ B) = P(A) + P(B) 􀀀 P(AB): Proof. Write A [ B = (ABc) [ (AB) [ (AcB) and note that the events are disjoint. Use repeatedly the fact that P is additive for disjoint events. P(A [ B) = P((ABc) [ (AB) [ (AcB)) = P(ABc) + P(AB) + P(AcB) + P(AB) 􀀀 P(AB) = P((ABc) [ (AB)) + P((AcB) [ (AB)) 􀀀 P(AB) = P(A) + P(B) 􀀀 P(AB) Pros Naval (University of the Philippines) Statistics Review 12 / 28

Sample Spaces and Events Probability Example Two coin tosses. Let H1 be the event that head occurs on toss 1 and let H2 be the event that head occurs on toss 2. If all outcomes are equally likely, then P(H1 [ H2) = P(H1) + P(H2) 􀀀 P(H1H2) = 1=2 + 1=2 􀀀 1=4 = 3=4: Pros Naval (University of the Philippines) Statistics Review 13 / 28

Sample Spaces and Events Probability Theorem (Continuity of Probabilities) If An ! A then P(An) ! P(A) as n ! 1: Proof. Suppose that An is monotone increasing so that A1  A2  :::. Let A = limn!1 An = S1 i=1 Ai: Define B1 = A1; B2 = f! 2
 * ! 2 A2; ! =2 A1g; B3 = f! 2
 * ! 2 A3; ! =2 A2; ! =2 A1g,...

It can be shown that B1; B2; ::: are disjoint. An = Sn i=1 Ai = Sn i=1 Bi for each n and S1 i=1 Bi = S1 i=1 Ai. From Axiom 3 P(An) = P [n i=1 Bi ! = Xn i=1 P(Bi) and hence, using Axiom 3 again, lim n!1 P(An) = lim n!1 Xn i=1 P(Bi) = 1X i=1 P(Bi) = P

1[ i=1 Bi ! = P(A) Pros Naval (University of the Philippines) Statistics Review 14 / 28

Sample Spaces and Events Probability on Finite Sample Spaces Suppose that the sample space = f!1; !2; :::; !ng is finite and each outcome is equally likely, then P = jAj j j where jAj denotes the number of elements in A. This is called the uniform probability distribution. Example Toss a die twice. = f(i; j) : i; j 2 f1; :::; 6gg  if each outcome is equally likely then P(A) = jAj=36  probability that sum of the die is 11 is 2/36 since there are two outcomes that correspond to this event, namely f(5; 6); (6; 5)g. Pros Naval (University of the Philippines) Statistics Review 15 / 28 Sample Spaces and Events Probability on Finite Sample Spaces To compute probabilities, we need to count the number of points in an event A. Given n objects, the number of ordering these objects is n! = n(n 􀀀 1)(n 􀀀 2)    3  2  1: Define 0! = 1 The number of distinct ways of choosing k objects from n is  n k  = n! (n 􀀀 k)!k! Note:  n 0  =  n n  = 1  n k  =  n n 􀀀 k  Example We have a class of 20 people and we want to select a committee of 3 students. There are 20 3  = 20! 3!17! = 20  19  18 3  2  1 = 1140 possible committees Pros Naval (University of the Philippines) Statistics Review 16 / 28

Sample Spaces and Events Independent Events Two events A and B are independent if P(AB) = P(A)P(B) and we write A q B. A set of events fAi : i 2 Ig is independent if P

\ i2J Ai ! = Y i2J P(Ai) for every finite subset J of I. Independence can arise in two ways 1 By assumption: we explicitly assume that two events are independent  example: toss a coin twice – assume that tosses are independent since coin has no memory of the first toss 2 By derivation: derive independence by verifying that P(AB) = P(A)P(B). Example Toss a fair die. Let A = f2; 4; 6g, B = f1; 2; 3; 4g, then A \ B = f2; 4g. P(AB) = 2=6 = P(A)P(B) = 3=6  4=6 Therefore, A and B are independent. Pros Naval (University of the Philippines) Statistics Review 17 / 28

Sample Spaces and Events Independent Events Suppose that A and B are disjoint events, each with positive probability. Can they be independent ? NO.  here P(A)P(B) > 0 yet P(AB) = P = 0 Example Toss a fair coin 10 times. Let A = “at least one head". Let Tj be the event that tails occur on the j-th toss. Then P(A) = 1 􀀀 P(Ac) = 1 􀀀 P(\all tails") = 1 􀀀 P(T1  T2    T10) = 1 􀀀 P(T1)P(T2)    P(T10) = 1 􀀀  1 2 10  0:999 Pros Naval (University of the Philippines) Statistics Review 18 / 28

Sample Spaces and Events Independent Events Example Two persons take turns trying to sink a basketball into a net. Person 1 succeeds with probability 1=3 while Person 2 succeeds with probability 1=4. What is the probability that Person 1 succeeds before Person 2 ? Let E denote the event of interest. Let Aj be the event that the first success is by Person 1 and that it occurs on trial j. Note that A1; A2; ::: are disjoint and that E = 1[ j=1 Aj: Hence P(E) = 1X j=1 P(Aj) P(A1) = 1=3. A2 occurs if we have the sequence Person 1 misses, Person 2 misses, Person 1 succeeds. P(A2) = (2=3)(3=4)(1=3) = (1=2)(1=3). Following this logic we see that P(Aj) = (1=2)j􀀀1(1=3). Hence P(E) = 1X j=1 (1=3)(1=2)j􀀀1 = (1=3) 1X j=1 (1=2)j􀀀1 = 2=3: (We used the fact that if 0 < r < 1 then P1 j=1 rj = rj=(1 􀀀 r):) Pros Naval (University of the Philippines) Statistics Review 19 / 28

Sample Spaces and Events Independent Events Summary of Independence 1 A and B are independent if and only if P(AB) = P(A)P(B). 2 Independence is sometimes assumed and sometimes derived. 3 Disjoint events with positive probabilities are not independent. Pros Naval (University of the Philippines) Statistics Review 20 / 28 Sample Spaces and Events Conditional Probability Definition (Conditional Probability) If P(B) > 0 then the conditional probability of A given B is P(AjB) = P(AB) P(B)  think of P(AjB) as the fraction of time A occurs among those in which B occurs. For any fixed B such that P(B) > 0 then  P(jB) is a probability  P(AjB)  0  P( jB) = 1  if A1; A2; ::: are disjoint P

1[ i=1 AijB ! = 1X i=1 P(AijB) Pros Naval (University of the Philippines) Statistics Review 21 / 28

Sample Spaces and Events Conditional Probability In general, it is NOT TRUE that P(AjB [ C) = P(AjB) + P(AjC)  rules of probability apply to events on the left of the bar Also NOT TRUE: P(AjB) = P(BjA) Example Probability of spots given that you have measles is 1 but the probability that you have measles given that you have spots is not 1.  often made in legal cases that is is sometimes called the presentor’s fallacy Pros Naval (University of the Philippines) Statistics Review 22 / 28

Sample Spaces and Events Conditional Probability Example A medical test for disease D has outcomes + and 􀀀. The probabilities are D Dc + 0.009 0.099 - 0.001 0.891 From definition of conditional probability P(+jD) = P(+ \ D) P(D) = 0:009 0:009 + 0:001 = 0:9 and P(􀀀jDc) = P(􀀀 \ Dc) P(Dc) = 0:891 0:891 + 0:099  0:9 Suppose you go for a test and get a positive. What is the probability that you have the disease ? (Most people would answer 0:9.) The correct answer is P(Dj+) = P(+ \ D) P(+) = 0:009 0:009 + 0:099  0:08: (Lesson: Don’t trust your intuition !) Pros Naval (University of the Philippines) Statistics Review 23 / 28

Sample Spaces and Events Conditional Probability Lemma If A and B are independent events then P(AjB) = P(A): Also, for any pair of events A and B, P(AB) = P(AjB)P(B) = P(BjA)P(A): Another interpretation of independence: knowing B does not change the probability of A. Example Draw two cards from a deck without replacement and let B be the event that the second draw is the Queen of Diamonds. Then P(AB) = P(A)P(BjA) = (1=52)  (1=51): Pros Naval (University of the Philippines) Statistics Review 24 / 28

Sample Spaces and Events Conditional Probability Summary of Conditional Probability 1 If P(B) > 0 then P(AjB) = P(AB) P(B) 2 P(jB) satisfies the axioms of probability for fixed B. (In general, P(Aj) does not satisfy the axioms of probability for fixed A.) 3 In general, P(AjB) 6= P(BjA): 4 A and B are independent if and only if P(AjB) = P(A): Pros Naval (University of the Philippines) Statistics Review 25 / 28

Sample Spaces and Events Bayes’ Theorem We need a preliminary result. Theorem (The Law of Total Probability) Let A1; A2; :::; Ak be a partition of . Then for any event B, P(B) = Xk i=1 P(BjAi)P(Ai) Proof. Define Cj = BAj and note that C1;C2; :::;Ck are disjoint and that B = Skj =1 Cj. Hence P(B) = X j P(Cj) = X j P(BAj) = X j P(BjAj)P(Aj) since P(BAj) = P(BjAj)P(Aj) from the definition of conditional probability. Pros Naval (University of the Philippines) Statistics Review 26 / 28

Sample Spaces and Events Bayes’ Theorem Theorem (Bayes’ Theorem) Let A1; :::; Ak be a partition of such that P(Ai) > 0 for each i. If P(B) > 0, then for each i = 1; :::; k P(AijB) = PP(BjAi)P(Ai) j P(BjAj)P(Aj)  P(Ai) - prior probability of A  P(AijB) - posterior probability of A Proof. Apply definition of conditional probability twice, followed by the law of total probability: P(AijB) = P(AiB) P(B) = P(BjAi)P(Ai) P(B) = PP(BjAi)P(Ai) j P(BjAj)P(Aj) Pros Naval (University of the Philippines) Statistics Review 27 / 28

Sample Spaces and Events Bayes’ Theorem Example I divide my email into three categories: A1 = \spam", A2 = \low priority", A3 = \high priority". From previous experience, I find that P(A1) = 0:7, P(A2) = 0:2, P(A3) = 0:1. Let B the event that the email contains the word "free". From previous experience P(BjA1) = 0:9, P(BjA2) = 0:01, P(BjA3) = 0:01. (note: 0:9 + 0:01 + 0:01 6= 1.) Answer: P(A1jB) = (0:9)(0:7) (0:9)(0:7) + (0:01)(0:2) + (0:01)(0:1) = 0:995 Pros Naval (University of the Philippines) Statistics Review 28 / 28

=Probability=

How Likely
In the real world events can not be predicted with total certainty. The best we can do is say how **likely** they are to happen, using the idea of probability. When a coin is tossed, there are two possible outcomes: We say that the probability of the coin landing **H** is 1/2. Similarly, the probability of the coin landing **T** is 1/2. ||
 * [[image:http://www.mathsisfun.com/data/images/head-tails.jpg width="81" height="70" caption="pair of dice"]] || ===Tossing a Coin===
 * heads (H) or
 * tails (T)

When a single [|die] is thrown, there are six possible outcomes: 1, 2, 3, 4, 5, 6. The probability of throwing any one of these numbers is 1/6. ||
 * [[image:http://www.mathsisfun.com/geometry/images/pair-dice2.jpg width="150" height="112" caption="pair of dice"]] || ===Throwing Dice===

Probability
In general:
 * Probability of an event happening = ||  || Number of ways it can happen ||
 * ^  ||^   || Total number of outcomes ||
 * ^  ||^   || Total number of outcomes ||

Example: the chances of rolling a "4" with a die Example: there are 5 marbles in a bag: 4 are blue, and 1 is red. What is the probability that a blue marble will be picked?
 * Number of ways it can happen: 1** (there is only 1 face with a "4" on it)
 * Total number of outcomes: 6** (there are 6 faces altogether)
 * So the probability = || 1 ||
 * ^  || 6 ||
 * ^  || 6 ||
 * Number of ways it can happen: 4** (there are 4 blues)
 * Total number of outcomes: 5** (there are 5 marbles in total)
 * So the probability = || 4 || = **0.8** ||
 * ^  || 5 ||^   ||
 * ^  || 5 ||^   ||

Probability Line
You can show probability on a [|Probability Line]: The probability is always between 0 and 1

Probability is Just a Guide
Probability does not tell us exactly what will happen, it is just a guide Example: toss a coin 100 times, how many Heads will come up? Probability says that heads have a 1/2 chance, so we would **expect 50 Heads**. But when you actually try it out you might get 48 heads, or 55 heads ... or anything really, but in most cases it will be a number near 50.

Words
Some words have special meaning in Probability:

Tossing a coin, throwing dice, seeing what pizza people choose are all examples of experiments.
 * Experiment:** an action where the result is uncertain.

Example: choosing a card from a deck There are 52 cards in a deck (not including Jokers) So the **Sample Space is all 52 possible cards**: {Ace of Hearts, 2 of Hearts, etc... }
 * Sample Space:** all the possible outcomes of an experiment

The Sample Space is made up of Sample Points:

Example: Deck of Cards "King" is not a sample point. As there are 4 Kings that is 4 different sample points.
 * Sample Point:** just one of the possible outcomes
 * the 5 of Clubs is a sample point
 * the King of Hearts is a sample point

Example Events: An event can include one or more possible outcomes:
 * Event:** a single result of an experiment
 * Getting a Tail when tossing a coin is an event
 * Rolling a "5" is an event.
 * Choosing a "King" from a deck of cards (any of the 4 Kings) **is** an event
 * Rolling an "even number" (2, 4 or 6) is also an event

A Sample Point is just one possible outcome. And an Event can be one **or more** of the possible outcomes. ||
 * [[image:http://www.mathsisfun.com/data/images/probability-sample-space.gif width="272" height="128"]] ||  || The Sample Space is all possible outcomes.

Hey, let's use those words, so you get used to them:

Example: Alex decide to see how many times a "double" would come up when throwing 2 dice.
Each time Alex throws the 2 dice is an **Experiment**. It is an Experiment because the result is uncertain.

The **Event** Alex is looking for is a "double", where both dice have the same number. It is made up of these **6 Sample Points**: {1,1} {2,2} {3,3} {4,4} {5,5} and {6,6}

The **Sample Space** is all possible outcomes (**36 Sample Points**): {1,1} {1,2} {1,3} {1,4} ... {6,3} {6,4} {6,5} {6,6}

These are Alex's Results:
 * Experiment || Is it a Double? ||
 * {3,4} || No ||
 * {5,1} || No ||
 * {2,2} || **Yes** ||
 * {6,3} || No ||

After 100 **Experiments**, Alex had 19 "double" **Events** ... is that close to what you would expect?

[|An Experiment with a Die] [|An Experiment with Dice] = = =Reference=

http://www.mathsisfun.com