POSC 8410: Random Variables and Discrete Distributions

Bernoulli and Binomial Distributions

Will Horne

Where We Are

We can now talk about events and their probabilities
But events can be tricky. “The event that at least 3 of 5 respondents turn out” is a mouthful, and hard to do arithmetic on it.
Today we attach numbers to outcomes
- Precise probabilities
- Averages
- Descriptive Stats

Random Variables

From Outcomes to Numbers

A random variable is a function that assigns a number to every outcome in the sample space
That is really all it is: a rule for turning “what happened” into “a number”

\(S\) = all ways 5 sampled voters could turn out or not
\(X\) = how many of them turned out
The outcome (Yes, No, Yes, Yes, No) becomes \(X = 3\)

Random Variables vs. Outcomes

The outcome is what actually happened in the world
The random variable is a summary of it that we chose
Different questions call for different summaries of the same outcome

From one survey we might define \(X\) = number who turned out, \(Y\) = number who turned out and voted third party, \(Z\) = 1 if anyone did. Same sample space, three random variables.

You choose the random variable. It is a measurement decision.

Discrete and Continuous

A random variable is discrete if the values it takes with positive probability are finite or countably infinite
- Binary (turned out / did not), categorical, and count variables
Continuous variables can take any value in an interval
- Income, vote share, feeling thermometers (arguably)

Today and next time: discrete random variables.

What Is Random Here?

Uncertainty over the sample space becomes uncertainty over the value of \(X\)
The distribution of \(X\) describes that uncertainty completely
If \(X\) = days a randomly chosen student was absent, the distribution answers every question: \(P(X = 0)\), \(P(X > 10)\), and so on

Once you know the distribution, you know everything probabilistic about the variable.

Probability Mass Functions

Four Coins

Flip four fair coins. Let \(X\) be the number of heads.

What is \(P(X = 1)\)?
What is \(P(X = 2 \text{ or } 3)\)?
What is \(P(X > 4)\)?

The Distribution of X

Note the shape: outcomes in the middle are more likely, because there are more ways to get them. That is the counting rule showing up.

The Probability Mass Function

The PMF of a discrete random variable is

\[p_X(x) = P(X = x)\]

\(X = x\) is an event, so this is just a probability
The support of \(X\) is the set of values it takes with positive probability
The PMF assigns probability “mass” to each point in the support

What Makes a PMF Valid

Write the support of \(X\) as \(x_1, x_2, \ldots\) (for four coins, the values \(0, 1, 2, 3, 4\)).

Non-negative: \(p_X(x_j) > 0\) for every \(j\), and \(p_X(x) = 0\) for any \(x\) outside the support
Sums to one: \(\displaystyle \sum_{j} p_X(x_j) = 1\)
Any set of values \(A\): add up the mass at the support points inside \(A\), \(\displaystyle P(X \in A) = \sum_{j:\, x_j \in A} p_X(x_j)\)

These are the axioms from Week 3, rewritten for a random variable. Nothing new here!

The Other Way to Describe a Distribution

Often we do not want \(P(X = k)\). We want \(P(X \leq k)\) — at most three callbacks, no more than two defections.

The cumulative distribution function answers that:

\[F_X(x) = P(X \leq x)\]

Just add up the PMF from the bottom — it is the last slide’s \(P(X \in A)\) with \(A = (-\infty, x]\): \(\displaystyle F_X(x) = \sum_{j:\, x_j \leq x} p_X(x_j)\)
It is defined for every real \(x\), not only the values in the support: for four coins, \(F_X(2.5) = F_X(2) = 0.6875\)

PMF and CDF, Side by Side

Four fair coins. The CDF is a step function: flat, then it jumps by exactly the PMF’s height at each value in the support.

Reading the CDF

It never decreases, starts at 0, and ends at 1
The size of each jump is the PMF. When \(X\) takes integer values, like our coin counts: \(p_X(k) = F_X(k) - F_X(k-1)\)
Intervals are easy: \(P(a < X \leq b) = F_X(b) - F_X(a)\)

The two contain exactly the same information. The PMF is easier to look at; the CDF is easier to compute with — which is why R gives you both.

Designing a Randomization

You are designing an educational policy experiment with three arms: Control, T1, T2.

You decide to randomize schools by flipping four coins. Let \(X\) be the number of heads:

\(X \in \{0, 1\}\) → Control
\(X = 2\) → T1
\(X > 2\) → T2

Look at the PMF two slides back. What share of schools lands in each arm? Would you use this scheme?

The Bernoulli

The Simplest Random Variable

\(X \sim \text{Bern}(p)\) if

\[P(X = 1) = p, \quad P(X = 0) = 1 - p\]

Story: an indicator of success in a single trial with two outcomes.

Everything Binary Is Bernoulli

Any event \(A\) has an indicator variable \(I(A) \sim \text{Bern}(P(A))\)
Did this respondent turn out? Did this bill pass? Did this résumé get a callback?
A single résumé in the Bertrand and Mullainathan study is a Bernoulli trial with \(p\) = the callback probability for that name

The Bernoulli is where binary social science data starts. Logistic regression is a model for this type of data.

The Binomial

Many Trials at Once

Let \(X\) be the number of successes in \(n\) independent Bernoulli trials, each with success probability \(p\)
Then \(X \sim \text{Bin}(n, p)\)
\(\text{Bin}(1, p)\) is just \(\text{Bern}(p)\)
And if \(X \sim \text{Bin}(n, p)\), then \(n - X \sim \text{Bin}(n, 1-p)\) — counting failures instead of successes

Building the PMF

We want \(P(X = k)\): the probability of exactly \(k\) successes in \(n\) trials. Let’s build it in two.

First: how likely is one specific sequence with \(k\) successes?

The trials are independent, so multiply: \(k\) successes at probability \(p\) each, and \(n-k\) failures at \(1-p\) each.

\[p^{k}(1-p)^{n-k}\]

Building the PMF

Second: how many such sequences are there?

We need the number of ways to choose which \(k\) of the \(n\) trials succeeded. Order within the sequence does not matter — we only care which trials succeeded.

\[\binom{n}{k}\]

That is the binomial coefficient from Week 3. Counting never goes away!

The Binomial PMF

Every sequence with \(k\) successes is equally likely, and there are \(\binom{n}{k}\) of them:

\[P(X = k) = \binom{n}{k} p^{k}(1-p)^{n-k}, \qquad k = 0, 1, \ldots, n\]

\(p^{k}(1-p)^{n-k}\) — the probability of one specific sequence
\(\binom{n}{k}\) — how many such sequences exist

Binomials in R

dbinom(2, size = 4, prob = 0.5)   # the PMF:  P(X = 2)

[1] 0.375

pbinom(2, size = 4, prob = 0.5)   # the CDF:  P(X <= 2)

[1] 0.6875

rbinom(10, size = 4, prob = 0.5)  # ten draws from Bin(4, 0.5)

 [1] 3 2 2 3 1 1 2 0 4 2

The d / p / r prefixes work for every distribution in R: density (or mass), cumulative probability, and random draws.

Different Shapes

\(p\) controls where the mass sits; \(n\) controls how many outcomes are possible. Symmetric only when \(p = 0.5\).

Back to the Résumés

4,870 Bernoulli Trials

Each résumé either got a callback or did not. That is a Bernoulli trial, and the count per group is binomial.

resumes |>
  group_by(ethnicity) |>
  summarise(n = n(), callbacks = sum(callback), rate = round(mean(callback), 4))

# A tibble: 2 × 4
  ethnicity     n callbacks   rate
  <chr>     <int>     <int>  <dbl>
1 afam       2435       157 0.0645
2 cauc       2435       235 0.0965

235 callbacks against 157, out of 2,435 résumés each.

Suppose There Were No Difference

Imagine names truly did not matter, and every résumé had the same callback probability: the pooled rate:

pooled <- mean(resumes$callback)
round(pooled, 4)

[1] 0.0805

round(2435 * pooled, 1)   # expected callbacks per group

[1] 196

We would expect about 196 in each group. We saw 235 and 157.

How Surprising Is That?

We have a distribution for this. If \(X \sim \text{Bin}(2435, 0.0805)\):

1 - pbinom(234, size = 2435, prob = pooled)   # P(X >= 235)

[1] 0.002565257

pbinom(157, size = 2435, prob = pooled)       # P(X <= 157)

[1] 0.001585065

Each tail is roughly 2 in 1,000. If names truly did not matter, a split this lopsided would be very strange.

Seeing It

The grey distribution is what “no difference” predicts. Both observed values sit far out in the tails.

What We Just Did

We assumed something was not true: that names do not matter
We worked out what the data would look like if that assumption held
We found the observed data would be very unlikely under it

That is a (frequentist) hypothesis test. We will formalize these soon.

You Try

A committee of 8 senators votes independently, each supporting a bill with probability 0.6. What is the probability exactly 5 vote yes?
What is the probability the bill gets a majority (5 or more)?
A survey firm calls 20 households, each answering with probability 0.3. What is the expected number of responses? What is \(P(\text{no responses at all})\)?
For question 3, why might the independence assumption fail in practice?