Lectures on Graphical Models

Christopher Bishop has an excellent set (1, 2, and 3) of introductory lectures on "Probabilistic Graphical Models". They are well-motivated and cover topics that include:

• directed and undirected graphs
• conditional independence
• factor graphs
• inference using factor graphs and sum/product rules

Joints from Marginals: Compilation

For convenience, here is a link to the three blogs in this series in one place.

1. A technique for solving the problem in a special case

2. The reason this technique works

3. The corners/edges of this technique, or how it fails for non-Gaussian marginals

Joint from Marginals: non-Gaussian Marginals

In a previous post, I asked the question if the method described here can be used with non-Gaussian distributions.

Let us explore that by considering two independent zero mean, unit variance distributions that are not Gaussian. Let us sample \(x_1\) from a triangular distribution, and \(x_2\) from a uniform distribution.

We consider a triangular distribution with zero mean and unit variance, which is symmetric about zero (spans -sqrt(6) to  +sqrt(6)). Similarly, we consider a symmetric uniform distribution, which spans -sqrt(3) to  +sqrt(3).

Samples from these independent random variables are shown below.

When we use a correlation coefficient of 0.2, and use the previous recipe, we get correlated random variables with zero mean and the same covariance matrix, but ...

... the marginals are not exactly the same!

This is evident when we increase the correlation coefficient to say 0.5.

The sharp edges of the uniform distribution get smoothened out.

Did the method fail?

Not really. If you paid attention, the method is designed to preserve the mean and the covariance matrix (which is does). It doesn't really guarantee the preservation of the marginal distributions. 

Joint from Marginals: Why?

In the previous blog post, we saw a special example in which we were able to sample random variables from a joint 2D-Gaussian distribution from the marginals and the correlation coefficient.

I listed a simple method, which seemed to work like magic. It had two simple steps:

• Cholesky decomposition of the covariance matrix, C(Y)
• Y = LX, where X are independent random variables

The question is, why did the method work?

Note that the covariance matrix of random variables with zero mean and unit standard deviation can be written as, \(C(Y) = E(Y Y')\), where \(E()\) denotes the expected value of a random variable. Thus, we can write the expected value of the Y generated by the method as, \[\begin{align*} E(Y Y') & = E\left(LX (LX)'\right)\\ & = L E(XX') L' \\ & = L I L'\\ & = LL' = C.\end{align*}.\] Here we used the fact that the covariance of X is an identity matrix by design.

Note that this method preserves the covariance matrix (and hence the standard deviation of the marginals).

Does it preserve the mean?

Yes. \(E(Y) = E(LX) = L E(X) = 0.\)

Do the marginals have to be normal for this method to work? Would this work for any distribution (with zero mean, and unit standard deviation)?

We will explore this in a subsequent blog.

Randomness, Gullibility, and Fraud

On a recent EconTalk episode on randomness, the guest, Campbell Harvey provided an interesting example at the intersection of probability and gullibility. I paraphrase:

Suppose you get an email predicting the winner of a football game. You ignore it, but after the game, you notice that the prediction did come out true.

Probably lucky, you say to yourself.

The next week you get a similar email predicting the winner of another football game. You later find out that that prediction also came true.

This happens for ten weeks in a row.

The emailer correctly predicts the winner in every game.

You sign up for the newsletter!

But what's going on behind the scenes. The emailer spams 100,000 email address; in half of them he predicts Team A is going to win, and in the other half he predicts that Team B is going to win.

One set gets the "correct prediction".

The following week he spams the 50,000 recipients who got the correct prediction. Again, he goes 50-50.

You see where this is going.

After 10 weeks, by pure chance, there will be 100 "suckers" who think you got 10/10.

Don't be a sucker!