Estimate Gaussian Mixture Model (GMM) - Python Example

In this repository, I'll introduce 2 methods for Gaussian Mixture Model (GMM) estimation - EM algorithm (expectation-maximization algorithm) and variational inference (variational Bayes).<br> To make you have a clear picture, I'll also give you mathematical descriptions, with several lines of code in Python.

• EM Algorithm
• Variational Inference (Variational Bayesian)

GMM formula has summation (not multiplication) in distribution, and the log likelihood will then lead to complex expression in regular maximum likelihood estimation (MLE). These 2 methods will then address this concern by procedural iterative algorithms (which approximate the optimal solutions).

EM algorithm is an iterative method based on maximum likelihood framework.<br> This iterative algorithm is simple and light-weight compared to variational Bayesian.<br> In complex system, however, it will be infeasible (intractable) to evaluate the posterior distribution or compute the expectation of the complete-data log likelihood with respect to the posterior distribution of the latent variables, so EM algorithm cannot be used in such cases.<br> Variational Bayesian (variational inference) addresses this problem by finding an approximation for the posterior distribution as well as for the model evidence.

Note : EM algorithms also have difficulties in specific cases, because it depends on likelihood approach.<br> For instance, when some data (observation) is exactly same as a mean of some element in GMM, it might have a singularity, in which the likelihood goes to infinity at σ = 0. (This won't occur in a single Gaussian distribution.) When the number of data is insufficient, EM algorithm might also over-estimate the results.<br> In such cases, variational Bayesian avoids over-estimating and over-fitting by applying Bayesian.<br> See here for the caveat of likelihood approach.

Here I show you the implementation from scratch in Python with mathematical explanations. But, with Scikit-Learn package in Python, you can also use functions for both EM algorithm (sklearn.mixture.GaussianMixture) and variational Bayesian (sklearn.mixture.BayesianGaussianMixture) in GMM.<br> By knowing mathematical background and implementation, these methods can also be applied to other distributions - such as, mixtures of Bernoulli distributions, hidden Markov Models (HMM), etc. (See here for applying EM algorithms in HMM and LDS.)