Constructing probabilistic models

To use the techniques of probabilistic modelling, one must usually somehow specify the likelihood of the data given some parameters. Not all models are, however, probabilistic in nature and have naturally emerging likelihoods. Many models are defined as generative models by stating how the observed data could be generated from unknown parameters or latent variables.

For given data vectors $\mathbf{x}(t)$, a simple linear generative model could be written as

$$\mathbf{x}(t)= \mathbf{A}\mathbf{s}(t)$$ (3.4)

where $\mathbf{A}$ is an unobserved transformation matrix and vectors $\mathbf{s}(t)$ are unobserved hidden or latent variables. In some application areas the latent variables are also called sources or factors [6,35].

One way to turn such a generative model into a probabilistic model is to add a noise term to the Equation (3.4). This yields

$$\mathbf{x}(t)= \mathbf{A}\mathbf{s}(t)+ \mathbf{n}(t)$$ (3.5)

where the noise can, for example, be assumed to be zero-mean and Gaussian as in

$$\mathbf{n}(t)\sim N(0, \boldsymbol{\Sigma}_{\mathbf{n}}).$$ (3.6)

Here $\boldsymbol{\Sigma}_{\mathbf{n}}$ denotes the covariance matrix of the noise and it is usually assumed to be diagonal to make the different components of the noise independent.

This implies a likelihood for the data given by

$$p(\mathbf{x}(t)\vert \mathbf{A}, \mathbf{s}(t), \boldsymbol{\Sigm... ... N(\mathbf{x}(t);\; \mathbf{A}\mathbf{s}(t), \boldsymbol{\Sigma}_{\mathbf{n}}).$$ (3.7)

For a complete probabilistic model one also needs priors for all the parameters of the model. For instance hierarchical models and conjugate priors can be useful mathematical tools here but in the end the priors should be chosen to represent true prior knowledge on the solution of the problem at hand.