Dropout as Bayesian procedure

Loading...

From the course by National Research University Higher School of Economics

Bayesian Methods for Machine Learning

210 ratings

National Research University Higher School of Economics

Bayesian Methods for Machine Learning

210 ratings

Course 3 of 7 in the Specialization Advanced Machine Learning

Bayesian methods are used in lots of fields: from game development to drug discovery. They give superpowers to many machine learning algorithms: handling missing data, extracting much more information from small datasets. Bayesian methods also allow us to estimate uncertainty in predictions, which is a really desirable feature for fields like medicine. When Bayesian methods are applied to deep learning, it turns out that they allow you to compress your models 100 folds, and automatically tune hyperparametrs, saving your time and money. In six weeks we will discuss the basics of Bayesian methods: from how to define a probabilistic model to how to make predictions from it. We will see how one can fully automate this workflow and how to speed it up using some advanced techniques. We will also see applications of Bayesian methods to deep learning and how to generate new images with it. We will see how new drugs that cure severe diseases be found with Bayesian methods.

From the lesson

Variational Autoencoder

Welcome to the fifth week of the course! This week we will combine many ideas from the previous weeks and add some new to build Variational Autoencoder -- a model that can learn a distribution over structured data (like photographs or molecules) and then sample new data points from the learned distribution, hallucinating new photographs of non-existing people. We will also the same techniques to Bayesian Neural Networks and will see how this can greatly compress the weights of the network without reducing the accuracy.

Learning with priors5:49

Dropout as Bayesian procedure5:57

Sparse variational dropout5:42

Meet the Instructors

Daniil Polykovskiy
Researcher
HSE Faculty of Computer Science
Alexander Novikov
Researcher
HSE Faculty of Computer Science

[MUSIC]

In this section we will review Dropout, and

review its connections with a Bayesian framework.

So Dropout was invented in 2011, and became popular regularization technique.

We know that it works.

We know that its params are fitting.

And the essence of Dropout is actually just injection of the noise to

the variance, or to the activations at each iteration of our training.

The magnitude of this noise is defined by user, and is usually called dropout rate.

The noise can be different.

For example, it can be Bernoulli noise, then we are talking about binary dropout.

Or it can be Gaussian noise, then we tell about Gaussian dropout.

Let us review Gaussian dropout in details.

At each iteration of training, we generate Gaussian noise with a mean of 1 and

variance Alpha.

Then I multiply each weight Theta ij to Epsilon ij,

Epsilon ij is this noise generated from Gaussian distribution,

and obtain noisified versions of the waves, wij.

Let us consider Gaussian dropout in details.

Additional iteration of training,

we multiply always Theta ij by Gaussian noise, Epsilon ij.

Epsilon ij goes from a Gaussian distribution with a mean of 1 and

variance Alpha.

Then I obtain noisified versions of the weights, wij.

And finally, we compute stochastic gradients of a triangle

gradient given these noisified weights, w.

But then we obtain exactly the same stochastic gradient as it would

be if we optimized the expectation with respect to Gaussian distribution over w.

With weight Theta, and variance Alpha Theta squared, obtaining our likelihood.

So the distribution itself is fully factorized.

To show it, let us first perform a little regularization trick.

So we change the distribution over w to the distribution over Epsilon.

Epsilon now has a min of 1 and variance of Alpha, and it is still fully factorized.

And the old likelihood is computed at the point Theta times Epsilon.

Now our probability density doesn't depend on theta, and

we may move the differentiation inside of our integral.

And then we may change the integral to its Monte Carlo estimate, and

obtain exactly the same expression as it was on the previous slide.

So now we know that Gaussian dropout optimizes the following objective.

This expectation with respect to the distribution over w,

the distribution is fully factorized and its Gaussian, with a mean of Theta ij and

variance Alpha Theta ij squared.

And the expectation is computed from obtaining a likelihood.

So this looks pretty much the same like the first term in ELBO, where as

an regularization approximation we used fully factorized Gaussian distribution.

But where is the second term?

Where is the KL divergence?

Remember that ELBO consists of 2 terms, the data term and

the negative KL divergence, that is our regularizer.

In Gaussian dropout,

we have shown that we are optimizing with respect to Theta of just the first term.

So we managed to find such prior distribution, p(W),

that the second term will depend only on Alpha, and it will not depend on Theta.

Now I've managed to prove that this two procedures are exactly equivalent.

So remember that in Gaussian dropout, Alpha is assumed to be fixed.

And if Alpha is fixed, then optimization of ELBO is equivalent to

the optimization of just the first term with respect to Theta.

Surprisingly such prior distribution exists, and

it is known from information theory.

So this is a so-called improper log-uniform prior.

It is fully factorized again,

and each of its factors is proportional to 1 over absolute value of wij.

This is improper distribution, so it can not be normalized.

Nevertheless, it has several quite nice purpose.

For example, if we consider the algorithm of absolute value of wij, in other words,

it is to show that it will be uniformly distributed from minus to plus infinity.

And again, this is improper probability distribution.

For us, it is important that this prior distribution, roughly speaking,

analysis the precision with which we are trying to find wij.

We may easily show that the counter divergence between our Gaussian numeration

approximation and such kind of prior distribution will be dependent only

on Alpha, and will not depend on Theta.

The counter divergence is still intractable function, but

now it is a function of just one-dimensional parameter Alpha,

and it can be easily approximated by smooth, differentiable function.

So in the figure you see the black dots.

This is the exact values of our KL divergence,

given different values of Alpha.

And the red curve is our smooth, differentiable approximation.

And the existence of this smooth, differential approximation means that

potentially we may optimize the KL divergence with expect to Alpha.

And hence, we optimize the ELBO with respect to both Theta and Alpha.

And this is what we are going to do in the next lecture.

So to conclude, dropouts is a popular regularization technique.

The essence of dropout is simple injection of the noise in each iteration you obtain.

In this lecture, we have shown that one of the popular dropouts,

so-called Gaussian dropout,

is exactly equivalent to a special kind of generalization Bayesian procedure.

And this understanding, the understanding that dropout is a particular case

of Bayesian inference, allows us to construct various generalizations of

dropout that may possess several quite interesting properties.

We'll review one of them in the next lecture.

[MUSIC]

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2018 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play