0:00

Hello, neuromusketeers from around the world.

Â In the previous lecture, we learned that neurons don't use

Â Facebook or LinkedIn to connect among themselves.

Â Rather, they use synapses.

Â We learned how to model a synapse on a computer and we were even entertained by

Â two neurons that performed a ballroom dance by following each other and then,

Â danced in synchrony by inhibiting each other.

Â In this lecture, we will learn how to model larger networks of neurons.

Â When modeling networks of neurons,

Â one of the choices you'll have to make as

Â a computational neuroscientist is deciding whether you should

Â use spiking models of neurons or firing rate based models for neurons.

Â Let's look at the advantages and disadvantages of these two choices.

Â If you decide to use spiking neurons for your network model,

Â the advantages that you get include the fact that you can now

Â model computation and learning based on spike timing.

Â So, you can model phenomena such as synchrony between neurons,

Â which you would not be able to model if you did not have spikes as the output of neurons.

Â On the other hand, the disadvantages include the fact that now

Â you have to model these differential equations and so if you have a very,

Â very large network of these neurons,

Â than it might be computationally expensive or in

Â some cases maybe even impossible to simulate, very,

Â very large networks of perhaps millions and millions of

Â neurons because you have to simulate these differential equations on your computer.

Â On the other hand, you might decide to use neurons with firing-rate outputs.

Â So, this means that the output of the neurons is not zero or one or a spike,

Â but real valued outputs denoting the firing-rate of the neuron.

Â The advantages include the fact that now you can have

Â greater efficiency and so you will be able to simulate for example,

Â very large networks of neurons.

Â On the other hand, since you are using firing rates as the output,

Â you will not be able to model phenomena based on spike timing or synchrony.

Â The question that we can ask now is,

Â how are these two approaches related?

Â The firing rate model can be related to the spiking model by

Â using the linear filter model of a synapse that we covered in the last lecture.

Â If you recall, we had a synapse b that was

Â receiving an input Spike train that we called Rho_b,

Â and the Rho_b was just a representation of a spike train such as this one.

Â And if we can model the synapse b using a filter,

Â which we call K(t), so for example,

Â K(t) could be an exponential filter such as this,

Â then the synaptic conductance at b could be modeled using

Â a linear filter equation that is given down here.

Â And so, this particular equation just implements the filtering operation,

Â so you simply replace each spike by the filter for

Â the synapse and you get a conductance that looks something like this.

Â We can now go from the basic synapse model that we had on

Â the previous slide to the model for multiple synapses.

Â So, here's a cartoon of a neuron receiving inputs from n different other neurons.

Â And so each of these synapses has a synaptic weight that we're calling W_1,

Â W_2 all the way up to W_N.

Â And each of these synapses now gets spike trains given by Rho_1,

Â Rho_2, all the way up to Rho_N.

Â And if we assume that there are no known linear interactions

Â between these synapses then the total synaptic current,

Â the total input to the neuron,

Â is given by just a summation of each of

Â the individual inputs coming in from each of these synapses.

Â And so the total input to the neuron then is

Â just the summation given by this particular equation.

Â So are you ready to make that leap from the spiking model to the firing rate model?

Â Some people say it's almost a leap of faith

Â and they compare it to converting from one religion to the other.

Â But I think that's a little bit extreme.

Â In any case here's one network and here are these spike trains from the input neurons.

Â And here's the equation that we had in the previous slide which

Â characterizes the total amount of input that the neuron is getting from all its synapses.

Â Now the leap that we make is going from these spike train functions,

Â the Rho_b that we have at each of the individual synapses

Â to the instantaneous firing rate that we have for each of the input neurons.

Â Now when can this replacement fail?

Â Well this would fail if, for example there are correlations between all the input neurons

Â or when there's synchrony for example between pairs or more than pairs of neurons.

Â So in those cases we cannot make this replacement.

Â But it turns out that in many,

Â many cases it is quite possible to replace the spike train function,

Â Rho_b, with the firing rate function,

Â the instantaneous firing rate for the individual input neurons.

Â Now I know many people don't like integrals in their equations and some people even

Â get nightmares when they have integrals such as this one in an equation.

Â So for those integral-phobes among you.

Â Is there any way to simplify

Â the input current equation so it does not contain this integral?

Â Well it turns out that if you choose the synaptic filter to be

Â an exponential filter then when you substitute this value for K(t)

Â this particular exponential expression for K(t) in

Â our equation for the input current and when you take the derivative

Â of this equation on both sides

Â then you have an equation that does not contain an integral.

Â So here's how the input current, Is,

Â changes as a function of time.

Â And you'll notice that it's very similar in form to

Â the equation we had for the RC circuit.

Â Except that now this equation tells you how

Â the input current changes as a function of time and there's a particular time constant,

Â Tau_s, that determines how rapidly the input current reacts to changes in the input.

Â And the input component itself is nothing but

Â just a linear weighted sum of

Â all the inputs weighted by the corresponding synaptic weight.

Â And if you want to simplify this even further you can write this weighted sum

Â as just the dot product of the rate vector with the input vector.

Â Okay, it's time to uncork that champagne bottle.

Â We've arrived at our first firing rate based network model.

Â Here's a picture of the network we're going to look at.

Â The network has a single output neuron.

Â The firing rate of the output neuron is given by V. The neuron receives

Â inputs from these input neurons whose firing rates are denoted by this vector,

Â U, and the synaptic connections between

Â the input neurons and the single output neuron is given by the rate vector,

Â the synaptic rate vector,

Â W. We're going to assume that the firing rate of the output neuron follows an equation

Â that is quite similar to the equation we had for

Â the membrane potential from the previous lecture.

Â The upward firing rate has a particular time constant Tau_r which determines

Â how rapidly the firing rate is able to follow

Â inputs that the neuron is getting from other neurons.

Â And there is an optional non-linear function, F,

Â that we can pass the input current through.

Â And finally the input current due to

Â the synaptic inputs is given by the same equation as we had on the previous slide.

Â Now let's look at how this network behaves when you compare the magnitude

Â of Tau_s with the magnitude for Tau_r.

Â If Tau_s is much smaller than Tau_r this means that the synaptic input converges

Â quickly which means that we are going to have the input current be equal to just w.u,

Â as opposed to the network dynamics which takes a long time to converge.

Â And so the equation that we have then would basically

Â replace the Is that we have here with w.u.

Â And so the equation we have then for the network is

Â just Tau_r dv/dt for the change of the firing rate as a function of time

Â is equal to minus V plus the potential non linear function and then just F of w.u.

Â On the other hand if the synapses are slow compared to the output dynamics.

Â In other words if Tau_r as much lesser than Tau_s then we can set V equal to F of

Â Is(t) because the upward dynamics is much

Â faster compared to how the synaptic current is changing over time.

Â And so we have an equation where the upward firing rate is equal to

Â some potentially non-linear function of

Â the input synaptic current where

Â the synaptic current is given by this differential equation.

Â And finally in the case of static inputs and by static we mean

Â that the input does not change as a function of time for long periods of time.

Â Then we can look at

Â the steady state output of the network so Vss denotes the steady state of the network.

Â So how do we get the steady state output of the network?

Â Well we can set dv/dt equal to zero as well as dIs/dt equal to zero.

Â So when we set dvdt equal to zero we get V steady state is equal to f of

Â Is and since Is is also not changing as a function of time Is is equal to w.u.

Â So those of you who are aficionados of

Â artificial neural networks you should find this equation quite familiar.

Â This is in fact the equation that one uses an artificial neural networks to

Â model neurons where we replace F with a threshold function or a sigmoid function.

Â And now you know that this equation that people in

Â the artificial intelligence community have been using for a very long time is in fact

Â a simplification of the rich dynamics that one has in

Â the synaptic current as well as the dynamics of the output firing rate.

Â Let's move to the case where there are multiple output neurons.

Â So in the case of a single output neuron as in the previous slide we use

Â an equation that looks like this to model how

Â the output firing rate changes as a function of time.

Â And if we make the assumption that the synapses are relatively fast then we can simply

Â set the input current due to the synapses to just w.u.

Â So weighted sum of the inputs.

Â And so this equation captures how the output

Â firing rate changes as a function of the inputs.

Â Now what do we do if we want to extend this to multiple output neurons?

Â So here's what the network looks like.

Â And in this case instead of having a single output we have an output vector.

Â And so the equation then looks something like this.

Â So instead of a rate of vector,

Â w, we now have a rate matrix,

Â W. And the equation is now

Â a differential equation that includes a vector output so V is now

Â a vector and this product instead of being

Â w.u is now just the matrix W multiplied with the input vector.

Â The matrix W has the components W_ij.

Â W_ij denotes the synaptic rate from neuron j to neuron i,

Â so if this is neuron j and if this is neuron i,

Â then this connection here would have a weight that is given

Â by W_ij and the synaptic weight matrix

Â W would capture all of these connections from

Â the input layer of neurons to the output layer of neurons.

Â We have so far been considering networks that take

Â inputs and feed them to a layer of output neurons.

Â These type of networks are called feedforward networks.

Â Now if our brain had only feedforward networks then we would be

Â constantly reacting to stimuli and saying things that we're not supposed to say.

Â Luckily the networks in our brains have recurrent feedback connections.

Â This means that for any particular layer of neurons or neurons in a brain area,

Â the neurons make connections with each other.

Â So for example, in this network we have

Â an output layer of neurons and we can allow connections between

Â the output layer neurons and we can characterize

Â the strength of these connections between the output layer of neurons using

Â a set of synaptic weights which we are going to call M. So

Â this matrix M denotes the strength of

Â the connections between the layer of output neurons.

Â So now we have a new equation that characterizes how

Â the firing rates of the output neurons change as a function of time.

Â We have as before a time constant and as before

Â we have a feedforward input given by w times u.

Â But now we also have the feedback given

Â by M times V. So the matrix M is multiplied by the vector V to give

Â you a new vector that characterizes the feedback

Â from past activities of the output neurons.

Â You'll notice that if you set the matrix M to a matrix of zeros,

Â which means that there are no feedback connections then we

Â have any equation but you don't have

Â the M times V component and that is equivalent to

Â the equation for a feedforward network that you saw in the previous slide.

Â Okay let's begin a journey into the land of networks by

Â looking at a simple linear feedforward network.

Â So here is the equation for a linear feedforward network and for the sake of simplicity

Â let's assume that the input doesn't change as

Â a function of time so if we have a static input, u.

Â And so we can look at the steady state value of

Â the output and so that means we just set dv/dt equal to zero.

Â And so we get the equation for steady state output of

Â the network as Vss equal to w times u.

Â Now suppose the feedforward weights are given by this matrix W,

Â and suppose I give you the input u to be this vector here.

Â What is the output of this linear feedforward network?

Â I'll give you a couple of seconds to think about that.

Â Or maybe you want to pause the video at this point?

Â Okay, I hope you found the answer. Here it is.

Â So what we had here was a six by five matrix multiplied by a five by

Â one vector and as you might expect you'll get a six by one vector that looks like this.

Â Now the question that I would like to ask you

Â is: what do you think this network is doing?

Â In other words how is it transforming the input u to the output Vss?

Â What is the function that this network is implementing?

Â As you might have guessed this network is performing linear

Â filtering in order to detect edges in the input.

Â So what do we mean by detect edges?

Â Well if you look carefully at the Matrix W,

Â you'll notice that the rows of the matrix W contain

Â shifted versions of this particular filter and the filter looks something like this.

Â And if you take the input,

Â so this was our input 1,

Â 2, 2, 2, and 1.

Â Here is a picture of what the input looks like.

Â The operation of multiplying the matrix W with

Â the input gives you an output that looks like this.

Â And you'll notice that the output has detected wherever there are transitions from,

Â in this case, 1 to 2 and then back from 2 to 1.

Â And so it has detected this sudden change in the input.

Â And so if we imagine that we have an input image then

Â the filter is essentially detecting changes in the brightness values of the image.

Â In this case is going from a brightness value,

Â or a pixel value, of one to a pixel value of two.

Â And then again from a pixel value of two back to a pixel value of one.

Â Now we can also apply the linear filtering network to

Â two dimensional images such as the one shown here.

Â So how would you transform a two dimensional image to a one

Â dimensional vector u that you feed to the linear filtering network?

Â One way you can do that is by collapsing each row of this two dimensional image.

Â And so if this is row number one,

Â row number two, row number three, and so on,

Â you would simply collapse all the rows to form

Â one very big long vector where the first part of the vector is row number one,

Â the second part of the vector is row number two, and so on.

Â And so there you have your input vector U.

Â And here is an example output.

Â So in this case we are not using the same filter as the one in

Â the previous line but something that is used in this particular website.

Â And so you can see how the input has been transformed to

Â enhance or detect the edges of this particular image.

Â At this point you might be wondering does the brain detect edges?

Â And you already know the answer to this because in week one

Â we discussed the concept of receptor fields and we noted

Â that cells in the primary visual cortex or

Â V1 have receptive fields that look something like this.

Â And this type of receptive field is indicative of edge detecting neurons.

Â For example this receptive field suggests that the neuron detects,

Â or gets excited by,

Â a transition from a dark to a bright edge in the visual image whereas

Â this neuron here would be sensitive to dark bars embedded within a bright background.

Â The brain is in fact not just detecting edges it is actually doing calculus.

Â The brain was doing calculus even before Newton and Leibniz invented calculus.

Â Sorry Newton and Leibniz.

Â Here's what I mean when I say that the brain is doing calculus.

Â Here is the receptive field of a vival neuron and this

Â as you recall is the linear filter that

Â we used in our feedforward matrix W. So if

Â this is some particular location X this would be the next location,

Â X plus 1, in the image.

Â And now if you look at the definition of a derivative of a function, f,

Â with respect to the value X,

Â then here is the definition of the derivative from your calculus textbook.

Â And if you look at a discrete approximation of this derivative you'll find that it

Â amounts to just a difference between f of X plus 1 minus f of X.

Â And if you look at our feedforward matrix it's simply multiplying w with u.

Â And so if u is the image the f of x then W which is essentially shifted versions of

Â this filter performs exactly this particular operation of

Â subtracting neighboring pixel values at the location X and X plus 1.

Â Now here's something that's even more interesting,

Â if we look at the other type of receptive field for

Â vival neurons we call this the bar detector type receptive field.

Â We can implement this type of receptive field by

Â using a linear filter that looks something like this.

Â So if this is location X this would be location X minus one.

Â This would be location X plus one in the image and we would use a matrix W in

Â our feedforward network that would have shifted versions of this linear filter.

Â Now let's look at the definition of the second order derivative of a

Â function f. So D squared F over Dx squared is given by this expression.

Â If we approximate this expression

Â using a discrete approximation we would have something that looks like this.

Â Now you'll notice that this approximation,

Â so f of X plus one minus two times f of X plus f of x minus one,

Â has the same coefficients as our linear filter.

Â So one minus two and one here's one minus two and one.

Â And so when you multiply the feedforward matrix with

Â the input u using this linear filter what are you going to get?

Â Well you're going to get an approximation to

Â the second order derivative of the image if u represents an image.

Â And since the image is two dimensional the brain computes derivatives along

Â multiple directions in the image using oriented receptive fields such as these.

Â That wraps up our first expedition into the world of networks.

Â In the next lecture we'll meet those wild and crazy creatures called recurrent

Â networks and will also enjoy the company of

Â our good old friends eigenvectors and eigenvalues.

Â Until then Ciao and Alvida.

Â