The primary topics in this part of the specialization are: greedy algorithms (scheduling, minimum spanning trees, clustering, Huffman codes) and dynamic programming (knapsack, sequence alignment, optimal search trees).

Loading...

From the course by Stanford University

Greedy Algorithms, Minimum Spanning Trees, and Dynamic Programming

439 ratings

The primary topics in this part of the specialization are: greedy algorithms (scheduling, minimum spanning trees, clustering, Huffman codes) and dynamic programming (knapsack, sequence alignment, optimal search trees).

From the lesson

Week 1

Two motivating applications; selected review; introduction to greedy algorithms; a scheduling application; Prim's MST algorithm.

- Tim RoughgardenProfessor

Computer Science

So at this point we understand, Prim's algorithm and we also know why it's

Â correct. That is why it always computes the

Â minimum cost spanning tree of a graph. So in this video, we're going to turn to

Â implementation issues and running time analysis.

Â We'll begin by just analyzing the straightforward implementation of Prim's

Â algorithm. That's already reasonable.

Â It's polynomial running time, but not especially close to linear.

Â Then we'll see how a suitable deployment of heaps very much in the way that we did

Â for Dijkstra's algorithm leads to a blazingly fast, near linear running time.

Â So let's briefly review the pseudocode for Prim's algorithm.

Â Recall that Prim grows a tree one edge at a time spanning one new vertex at each

Â iteration. So it maintains two sets,

Â capital X, a set of vertices that have spanned so far,

Â and capital T, these are the edges we've committed to thus far.

Â They start out by just being some arbitrary vertex, little s, and the empty

Â set, and in each iteration of this main while-loop, we add one new edge to the

Â tree. And whatever new vertex that edge spans,

Â we add that to capital X. The algorithm terminates when we're

Â spanning all of the vertices and as we've seen, it halts not just with a spanning

Â tree but with a minimum cost spanning tree.

Â So suppose we just literally implemented this algorithm as is, what would be the

Â running time? Well, the initialization stick, step

Â takes only constant time, so let's ignore that.

Â So let me just have this one loop. So let's just ask how many iterations

Â does the loop take and how much time is needed to execute each iteration?

Â Well, the number of loop iterations is going to be exactly n - 1, so, where n is

Â the number of vertices. X starts out with one vertex and

Â terminates when it has all n vertices. How much work do we need to implement

Â each iteration? Well, essentially, what each iteration is

Â doing is a brute-force search through the edges.

Â It's looking for the edges that have one endpoint inside X and one endpoint

Â outside, and amongst those, it just remembers the

Â cheapest. And it's easy to see that you could

Â implement each iteration in O of m time, where M is the number of edges.

Â For example, you can just, with each vertex associate a Boolean variable that

Â keeps track of whether or not it's in this capital X, that way when you see an

Â edge, you know whether it's crossing the frontier or not in constant time.

Â So putting it together, O of m iterations with O of m works for each gives us a

Â running time of O of m times n. So this running time is already nothing

Â to sneeze at. As we discussed, graphs can have an

Â exponential number of spanning trees. So, this algorithm is doing far less work

Â than examining each of these spanning trees.

Â It's homing in in polynomial time, to the minimum cost point.

Â So that's pretty cool. But remember the mantra of any algorithm

Â designer worth their salts, confronted with a solution, you should

Â always ask but can we do better? And can we do better than running time O

Â of m times n? We can as we'll see in the rest of this

Â video. The big idea for speeding up Prim's

Â Algorithm is exactly the big idea we used in part 1 to speed up Dijkstra's

Â algorithm, namely we're going to deploy a suitable

Â data structure. So, what data structure seems like it

Â might be a good idea for making Prim run faster?

Â Well, what's happening in the main workhorse while-loop of Prim's algorithm?

Â Over and over again, we keep meaning to do a minimum computation amongst all

Â edges crossing the frontier, we need to find the cheapest one.

Â So, the question we should ask ourselves is what kind of data structure would

Â facilitate, would speed-up repeated minimum computations.

Â And if you recall from part 1, we have a data structure where that's exactly what

Â it's raison d'etre is, the heap, the meaning of life for a heap is to

Â speed-up repeated minimum computations, just like in Prim's algorithm.

Â So let me just remind you briefly, what are the operations exported by heap data

Â structure and what is the running time? So first recall that a heap contains a

Â bunch of objects, and each of those objects should have some key value from

Â some totally ordered set, like a number, like for example, an edge cost.

Â So what can you do with a heap? Well, the salient operations for our

Â purposes today are, first of all, you can insert new stuff into the heap with

Â their, whatever their key value is. You can extract the object with the

Â minimum key value. And you can also delete stuff from the

Â middle of the heap. And all of these can be done in

Â logarithmic time, logarithmic in a number of objects stored by the heap.

Â So it's not going to be important for us today to know how heaps are implemented

Â and what they look like under the hood. We're just going to be clients of them.

Â We're just going to make use of these operations and the fact that they run in

Â logarithmic time. But you know, just for those of you who

Â are curious, and/or want to have your memory jogged. Under the hood, heaps are

Â implemented logically as complete binary tree.

Â They're actually laid out in an array, but you sort of think of them

Â conceptually as being in a complete binary tree. And they, they, they satisfy

Â what's called the heap property. And the heap property is to make sure that you

Â know where the object with the minimum key value is.

Â So the actual definition is, every parent should have a key value which is less

Â than that of its children. So as you go up the tree, the key value

Â can only drop and that means you know where the minimum is got to be.

Â It's got to be at the root of this tree orr the front of the array.

Â So that's great. That's how you locate the minimum so

Â quickly in a heap. Now, what do you do when you want to

Â extract the minimum? So you rip off the root of this tree,

Â and now, you have to rearrange the tree to restore the heap property.

Â So you swap the last leaf up to where the root was, you bubble-down as needed to

Â restore the heap property. how do you insert?

Â You put the new object as the new last leaf and you bubble it up as needed to

Â restore the heap property. To delete from the middle of a heap, you

Â just sort of rip it out and then bubble things up or down as necessary to restore

Â the heap property. Again, that's not supposed to, if you're

Â hearing this for the first time, I know this is too fast,

Â this is just to jog your memory for those of you who already learned this in part 1

Â of the course. For more details, you can go review the

Â appropriate videos there. So now that I've reminded you about the

Â salient properties of heaps. Let's return to the question of how do we deploy them

Â cleverly to speed-up Prim's algorithm. So our intuition is that because we're

Â doing repeated minimum computations in Prim's algorithm, each time that it's

Â while-looped, compute the cheapest edge cross in your frontier, that's sort of in

Â the wheelhouse of heaps. So how should we use heaps? Well, the

Â first idea, which is a pretty good idea, is to use the heap to store edges, right?

Â Because our minimum computation should result in us choosing an edge, so when we

Â EXTRACT-MIN from a heap, we want it to hand us an edge on a silver platter. So

Â it would seem this would be your first thought,

Â that the heap should store edges and that the key value that you use should just be

Â the edge cost, because you want to find the cheapest edge.

Â So this already a quite good idea using heaps in this manner.

Â We'll already definitely speed-up Prim's algorithm relative to the naive

Â implementation. And in fact.

Â and I'll leave this as an exercise for you to work out.

Â using heaps in this way results in an implementation that has, that runs in

Â time big O of m log n. What I'm going to show you instead is not

Â that implementation, but an even cleverer implementation of Prim using heaps.

Â We're not going to see a benefit in the asymptotic running time.

Â This more sophisticated version will also give us m log n running time, but it

Â would give you better constants and it is the version you would want to implement

Â in practice. [SOUND] So, the one slightly tricky point

Â in this exercise is remembering Prim's algorithm, you don't just want the

Â cheapest edge overall [INAUDIBLE] You want the cheapest edge which crosses the

Â current cut that has one endpoint in each of x and v - x.

Â And, when you use heaps in this way, it might hand you in a silver platter and

Â edge which is cheap, but isn't necessarily crossing the frontier.

Â So, you need some extra checks to ensure that you're always finding the minimum

Â edge and that that edge crosses the frontier between x and v - x.

Â So I'll leave it to you to work out the details of this implementation in the

Â privacy of your own home. What I want to spend our time together on

Â instead is this somewhat more sophisticated, more practical way to use

Â heaps. And for those of you who remember our

Â fast implementation of Dijkstra, this will be very familiar to you.

Â It will be the same kinds of ideas that we used for Dijkstra,

Â and the keypoint is, instead of using the heap to store edges, were going to use it

Â to store vertices. So, in a bit more detail, our plan is

Â going to be to maintain two invariants. The first invariant is going to describe

Â what the heap contains. The second invariant is going to be what

Â the key values of those heap object are. So as I said, we're now going to be

Â starting at vertices in the heap, not edges.

Â Which vertices? Exactly the vertices that we don't yet

Â span. The vertices of v - x.

Â The motivation here is that rather than getting on a silver platter, the edge in

Â which to add next to the tree, we're going to get from a heap on a silver

Â platter, the vertex, that we're next going to add to capital X.

Â So the second invariant tells us what the key values of these vertices in v - x are

Â supposed to be. And we're going to define them to be the

Â cheapest cost of an edge incident of this vertex that crosses the current frontier.

Â So, I think a picture will make this definition clear.

Â So, consider some snapshot of Prim's algorithm at some iteration.

Â We have our vertices X that were already spanning.

Â We have our vertices v - x that were not spanning.

Â And remember, the elements of the heap by invariant 1 are exactly the vertices on

Â the right-hand side, the vertices of v - x.

Â So were trying to find the key value for some vertex in the heap.

Â So some vertex v, which is on the right side, which is not in x.

Â And so, what we do is we look at the edges incident on this vertex v that go

Â back to the left-hand side, so, edges incident to v that are crossing

Â the frontier and there may be of course be many such edges.

Â And the invariant we want to maintain is that the key value for this vertex V is

Â the cheapest of all the incident edges crossing their frontier or in this

Â picture the key should be equal to two. There is the niggling issue of how do you

Â define the key if there are no incident edges at all that are crossing the

Â frontier. So maybe you have a vertex w, which is

Â buried deep inside of v - x, and actually, none of the incident edges go

Â back to the left blob at all. So in that case we just define the key to be plus

Â infinity. So given this high level approach to

Â implementing Prim's algorithm using heaps, we now have a few things to think

Â through. So first of all we have to think about

Â how to initialize the heap so that these invariants are satisfied at the beginning

Â of Prim's algorithm. Second of all, we have to check that if

Â these invariants are satisfied, then we can faithfully simulate each iteration of

Â the while-loop in Prim's algorithm, hopefully very quickly.

Â And then third, we have to think about how do we make sure these invariants are

Â maintained throughout the course of Prim's algorithm,

Â so let's do those in turn. So the first thing is how do we set up the heap at the

Â beginning of Prim's algorithm and a preprocessing step, so that both of these

Â invariants are satisfied. Well, at the beginning,

Â X consists just of this single arbitrary star vertex S.

Â V minus X contains the other n - 1 vertices.

Â The key value of a vertex other than S at the beginning of Prim's algorithm, is

Â just the cheapest edge between that vertex and S if there is one, or plus

Â infinity otherwise. So, the thing to think through and make

Â sure you believe this, is that first of all, with a single scan through the

Â edges, so an O of m time, we can compute the key value for each vertex that needs

Â to go in the heap. And then, we just have to insert those n

Â - 1 vertices into the heap. So that's going to cost us O of m time for an edge

Â scan, and then, m log n for the inserts.

Â In fact, for those of you that really know heaps very well, you might know

Â about a heapify operation which allows you to do a batch of n inserts in O of n

Â time because we can do this even faster in linear time but we're not going to

Â need that in this lectures. And also, I claim that this expression m

Â + n log n is bounded above by the expression m log n, at least an

Â asymptotic notation. To see that, remember two things. First

Â of all we're assuming that the input graph is connected, otherwise there's no

Â spanning trees and it's not interesting to talk about minimum spanning trees.

Â Second of all, in any connected graph, the number of edges m is at least n - 1.

Â So asymptotically, m is always at least as big as n and it can be bigger. So you

Â can always replace an n by an m and get a valid upper bound, so that's what we're

Â doing here. The second issue we need to think through

Â is how do we faithfully simulates each iteration of the while loop in Prim's

Â Algorithm, given that these two invariants halt.

Â So this issue is going to work out beautifully really, by construction.

Â We set up our heap and we set up our definition of keys,

Â so that extracting min from the heap and iteration is a faithful simulation of the

Â brute-force search in the naive implementation of Prim's alogrithm.

Â So specifically, assuming that these two invariants hold when we invoke

Â EXTRACT-MIN from this heap, what it provides to us on a silver platter is the

Â next vertex, not yet in X.

Â The next vertex that we should add to X in this iteration.

Â And moreover, the cheapest edge incident to that vertex crossing the frontier is

Â the one that we should be adding to the set T in this iteration.

Â And the way to think about this fact is to think of us as essentially simulating

Â the brute-force search and the naive implementation using a 2-round knockout

Â tournament. So, in the straightforward implementation

Â of Prim, the way we think of it is we just do a

Â scan through all the edges crossing the frontier and we remember the winner, we

Â remember the smallest cost of them all. Here,

Â with a heap, we're doing it in two steps. So first of all, for each vertex on the

Â right-hand side of the cut, for each vertex in v - x, it locally remembers

Â what is its best candidate so what is the cheapest edge incident on that vertex

Â crossing the frontier. So that's kind of round one, so for an

Â edge to be chosen as the winner, at the very least, it'd better be a local

Â winner. It'd better be the cheapest edge crossing

Â the cut that ends at this particular vertex on the right-hand side of the cut.

Â So that's just in a definition of the key of each vertex and encodes the value of

Â the winner localed in that vertex. And then this EXTRACT-MIN is envoking the

Â second round of this 2, 2-round elimination tournament.

Â It's saying, well, amongst all the proposals from the 1st round, amongst all

Â the crossing edges that are locally minimum given it's endpoint, which of

Â them is the cheapest overall? And that's going to be the cheapest edge

Â crossing this cut, the result of this exact min computation.

Â Coursera provides universal access to the worldâ€™s best education,
partnering with top universities and organizations to offer courses online.