The primary topics in this part of the specialization are: greedy algorithms (scheduling, minimum spanning trees, clustering, Huffman codes) and dynamic programming (knapsack, sequence alignment, optimal search trees).

Loading...

From the course by Stanford University

Greedy Algorithms, Minimum Spanning Trees, and Dynamic Programming

437 ratings

The primary topics in this part of the specialization are: greedy algorithms (scheduling, minimum spanning trees, clustering, Huffman codes) and dynamic programming (knapsack, sequence alignment, optimal search trees).

From the lesson

Week 1

Two motivating applications; selected review; introduction to greedy algorithms; a scheduling application; Prim's MST algorithm.

- Tim RoughgardenProfessor

Computer Science

Today, we're going to embark on the discussion of a new algorithm design

Â paradigm. Namely, that of designing and analyzing

Â greedy algorithms. So to put this study of greedy algorithms

Â in a little bit of context, let's just zoom out.

Â Let's both review some of the algorithm design paradigms that we've already seen,

Â as well as look forward to some that we're going to learn later on, in this

Â course. So it's sort of a sad fact of life that

Â in algorithm design, there's no one silver bullet.

Â There's no magic potion that's the cure for all your computational problems.

Â So instead, the best we can do, and the focus of these courses, is to discuss

Â general techniques that apply to lots of different problems that arise and lots of

Â different domains. So that's what I mean by algorithm design

Â paradigms. High level problem solving strategies

Â that cut across multiple applications. So let's look at some examples.

Â Back in part one, we started with the divide and conquer algorithm design

Â paradigm. A canonical example of that paradigm

Â being the merge sort algorithm. So remember in divide and conquer what

Â you do is you take your problem you break it into smaller sub problems, you solve

Â the sub problems recursively and then you combine the results into a solution for

Â the original problem. Like how in merge sort you recursively

Â sort two sub arrays and then merge the results to get a sorted version of the

Â original input array. Another paradigm that we touched on in

Â part one, although we didn't discuss it anywhere

Â near as thoroughly, is that of randomized algorithms.

Â So the idea that you could have code flip coins,

Â that is, make random choices, inside the code itself.

Â Often, this leads to simpler, more practical, or more elegant algorithms.

Â A canonical application here is the quick sort algorithm using a random pivot

Â element. But we also saw applications for example,

Â to the design of hash functions. So the next measure paradigm we're going

Â to discuss is that of greedy algorithms. So these are algorithms that iteratively

Â make myopic decisions. In fact, we've already seen an example of

Â a greedy algorithm in part one namely Dijkstra's shortest path algorithm.

Â And then, the final paradigm we're going to discuss in this class is that of

Â dynamic programming, a very powerful paradigm which solves, in

Â particular, two of the motivating questions we saw earlier namely, sequence

Â alignment, and distributed shortest paths.

Â So what is a greedy algorithm anyways? Well, to be honest, I'm not going to

Â offer you a formal definition. In fact, much blood and ink has been

Â spilled over which algorithm is precisely greedy algorithms.

Â But, I'll give you a sort of informal description.

Â A sort of rule of thumb for what greedy algorithms usually look like.

Â Generally speaking, what a greedy algorithm does, is make a sequence of

Â decisions with each decision being made myopically.

Â That is, it seems like a good idea at the time and then you hope that everything

Â works out at the end. The best way to get a feel for greedy

Â algorithms is to see examples and the upcoming lectures will give you a number

Â of them. But I want to point out we've actually

Â already seen an example of a greedy algorithm in part one of this course,

Â namely Dijkstra's shortest path algorithm.

Â So in what sense is Dijkstra's algorithm a greedy algorithm?

Â Well if you recall the psuedo code for Dijkstra's algorithm, you'll recall

Â there's one main wild loop and the algorithm process's exactly one new

Â destination vertex in each iteration of this wild loop, so there's exactly N - 1

Â iterations overall, where N is the number of vertices.

Â So the algorithm only gets one shot to compute the shortest path to a given

Â destination. It never goes back and revisits the decision, in that sense the

Â decisions are myoptic, irrevocable and that's the sense in which Dijkstra's

Â algorithm is greedy. So let me pause for a moment to discuss

Â the greedy algorithm design paradigm generally.

Â Probably this discussion will seem a little abstract so I recommend you

Â revisit this discussion on the slide after we've seen a few examples so at

Â that point I think it will really hit home.

Â So let me proceed by comparing it and contrasting it to the paradigm we've

Â already studied in depth. That of divide and conquer algorithms.

Â So you'll recall that in a divide and conquer algorithm what you do is, you

Â break the problem into sub-problems. So, maybe you take an input array and you

Â split it into two sub-arrays. Then you solve the smaller sub-problems

Â recursively, and then you combine the results of the sub-problems into a

Â solution to the original input. So the greedy paradigm is quite different

Â in several respects. First, both a strength and a weakness of

Â the greedy algorithm design paradigm is just how easy it is to apply.

Â So it's often quite easy to come up with plausible greedy algorithms for a

Â problem, even multiple difference plausible greedy algorithms.

Â I think that a point of contrast with divide and conquer algorithms.

Â Often it's tricky to come up with a plausible divide and conquer algorithm,

Â and usually you have this eureka moment where you finally figure out how to

Â decompose the problem in the right way. And once you have the eureka moment,

Â you're good to go. So secondly, I'm happy to report that

Â analyzing running time of greedy algorithms will generally be much easier

Â than it was with divide and conquer algorithms.

Â For divide and conquer algorithms it was really unclear whether they were fast or

Â slow, because we had to understand the running time over multiple levels of

Â recursion. On the one hand problems were size was

Â getting smaller, but on the other hand, the number of some problems was

Â proliferating. So we had to work hard, we developed

Â these powerful tools like the master method, and some other techniques, for

Â figuring out just how fast an algorithm like Merge Sort runs, or just how fast an

Â algorithm like Strassen's fast matrix multiplication algorithm runs.

Â In contrast with greedy algorithms, it will often be a one liner.

Â Often it will be clear that the work is dominated by say, a sorting sub routine

Â and of course we all know that sorting takes n log and time if you use a

Â sensible algorithm for it. Now the catch,

Â and this is the third point of comparison,

Â is we're generally going to have to work much harder to understand correctness

Â issues of greedy algorithms. For divide-and-conquer algorithms we

Â didn't talk much about correctness. It was generally a pretty straightforward

Â induction proof. You can review the lectures on Quicksort

Â if you want an example of one of those canonical inductive correctness proofs.

Â But the game totally changes with greedy algorithms.

Â In fact, given a greedy algorithm we often won't even have very good intuition

Â for whether or not they are correct. Let alone how to prove they're correct.

Â So even with a correct algorithm, it's often hard to figure out, why it's

Â correct. And in fact, if you remember only one

Â thing from all of this greedy algorithm discussion many years from now,

Â I hope one key thing you remember is they're often not correct.

Â Often, especially if it's one you proposed yourself which you're very

Â biased, in favor of. You will think the algorithm, the greedy

Â algorithm must be correct because it's so natural.

Â But many of them are not, so keep that in mind.

Â So to give you some immediate practice with the ubiquitous incorrectness of

Â natural algorithm. Let's review a point that we already

Â covered in part one of this class concerning Dijkstra's algorithm.

Â Now, in part one we made a big deal of what a justly famous algorithm Dijkstra's

Â shortest path algorithm is, it runs brazenly fast and it computes all the

Â shortest paths. What else do you want?

Â Well remember there was an assumption when we proved that the Dijkstra's

Â algorithm is correct. We assumed that every edge of the given

Â network has a non negative length. We did not allow negative edge lengths.

Â And as we discussed in part one, you know,

Â for many applications, you only care about non negative edge lengths.

Â But there are applications where you do want negative edge lengths.

Â So let's review on this quiz why Dijkstra's is actually incorrect, despite

Â being so natural. it's incorrect when edges can have

Â negative lengths. So I've drawn in green, a very simple

Â shortest path network with three edges and I've annotated the edges with their

Â links. You'll notice one of those edges does

Â have a negative length, the edge from V to W with length minus two.

Â So the question is consider the source vertex S and the destination vertex W.

Â And the question is, what is the shortest path distance computed by Dijkstra's

Â algorithm and you may have to go and review just a pseudo code in part one or

Â on the web. to answer that part of the question and

Â then what is in fact the actual shortest path distance from S to W where as usual

Â the length of a path is just the sum of the lengths of the edges in the path.

Â All right, so the correct answer is D. So let's start with the second part of

Â the question, what is the actual length of a shortest path from S to W when

Â there's only two paths at all in the graph?

Â The one straight from S to W that has length 2, and the one that goes by the

Â intermediate point V that has length 3 + -21, = 1 which is shorter.

Â So, SVW is the shortest path that has length 1. Why is Dijkstra incorrect? Well

Â if you go back to the pseudo code of Dijkstra, you'll see that in the very

Â first iteration it will greedily find the closest vertex to S in that case this is

Â W, W is closer then V. It will greedily compute the shortest path distance to W

Â knowing the information it has right now and all it knows is there's this one hot

Â path from S to W, so it will irrevocably commute to the shortest path distance

Â from S to W as 2. Never reconsidering that decision later.

Â So Dijkstra will terminate with the incorrect output that the shortest path

Â link from S to W is 2. This doesn't contradict anything we

Â proved in part one, because we established correctness of Dijkstra only

Â under the assumption that all edge links are non-negative, an assumption which is

Â violated in this particular example. But again, the takeaway point here is

Â that, you know, it's easy to write down a greedy algorithm, especially if you came

Â up with it yourself. You probably believe deep in your heart

Â that it's got to be correct all the time, but more often than not, probably your

Â greedy heuristic is nothing more than a heuristic.

Â And there will be instances in which it does the wrong thing.

Â So keep that in mind in greedy algorithm design.

Â So now that my conscience is clear, having warned you about the perils of

Â greedy algorithm design, let's turn to proofs of correctness.

Â That is if you have a greedy algorithm that is correct.

Â And we will see some notable examples in the coming lectures.

Â How would you actually establish that effect?

Â Or if you have a greedy algorithm, and you don't know whether or not it is

Â correct, how would you approach trying to

Â understand which one it is, whether it's correct or not?

Â So let me level with you. Proving greedy algorithm is correct.

Â Frankly, is sort of, more art than science.

Â So, unlike the divide and conquer paradigm, where everything was somewhat

Â formulaic. We had these black box ways of evaluating

Â recurrences. We had this sort of, template for proving

Â algorithms correct. Really, proving correctness of greedy

Â algorithms takes a lot of creativity. And it has a bit of an ad hoc flavor.

Â That said, as usual, to the extent that they are recurring themes.

Â That is what I will spend our time together emphasizing.

Â So let me tell you just again about very high level.

Â How you might go about this. You, again, might want to revisit this

Â context aft-, content after you've seen some examples where I think it'll make a

Â lot more sense. So method one is our old friend or

Â perhaps nemesis depending on your disposition, namely proofs by induction.

Â Now for a greedy algorithms remember what they do, they sequentially make a bunch

Â of irrevocable decisions, so here the induction is going to be on decisions

Â made by the algorithm. And if you go back to our proof of

Â correctness of Dijkstra's algorithm, that in fact is exactly how we proved

Â Dijkstra's algorithm correct. It was by induction of the number of

Â iterations, in each iteration of the main wild loop.

Â Computed the shortest path to one new destination.

Â And we always proof that assuming all of our previous computations were correct,

Â that's the inductive hypothesis. Then so is the computation in the current

Â iteration. And so then by induction, everything the

Â algorithm ever does is correct. So that's a greedy

Â proof by induction that a greedy algorithm can be correct.

Â And we might see some more examples of those in, for other algorithms in the

Â lectures to come. Some of the text books call this method

Â of proof greedy stays ahead, meaning you always proof greedy's doing

Â the right thing iteration by iteration. So a second approach to approving the

Â correctness of greedy algorithms which works in a lot of cases is what's called

Â an exchange argument. So you haven't yet seen any examples of

Â exchange arguments in this class so I can't really tell you what they are but

Â that's what we're going to proceed next. I'm going to argue by an exchange

Â argument that a couple of difference famous greedy algorithms are in fact

Â corrected. It has a couple of different flavors one

Â flavor is to approach it by contradiction.

Â You assume for contradiction that a greedy algorithm is incorrect and then

Â you show that you can take an optimal solution and exchange two elements of

Â that optimal solution and get something even better which of course contradicts

Â the assumption that you started with an optimal solution.

Â a different flavor would be to gradually exchange an optimal solution into the one

Â output by a greedy algorithm without making the solution any worse.

Â That would show that the output of the greedy algorithm is in fact optimal.

Â And formally that's done by an induction on the number of exchanges required to

Â transfer an optimum solution into yours. And finally, I've already said it once,

Â but let me say it again, there's not a whole lot of formula behind proving

Â greedy algorithms correct, you often have to be quite creative, you might have to

Â stitch together aspects of method one and method two, you might have to do

Â something completely different. Really, any rigorous proof is fair game.

Â Coursera provides universal access to the worldâ€™s best education,
partnering with top universities and organizations to offer courses online.