A lot of the action in machine learning has focused on what

algorithms are the best algorithms for

extracting information and using it to predict.

But it's important to step back and look at the entire prediction problem.

This is a little diagram that I made to illustrate

some of the key each, issues in building a predictor.

So you start of with, suppose I want to

predict for these dots whether they're red or blue.

Well, what you might do is have a big group of dots that you

want to predict about, and then you use

probability and sampling to pick a training set.

The training set will consist of some red dots and some blue

dots, and you'll measure a whole bunch of characteristics of those dots.

Then you'll use those characteristics to build what's called a

prediction function, and the prediction function will take a new dot,

whose color you don't know, but using those characteristics that

you measured will predict whether it's red or whether it's blue.

Then you can go off and try to

evaluate whether that prediction function works well or not.

One thing that I think is very important

and often underappreciated about building a machine learning

algorithm is to look at probability and sampling

step of building the training and test stats.

This is always a required component of building every machine learning algorithm

is deciding which samples you're going to use to build that algorithm.

But sometimes it's over-looked, because all of the action that you hear about for

machine learning happens down here when you're

building the actual machine learning function itself.

One very high profile example of the ways that this

can cause problems is the recent discussion about Google Flu trends.

Google Flu trend is tried to use the terms that people were typing into

Google, terms like, I have a cough, to predict how often people would get flu.

In other words, what was the rate of flu that was going

on in a particular part of the United States at a particular time?

And they compared their algorithm to approach taken

by the United States government, where they went out

and they actually measured how many people were

getting the flu, in different places in the US.

And they found in their original paper that the

Google Flu trends algorithm was able to very accurately represent

the number of flu cases that would appear in

various different places in the US at any given time.

But it was quite a bit faster and quite a

bit less expensive to measure using search terms at Google.

The problem that they didn't realize at the time, was that

the search terms that people would use would change over time.

They might use different terms when they were

searching, and so that would affect the algorithm's performance.

And also, the way that those terms were actually

being used in the algorithm wasn't very well understood.

And so when the function of a particular search

term changed in their algorithm, it can cause problems.

And this lead to highly inaccurate results for the Google

Flu trends algorithm half over time as people's internet usage changes.

So this gives you an idea that choosing

the right dataset and that knowing what the specific

question is are again paramount, just like they have

been in other classes of the data science specialization.

So here are the components of a predictor.

You need to start off as always in all, any problem

with data science with a very specific and well defined question.

What are you trying to predict and what are you trying to predict it with?

Then you go out and you collect the best

input data that you can to be able to predict.

And from that data you might either use measured

characteristics that you have or you might use computations

to build features that we'd think you might be

useful for predicting the outcome that you care about.

At this stage then you can actually start to use the machine learning

algorithms you may have read about, such as Random Forest or Decision Trees.

And then what you can do is estimate the

parameters of those algorithms, and use those parameters to

apply the algorithm to a new data set and

then finally evaluate that algorithm on that new data.

So I'm going to just show you one quick little

example, to show you how this little process works.

So this is obviously a trivialized version of what would happen in a

real machine running algorithm, but it gives you a flavor of what's going on.

So you start off with asking something about the question.

So you start with a in general

people usually start with a quite general questions.

So here is, can I automatically detect emails

that are SPAM from those that are not?

So SPAM emails are emails that you got

that you, come from companies that get sent out

to thousands of people at the same time

and that you might not be interested in it.

So you might want to make your question a little bit more concrete.

You often need to when doing machine learning.

So, the question might be, can I use

quantitative characteristics of those emails to classify them as

SPAM, or what we're going to call HAM which

is the email that people would like to receive?

So once you have your question, then you need to find input data.

In this case, there's actually a bunch of data

that's available and already pre-processed for us in R.

So it's actually in the current lab

package K-E-R-N-L-A-B and it's the SPAM dataset.

So we can actually load that data set into R directly, and it has some information

that's been collected about SPAM and HAM emails already available to us.

Now we might want to keep in mind that that might

not necessarily be the perfect data, in fact, we don't have all

of the emails that have been collected over time, or we

don't have all the emails that are being sent to you personally.

So we need to be aware of the potential limitations of this

data, when we're using it to build an algorithm, a prediction algorithm.

Then we want to calculate something about features.

So, imagine that you have a bunch of emails.

And here's an example email that's been sent to me.

Dear Jeff, can you send me the address, so I can send you the invitation.

Thanks, Ben.

If we want to build a prediction algorithm,

we need to calculate some characteristics of

these emails that we can use to be able to build a predictive algorithm.

And so one example might be, we can

calculate the frequency with which a particular word appears.

So here, we're looking for the frequency that the word you appears.

And so in this case, it appears twice in this email so 2 out

of 17 words or about 11% of the words in this email are you.

We could calculate that same percentage for every single email that we have and

now we have a qualitative characteristic that we can try to use to predict.

So if the data in the current lab package that I've shown here are actually,

information just like that, for every email we

have the frequency with which certain words appear.

And so, for example if credit appears very often in the email or money appears

very often in the email, you might imagine that that email might be a SPAM email.

So, as one example of that, we looked at the frequency

of the word, your, and how often it appears in the email.

And so, I've got a plot here that's a density plot of the, that data.

And so, on the x-axis is the frequency

that with which, your, appeared in the email.

And on the y-axis is the density, or the

number of times the that frequency appears amongst the emails.

And so what you can see is that most of the emails that are SPAM, those are the

ones that are in red, you can see that

they tend to have more appearances of the word, your.

Where as all of the emails that are HAM, the

ones that we actually want to receive have a much higher peak

right over here down near 0, so there's very few

emails that have a large number of viewers that are HAM.

So, we can build an algorithm in this case let's build a very very simple algorithm.

We can estimate an algorithm where we want to just find a cut off a constant C, where

if the frequency of your is above C then

we predict spam and otherwise we predict that it's ham.

So going back to our data we can fig, try to figure out what

that best cut off is, and here's an example of a cutoff that you could

choose, so choose a cut off here that if it's above 0.5 then we

say that it's SPAM, and if it's below 0.5 we can say that it's HAM.

And so we think this might work because you can see that

the large spike of blue HAM messages are below that cut off.

Whereas the big, one of the big spikes of the SPAM messages is above that cut off.

So you might imagine that wil cache quite a bit of that SPAM.

So then what we do is we evaluate that.

So what we would do is calculate for

example predictions for each of the different emails.

We take a prediction in that says, if the frequency of yours

above 0.5, then you're spam and if it's below then you're nonspam.

And then we make a table of those predictions and divide

it by the length of the, all the observations that we have.

And so we can say is that, when you're nonspam about

45% of the time, 46% of the time, we get you right.

When you're spam about 29% of the time, we get you right.

So, total we get you write about 45% plus 29% is about 75% of the time.

So our prediction algorithm is about 75% accurate in this particular case.

So that's how we would evaluate the algorithm.

This is of course any same dataset where we actually calculated

it, the prediction function, and as we will see in later lectures.

This will be an optimistic estimate of the overall error rate.

So that's an overview of, the basic steps in building a predictive algorithm.

What is prediction?

Practical Machine Learning

Meet the Instructors