Using the Developer Console to Explore HTTP

Loading...

From the course by University of Michigan

Using Python to Access Web Data

19798 ratings

University of Michigan

Using Python to Access Web Data

19798 ratings

Course 3 of 5 in the Specialization Python for Everybody

This course will show how one can treat the Internet as a source of data. We will scrape, parse, and read web data as well as access data using web APIs. We will work with HTML, XML, and JSON data formats in Python. This course will cover Chapters 11-13 of the textbook “Python for Everybody”. To succeed in this course, you should be familiar with the material covered in Chapters 1-10 of the textbook and the first two courses in this specialization. These topics include variables and expressions, conditional execution (loops, branching, and try/except), functions, Python data structures (strings, lists, dictionaries, and tuples), and manipulating files. This course covers Python 3.

From the lesson

Networks and Sockets (Chapter 12)

In this section we learn about the protocols that web browsers use to retrieve documents and web applications use to interact with Application Program Interfaces (APIs).

12.1 - Networked Technology7:38

12.2 - Hypertext Transfer Protocol (HTTP)14:22

Worked Example: Sockets (Chapter 12)6:11

Using the Developer Console to Explore HTTP10:12

Meet the Instructors

Charles Severance
Professor
School of Information

So, in this video, we're going to talk a little bit about

how to use the developer console in your browser.

Really in this course, we're writing Python programs to talk to the Internet,

but sometimes you want to debug how a page is really working.

The Telnet program is a less and less Operating Systems,

they took it out of the Macintosh,

it's still there in Linux.

So, I want to tell you how to use this developer console to

find out various things about your request response cycle.

So, let's take a look here at this page.

So, this is our page1.htm,

and we've looked at this a lot of different ways,

whether it be Telnet or in a Python program,

and now I'm just going to type it in on a browser,

and then we're going to tear apart what's going on.

So, I did this in View,

Developer, Console, and under the Network tab,

and then I refresh the page.

So, what this is doing is showing each of the requests,

and the request-response cycle.

So page1.htm is the document.

Remember, we talked about that.

There is the protocol,

the server, and the document.

Page one is the document.

So, if I click on this,

then I get to see all kinds of things.

So, the preview, that's the HTML as it's rendered.

But the more interesting thing is the get request, the headers.

So, remember, it does a request.

In this case, it's a get, right?

Then it gets a response, and the 200,

and if you look at all the little things I had in the other lectures,

you'll see these 200 are coming out.

You see the IP address of where it's going to,

and then the response headers, cache control,

and I actually have an assignment where you're supposed to

read some of these things and type them in.

You'll get these exact same headers either in

Python or with Telnet or with the developer console.

So, in that assignment, you can do it any way that you want.

So, like the ETag or the Content-Type,

that says this is an HTML page.

That tells it when it gets the actual response.

The actual response is like that less thans and greater thans,

that's HTML, but says format this with HTML.

So, the request headers,

I wish they put these in a different order because

the browser makes the connection to port 80,

and then it sends the request on port 80,

then it sends the request headers on port 80,

and then it waits for the response.

Right? So, these are the things.

It's telling it various things like accept language,

for example, says that I prefer English.

If you can have multiple languages,

and you can have a priority for them,

so it's like English is my primary language and Spanish might be my secondary language.

The browser is doing this,

sending this information to the server on every request.

So, that's how some webpages know that they

should render themselves in Spanish if you're a Spanish-speaking student.

So, that's the request,

and then the response is this HTML,

and that's literally what came back from the server and then it shows up as this.

So, the way it's rendered is this H1 turns it into text, et cetera.

So, that's what it is. You see the request response cycle.

So, I'll go back to get rid of this.

So, I'm going to clear this and I'm going to hit "Refresh",

and that's going to trigger another get request and there you go.

You look at Headers, Preview,

Response, et cetera, and Timing.

So, that's the basic idea.

So, I want to show you a couple of other things about this request-response cycle.

You can stop watching now if you wanted to,

but I want to show you a few other things that

actually come from my other specialization,

web applications for everybody.

So, this is a guessing game that I wrote,

and you're supposed to guess a number,

and if you've taken this class long enough,

you know what the number is that will be the correct answer,

because it's always the right answer.

So, I want to show you the idea of putting parameters on the URL.

So, I can put a parameter, Guess equals 12.

So,I put a question mark,

and you'll see URLs.

If you're using Google Maps or whatever, you'll see these question marks.

These are parameters that go on in addition to the URL.

So, here we have the URL is this,

and then there's perimeters that being sent to

this interactive application on the server.

So, if we take a look at what happened,

we do a get request,

and if I look at the headers,

it sent in this whole URL,

the document to retrieve.

Question mark guess equals one,

web applications after the question mark,

see these as key value pairs for the parameters.

It comes like a little dictionary.

So, if we look here,

that's what the URL was,

we did a get request to that,

that was the IP address that we connected to, got a 200,

the response headers, we see that it's content type text HTML.

The request headers again,

these are the ones we are sending in.

There's a path and the scheme and all these other things,

and then there's query string parameters.

So, these parameters that are sitting there at the end

are called get parameters or query string parameters,

and you can put more than one in although this application doesn't care.

I could put ampersand X equals 14 ampersand,

ABC equals 22, and then all those W3 key value pairs that are being sent.

So, if we take a look, that's the URL that it hits.

But as it's perceived inside of the server,

it sees that there are three variables that have been sent in at the end of this URL,

so that's another thing that you're going to see.

Again, you don't need it for this class,

but as we talk about the request response cycle,

sometimes there's other information.

Now I want to show you one last little example.

I'm going to show you how the status code can change.

So, in this case, the 200 means I ask for a document,

the 200 says here's your document.

But 200 is not the only one.

Let's do another one.

Let's do this one, let's see if it works.

We'll see what happens here.

Go to a bad URL,

this one might be one that you know.

Okay. So, let's go to the very first one.

See this one? 404. It did it right.

So, if I click at this,

and I want to get this URL,

Zap is not there and the answer the server gave is not 200.

200 would mean you're good, here's your document.

404 means not found, 404 not found.

That's kind of another mean,

just like 42 is the mean,

so that's 404 not found.

So, that's in the response,

so that's part of the response headers.

The status is 404,

and the other ones that said it should the 200.

So, those are two status.

200 means good, 404 means not found.

Now, I want to show you one more status.

So, what this does is,

this is an application that does what's called a redirect.

If I do a get request and then the server says,

You know, you're at the wrong page.

I'd like to bounce you over to this other page".

It sends me back a response.

It's a 302, and it says,

"Hey, redirect to this other page."

302 means you're at the wrong place,

and then it gives me back a URL which is the right place.

So, I'm going to say two here,

and we'll see what happens.

I'm going to ask the server for a page,

and it's going to send me back a page,

and then my browser's going to grab the next page.

So, I'm going to ask for it.

So, I went after reader1.php,

and you'll notice I got nothing back on this particular one.

So, there's two, here's the second one, here's the first one.

If I look at the headers,

I did a post request which is like get is a different request to this thing.

I sent the number two,

I type number two in there,

and then I got back a 302.

302 means I did not give you a document.

What I actually gave you is location header,

which is you're at the wrong place,

this is the right place,

reader2.php is the right place.

So, your browser receives this 302,

and then immediately goes and does a get request on the URL it was redirected to,

so this is the redirected URL.

Okay. So, that was a get request and now I actually

have a real response and the previews, et cetera.

So, that gives you the basic idea of how the request response cycle.

I click on page two,

and then I do a get and it's okay.

So, that gives you a basic idea of the request response cycle,

and how you can look at those headers and see

the interaction between your browser and it's server,

how status responses work, how the output.

Let me show you one more thing.

Here's some files.

I'll click on cover3.jpg,

I'll clear this out first just so it's a little prettier.

Cover3.jpg, watch this.

So, that's the cover to the textbook.

Now, if I look at this,

I can see some things.

I did a get request to that.

JPG is a picture.

I got get request.

I got a 200 okay which means you asked for something and I gave it to you,

and everything else is the same,

but the thing that's different here is the content type, image/jpeg.

So, that means that the response,

we can't see it, it's a bunch of little numbers.

The numbers themselves are that picture.

But the browser knows that it's a picture

because it's been told it's a picture with this Content-Type header.

So, that's one of the things these headers do.

If you recall, headers come out first,

then the data comes out second.

So, I can look at the header,

say, "This I'm supposed to interpret a certain way."

I can show you one more, romeo.txt.

Okay. So, if I look at this,

romeo.txt is another document.

I got a 200, it's a text.

But you'll see that this is text/plane.

The other one we said was text/html.

So, the metadata in those headers tells the browser how to

interpret the body of the response document when it comes back.

Okay. So, with that, I'll let you go.

This is something very useful whether you're

a web application developer or whether you're playing with APIs,

to be able to go into this console and

find your way around and figure out what's happening.

Just because there's only one down here doesn't mean that there's not a lot.

There could be like 40 or 50,

and you notice I had to scroll up and down to find that 404 not found.

So, hope you found this useful.

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2019 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play