What is the use of http headers?

[adjacent]

For example, I am downloading an image file from example.com using a python script.
Is there any need to include the headers in the script? I don't see anything happening if I don't include the headers.
So what is the need of it?

 

[jedishrfu]

Here's a description of the headers:

http://en.wikipedia.org/wiki/List_of_HTTP_header_fields

I know some but not all are required, they give the browser info on how to display the content that follows. In general, if any value is missing the browser defaults or uses cached values.

 

[martiandawn]

As far as I'm aware of the only mandatory header in HTTP 1.1 is "Host: ". However it might be better if your script can also send appropriate headers to define the encoding, and request the TCP connection to be closed or kept open.

 

[Medicol]

[Edit: I thought you were only trying to download an image ]

You need http header options especially in case the website in contact needs e.g your credentials for access (username, password, apikey etc). Plus, it depends on which method you use to send your packet so as to include your appropriate request header i.e partial vs full download stream, request string from your client to the server in transaction etc.

If for example you would want to download an image from the site example.com, you can just need to GET the image location provided that you've been granted your access right to the site.
In case you would want to query something from the site which has supplied you with i.e public api methods, they sure also document them on their website, visit and follow their examples.

 

[adjacent]

Thanks for the answers. Can the site Identify me as a bot if I don't use the headers?

 

[martiandawn]

Can the site Identify me as a bot if I don't use the headers?

Some sites do use headers for that, but it's very few of them and in most cases that's just for protecting copyrighted material (e.g. videos). So most likely you won't have trouble.

If you notice that it doesn't work (and you're not doing any nasty things) I'd suggest you use netcat in server mode and send a browser request to it. You'll be able to see on screen all the headers that your browser sends. Alternatively, you can use packet capture software to inspect your browser's requests to the real site.

 

[adjacent]

If you notice that it doesn't work (and you're not doing any nasty things) I'd suggest you use netcat in server mode and send a browser request to it. You'll be able to see on screen all the headers that your browser sends. Alternatively, you can use packet capture software to inspect your browser's requests to the real site.

The 'firebug' add-on for firefox can also be used to find the request and response headers.

 

[martiandawn]

The 'firebug' add-on for firefox can also be used to find the request and response headers.

Thanks for the tip!

 

[Mark44]

All of the major browsers (Firefox, Chrome, IE) have debugging capabilities that can be accessed from the F12 key. Using the F12 debugging tools you can see the request and response headers and quite a bit more.

 

[jedishrfu]

All of the major browsers (Firefox, Chrome, IE) have debugging capabilities that can be accessed from the F12 key. Using the F12 debugging tools you can see the request and response headers and quite a bit more.

I've played with web apps for years and never knew about the F12 key. Thx.