What is the use of http headers?

  1. adjacent

    adjacent 1,533
    Gold Member

    For example, I am downloading an image file from example.com using a python script.
    Is there any need to include the headers in the script? I don't see anything happening if I don't include the headers.
    So what is the need of it?
     
  2. jcsd
  3. jedishrfu

    Staff: Mentor

    Here's a description of the headers:

    http://en.wikipedia.org/wiki/List_of_HTTP_header_fields

    I know some but not all are required, they give the browser info on how to display the content that follows. In general, if any value is missing the browser defaults or uses cached values.
     
    1 person likes this.
  4. As far as I'm aware of the only mandatory header in HTTP 1.1 is "Host: ". However it might be better if your script can also send appropriate headers to define the encoding, and request the TCP connection to be closed or kept open.
     
    1 person likes this.
  5. [Edit: I thought you were only trying to download an image :tongue: ]

    You need http header options especially in case the website in contact needs e.g your credentials for access (username, password, apikey etc). Plus, it depends on which method you use to send your packet so as to include your appropriate request header i.e partial vs full download stream, request string from your client to the server in transaction etc.

    If for example you would want to download an image from the site example.com, you can just need to GET the image location provided that you've been granted your access right to the site.
    In case you would want to query something from the site which has supplied you with i.e public api methods, they sure also document them on their website, visit and follow their examples.
     
    Last edited: Aug 29, 2014
    1 person likes this.
  6. adjacent

    adjacent 1,533
    Gold Member

    Thanks for the answers. Can the site Identify me as a bot if I don't use the headers?
     
  7. Some sites do use headers for that, but it's very few of them and in most cases that's just for protecting copyrighted material (e.g. videos). So most likely you won't have trouble.

    If you notice that it doesn't work (and you're not doing any nasty things) I'd suggest you use netcat in server mode and send a browser request to it. You'll be able to see on screen all the headers that your browser sends. Alternatively, you can use packet capture software to inspect your browser's requests to the real site.
     
    1 person likes this.
  8. adjacent

    adjacent 1,533
    Gold Member

    The 'firebug' add-on for firefox can also be used to find the request and response headers. :smile:
     
  9. Thanks for the tip!
     
  10. Mark44

    Staff: Mentor

    All of the major browsers (Firefox, Chrome, IE) have debugging capabilities that can be accessed from the F12 key. Using the F12 debugging tools you can see the request and response headers and quite a bit more.
     
    2 people like this.
  11. jedishrfu

    Staff: Mentor

    I've played with web apps for years and never knew about the F12 key. Thx.
     
Know someone interested in this topic? Share a link to this question via email, Google+, Twitter, or Facebook