Python httplib2 - Handling Cookies in HTTP Form Posts

I often need to automate tasks in web based applications.  I like to do this
at the protocol level by simulating a real user's interactions via HTTP.  Python
comes with two built-in modules for this: urllib (higher
level Web interface) and httplib (lower
level HTTP interface).

However, I usually don't use either of these.  I prefer to use Joe
excellent httplib2 module (btw,
I really wish this could make its way into Python's Standard Library). 
It is a much richer library and has a lot of nice features for dealing with HTTP.  

When automating something, you often need to "login" to maintain some sort of session/state
with the server.  This is usually achieved with form-based authentication. You
post a form to the server, and it responds with a cookie in the incoming HTTP header. 
You need to pass this cookie back to the server in subsequent requests to maintain
state or to keep a session alive.

Here is an example of how to deal with cookies when doing your HTTP Post.

First, lets import the modules we will use:

 import urllib import httplib2 

Now, lets define the data we will need: In this case, we are doing a form post with
2 fields representing a username and a password.

 url = '' body = {'USERNAME':
'foo', 'PASSWORD': 'bar'} headers = {'Content-type': 'application/x-www-form-urlencoded'} 

Now we can send the HTTP request:

 http = httplib2.Http() response, content = http.request(url,
'POST', headers=headers, body=urllib.urlencode(body)) 

At this point, our "response" variable contains a dictionary of HTTP header fields
that were returned by the server. If a cookie was returned, you would see a "set-cookie"
field containing the cookie value. We want to take this value and put it into the
outgoing HTTP header for our subsequent requests:

 headers['Cookie'] = response['set-cookie'] 

Now we can send a request using this header and it will contain the cookie, so the
server can recognize us.

So... here is the whole thing in a script. We login to a site and then make another
request using the cookie we received:

 #!/usr/bin/env python import urllib import httplib2 http
= httplib2.Http() url = '' body = {'USERNAME': 'foo',
'PASSWORD': 'bar'} headers = {'Content-type': 'application/x-www-form-urlencoded'}
response, content = http.request(url, 'POST', headers=headers, body=urllib.urlencode(body))
headers = {'Cookie': response['set-cookie']} url = '' response,
content = http.request(url, 'GET', headers=headers)