App Engine and httplib2

Joe Gregorio

The 1.1.9 release of the App Engine SDK includes support for httplib, urllib and urllib2. This is great since httplib2 depends on httlib. So far I've only found one problem, which is that httplib2 presumes that httplib returns all headers in lower case, and have already committed a fix to httplib2 for that. The change is currently in trunk and will appear in the next release.

Setting up is easy, just pull the httplib2 directory into your project.

The nice part is that you can hook up httplib2 to the memcache support in App Engine very easily:

import wsgiref.handlers
from google.appengine.ext import webapp

import httplib2
from google.appengine.api.memcache import Client

mem = Client()
http = httplib2.Http(mem)  

class MainHandler(webapp.RequestHandler):

  def get(self):
    headers, body = http.request("")

    self.response.headers['content-type'] = "text/plain"
    self.response.out.write("Reported status = %d\n" % headers.status)

The actual utility of httplib2 goes down on App Engine since the system that URLFetch runs through is an HTTP 1.1 compliant proxy and takes care of redirects and caching, but httplib2 still provides benefits such as authentication and lost update support. In addition you could supply a cache handler to httplib2 that persisted the cached pages to the datastore instead of memcache.

Maybe a little OT, but are there any plans to propose httplib2 for Python's stdlib?

Posted by Eduardo Padoan on 2009-02-10


I did some work on getting httplib2 into the stdlib in the past, but haven't had the personal bandwidth to pursue it.

Posted by Joe on 2009-02-10

Very good news indeed. This was the first question I asked when I saw urllib,urllib2, and httplib had landed in GAE.

Posted by Peter Keane on 2009-02-10

"The change is currently in trunk and will appear in the next release". Next release of httplib, but when will it be in GAE? In a similar vein, the first thing I thought when I noticed support for urllib2 and httplib was "great, I get to set timeouts on outbound HTTP requests". Except it's a 2.6 addition and GAE seems to still be on 2.5.2... :-(

Posted by William Vambenepe on 2009-02-11


I was talking about the next release of httplib2, not httplib which is a part of the Python standard library.

Posted by Joe on 2009-02-11

Joe: yes, of course, sorry I read too fast. In any case, my point was to whine about the lack of timeout (added in 2.6) in the version of urllib/httplib2 supported by GAE. But just today (or at least it was announced just today) you guys made an even better fix: no more "High CPU Requests". Avoiding these dreaded beasts is what was motivating my quest for timeout support. I am a happy man tonight because I can now emulate a long-running process on GAE. At least as a toy project. Thank you.

Posted by William Vambenepe on 2009-02-13

comments powered by Disqus