'Yield From' Project: use the Python stdlib web modules from Asyncio

The Asyncio framework is the best thing to happen to Python since, say, Unicode support. Hopefully it won't take as long for Asyncio to find whole-hearted community support as it took for Unicode.

Without going into the basics of Asyncio, I'll say that the appeal has a lot to do with the use of yield from to call coroutines, so you can control the sequence of asynchronous functions without having to chain callbacks or use promises.

There is a fledgling set of modules to support networking in Asyncio with the HTTP protocol, (see aiohttp) but I have not been overly pleased with using it as it does not seem very modular.

There is a very solid set of http networking modules in the standard library, and a slew of both standard library and PyPI modules rely on them. I believe these standard library modules can be updated to work in an asynchronous way in Asyncio without major changes to the API.

With these converted modules, it will be straightforward to port many of the depending modules to Asyncio. The class and method names do not need to change, but those methods that have input/output latency will become coroutines, and the yield from expressions propagate up the call stack, where methods and functions that call coroutines become coroutines themselves.

The aiohttp modules have a different API, and converting HttpLib dependent modules to use aiohttp under Asyncio would take a more substantial effort.

There is also a mental burden adapting to new APIs, having to remember the new method calls and how they differ from the old way. Kenneth Reitz's Requests module is a wonderful web client library, for example, and I want to use it under Asyncio in lieu of learning another client. You should want this as well. ;) We already know how to request a page with Requests:

import requests
r = requests.get('http://www.rdbhost.com', auth=(usr,pass))
pg = r.text

Under Asyncio, we would like to write:

from yieldfrom import requests
r = yield from requests.get('http://www.rdbhost.com', auth=(usr,pass))
pg = r.text

and have it work. Same options, same call sequence, same elegance.

Soon, this will be so.

I have started this effort, so far completing1 the conversion of the Python3 http.client module, including its test suite. That new module is on Github, at yieldfromHttpLib. It will eventually be on PyPI, but not until I have more done, and can organize a package with multiple modules.

Next, I will be converting the Urllib3 module, as that seems to be basis of the new cool web modules, such as Requests. Then Requests itself, and urllib.requests, as it is a dependency of those many PyPI web modules.

Eventually, I will be converting select portions of the Boto library for AWS.

Please use these converted modules as they are published, file bug reports or pull requests for patches, and tell people.


1 I say 'completing', but there is one known deficiency; the tunnel test does not consistently pass. The http.client tests used a FakeSocket, but Asyncio demands a true socket, so the test has been rewritten, and I do not understand the tunneling technology sufficiently to write a good test. The conversion only lightly touched the tunnel methods, so they may well work, but the test doesn't show it.

comments powered by Disqus