davidbau.com

Simple Cross-Domain Javascript HTTP with call.jsonlib.com

In Python, fetching any webpage in the world is a one-liner:

print urllib2.urlopen('http://davidbau.com/data/animals').read()

But Javascript's same origin policy prevents you from doing the same thing in Javascript unless your script happens to be running on a page from the same domain. The SOP is intended to protect the security of the user by limiting access to private resources such as:

1. Private browser state like cookies from other domains.
2. Private web pages that may be only accessible behind firewalls.

While this is a nice thing for protecting logged-in banking sessions and secrets on corporate LANs, it also means that plenty of perfectly safe and useful network code cannot be written with Javascript. You need to do your cross-domain networking server-side, and then bounce your requests off a server in your own domain.

JSONP Lets Servers Expose Services Cross-Domain

Interestingly, the "src" attribute of the <script> tag is not subject to the same origin restriction, which means that servers can intentionally expose services cross-domain by encoding their data as a javascript function call.

In 2005 Bob Ippolito proposed standardizing this convention as a format called JSONP. Today, JSONP can be used to make cross-domain calls to various useful APIs including Flickr, GData, Twitter, and YQL. JSONP is incredibly useful and is supported in all the major AJAX libraries such as JQuery.

YQL in particular is very versatile, and it actually allows you to make your own further HTTP requests to other domains. However, it does a lot more and is a bit complicated and slow as a result. So for the specific purpose of making secure cross-domain HTTP calls on the public internet, I have been using a service I've posted on Google App Engine, called call.jsonlib.com.

Cross-Domain Javascript HTTP Requests on the Public Web

The library at call.jsonlib.com lets you get very close to the python one-liner:

jsonlib.fetch('http://davidbau.com/data/animals', function(m) { alert(m.content); });

It is just a simple packaging of the python urllib2 library as a JSONP library. But it is tremendously useful. It allows you to:

• Make GET or POST requests to arbitrary public servers
• Set arbitrary HTTP headers
• Read all the HTTP headers of the response

Because all the requests are made from the context of a server on the public internet, it does not expose your private LAN or private cookies to security holes.

A few corners are smoothed out to make use from javascript easier. For example:

• jsonlib.fetch is intelligent about encodings and converts the content to unicode based on HTTP headers, content sniffing, and <meta> tags in the HTML content, if any.
  
• jsonlib.fetch can also do basic scraping tasks server-side such as extracting elements that match a particular css selector, or stripping HTML markup and returning just text.
  
• jsonlib.fetch is careful to access the proxy using https if the underlying URL being scraped is https, so that the whole path is encrypted.
  

More documentation here.

Some examples here.

Posted by David at January 29, 2011 06:47 AM