To most, the virtues of Web 2.0 are rather ephemeral; that’s always been one of its main criticisms. However, I like to think that one of the movement’s key aspects is a sense of community, an ability to create sites and applications that bring people together.
One of the most powerful ways to bring people together on the Web is to give them ownership of the sites that they frequent - make them active participants in the development and growth of the product they’re using. There’s many different approaches you can take to implementing this idea, but I’m going to talk about just one of them: setting your data free.
Data sharing isn’t a new idea. It’s a concept as old as the Web, in fact, the Web is all about sharing data. However, most of the data available on the Web is designed to be consumed by humans, through a browser. It’s encapsulated by HTML pages, trapped inside Flash files, and disguised by design. There used to be a time where, if you wanted to get any real data from a site that wasn’t your own, your only recourse would be to grab the raw HTML and scrape through that tag soup for the gem that you were looking for. Fancy writing a 15-line regular expression to get the weather off Yahoo!?
However, data providers are slowly realising that offering their data in much more flexible formats can only be an advantage to them - they get used by more applications, they get seen by more eyeballs, they get more exposure.
RSS is a great example of this. Strip all the images, colour, layout, font styles and browser dependence from a Web page, and what do you have? A lightweight, versatile data format that can be easily interpreted by pretty much any software out there. Two years ago it was barely on the radar. Now your green grocer has an RSS feed that tells you when the latest shipment of fresh guava has arrived.
RSS is good, but it’s still a bit restrictive for real application development - the content provider tells you exactly what you’re going to get, and when you’re going to get it. What if, instead of getting the last 10 blog posts from your favourite incendiary author, you could get their last 10 posts that were tagged with “free beer”; or you could get their last 10 posts that were tagged with “free beer” and were written in Melbourne?
What if you could take their last 10 moblog posts that were tagged with “free beer” and written in Melbourne, get the geolocation data of each post, plot the exact position of each free beer spot on a street map and show it in relation to your current location? That’s a good night out. And APIs can let you do it.
Application Program Interfaces are, broadly speaking, tools for building software applications. An operating system API might help you build applications with a consistent interface, but Web APIs are mostly about giving you access to data.
Even Web APIs aren’t a new idea. Google’s search API has been available via SOAP since 2002, and there’s definitely older services than that. However, the recent growth in Web API availability has been fuelled by two recent developments. The first, which I’ve already mentioned, was a philosophical change in the way that data is handled. The second was the introduction of AJAX. Again, not a new idea, or even a new technology, but sometimes it’s all about timing.
Some of the more interesting JavaScript-friendly Web APIs that are available at the moment are:
- Flickr (Photo sharing API)
- Del.icio.us (Social bookmarking API)
- Google Maps (Mapping API)
- Yahoo! Maps (Mapping API)
- Upcoming (Event planning API)
- Yahoo! Search (Search engine API)
- eBay (Online auction API)
- Amazon (E-commerce API)
The mapping APIs have received the most attention to-date, mainly through mapping “mashups” such as chicagocrime.org, or dartmaps.mackers.com. They’re also probably one of the least flexible, because you are constrained by the mapping interface that the provider gives you. When you include the Google Maps API on your page, you gain access to a number of JavaScript functions that allow you to move and zoom the map, or add your own data and visual components, but you aren’t able to manipulate any of the original data, beyond what the existing JavaScript methods allow you to do. This is perhaps just a constraint of maps themselves - how many different forms can mapping data take? A good tutorial on how to create an application with the Google Maps API is available at Hacking Maps with the Google Maps API.
Other forms of data, however, offer greater possibilities.
These web applications use AJAX to query the various non-map APIs and create new and unique ways to interact with data:
- Delicious Director (Uses the del.icio.us API)
- Yahoo! local events browser (Uses the Yahoo! search engine APIs)
- FamiliarFaces (Uses the upcoming.org API and the Yahoo! image search API)
- Mindsack integrated searching (Uses the Yahoo! web search API)
We’ll be taking a look at the how they access those APIs, so you can get right in and make your own data-sharing application.
Accessing Web APIs with AJAX
Most Web APIs are accessed via HTTP, so they’re always going to be available to server-side scripts, however, the nexus that AJAX enables between user interaction and data communication, as well as the ease with which it can be incorporated into webpages, means that the use of APIs by client-side scripts offers some of the most exciting possibilities for Web applications.
The API for Flickr has a number of methods that are specified by passing particular CGI parameters to a base script. For instance, to get a list of the most recently uploaded photos to all of Flickr, you would access their script:
http://www.flickr.com/services/rest/?method=flickr.photos.getRecent&api_key=XXXX
The “method” specifies exactly which method from the API you want to execute, and “api_key” is a way for Flickr to track who’s using what from their API. If you want to see the data that this call returns, simply sign up for your own Flickr API key, insert it into the string above, paste that string into your browser location bar, and you’ll get a nicely formatted XML file in return. Something like:
The same process applies for an XMLHttpRequest via JavaScript - you specify the URL above to your XMLHttpRequest object, and the client browser will go off and fetch it. (For a more in-depth look at creating XMLHttpRequest connections, see Remote Scripting with AJAX) Because you get XML in return, you’ll be able to use JavaScript’s in-built DOM functions to parse the data in much the same way as you do for HTML; navigating through elements, getting attributes, etc. However, as with any XMLHttpRequest, the data doesn’t necessarily have to be XML, so a Web API could send back a simple text string instead. In fact - as we’ll see later - that’s what JSON does.
XMLHttpRequest Proxies
Even though it’s theoretically possible to request this data from Flickr using XMLHttpRequest on the client-side, there’s one problem: cross site scripting. Currently, most browsers will not allow you to get data from other sites using XMLHttpRequest. Due to security restrictions that prevent cross site scripting (XSS), they will not let JavaScript access a URL that is outside of the domain that the current page is being served from. This even applies to sub-domains on the same site, for example www.example.com versus dev.example.com.
This is not an intractable problem, though. There are a couple of ways you can get around it, but they all require the server on the current domain to act as a proxy for the external data. This works because server-side scripts generally have more privileges than client-side scripts, so they are able to access any address on the Internet. With a server-side proxy, we create a script that accesses a remote server, gets the data and relays it back to the browser, making it look like the data was sent from www.example.com, when in reality it came from www.upcoming.org, or some other website which we proxy data for.
Apache Proxy
The easiest way to proxy data is to get your web server to do it for you. You can set up a proxy in Apache by making sure the mod_proxy extension is loaded and editing the httpd.conf file to include your new proxy:
ProxyPass /proxy/flickr/ http://www.flickr.com/
ProxyPassReverse /proxy/flickr/ http://www.flickr.com/
The first line forwards any requests from your server to Flickr’s, the second line handles any redirections that the Flickr server might respond with.
However, if you’re running a site off a shared server, you more than likely won’t be able to edit httpd.conf, so you can always use mod_rewrite to redirect requests by editing a .htaccess file. If I put this inside the .htaccess file for the directory /proxy/flickr/:
RewriteEngine on
RewriteRule ^(.*)$ http://www.flickr.com/$1 [P]
Anytime I make a call to a path inside http://www.example.com/proxy/flickr/, it will pass on all extra URL information to http://www.flickr.com/. So, when I reference http://www.example.com/proxy/flickr/services/rest/ it’s actually passing the request through to http://www.flickr.com/services/rest/. The [P] at the end of the RewriteRule means that the re-direction is passed through without informing the user’s browser that a change of URL has occurred.
Application Proxy
The second method to proxy data is to create a script that will do it for you - receive request URLs, create a connection, get the data, and return it to the requester. The exact code that you use to do this will differ depending upon what language you’re using, but if you are running PHP with the Curl extension installed (for creating external connections), then you could use this as your proxy script:
$remoteServer = "http://www.flickr.com";
$path = $_GET[”path”];
$proxyTarget = $remoteServer.$path;
$connection = curl_init($proxyTarget);
curl_setopt($connection, CURLOPT_HEADER, false);
curl_setopt($connection, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($connection);
curl_close($connection);
header(”Content-Type: text/xml”);
echo $data;
This script assumes that you are passing a CGI parameter called “path” that specifies what location you want to retrieve from the remote server. Note that we don’t directly pass on this parameter to the Curl connection, as we don’t want this script to become an open proxy. Instead, we specify the exact address that we are using the proxy for - www.flickr.com - and this will limit its potential for abuse. You’d call it from a browser using this URL:
http://www.example.com/proxy.php?path=/services/rest/
If you’re sending that over an XMLHttpRequest connection, you should remember to URL endcode it using escape(), so that any special characters inside the path are passed along properly.
When we initialise the Curl connection, we tell it that we don’t want it to pass on any header information, but we do want it to pass on the content of the request. Then, once the data has been received we write the appropriate header back to the client browser and also forward on the data which we retrieved from the remote server.
The downside of proxies is that you’re double handling your data every time - a request has to come to the current domain server, which then passes it along to the actual remote data server. Then the data has to go back to the current domain server before it can be sent to the client browser. This extra traffic will always introduce some delay; the amount of which depends on the network architecture and speed of the highway at any given time, so you’ll have to ask whether that delay (however minimal it is) is acceptable to your service.
JSON
Another way to avoid XSS issues with XMLHttpRequest is to not use it all. How do you do that? You use JSON.
JSON (short for JavaScript Object Notation) is an alternative format to XML. It actually stores data as an object literal - a structure that is natively available in JavaScript. So if you receive a JSON string you can use the JavaScript evaluation function eval() and it will automatically create an object structure that corresponds to the data that you want to retrieve. Although object literals are native to JavaScript, they’re farly easy for other languages to parse. Indeed, at json.org there are links to JSON parsers for over 20 different programming languages.
Using eval() on a text string that you receive from a 3rd party can be a little scary - it opens up all sorts of security holes that could be exploited in your code, so you really have to trust the data provider if you’re blindly using JSON. There’s some parsers available that have been written in JavaScript by Douglas Crockford - the creator of JSON - that will offer some protection against malicious code. These are also available at www.json.org.
Douglas calls JSON the “fat free alternative to XML”, mainly because it’s less verbose and requires less parsing than XML, particularly if you’re using it from inside JavaScript. If we translated the XML we had from Flickr earlier into a JSON structure, it might look like this:
{
"rsp": {
"stat": "ok",
"photos": {
"page": "1",
"pages": "10",
"perpage": "100",
"total": "1000",
"photo": [
{
"id": "169563197",
"owner": "98319521@N00",
"secret": "f4002ac03f",
"server": "67",
"title": "turkije021",
"ispublic": "1",
"isfriend": "0",
"isfamily": "0"
},
{
"id": "169563188",
"owner": "17208601@N00",
"secret": "fbf107bffb",
"server": "56",
"title": "PICT0221~1",
"ispublic": "1",
"isfriend": "0",
"isfamily": "0"
}
]
}
}
}
It’s entirely possible to use JSON via XMLHttpRequest - make a request to the server, receive a text string and evaluate it - but one of its real advantages is that you don’t need XMLHttpRequest in order to get it. Because JSON is a JavaScript string, if you specify a server-side script that returns JSON as the source for a
two things: json allows your objects to be created with data types intact i.e. everything doesn’t have to be a string. for example, “page”: “1″ could be defined as page: 1 which more closely represents the proper data type for a numeric.
second thing, how do you tackle versioning of an api? making a simple change to a method could cause you to break many clients that are consuming your api.
True, you can use any valid JavaScript type as the data value for a JSON property, but the Yahoo! API normalises most values into strings, so that’s the way it would be represented in the Flickr example.
Versioning of an API would be fairly easy to do via a URL structure, i.e. .com/services/v2/… but it’s mainly up to the provider to do this. True, if they have a singular API that they’re updating it can break third-party applications. This is more of a worry in higher level APIs such as Google Maps, where you’re relying heavily upon the code that Google give you, rather than a straight data API such as Upcoming or Flickr.
Great write-up, Cameron. This is something I’ve been researching lately. My route had been to go with a PHP script, but I love the mod_rewrite idea (and love the fact that you included it in addition to the httpd.conf solution — we little web design agencies can’t afford our own shiny servers).
I’m attempting to expand content on my website by gather new contributors. Problem is, most of these contributors have their own website and don’t like the idea of having to post the same stuff twice. So, my plan is to write various APIs for them to pull the data from my server and put it on their site. This article has defined a clear path for setting out and doing this.
Awesome! Thanks again.
[…] Go Forth And API - an awesome article from Cameron Adams over on Vitamin. […]
Great article and just in time! I am currently designing my blog and using Flickr API to users’ avatars rom flickr by email as well as ma.gnolia API to show recommended sites on the home page. Thanks, Cameron, for great article once again.
[…] Vitamin Features » Go forth and API (tags: ajax javascript web2.0 mashup api) […]
Great article. The point of view is exactly what I wanted to hear: from perspective of JavaScript developer. When I asked a question about API implementation on local development forum I got replies mostly from desktop developer or web developer that used only PayPal API, Authorize.net integration etc.
I really like the idea where Flickr API is used for async requests in Flickr interface (I’ll use something similar for activeCollab). When accessing API from other site (or desktop application or whatever) you’ll need to send API key request first with proper username and password. When you are accessing website through activeCollab interface API key is automatically generated by the script and provided to JavaScript (because your already logged in and there is no need for second authentication).
What do you think about that kind of API usage (to use application API for building its interface)? Possible problems?
Btw, sorry for my poor English.
One problem that this article doesn’t touch on is that of caching. Most APIs only respond to a limited number of requests - usually about 1 per second. For localised testing, there’s no need to worry about caching - but once you set things live, things become a different matter. API providers might stop serving requests temporarily, or may even block your API key if you make too many requests a second.
Also, the API provider may set limits on what you can cache - Amazon only let you cache product data for 24 hours, and price data for 1 hour, IIRC.
So make sure you have some form of caching solution in place, or else you may find your API data missing due to throttling or, worse, banning.
The other thing to consider is the suitablity of JavaScript as an API interface. Clients without Javascript, or clients which don’t notify users to updated elements (eg screenreaders) might miss out on some of your content. To get around this, consider implementing your display of data via APIs on the back-end, where the page will be rendered as a whole and then sent to the browser. PHP scripts still have their place.
Obviously, for dynamic functionality - AJAX livesearch, for instance, JS is more necessary, but you still need to implement a page-reload based alternative.
Tom -
One dirty hack I’ve been using for this is to throw every URL that gets pulled down (doesn’t matter whether its an API endpoint, HTML or RSS) into a temporary file based on the URLs md5 hash.
You can then expire the cache based on last-accessed times (reading it off the filesystem).
Well, I think it goes unsaid, perhaps, that you should never have Javascript solely providing content, no matter where the data is coming from.
[…] read more | digg story […]
[…] Vitamin Features » Print » Go forth and API (tags: tutorial software ajax javascript) […]
[…] Vitamin Features » Go forth and API […]
Good Article !!! Another way to avoid XSS while communicating data one way (from client to server) is to use the IMG tag. Google Analytics use 1×1 images to send analytics data to its servers.
[…] Backend scripts aren’t the only way to access the wealth of valuable web services out there. Anyone that’s done any playing around with Ajax can see the possibilities of combining the two - accessing the APIs directly from your client-side application. In this brief tutorial from ThinkVitamin.com, they follow this course and give you a simple example to get the ball rolling. […]
[…] Safari gets a Javascript debuggerSpeeding up Prototype’s $$ SelectorTuesday Morning Roundup - Prototype, Safari and commas, and LesserWiki, JS -> Rails Serialization…Ajax Activity IndicatorsMODx CMS - An Ajax/PHP Content SystemWeb API authentication for mashupsOPML IconExplaining AJAXEcho2 Widget PanelSlightly ThickerBoxJSON.NETJson.NET: Library to help with .NET - JS communicationuniAjax: an ajax framework focused on browser supportPayPerPost: Right or Wrong?Atlas June CTPRelay: Ajax File ManagerDojo Available in Ning ApplicationsIntelliJ IDEA Google Web Toolkit SupportGoogle CheckoutPrivate and Public Members in JavaScriptJavaRef: Ajaxified JavaDocInterview with ZK Creator Tom YehLondon Tube Route Finder Backend scripts aren’t the only way to access the wealth of valuable web services out there. Anyone that’s done any playing around with Ajax can see the possibilities of combining the two - accessing the APIs directly from your client-side application. In this brief tutorial from ThinkVitamin.com, they follow this course and give you a simple example to get the ball rolling. To most, the virtues of Web 2.0 are rather ephemeral; that’s always been one of its main criticisms. However, I like to think that one of the movement’s key aspects is a sense of community, an ability to create sites and applications that bring people together. Even Web APIs aren’t a new idea. Google’s search API has been available via SOAP since 2002, and there’s definitely older services than that. However, the recent growth in Web API availability has been fuelled by two recent developments. The first, which I’ve already mentioned, was a philosophical change in the way that data is handled. The second was the introduction of AJAX. Again, not a new idea, or even a new technology, but sometimes it’s all about timing. They mention several of the different APIs out there (including Flickr, Google Maps, and Amazon) as well as some of the mashups that take advantage of the merging of these APIs. To illustrate the point of the simplicity of these interfaces, they include a code example of connecting to the Flickr API to grab photo information. There’s also a bit on proxying your XMLHttpRequests and a brief look at using JSON to communitcate with the APIs that support it. July 11th, 2006 […]
Great article!! There’s also a client-side
mashup tool called xfy. See the following link..
xfy mashup
[…] API usage with AJAX Contains clear and concise information on getting around browser security restrictions that prevent cross site scripting. You can’t always xmlhttprequest() remote sites, so you’ll need to use some sort of proxy or JSON. […]
Hey, Great writeup. One thing you are missing is a closer look at Google’s newest search api. It offers parallel search across a number of properties including web, blog, video, and local. More to come as well. It’s 100% buzzword compliant based on AJAX, JSON Encoded REST-ful query, HTML Microformats, etc.
http://code.google.com/apis/ajaxsearch/index.html
[…] During a recent trip to the grocery store, I noticed “Pow’r Detergent” from HomeBest. I was quite intrigued. One of the hallmarks of the Web 2.0 revolution has been dropping the last vowel in the name and usually adding a single apostrophe in its place. After recently reading an article about API’s and the SOAP protocol, I was curious and decided to purchase* it and see how it improved my Web 2.0 browsing experience. […]
[…] Practical advice on how best to use APIsread more | digg story […]
[…] Pensate ancora che le API siano dei simpatici insetti che a tempo perso producono dell’ottimo miele? Allora è il momento di leggere “Go forth and API”, un ottimo articolo che spiega le famigerate Application Program Interfaces. […]
[…] I was wary of this one as I thought it might be a repetition of what Cameron Moll had to say about the same subject but Jeremy gave a very engaging talk about how much joy he gets out of using APIs and realizing that you can turn any web site into an API by embedding microformats. Jeremy showed several APIs (Amazon, Google Maps, Flickr, delicious) and matched them against in other in terms of fun, ease of use, documentation and power. He also pointed out that in order to really use APIs you need to know programming, and that this should change. As a remedy he pointed out that some services like Amazon (and soon Ebay, as I heard at barcamp) offer XSLT as an option, which means you don’t need to use PHP or JavaScript to use the APIs and you still get nice results you can format easily. He also showed that there are some sites that offer people the option to clone other API implementations. I see the point and ease of these XSLT options, but I do disagree to a certain extend that they are a good option. Having suffered XSLT on an enterprise level for some years I don’t see it as very effective (especially as it is very heavy on the server computation side). Having used APIs I also realized that it is very cool and sexy to implement them with some lines of PHP or JavaScript but to really use them in a secure and reliable fashion you will need a proper server side component to do much of the heavy lifting, caching and optimization for you. In a lot of cases pulling the API data on the server, cleaning it up and spitting out optimized JSON makes up for a much quicker and faster application than going to a lot of third party providers on every page load. There is no argument on the usefulness of Microformats and I am all over them, too. It was very much fun to see Jeremy give a “Microformats Picnic” talk in the park during lunch break, especially as the backdrop was a Indian Style Palace with Sitar music. Passers-by seeing Jeremy preach in the middle of a listening crowd must have gotten a really quaint impression and I wonder if Jeremy hasn’t started a cult of some kind. If you see confused looking people with cymbals chanting “hCard, hCarc, hCard” just point them to Clearleft office. […]
[…] Discover how to put the web’s finest free data to use in your own creations. […]
Pilot Online Training Solution’s powerful enterprise-class technologies have made it possible for educators, administrators, and Learning Service Providers to create, launch, and evaluate learning over the Internet using nothing other than a standard browser.
a pipes-like app, feedmashr.com mashes feeds, but also stores the links, so that users can fetch past feeds.
What exactly is a “JavaScript-friendly Web API”?
If AJAX can only be used with on-server proxies, aren’t all web services “JavaScript-friendly”/
A quite interesting idea is realized in this website! And a good and easy to handle design has been found too!
great site with very good look and perfect information…i like it
Very nice article from 2006 but I have just found it and I want to thank you a lot. Added to my bookmarks.
Good article, thanks a lot for putting all this together.
[…] An interesting approach is presented by Cameron Adams in his great article Go forth and API. Cameron suggests to take advantage of mod_rewrite or mod_proxy module in Apache in order to redirect our calls in external domains; a simple but ingenious solution! However, the most common solution is the application proxy which is accompanied by some advantages outlined perfectly well by Jonathan Snook: […] you have more control over the entire lifecycle. You can parse the data from the remote server, do with it what you will before sending it back to the client. If anything fails along the way, you can handle it in your own way. And lastly, you can log all remote calls. With that you can track success, failure and popularity. Cross Domain Ajax: a Quick Summary […]
This blog posting was of great use in learning new information and also in exchanging our views. Thank you.
vitamin…
Hello. Great post! I agree 100%!…
Very useful article
Thank you
good luck
Good work Cam,
I think the point that Tom Armitage raises is a valid and important one. The ability to fall back to a cached response is a simple and effective solution. I wonder if there are other solutions that are being thrown about and how they stack up. I personally haven’t gone down this route yet but I have seen the warning signs from a mile off!
Great work, prompt delivery would buy again A+++
good job
good perfectt
Pilot Online Training Solution’s powerful enterprise-class technologies have made it possible for educators, administrators, and Learning Service Providers to create, launch, and evaluate learning over the Internet using nothing other than a standard browser.
[…] Practical advice on how best to use APIsread more | digg story […]
Ajax or similar technologies will revolutionize the web. There’s no doubt about it, this ist the future.
Nice
thnks
Really Great
Fantastic creation
Good job Done
Well done
Fell free to comment
Great post
write up more
I think a balance between Marketplace and your own site will be the key. But no details about WP Marketplace have been revealed yet? So it might not even be possible to link back to your site
great site with very good look and perfect information…i like it
It is a very interesting article and thanks a lot for putting together such competent information.
well done, nice and simple method with mod_rewrite
regards
[…] Go forth and API Jul 03, 06 Application Program Interfaces are, broadly speaking, tools for building software applications. An operating system API might help you build applications with a consistent interface, but Web APIs are mostly about giving you access to data. Even Web APIs aren’t a new idea. Google’s search API has been available via SOAP since 2002, and there’s definitely older services than that. However, the recent growth in Web API availability has been fuelled by two recent developments. The first, which I’ve already mentioned, was a philosophical change in the way that data is handled. The second was the introduction of AJAX. Again, not a new idea, or even a new technology, but sometimes it’s all about timing. […]