Inside Amazon Web Services

From storage to payment, the king of clouds is dangling an array of low-cost services. We take a close look at the tools for IT and developers.

Two aspects of FPS are especially interesting. First, it supports micropayments, those that involve cents – or even fractional cents. This is useful when business activities involve piles and piles of transactions, each having little monetary value, but the sum of which has measurable value. Imagine selling bubblegum for 10 cents. That doesn't seem like much – unless you're selling, say, 100,000 pieces a month. Amazon FPS lets you aggregate micropayments into a single transaction, thus eliminating the problem of transaction costs swamping whatever profits the transactions involve.

FPS's other interesting aspect is its support for "middleman" operations. That is, you can facilitate a transaction in which you participate neither as a sender (buyer) or recipient (seller). You can, however, take a cut of the action.

There are two ways to employ FPS in your Web application: using an Amazon-supplied "widget" (of which there are two), or hard-coding an interface. The two available widgets are Pay Now and Marketplace (both designed to be easily added to a Web site's UI).

Amazon has automated the creation of Pay Now widgets. Connect to the online Pay Now Widgets Implementation Guide, and it walks you through the process of building a widget by prompting for various parameters (for example, the destination URL after the payment has been placed), then generates the HTML that you cut and paste into your Web site's code. The Marketplace widget lets you act as a third party between buyer and seller. In essence, it turns you into an instant reseller. You can use a MarketPlace Widget to let sellers do business on your Web site and pay you for the privilege.

The hard-coded approach is more difficult, but more flexible, as it enables any application that can communicate with a Web service to tap into FPS. You have to express the parameters and processes for payment transactions in a specialized mini-language called Gatekeeper. Once you've done that, you install those instructions into the Amazon FPS, which returns a token that is essentially a handle to the Gatekeeper code. Future transactions that employ that token are shepherded by your Gatekeeper program. Details for this process can be found in the online Amazon FPS Technical Documentation.

Amazon DevPay. Suppose you've written an amazing application that runs in Amazon EC2. You're convinced that people would be willing to pay you to use your application. Enter the Amazon DevPay Service.

Amazon DevPay is built on the same payment management infrastructure as Amazon FPS. But DevPay – as its name attests – is designed specifically to let developers charge for the use of their EC2- or S3-based applications.

Interaction with DevPay takes place via tokens (unique identifiers). One token identifies your application; the other identifies a specific user allowed to employ your application. The first, the product token, is generated by Amazon when you register your product with DevPay. That token, combined with a user's activation key (created when the user signs up with AWS), is implemented during product installation to generate credentials that include the second token, the user token. Your product embeds these tokens in service calls it makes to AWS, and in that way, DevPay tracks your application's usage by a given customer.

When you register your application with DevPay, you establish how your application is priced. Users can be billed on a metered (pay for what they use) basis, they can be charged monthly, or they can pay a one-time up-front fee. Of course, you have to be careful how you structure your billing. While your clients pay you for the use of your application, you must pay Amazon for the use of its services. So, at the very least, you have to make sure that your customers pay you more than you pay Amazon. Unfortunately, Amazon does not provide a sandbox for testing your application's integration with DevPay, so you have to do your testing with real money. Fortunately, the cost of Amazon services is low enough that this is not a substantial problem.

Amazon Associates and Amazon Fulfillment Web Service (FWS). Anyone who has clicked through a site to order something from Amazon has used Amazon Associates: It's the service that lets you sell Amazon stuff from your Web site. You get a percentage – a referral fee – for each sale. There is not much more to be said about Amazon Associates.

A more interesting Amazon e-commerce service, however, is a remarkable kind of inverse of Amazon Associates: Amazon Fulfillment Web Service. With FWS, instead of your selling Amazon stuff, Amazon sells your stuff. Not only that, but Amazon will also warehouse, package, and ship your stuff.

FWS is actually two Web services: inbound and outbound. You use the inbound system to inform Amazon of incoming shipments bound to their warehouse. When a customer orders one of your products, you use the outbound service to inform Amazon of the sale. Based on the details of the order, Amazon packages and ships the product, and even provides tracking information that you and your customer can use to monitor the shipment's status.

Of course, there are warehousing and handling fees involved, but it's a compelling model. A small company, unable to afford warehousing and shipping costs, can "virtualize" those components with Amazon FWS, until that company is large enough to provide them for itself. And any developer interested in exploring the mechanics of the inbound and outbound services will be happy to discover that Amazon has provided "scratchpad" applications – tools that let you exercise simulations of the services.

Mechanical Turk. Amazon's Mechanical Turk is a peculiar service. (It is difficult to categorize; I have listed it with the other e-commerce services.) Its name comes from the famous 18th-century robotic chess player invented by Wolfgang von Kempelen. The robot, however, was no robot; inside the machine was a human chess player who operated the mechanism, unbeknownst to the human opponent. The idea of Mechanical Turk, then, is an automated front end, behind whose machinery hides a human.

Only, in this case, it's not just one human; there're lots. Whereas EC2 provides an elastic cloud of computers, Mechanical Turk provides an elastic cloud of humans. But this analogy goes only so far; the computers in EC2 are virtual, the humans of Mechanical Turk are not.

Here's how it works. Suppose you have a big pile of identical tasks that must be performed by humans. Perhaps you have a large quantity of text files that must be translated from one language to another. In the world of Mechanical Turk, you are a requester; you submit your tasks to the Mechanical Turk service, which places them on a kind of global bulletin board. Using that same service, workers log onto this bulletin board, select tasks, perform them, and post the results back to the service. Later you return to the Mechanical Turk, review the posted results, select those that are acceptable, and release funds to pay the workers. In short, the Mechanical Turk service is a middleman between employers and employees.

When I first read Mechanical Turk's description, I thought it was a great idea. It may yet be, but if my perusal of the tasks that are available is any indication, this is not a way to make any appreciable amount of money. Most of the HITS ("Human Intelligence Task," referring to a unit of work) posted paid mere pennies, and reading some of the descriptions gave me the uneasy feeling that workers would be used as human spam-bots.

It is possible that, in the future, Mechanical Turk will become a marketplace of decent work for reasonable money. For now, though, I am confident that I can make more money in less time – and do more good – by mowing the old lady's lawn next door.

Wading into Web Information Services
Amazon's Web Information Services are essentially query interfaces into extensive databases generated by a mixture of Web crawlers and Web traffic monitors. Data-mining organization can tap into the crawler-produced data to sift through information that is as wide-ranging as the Web itself. The utility of Web traffic data is self-evident to any company or individual interested in user visitation trends to their sites – as well as to related or competing sites.

AlexaWeb Search. Amazon's Alexa Web Search is the result of partnering between Amazon and Alexa, and it lets you query the information gathered by Alexa's Web crawler bots. The quantity of information available is difficult to gauge; Alexa has been crawling the Web for over a decade, and the Internet is in nonstop growth. Alexa's site says that, while its bots are working constantly, it takes about two months for a complete cycle through the Internet.

When Alexa adds a new Web site document to its database, it indexes about 50 attributes associated with that document. Attributes include the document's language, its Open Document Category, various parsed components of the URL, geographic location of the hosting server, and more. Also available is the document's text, the first 20KB of which is text-indexed. All this is available for searching.

Naturally, searches on such a large database can take time. The Alexa Web Search service is architected so that when you issue a search, the service returns a request ID. You use this ID to track the status of your search's progress. When the search is complete, results are stored in a (possibly gigantic) text file. The text file can be downloaded and "mined" locally.

Alexa Web Information Service (AWIS). The Alexa Web Information Service lets you dip into traffic data gathered by various Alexa tools deployed about the Internet. You can query information for a specific URL, such as site contact information, traffic statistics (going back five years), and more. You can also discover how many links are on a given page, how many URLs are embedded in JavaScript, or the more interesting statistic of how may other sites link to the target ("inward-pointing" links). You can also use AWIS to fetch a thumbnail image of a Web page, useful for displaying pop-ups in response to a cursor hovering over links.

The accuracy of Alexa's data is unclear. The Alexa Web site states that the "traffic data are based on the set of toolbars that use Alexa's data, which may not be representative of the global internet population." Meanwhile, an Amazon Web services representative informed me that Amazon "aggregate[s] data from multiple sources to give you a better indication of Web site popularity." In any case, the ability to scour the text content of whole swaths of the Internet makes the Alexa Web service a profitable vein for Web data spelunkers.

Ready for the big time?
Amazon's Web Services are at once exciting and troubling. The infrastructure services adopt a sort of "mercenary" model of hardware and software horsepower; in theory, you can employ as large an army of computing power as your pocketbook can withstand. All the services offer universal availability – if your network connection can reach Amazon, it can reach AWS. These are two powerful isotopes for fueling large-scale, on-demand, software services.

On the other hand, however, some of the important components are still in beta. SimpleDB, in fact, was in limited beta and not accepting new users at the time of this writing. The description of "beta" is off-putting, as it implies an architecture whose foundation has not yet solidified. And this implication became hard reality when, in June, Amazon's S3 suffered a temporary power outage that affected such high-profile users as the New York Times, whose archives were crippled.

Furthermore, the long-term security of the entire AWS remains to be seen. We can only take Amazon's word that its systems guarantee isolation of one user's applications from another's. Put simply, AWS is only going to work if its users' trust in it is complete. A security breach of any sort would likely be a mortal wound.

Programmers and architects of distributed systems will find the infrastructure pages on the AWS site to be nothing short of a playground. You can spend hours perusing the documentation, tutorials, examples, and references to community-supplied tools and libraries.

The "cloud" services – EC2, S3, SQS, and SimpleDB – are certainly compelling. Real applications are being built atop these virtual technologies. Examples can be found at the Amazon Web Services Elastic Compute Cloud resources page.

Some of the AWS components are of questionable utility. In particular, Mechanical Turk seems to create a built-in incentive to cause tasks to be priced below what they otherwise would. However, even the Turk might be a case of a technology ahead of its time. As the ability to conduct business over the Net continues to improve, perhaps Mechanical Turk will also.

Whether the notion of Amazon's "rentable infrastructure" catches on is unknown. Its failure (should it fail) will not be for lack of information and tools. I will be eagerly prowling the AWS Web site and AWS-relevant blogs to see what creations arise from the enticing techno-tinker-toy set that AWS represents.

| 1 2 Page 3
From CIO: 8 Free Online Courses to Grow Your Tech Skills
View Comments
Join the discussion
Be the first to comment on this article. Our Commenting Policies