The engineering team at DreamFactory designed and built some of the very first applications that used Web services. Over the years, we learned many lessons trying to create the perfect back end for these applications. We also realized we encountered many of the same server-side problems that other companies deal with when they build custom REST APIs from scratch. Let’s take a look at some of the technical characteristics and architectural advantages of a back end that provides reusable REST APIs.
When a company starts a new mobile or Web project, it usually fleshes out some of the requirements before jumping in and building the actual software. Often there is a client-side team that designs the front end of the application and a server-side team that builds the APIs to integrate with back-end infrastructure.
These two teams must work together to develop a REST API that connects the back-end data sources to the client application. One of the most time-consuming and expensive aspects of the development process is the “interface negotiation,” or API implementation ping-pong match, that occurs between these two groups. API ping-pong can take many weeks to several months before the APIs are solid and have the right capabilities. If teams are trying to use agile development processes, the lack of reliable APIs at the start of the project interferes with the first few sprints.
The modern enterprise may need to build dozens or even hundreds of mobile and Web applications. New mobile projects typically have new requirements that were not anticipated by the company and are not met by existing back-end services. You could go back and try to expand the scope of the old services, but they are already in production, so in many situations new REST APIs are built for each new project.
Which means the API building and ping-pong process continues over time with various developers, consultants, and contractors. The REST APIs are often written with different tools and developer frameworks. They run on different servers or in the cloud. They are tied to different databases and file storage systems. Each new service has different security mechanisms, credential strategies, user management systems, and API parameter names.
The result, as you can imagine, is a giant mess. Data access rights are confused, user management is complex, and application deployment is cumbersome. The system is difficult to manage, impossible to scale, and full of security holes. API documentation is inconsistent. Often, companies can’t define what all the services do or even where all of the endpoints are located.
Reducing REST API complexity
The mistake here is that development activity started with business requirements and application design, then worked back to server-side data sources and software development. This is the wrong direction. It’s better to first identify the data sources that need to be mobilized, then create a comprehensive and reusable REST API back end that supports general-purpose application development.
There are many benefits to adopting a reusable REST API strategy. There is no need to keep building server-side software for each new project. Client-side application design is decoupled from security and administration. The API “interface negotiation” is simplified. Development time and costs are dramatically reduced. Developers don’t have to learn a different API for each project. RESTful services are no longer tied to specific pieces of infrastructure. Applications can be moved between clouds or from testing to production with ease.
At this point some of you may be wondering about the technical characteristics of reusable REST APIs and how they should actually be implemented. Unfortunately, you can’t learn this until you have built many RESTful APIs incorrectly, and at that point it’s usually too late. We have been working on this problem at DreamFactory Software for more than 10 years now, and we had to start over many times before hitting on the right design pattern to enable developers to build applications out of general-purpose interfaces.
Characteristics of a reusable API
There are some basic characteristics that any reusable REST API needs to have. The API needs to support both HTML5 and native mobile access patterns. Requests and responses should include JSON or XML with objects, arrays, and subarrays. Noun-based endpoints should be automatically generated depending on the database schema. All HTTP verbs (GET, PUT, DELETE, and so on) need to be implemented for every use case. Support for Web standards like OAuth, CORS, GZIP, and SSL is also important.
There needs to be a consistent URL structure for accessing any back-end data source. The file storage API should be a subset of the NoSQL API, which should be a subset of the SQL API. Parameter names should be reused across services where possible. This presents developers with a familiar interface for any data source. The API should include interactive documentation that allows developers to quickly understand and try an API as well as experiment with different parameters.
In general, the structure of the request URL and associated parameters needs to be very flexible and easy to use, but also comprehensive in scope. Looking at the example below, there is a base server, an API version, the back-end database, and a particular table name in the request URL string. Then the parameters specify a complex filter with a field name, operator, and value. Lastly, an additional parameter sorts the returned JSON data array.
A huge number of application development scenarios can be implemented with only the filter parameter. This allows any subset of data to be identified and operated on. For example, objects in a particular date range could be loaded into a calendar interface with a filter string. Complex logical operations should also be supported. The filter string interface needs to protect against SQL injection attacks. Other database specific capabilities include the following:
- Pagination and sorting
- Complex logical operations
- Rollback and commit
- Role-based access controls
- Record-level access controls
- Stored functions and procedures
The open source REST API back end that DreamFactory built supports operations on arrays of objects, but you can also specify related objects as a URL parameter. This allows complex documents to be downloaded from a SQL database and used immediately as a JSON object. The data can be edited along with the related objects. When committed back to the platform, all of the changes are updated, including parent, child, and junction relationships. This flexibility supports a huge number of very efficient data access patterns.
A reusable REST API platform
- Custom business logic
- Workflow triggers
- Formula fields
- Field validation
- Web service orchestration
REST API complexity is a challenging problem for development teams building mobile, Web, and IoT applications. The tendency to build new APIs for each new project has negative consequences for each project and leads to bigger back-end complexity problems over time. Adopting a REST API platform strategy with reusable and general-purpose RESTful services addresses these problems and provides many benefits in terms of more agile development, decreased development costs, and faster time to completion.
Bill Appleton is the co-founder and CEO of DreamFactory, the Silicon Valley-based creator of the leading open source back end for enterprise mobile and Web application developers. Connect with him on @bill_appleton.
New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to email@example.com.