Following the thrill of quickly onboarding your team to a new cloud application, following the excitement of being able to experience instant productivity, comes the realization of an additional silo that is disconnected from everything else -- from other users, other business applications, and other processes. How will this new system work with all other existing applications? How will your team share information and activities provided by this new application with others in your organization?
And once again you start searching various marketplaces for point-to-point connectivity solutions.
The rapid rise and adoption of cloud applications has amplified the silos within the enterprise, further disconnecting teams, processes, and information. It also has made cross-departmental collaboration considerably more difficult. Cloud-based applications provide the business users with a fast path toward productivity, typically offering a feature-rich environment with relatively easy onboarding, especially when compared with on-premise deployments. They offer features targeting a spectrum of enterprise functions. Collectively, they support almost all facets of an enterprise ranging from workflows, collaboration, analytics, to specific departmental activities.
The need for integration
However, this range and collection of cloud-based products and services also exacerbates the problems of integration, the sharing of process and information between applications, and collaboration across teams. It introduces a lot of gaps along the pathway from sales to customers to product development to support. The gaps are critical to efficiencies of operations within the enterprise, since they contribute to everything from increased effort in tracking work across teams, poor collaboration, and ineffective decision making. It is no wonder that DevOps managers spend most of their cycles trying to catch up to these gaps and look for solutions to address them to improve business operations.
Since the cloud-based application spectrum covers a wide range of application types and feature sets, the need for integration is also much more complex and multi-dimensional. Furthermore, the integrations are also influenced often by the type of interaction desired with the application based on who the constituents will be. There are other factors that also influence this, including security, access control, API limits, execution threshold, etc, although those are out of scope for this discussion. Suffice it to say, the solutions to address them also continue to evolve in profoundly creative ways.
Real-time replication/mirroring
Primarily to address API limits as well as to optimize cloud-based applications' resources in scenarios that involve large scale access, often by customers and end-users, real-time replication of an application's data to an external source is provided. Heroku Connect is an example of such an implementation that allows its customers to select a subset of their Salesforce data for replication/mirroring. The technology behind Heroku Connect ensures the real-time replication of the data set in the background. Its sole purpose is to make this subset accessible to a broader audience (customer portals, end-user applications, mobile and IoT devices, etc.) without the added expense of a Salesforce account, as well as its API and other limits.
Virtual access to external data
Rather than moving and copying data across applications using legacy export formats, ETL tools, etc. that also do not offer any options for real-time or on-demand access, the REST-based Open Data Protocol provides access to an external application's data and resources in a variety of ways, for both read and write operations, based upon the specific implementation. Salesforce Connect is an example of such an integration where live external data (and metadata) can be available as Salesforce objects for direct internal consumption, be it via queries, reports, Apex code, search, etc. without the need to move or copy it outside the hosting application that owns it. Salesforce Connect offers out-of-the-box integration with a variety of data sources. It is also possible to write OData-compliant connectors for other applications to enable their data to be seamlessly shared within the Salesforce context. The same paradigm can also be applied to other cloud applications via pre-built or custom connectors.
Publishing structured data on the web
To facilitate access to structured information, a number of technologies and specifications have evolved that allow for the publication of structured content on the web. The consumers are primarily search engines, but it's also possible to develop applications that can parse this content in a meaningful way and process it as if it were retrieved directly from the host application. For example, by embedding RDFs micro format using GoodRelations schema (or ontology), BestBuy incorporated rich product information into their website content to allow for search engines to become more precise and comprehensive when displaying product information related to the search. This paradigm extends further with Linked Data and Linked Open Data, which allow for the structured publishing, via RDF, of information across the web. This way of publishing content makes it machine-readable and may be used to not only search but also query using standard query languages and tools.
Extending an application's interfaces
Most cloud applications provide published APIs for authenticating and interacting with the application and its resources. However, due to the varying and complex needs of integrations from disparate applications, they also allow for the interfaces to be extended and new interfaces to be implemented via add-on components. These add-ons may be installed directly into the application to extend its functionality, as well as provide new interfaces to external applications. They may also be separately hosted components that run independent of the host application but still provide a seamless interface to the end user. Cloud application vendors such as Atlassian provide specifications of these add-ons that function as application components that may be hosted externally but are integrated within the host application. Similarly, Salesforce supports the implementation of managed and unmanaged packages which are installed within an instance and execute in the context of the host application, often as part of a single transaction related to a process or workflow.
Bots
While the concept of bots isn't new, the emergence of certain technologies recently has made them more effective. Bots are gaining widespread adoption, as they offer unique interactions with host applications from a variety of devices and may also be used to perform actions on other applications via the host. For example, Slack supports the development of custom bots that have been used to make lunch reservations, order Uber, or process expense invoices. Bots offer a casual interface compared to conventional application interfaces. They can understand and interpret natural language commands. There is now at least one Bot Framework to promote the development and integration of bots within enterprise applications.
Search and indexing
Finally, for content repositories and other sources of searchable data, there are also connector frameworks, such as Apache ManifoldCF, that facilitate the connection between the repositories and various destinations (primarily search servers). Support for a wide variety of repositories, such as Documentum, Alfresco, Sharepoint, etc. is already available. Other custom connectors may also be similarly developed. On the other side, search servers such as ElasticSearch and Apache Solr are supported, amongst others.
Bringing it together
There are many more approaches that are similarly unique in the types of interactions they offer with cloud applications, the use cases they address, as well as facilitate integration with and across applications. In future posts, I will dive further into the integration issues that teams face as they turn increasingly towards cloud applications for performing specific functions, as well as the evolving solutions that are vital to the success of cloud application adoption in the enterprise.
When it comes to the rapidly evolving landscape of cloud-based applications, one size does not fit all and it is no longer sufficient to be constrained by legacy paradigms of application or data integration. Rather, I propose that application vendors consider all of these use cases when developing external interfaces and DevOps managers consider the broad range of needs of their users before deciding on integration strategies for their company.