As more data and applications move to the public cloud, the need to synchronize that data between the enterprise and the cloud grows. Many tried to solve this integration problem by using traditional approaches, such as enterprise data integration (EDI) technologies. However, others are reverting to ad hoc and primitive approaches, such as simple FTP jobs or even FedEx'd USB drives.
It's becoming a mess. We should know better.
I see enterprises consistently erring on three points with cloud integration:
- The inability to define the metadata or what the data truly means. This makes it very difficult to figure out the right approaches to exchange data with systems hosted in public clouds and those in the local data centers. A fundamental step in defining a data-integration strategy, no matter if it's in the cloud or otherwise, is to define what the data means and how the data should be managed.
- The lack of understanding of bandwidth limitations. Although a few megabytes of data an hour is fine for transmission over the open Internet, when you're approaching the gigabyte range, you're getting to the point where latency and even bandwidth costs become an issue. This is a key issue as big data systems are built on public clouds, but the operational data required to populate those big data systems remains in the enterprise.
- The inability to deal with error-handling issues. Data-integration solutions fail. You're linking systems on public cloud providers with those in enterprise data centers, so the likelihood of failure goes way up. These issues can lead to data quality and availability issues that must be corrected by people -- not by automated processes. Thus, there is a need to consider error-handling routines in your cloud-to-enterprise data integration tools, not just on-premises.
If these three issues arise in your cloud-integration approach, all is not lost. Take a few steps back, do some planning, rethink your technology, then test and implement. Spend a few bucks, invest a few weeks, and you'll see a big difference.