Across scripting, ETL, data prep, application integration services, and big data platforms, there’s plenty of hands-on, manual data integration work for developers, data scientists, data stewards, and analysts. Vendors know this, and some of the next-generation data-integration tools and features will include artificial intelligence (AI) capabilities to help automate repetitive tasks or to identify hard-to-find data patterns. For example, Informatica is marketing Claire, “the intelligent data platform,” and Snaplogic is marketing Iris, which “powers self-driving integrations.”
Finding the right mix of data-integration tools
The list of data-integration options can be daunting considering the types of platforms, the number of vendors competing in each space, and the analyst terminology used to categorize the options. So, how do you go about deciding the right mix of tools for today and future data integration requirements?
The simple answer is that it requires some discipline. Start by taking inventory of the tools already in use, cataloging the use cases of where they are successfully being applied, and capturing the people successfully working with these tools. Ask them for additional example use cases where they had difficulty implementing solutions and thus where seeking other tools may be beneficial.
See how the data-integration subject matter experts feel. Maybe there are data-integration scripts that need ongoing maintenance, and the finance team is frustrated with the repetitive work, or perhaps developing with the ETL solutions is too slow for the marketing team’s needs. Maybe the data scientists are spending significant time wrangling data with a programming language and creating a large code base. Perhaps many of your data-integration requirements are with a few standard platforms and a standardized integration approach will yield operational benefits.
With an inventory in place, a team of data-integration specialists can review implementation options when new or enhanced data integrations are requested. If a new request is like one that is already implemented and working, the team should have confidence to apply it again. If not, it can elect to experiment making the implementation with an existing tool or consider doing a proof of concept with a new tool if it’s a highly divergent data-integration job.
This discipline of consolidating use cases and reviewing the implementation new ones is a best practice when there are new business needs and an evolving technology landscape.