With the amount of time and energy that’s spent on perfecting and creating an analytic model, you’d think it would come at the hands of a wizard or sorceress. Unfortunately, there’s no magic wand that can successfully bridge IT and data science teams when it comes to model deployment.
These models are created to generate insights that enable us to make better decisions, faster. To maintain that competitive edge that’s crucial to survive, the value of the data generated must be recognized as quickly as possible to be relevant, and it must always remain as accurate as possible.
This can be an easy task when you’re working with one model, but as systems, problems and team complexity increases, this task can be paralyzing, not to mention cost-prohibitive, for an organization. I’ve spent the past few articles on this column and in our research at Open Data Group detailing just how important it is for the data science and IT teams find solutions that allow them to come together every time a model is pushed into production. The importance of this step can’t be overstated. It’s a process that is further complicated when making updates to the model or the system surrounding it.
The problem
The relationship between data science and IT teams can be quite complex and sometimes damaging to the goal of producing value through analytics. The data science world is experimental and research-oriented. Solutions are often a result of a deep understanding of the problem domain and the data available. This experimental nature requires a modeling environment with the freedom to try new tools and libraries/packages (model dependencies). Once the models are designed, trained and evaluated, candidates are selected that will satisfy requirements for performance and accuracy, and be ready to graduate to final pre-production validation.
At this point the organization is eager to use the selected model and you’d think it would be pushed into production. Shouldn’t the model creator be able to just give it to IT? Unfortunately, this isn’t the case for most organizations. You need to test the model to access all of its dependencies while being executed in the production environment, the model must receive the correct production data and send the scores to the right place, the model must pass testing on all fronts, and the system must be set up for monitoring and scalability. This process is made even more complicated if this is a first-time model, where the luxury to adapt isn’t at your disposable.
The production
For the model to run properly in the production environment, it must have access to the required dependencies. If this is not already the case, it needs to be addressed either in the model or in the production environment. Changing the production environment may both be risky and work intensive. More often than not, changing the production environment requires approval.
Approval processes take time, which causes many companies to opt to change the model entirely. In some cases, the model is actually rewritten into a language that the production environment already supports, such as from R to Java. Some models are sent back to the data science team and told that they must create a model that does not require certain libraries.
This puts the data scientist back to square one with even more constraints than before. In either case, this process rarely finds success the first time through, which results in an ongoing and time-intensive back and forth between IT and data science. Overcoming this problem would require a way for data scientists to test the execution of their model in an environment identical to that of the production environment without having to involve IT.
By nature, this environment must support an ever-growing list of libraries and languages that data scientists use and in an isolated way as to not disrupt other applications on the production servers. The solution must be scalable and include the necessary tools for testing and quality assurance.
The Docker ecosystem lays the groundwork to build such a solution: Using a containerized analytic engine to run analytics provides a portable way to deploy, test, and push models into production. Container portability means that models can be validated early, by the data science team, reducing the back and forth between data science and IT.
Organizationally, the process can be peppered with the appropriate approval steps.
The roles
Running a model in production is often lumped into one task that is either entirely done by IT, by an IT-savvy data scientist, or a combination of both IT and data science. Depending on how the work is distributed, it often results in lack of ownership for success along the way.
But to find success, it’s best to start breaking up tasks into individual components and assigning clear operational tasks generates responsibility for work that can be done in parallel by an individual with appropriate knowledge and expertise.
Here are some questions we ask to segment this process:
- What data does the model expect to ingest and produce?
- What infrastructure is required to run this model at scale?
- Does the model have everything it needs to execute?
- How easy is it to identify points of failure if the model is not running properly?
The testing
With the handoff process now broken into individual components, it’s absolutely imperative to make sure they will work as intended before they’re put together. This forces validation on smaller pieces which makes troubleshooting easier. It also ensures that the components are fully tested by the creator which reduces the amount of time to find problems and fix them.
In most cases the model is validated in the creation environment and thrown over the wall to IT to test the execution and accuracy in a stage environment that is identical to production.
Once the model is validated and ready for production, the team must think through how to track metrics about the performance and accuracy to ensure problems are caught early and are easy to identify.
The monitoring
Setting up the right measurement and reporting methods throughout the components involved in executing a model will provide metrics capturing model accuracy, data throughput, and performance/resource utilization. IT will generally care most about scalability, capacity planning, and quality of service (fast and reliable delivery of model execution results). Data scientists will care about the performance of the model code and the results the model is producing.
IT should not be responsible for writing the logic to calculate accuracy or any other metric that evaluates the model’s quality over time. Similarly, the data scientist, unless they have a background in systems/IT architecture, should not be responsible for computing metrics on resource consumption for each individual piece of the operation.
The handoff
The handoff is such a delicate process in model deployment. All teams need to come together from the onset to lay out clear expectations and testing practices for each team member and component involved. A solution like this can be incrementally integrated into legacy systems and provide the ability to expand the technologies needed to generate top performance.
This breaks away from traditional monolithic thinking, with a solution that is more flexible, highly configurable, easy to implement, and introduces best practices and standards. In fact, the handoff should be replicated over time to ensure a model is long-lived and a durable asset within the organization. And it starts with understanding the responsibilities of each team between IT and data scientists, and opening up lines of communication to ensure that the process is running without wasting valuable time or resources.