We are now part of Eviden, discover more...

7 Steps for Successfully Building a Human-in-the-Loop Machine Learning Model

Even though human-in-the-loop is still in its “early innings,” companies can generate considerable business value in the near-term by implementing this emerging technology. At most workplaces, it’s now expected that employees across a wide array of functions are able to automate their routine daily tasks through the use of predictive analytics. This by no means insinuates employers want their staff to automate themselves out of a job; rather, they want every employee to be functioning in the most productive way possible. Given the capabilities available today, it’s best to think of human-in-the-loop as something that makes your life easier by augmenting a built-in feedback loop.


Companies should embrace predictive analytics with the goal to develop assistive technologies, which boost or maintain the functional capabilities of people living with disabilities. Categorical recommendation systems are a particularly valuable type of assistive technology. Also known simply as “tagging,” they essentially save people from having to sift through large amounts of data when seeking specific values by predicting what those desired values are. The predicted answers are delivered through some sort of user-centric interface, and the system even lets you know its level of confidence based on the accuracy of its prediction. Users must then be allowed to correct wrong predictions to enable the system to “learn” over time, the re-enforcing human-in-the-loop advantage. The system would then re-train on that data and provide more accurate results over time, which is the very essence of machine learning.

In order to enable employees to take advantage of the productivity benefits of assistive technologies, companies must first establish a foundational model for predictive analytics. Having assisted many companies in creating self-learning predictive analytics models, Maven Wave recommends that any organization wishing to roll out assistive technologies, especially categorical recommendation systems, adhere to the following steps to maximize success.

Step 1: Clearly Define the Problem That Needs Solving

Before investing time and resources on a predictive analytics program, it’s critical you define your objective clearly. Optimizing operations, mitigating risk, and providing insights are all worthwhile goals, and they should all save humans time. With a firmly solidified goal in mind, you are better positioned for success right off the bat.

Step 2: Harness the Power of the Cloud

Traditional IT infrastructure does not have the required power or agility to adequately fuel a predictive analytics model. To ensure you have ample compute power to facilitate machine learning and enough elasticity to minimize cost, it’s necessary to secure resources in the public cloud for your foundational IT infrastructure. 

Step 3: Bring in a Data Scientist

Considering they’ve pursued what’s considered one of tomorrow’s most promising career paths, data scientists don’t come cheap. But they are necessary for building a functioning predictive analytics model with self-teaching capabilities. Once you enlist a data scientist and they develop an initial algorithm, they should conduct Exploratory Data Analysis (EDA) using historical data. This provides your company an overview of its existing data landscape and shares some of the preliminary results of various queries. Once there’s some indication of how well the initial model works, have the data scientist present the results to all stakeholders slated to use the model upon production and discuss what impact the results may have on your operations.

Step 4: Trial Production with a Controlled Experiment Group

Even after EDA and some subsequent fine-tuning, the model is not yet ready to start making forward-looking predictions on all of your data. Remember, the machine teaches itself and its efficacy will improve over time – it needs to walk before it runs. Have the data scientist take a controlled experiment group and run it through future-looking predictive analysis. This test provides an opportunity to iterate the platform to arrive at the best model that will improve on its own over time. Three key performance indicators should be securitized: accuracy, stability, and transparency.

Step 5: Involve Your SMEs

The performance of the controlled experiment should be assessed not just by the data scientist, but also the subject matter experts for the specific knowledge being run through the model. The team should provide a frank and honest assessment on how that model will perform in generally available production. During that conversation, you may need to have some basis of understanding on how the accuracy of a predictive model is calculated. In rare cases, the resulting model will not perform to your level of satisfaction. This is usually due to the quality of the data. It’s therefore crucial to establish a “ground truth” that can be used to train a model. If this cannot be established, more work will need to be done.

Step 6: Production Time Has Come

Once the data is solidified and final adjustments to the model are executed based on the controlled experiment, it’s time to release the model for general availability and let machine learning do its thing. You should begin to see the amount of time employees save—the ultimate measure of efficacy for predictive analysis—and it should improve over time. With human-in-the-loop the model will get “smarter” and “smarter” over time.

Step 7: Don’t Forget About Maven Wave!

This process may seem daunting for companies whose core competency isn’t data science, which is most. But don’t fret; since predictive models teach themselves, most of the heavy lifting is done upfront. With a dedicated Data Analytics and Machine Learning practice, Maven Wave is ready to help take companies of all types through the steps we’ve outlined above. Our team of infrastructure specialists and data scientists can help deploy the underlying IT infrastructure and create and execute the modeling, so you arrive at measurable success. Contact us to get started today.

About the Author

Brian Ray
Brian Ray is a Managing Director and Data Science ML/AI Horizontal Practice Lead for Maven Wave Partners. Mr. Ray is heading the group’s mission of solving complex analytical problems for major businesses worldwide through the power of Data Science with enablement in the cloud. Prior to Maven Wave, Brian has 20 years of hands-on experience in Engineering around the Sciences. A big picture strategist, team builder, and influential top technologist, he has extensive expertise in agile delivery of Data Science — from ideation/discovery, feasibility, exploration, modeling, to engineering and architecture, to hands-on integration and deployment of best-in-class solutions.
October 16th, 2019

Get the latest industry news and insights delivered straight to your inbox.

Sign up for our Newsletter