Should you use machine learning in your product?

Today, everyone builds an AI product, don’t they?

Do you always need machine learning? What is the reason behind the surge of AI projects?

Let’s figure out when machine learning is overkill and how to introduce AI in your project without shooting yourself in the foot.

AI everywhere

I signed up for the co-founder match on YCombinator. Every founder with whom I talked had an idea for an AI product. Everyone! Why?

They don’t need AI to build those products. Why is everyone building needless AI systems? I was told why they were doing it. To get founding. More people want to buy a share of your company when you’re doing something related to AI. Whatever it is.

Apparently, the product doesn’t matter as long as you can say you’re doing AI. Once again, bootstrapped startups turn superior. You don’t sell your company (cheap!). You build a profitable product from day one (because you have to). But also, you don’t base your technology choices on what VCs find fashionable.

VCs, however, aren’t the only people to blame for the needless usage of AI.

Fashionable technologies

A few years ago, we were introducing functional programming or streaming into the software.

Programmers went to conferences, heard the speakers were doing it and thought they must do that too! Otherwise, you will be left behind! Otherwise, you will be stuck maintaining a legacy Java monolith until the day you retire.

In my opinion, every data engineer should learn functional programming. It makes you better at your job, even when you write Python. However, I still remember writing CRUD microservices in Scala for two years. We could use Java, add the Spring Framework to the codebase, and automatically generate 80% of the code. We would finish the projects faster. We could easily find new developers because many people know Java and Spring Framework. Instead, we used Scala.

From the programmer’s perspective, it was a fun learning opportunity. In reality, however, the business was burning money.

Of course, when the entire project consists of (glorified) CRUD services, you must give the programmers something cool to do. Otherwise, they will quit when you tell them to implement the hundredth CRUD. Functional programming may be such a cool thing.

I feel we’re doing the same with AI right now. First of all, everyone calls it AI - machine learning is not a cool term anymore. Second, how many companies put their AI systems in production? When we interviewed people for a machine learning engineer position, we met a person who had been training models for three years. None of his models ever end up in production. What’s the point of doing it, then?

It is nearly impossible to find an engineer with experience in running machine learning in production. Of course, perhaps such people aren’t looking for new jobs because they may end up in a place where you train models for years to store them in S3 and forget about them, so it is safer to stay where they currently are.

Not everything needs AI

Most likely, you aren’t working on the autopilot project at Tesla.

Do you need AI in your product? If you need it, perhaps only a tiny portion of the system should use AI.

What’s wrong with rule-based systems? What’s wrong with systems you can easily debug and test? You can write automated tests for every possible case in such a system. How would you do that in the case of machine learning models? Sure, you can run the model verification on the test dataset and say it is correct 96% of the time. Well… a rule-based system wrong 4% of the time is considered broken.

Of course, that’s the nature of machine learning. Being wrong some of the time is the price we pay for generalizing from specific examples. However, I suggest doing it only when there is no other option.

If people can’t even articulate the rules, you may start using machine learning. Otherwise, you can use a bunch of if statements instead.

Building a rule-based system

A rule-based system in production getting results for the users and making money for the company beats a perfect AI model existing only in a Powerpoint presentation filled with buzzwords.

You can’t build a rule-based system in the same way we train machine learning models. You can’t take a bunch of data points and hope to get results. Data without knowledge isn’t enough.

We need someone who understands the rules behind the data. This may be the first problem. Programmers will need to talk with people! However, this shouldn’t be an issue. If you’re building software for people to use, you must be able to speak with them. The years of nerds living in the basement have ended. If you can’t cooperate with people, perhaps you shouldn’t be a programmer.

By the way, if you need a fancy buzzword, here is a buzzword for you: expert system. We put the expert’s knowledge in the code, so we call those rule-based systems expert systems.

Extending rule-based systems with AI

How would you write if statements to recognize an object in an image or extract entities from the text? You can’t.

Here is where we extend an expert system with AI. This is the small portion of the code where we trade certainty for extra capabilities. A machine learning model may be wrong sometimes, but we cannot solve the problem without it. Hence, we introduce potential bugs into our otherwise perfect (because properly tested) rule-based system.

We must contain the machine learning part. We shouldn’t mix the code handling rules with AI predictions. When we use ML, we need additional rules in our system. What if the score is close to the threshold? Should a human review the result? What if we know the model is wrong? After all, the prediction may contradict some of the rules we have already applied to the input data. What should happen in this case?

Last but not least, how will we improve the model? We may need to A/B test multiple model versions. How should the expert system handle getting two different predictions for the same input data?

There is no easy answer I can give. It depends on the business context. Once again, you will have to talk with the domain expert.

Automating human tasks

Sometimes we have a system with a portion of tasks done manually by people, but the volume of data makes it impossible for them to keep up.

From the perspective of building a rule-based system, this doesn’t make much difference. People can be wrong too. We can treat human input in the same way we handle predictions of an artificial neural network.

From the perspective of the expert system, any improvement will look like deploying a new version of the machine learning model. The quality of data will improve, but the coordination mechanism stays the same. If we separate the rule-based code from the code handling human tasks or ML predictions (as we should!), we won’t notice the difference.

We may wonder whether we should automate human tasks. What about those people? Will they be out of the job?

In human history, we have never seen automation decreasing the number of jobs in the long term. It didn’t happen during the industrial revolution. It didn’t happen when we invented computers or the Internet. All of those people will do something else.

What will they do? I have no clue. I work in a position that didn’t exist ten years ago! (ok, fine… In many companies, MLOps didn’t exist three years ago…) It wouldn’t be possible without automation.

Also, should we turn people into robots? Imagine doing a job that could be automated. Every day. For years. Would you enjoy your life?