▮ Deployment Myths
For machine learning engineers who have never deployed a model before, deployment itself can be quite intimidating and it may be hard for them to estimate the time and effort required.
Here are several mindsets beginners should have when deploying their first model.
Reference: Designing Machine Learning Systems
▮ Myth 1：We only deploy 1 or 2 models at a time
Almost every app we use daily has many features with specific models dedicated to each of those features.
Let’s take a look at a few examples.
Netflix
Filters over 3,000+ titles using 1,300 recommendation clusters.
Statistics
A 2021 study by Algorithmia shows that among organizations with over 25,000 employees, 41% have more than 100 models in production.
Unless you are doing academic research, it is rather rare to deploy only 1 or 2 models.
▮ Myth 2: Model performance won’t change
Unfortunately, models do not become better as time passes like wine. In most cases, it gets worse(usually called “software rot” or “bit rot”). The reason why that happens is that the data distribution that is inputted to the model always gradually changes. This means, the more time passes, the inputting data is far away from the original training dataset, leading to performance decay. This myth leads to the next myth.
▮ Myth 3: You don’t have to update your model that often
As we’ve learned from the previous myth, models degrade as time passes. This means we need to be able to update the model as fast as possible.
While most companies update their models once per month or even once per quarter, some companies such as Weibo, Alibaba, and ByteDance(the company behind TikTok) do this iteration once every ten minutes. You can clearly see how much it takes to be on top of the game.
Considering the above, the question we should be asking ourselves may not be
“How often SHOULD we update our model?” but “How often CAN we update our model?”
▮ Myth 3:ML engineers don’t have to worry about scale
According to the Stack Overflow Developer Survey(2019), about half of the companies with ML engineer positions consist of more than 100 people.
If more than 100 people are working, the number of customers they are serving should be large to some extent. Even though this fact may still be a little weak to point out, we might be able to say that we should consider scaling, statistically, during our development in most cases.
