Every now and then, I’m reminded of a project that we did a few years ago. We had a 6-week deadline and our client asked us to implement a machine learning model to predict the demand for something they produced. We were competing with a heuristic model that was honed over 20 years with a lot of domain knowledge.
Initially, we used the Mean Squared Error (MSE) as the optimisation metric which is a very common metric to use when training models. The final score of our model was slightly better than the original model, but had a major flaw. Implementing it would cost the client a lot of money!
The company needed to minimise underproduction since “just in time” production was much more expensive than preparing in advance. Using the MSE was the wrong choice because it weighs both over- and under-prediction the same.
I don’t remember exactly what metric we ended up using, but that doesn’t really matter. The main thing that we learned from this experience is that it’s super important to think about your metrics and what you’re trying to optimise. Consider the context in which you will apply your model.
(Also posted on my LinkedIn feed)