▮ My Mistakes
I’ve finished a couple of POCs(Proof Of Concept; a phase where we create a prototype to check the feasibility of the machine learning project) this year as a Data Scientist/Project Manager for computer vision tasks, and I’d like to share my mistakes and what I’ve learned from them.
I hope my learning will help you too when you tackle your machine-learning project.
▮ Workflow
Before I start, I’d like to first talk about the basic project workflow.
In most of the computer-vision-related projects I’ve been involved in, the main objective is to automate a task that would otherwise be done by people. To achieve this objective, we can break down the steps to the following.
- Listen to client’s requirements
- Choose a metric to measure the project’s “success criteria” that suffices the client’s requirements
- Data Preparation
- Create a prototype (machine learning model)
- Implement an algorithm that inputs the machine-learning model’s output and outputs results that helps automate the task.
- User Interface Development to integrate into the current workflow
- Check the feasibility by using the “success criteria”.
Now that we know the workflow, let’s get to the mistakes.
▮ Mistake 1: Scheduling
When doing a POC, since it is just creating a prototype, the client often wants to know the results as fast as possible. No matter how much they rush you, you should always consider the following.
- How much data preparation is required
- Do you need to annotate the data?
- How much do you have to clean your data?
- How much data do they have?
- How fast can they provide the data?
- How difficult is it to implement the machine-learning model
- Especially if you are using a research paper’s Github repository
- Are there any additional preparation required to use the code
- How much time the client has for this project
- Do they have enough time to give response to the question you asked?
One of the POCs I was doing was a month long. We’ve set a weekly meeting but considering that A) There are only 4 meetings, B) You need to use the last meeting to present the final results, which meant that I only had 3 weeks to do everything. In addition, the client was not available most of the time, so that made it even harder for me.
▮ Mistake 2: Success Criteria
Setting success criteria that satisfy the client’s requirements may be one of the most important things to succeed a POC, but demonstrating HOW you use the success criteria may be equally important as well.
If you don’t concretely demonstrate how you expect the client to use the success criteria, there may be a chance that you create a deliverable that the client is not expecting.
What I recommend is to create a mock version of the final deliverable beforehand(ideally presenting it in the last meeting before the last meeting) and provide the exact procedures for how the client is going to use the final deliverables to decide whether to move on to the next phase.
Doing so can help you reorganize what you need to deliver, while helping the client imagine what to expect.
▮ Mistake 3: Understanding Phases
As we’ve discussed in the “workflow” section, there are often several phases after the model provides an output, like implementing an algorithm in step 5.
I recommend creating a workflow diagram and provide which success criteria you are going to use and how for each phase.
In one of the machine-learning POCs I was involved with, we were trying to do an image classification task and the client wanted set a threshold so that if the predicted probability is below it, the image will be classified as “unknown” to avoid unnecessary errors.
So for this case, the workflow would look something like this.
This visualization is important because, in the case of the image above, I wrote that we will use the F1-score for Phase 3, but you can also use the F1-score for Phase 1 as well(to find the appropriate threshold). This can easily become confusing. What happened for me was that, the client was talking about the F1-score assuming that it is for phase 1, but I was talking about the F1-score for phase 3. Because of this, I wasn’t able to understand what my client was trying to discuss. I was finally able to understand it after I visualized each phase.
