392. Artifacts To Keep Track During Model Development

▮ Keeping Track

When developing an ML model, there is a lot of information to keep track of. Data scientists have to constantly experiment and the information can get easily chaotic. Ideally, all the information should be organized so that even after a while, let’s say 6 months later, you can still immediately reproduce the model.
Here are the 8 types of artifacts to keep track of when developing an ML model.

1. Model Definition

This is the information about the architecture of the network. This also includes information such as loss function, batch size, etc.

2. Model Parameters

You will need the information on the weights of the model to reproduce the prediction. Depending on your model’s saving methods, you can save both the architecture and the weights in a single file.

3. Feature Extraction

The feature extraction pipeline should also be stored so that you can use them whenever you want to preprocess raw data before inputting them into the model.

4. Dependencies

In most cases, ML models are packaged into containers, so information such as Python version and Python packages are usually stored within that container.

5. Data

It is quite common to re-split the datasets(training, validation, and test sets). Keeping track of which data you used for which dataset on which experiment can easily get confusing as you experiment repeatedly. You should name or version control your data to have reproducibility.

6. Model Generation Code

This information includes:
1. Frameworks(Pytorch, Tensorflow, etc.)
2. How it was trained
3. Dataset division method

The codes can be a Jupyter notebook or a simple python file.

7. Experiment Artifacts

These are the artifacts generated throughout the development process; loss/epoch curve, task-specific metric calculation results, etc.

Fig.1 – Loss Curve
8. Tags

This will help with model discovery and filtering.