Nowadays almost anyone can make use of machine/deep learning just by copying and pasting codes from the internet. But I think it is still important to understand the math behind the inference.
Ok, Let’s consider a simple neural network that has 3 inputs and 1 output like below.
First, the model will calculate z1~z3 (For simplicity, we will not consider activations)
So, the model was able to successfully calculate z1~z3. From the information above, the model can express z1~z3 with a single matrix Z like below.
Since we’ve got this far, let’s add another layer and see how the model calculates Z’1~Z’3
The calculation process is the same as in the previous example.
The model will then again be able to express the above into a single matrix Z’.
Since we already know z1~z3 from the previous example, let’s insert that to the equation above
And now you know how the model uses matrices! If there are n layers, that would mean n matrices would be stacked to the equation above.
Why do we want to utilize vectorization..?
The main reason is to effectively use the GPU(which is great at doing multiple tasks at once).
Like the example above, when calculating the output for a neural network, there will be A LOT of matrix multiplications (Also Additions) going on. And when doing those calculations, if you are using matrices, you can divide them into blocks(Like below) and calculate them separately. Since GPUs are great at doing tasks in parallel, it is more likely to FULLY use the GPU.
