Gnu-Screen can help your process keep running even when you disconnect from your gpu.
This is useful when you are using ssh to connect to your GPU machine, AND the model training process is taking a LONG time. I mean, I’m sure you don’t want to be worrying all the time about whether your connection might fail at some point and stop the process.
I used to run processes right before going to sleep hoping the training process would end by tomorrow morning, just to find out my pc’s connection was lost during at some point, which really sucked…
Here is how you can implement it using Linux.
- Install Gnu-Screen
apt-get install screen
- Run Process
#Create Session screen #Name session session -S {SESSION_NAME} #RUN YOUR PROCESS python main.py #Detach from session: Now you can close your connection any time ctrl+a d #Reattach: If there are several SESSIONS, it will ask you which SESSION to connect again screen -r