PyTorch data to device
what is the best practice to move data to gpu? Shall I move it before model training or move it while model training like the code you provided
The best practice is to move the data to the GPU in the training loop itself, for each batch, just like in the code I provided. The reason for this is that, in most cases, you cannot fit the entire dataset into GPU memory at once, especially when dealing with large datasets typically used for deep learning.
By moving the data to the GPU batch by batch, you only need to have enough GPU memory for one batch of data and the model itself. After each batch, the GPU memory used by that batch’s data can be freed, making room for the next batch.
Here is how you do it in the training loop:
for i, data in enumerate(train_loader, 0):
# get the inputs; data is a list of [inputs, labels]
inputs, labels = data[0].to(device), data[1].to(device)
...
It is important to note that moving data to GPU is not an instantaneous operation. It takes some time to transfer the data, so doing it batch by batch can introduce some additional overhead. However, this overhead is usually acceptable and far outweighs the benefit of being able to utilize a GPU for training, even with a large dataset.
If you have a very small dataset that can entirely fit into the GPU memory, you could technically move it all at once, but it’s generally better to stick to the batch-by-batch approach for consistency and to make your code more flexible and adaptable to different dataset sizes.