for what its worth I watched the mem usage with a cpu backend for a while and saw it spike to 32GB used in the second epoch.. maybe I just underestimated how memory hungry this training process is.
for what its worth I watched the mem usage with a cpu backend for a while and saw it spike to 32GB used in the second epoch.. maybe I just underestimated how memory hungry this training process is.