I have successfully fine-tuned the BERT model using my CPU. I would like to try fine-tuning using my GPU an RTX4090 with 24GB RAM. I am getting an out-of-memory error. I am a little surprised but I would like to make sure the memory is truly the issue and it’s not a misconfiguration. The out-of-mem…

I have managed to get it to run using XLA_TARGET=cuda so it will compile it’s own version. To get it to work, the batch size needs to be decreased, on my GPU it’s a batch size of 4.

have you configured it to run on GPU and not CPU? eg the appropriate XLA_TARGET ? maybe post more code..

The code is the one from the example here with the difference I have updated the libraries in order to be able to use cuda 12. XLA_TARGET is set to cuda120 with my version of cuda being cuda 12.2. Should I downgrade to cuda 12.0? From my understanding cuda120 is CUDA 12.0+ so 12.0 and above. Runni…

@steffel this is very interesting, the model is like 0.5gb, so loading it should be far from running out of memory. Note that XLA preallocates memory upfront, so your GPU memory usage will bump to a high value, but an OOM is definitely unexpected. What is the CUDA and cuDNN version you built with l…

@steffel can you please run with XLA_ARCHIVE_URL=https://static.jonatanklosko.com/builds/xla_extension-x86_64-linux-gnu-cuda120.tar.gz and see if it makes a difference?

Regarding the versions: Cuda compilation tools, release 12.2, V12.2.140 Build cuda_12.2.r12.2/compiler.33191640_0 cudnn: v8

The model loads without any memory issues. Now when it comes to training the model I am using mrm8488/codebert-base-finetuned-detect-insecure-code that is based on roberta-base when compiling myself I am not able to push the sequence_length up to 1024 (batch_size=1) with your version it works. I w…

Oh, so just to make sure we are on the same page, the original OOM was during loading or training? The archive I sent is precompiled using CUDA 12.1, rather than 12.0. According to Jax README it should still be compatible with CUDA 12.0, in which case we would be fine precompiling with 12.1. I will…

First I had an OOM during loading. I managed to solve it using my own built version of xla extension. I was able to train using a batch_size=1 and sequence_length=64 or 128. With sequence_length above 128, I am getting an OOM error during training. Something like that: ** (RuntimeError) Out of …

The OOM while training seems to be back, I am a little confused. Here is my livemd if that can help. [image] PrivateBin Visit this link to see the note. Giving the URL to anyone allows them to access the note, too.

First I had an OOM during loading. Ok, so there must be buggy behaviour in the archive when compiling with 12.0, and compiling with 12.1 fixes it. The OOM while training seems to be back If you restart the runtime and run the whole notebook once, is the OOM behaviour consistent or is it rando…

Trying the BERT fine tunning on GPU

Questions & Help Questions

steffel September 11, 2023, 4:49pm 11

The OOM while training seems to be back, I am a little confused.
Here is my livemd if that can help.