I compiled ggml with -DGGML_CUBLAS=ON and then clip.cpp, and used it to get text encodings, but the GPU is not being used. The code takes the same amount of time as it did with CPU-only. Is this expected? Does clip_text_encode always use the CPU no matter what? Or did I forget to do something?
Details:
ggml is detecting the GPU without problem (Nvidia AGX Orin):
$ ./myapp
ggml_init_cublas: found 1 CUDA devices:
Device 0: Orin, compute capability 8.7
Simplified version of my code:
#include "clip.h"
// ...
string model = "clip-vit-base-patch32_ggml-text-model-f16.gguf";
clip_ctx *ctx = clip_model_load(model.c_str(), verbosity);
for (int i = 0; i < 1000; i++)
clip_tokenize(ctx, "person".c_str(), &tokens);
float txt_vec[512];
clip_text_encode(ctx, /*threads:*/4, &tokens, txt_vec, true);
}
This takes 8 seconds to finish. While this runs, I have jtop open, and I see the GPU is only active during the first 3 seconds, when ggml gets the GPU name and compute capability to print them. After that, the GPU goes offline. GPU usage is always 0%.
I compiled ggml with
-DGGML_CUBLAS=ONand then clip.cpp, and used it to get text encodings, but the GPU is not being used. The code takes the same amount of time as it did with CPU-only. Is this expected? Doesclip_text_encodealways use the CPU no matter what? Or did I forget to do something?Details:
ggml is detecting the GPU without problem (Nvidia AGX Orin):
Simplified version of my code:
This takes 8 seconds to finish. While this runs, I have jtop open, and I see the GPU is only active during the first 3 seconds, when ggml gets the GPU name and compute capability to print them. After that, the GPU goes offline. GPU usage is always 0%.