Will an NVIDIA Data Center GPU (aka Tesla) work?

Tesla should have model numbers ( some of Tesla cards are from 2007 as per wiki ) so need to know which model you have so you can get right drivers and second - AI, what is AI? You mean LLM`s ? if yes - and if u use something like ollama or lama.cpp - it will use Vulkan as no CUDA in FreeBSD. Vulkan is much slower than CUDA. My volta was up to x4+ times slower on Vulkan than on CUDA in Linux.
 
GlitchyDot said:
Tesla should have model numbers ( some of Tesla cards are from 2007 as per wiki ) so need to know which model you have so you can get right drivers and second - AI, what is AI? You mean LLM`s ? if yes - and if u use something like ollama or lama.cpp - it will use Vulkan as no CUDA in FreeBSD. Vulkan is much slower than CUDA. My volta was up to x4+ times slower on Vulkan than on CUDA in Linux.
Click to expand...
Yes this would be for AI inference, specifically with the lama.cpp port as you mentioned. As for the GPU model itself, I still haven't decided which one, but it would certainly be something a little older, as far back as a T4 for example. Though, I think I may actually have a Tesla P100 laying around somewhere. I'm considering a Lenovo SR650/630 (v1) as the system itself. I'm not ready to invest too much into this hardware yet.
 
Dont know anything about pc, but p100 is pascal , t4 is turin - so yes, these will work. I dont know how long it will take, but nVidia stopping with pascal,volta gpus on driver developing as they achieved maximum ... dont know about T ones so for now nvidia-drivers should work as i dont see the reason why not and later it might be moved to different naming like nvidia-drivers-XXX.
Well, deppends how many gpu`s you will use but one thing to know that cpu speed ( i think single core ) is in the game as well for faster t/s . so do the research. if you planing on 1 or 2 maybe you just enough normal "gaming rig" hardware with gpu or two to have some fun.
P.s. all these tesla and all others - designed for data center/ai stuff really not that important much for normal person work. i have volta with dual precision - which is not needed for my needs ... and 3090 no ti with a bit of undervolt is a golden middle if you planing to use nvlink - try to get 2 same ones but i dont think nvlink will help much.
P100 if you have one - its ok , but if you dont - 30xx series with 16gb VRAM will be much much faster on llm`s than your p100... just saying :)
 
GlitchyDot said:
Dont know anything about pc, but p100 is pascal , t4 is turin - so yes, these will work. I dont know how long it will take, but nVidia stopping with pascal,volta gpus on driver developing as they achieved maximum ... dont know about T ones so for now nvidia-drivers should work as i dont see the reason why not and later it might be moved to different naming like nvidia-drivers-XXX.
Well, deppends how many gpu`s you will use but one thing to know that cpu speed ( i think single core ) is in the game as well for faster t/s . so do the research. if you planing on 1 or 2 maybe you just enough normal "gaming rig" hardware with gpu or two to have some fun.
P.s. all these tesla and all others - designed for data center/ai stuff really not that important much for normal person work. i have volta with dual precision - which is not needed for my needs ... and 3090 no ti with a bit of undervolt is a golden middle if you planing to use nvlink - try to get 2 same ones but i dont think nvlink will help much.
P100 if you have one - its ok , but if you dont - 30xx series with 16gb VRAM will be much much faster on llm`s than your p100... just saying :)
Click to expand...
Yes, it's the data center hardware I'm after sicne I have the rack space and power to accommodate it. The only gaming GPU's I have sitting around idle are some very old GTX 790's, and then some stuff from the early 2000's. I do have a 3050 I might be replacing soon with an Intel GPU, but I don't think that will fit in a rack server chassis due to the GPU fans height.
 
Actually, a used L4 doesn't look too bad price wise. I might consider that one so long as it works on FreeBSD.

EDIT: It looks like I can get away with using an Intel GPU by using llama.cpp with lang/opensycl.
 
If i was you - i would use FreeBSD as main OS, passtrough nvidia gpu to bhyve ( linux of your choice ) so you could use cuda.
P.s. Vulkan is intels stuff , rocM - amds stuff, cuda - nvidia.
 
T-Aoki said:
FYI:
List of supported GPUs on driver version 570.133.07 for FreeBSD.
570.133.07 can be used on FreeBSD by overriding driver version.
And list of supported GPUs on driver version 570.124.04 for FreeBSD (currently in-tree version of main branch).

FreeBSD version of driver itself does not support CUDA natively (lacking libraries for CUDA), but reported to be possible using Linuxulator with x11/linux-nvidia-libs installed.
Click to expand...
Hmm, so neither a T4 or an L4 are on that list. It's starting to look like this might be a dead end.
 
GlitchyDot said:
If i was you - i would use FreeBSD as main OS, passtrough nvidia gpu to bhyve ( linux of your choice ) so you could use cuda.
Click to expand...
I have a very strong dislike of Linux for various reasons. I'd rather run Windows if FreeBSD isn't an option (I assume I can do passthrough to a windows vm?). I'll do that for the moment until the hardware support is available. I think the SYCL/OneAPI is the future of all this stuff. It just needs time. I hope FreeBSD can get ahead of the curve and perhaps be a go to platform for AI.
 
tuaris said:
I have a very strong dislike of Linux for various reasons. I'd rather run Windows if FreeBSD isn't an option (I assume I can do passthrough to a windows vm?). I'll do that for the moment until the hardware support is available. I think the SYCL/OneAPI is the future of all this stuff. It just needs time. I hope FreeBSD can get ahead of the curve and perhaps be a go to platform for AI.
Click to expand...
If i understand correctly - if you modify/patch FreeBSD kernel for nVidia gpu passtrough for bhyve - you can not pass nVidia gpu to Windows VM.
But in you case - if you so much dislike Linux, just use Windows and have VM for FreeBSD.
I dont care about Windows so my setup getting back to patched FreeBSD kernel with nVidia gpu passtrough to Linux bhyve ... as its only for llm`s - i just need minimal Linux install and ollama running as everything else is done trough web interface or terminal - ( mostly oatmeal ) . I do have neovim with llm option ( oatmeal or gen.nvim ) but i do have oatmeal launchers for specific llms on my qtile.

tuaris said:
I hope FreeBSD can get ahead of the curve and perhaps be a go to platform for AI.
Click to expand...
There are talks with nVidia and FreeBSD for CUDA support in FreeBSD.
I cant wait for it as it would be the end for Linux to me too. Currently only one thing is stopping me from completely ditching Linux is CUDA support.
I agree - within AI world and all the craze it brings and usefulness / toy in OS - FreeBSD should have an ability to have it in the same power as Windows/Linux/Mac. Yes - Vulkan ( Intels equivalent to CUDA ) exist maybe even possible to have rocM ( AMD`s equivalent to CUDA ) on FreeBSD but the speed difference on some LLM`s is to big to ditch CUDA plus ollama is a bit wonky on FreeBSD.
P.s. im very interested with AMD`s rocM - as in the past AMD Gpus used to be better for image generation due to their algoritm/mathematical/what ever they use to make it done so i think this would be nice if everything would transform in better/same performance for LLM`s as nVidia. Why im saying in the past - i have not checked, researched it ... Ive seen LevelOne video where they were testing AMD`s gppu and generating images with AI and Wendel was saying AMD was better for it than nVidia.
 
tuaris said:
I have a very strong dislike of Linux for various reasons. I'd rather run Windows if FreeBSD isn't an option (I assume I can do passthrough to a windows vm?).
Click to expand...
I'm in the same boat (albeit crunching different data). I can actually recommend the following for CUDA tasks:
  1. Windows (incl Server) host
  2. Hyper-V
  3. Exclusive external network adapter for the Guest (and so Windows is *isolated* / *offline*)
  4. Internal Host <-> Guest network adapter
  5. Use the FreeBSD guest to do all online / management / infrastructure tasks
  6. Delegate *just* the number crunching to the host
  7. (optional) Tweak FreeBSD to allow it to run read-only / tmpfs. Less risk on loss of power, etc.
  8. (rarely necessary) Use shadowsocks, dante or ssh socks5h to expose *select* programs on Windows to access the network.
So long as you keep Windows offline, tragically I very much prefer working with it compared to a random collection of freeware with a kernel called Linux bolted on.

(Note: You will need one of the many open-source cracks to activate Windows *offline*. Bypassing this kind of DRM is allowed as per the law (interop related) in the UK but I am not sure about the rest of the world, the DMCA is very broken and requires exemption requests).

As you probably know, ollama is a tiny go-lang shim around llama-cpp, but it separates it as a client/server. You might find this works well across the VM boundary?
 
As an experiment, I was able to get my hands on a K80 (less than $40 for 10 year old tech), which I was able to install on one of my ESXi hosts. I used the PCI pass through feature and spun up a FreeBSD 14.2 machine on it. I don't expect much out of it, but I wanted to just 'see' how everything fits together.

I installed the 470 driver, enabled it, installed llama-cpp, downloaded a random model, and attempted to start it up...

...and it exploded

Code: 
llama-cli -m Meta-Llama-3-8B-Instruct.Q2_K.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Tesla K80 (NVIDIA) | uma: 0 | fp16: 0 | warp size: 32 | matrix cores: none
build: 0 (unknown) with FreeBSD clang version 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15a9eac9) for x86_64-unknown-freebsd14.1
main: llama backend init
main: load the model and apply lora adapter, if any
llama_load_model_from_file: using device Vulkan0 (Tesla K80) - 12288 MiB free
llama_model_loader: loaded meta data with 27 key-value pairs and 291 tensors from Meta-Llama-3-8B-Instruct.Q2_K.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Models
llama_model_loader: - kv   3:                         general.size_label str              = 8.0B
llama_model_loader: - kv   4:                            general.license str              = llama3
llama_model_loader: - kv   5:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   6:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv   7:                          llama.block_count u32              = 32
llama_model_loader: - kv   8:                       llama.context_length u32              = 8192
llama_model_loader: - kv   9:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv  10:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv  11:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  12:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  13:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  14:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  15:                          general.file_type u32              = 10
llama_model_loader: - kv  16:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  17:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  18:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  19:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  20:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  21:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  22:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  23:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  24:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  25:                    tokenizer.chat_template str              = {% set loop_messages = messages %}{% ...
llama_model_loader: - kv  26:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q2_K:  129 tensors
llama_model_loader: - type q3_K:   64 tensors
llama_model_loader: - type q4_K:   32 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.8000 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 128256
llm_load_print_meta: n_merges         = 280147
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 8192
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 500000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 8192
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 8B
llm_load_print_meta: model ftype      = Q2_K - Medium
llm_load_print_meta: model params     = 8.03 B
llm_load_print_meta: model size       = 2.95 GiB (3.16 BPW)
llm_load_print_meta: general.name     = Models
llm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token        = 128009 '<|eot_id|>'
llm_load_print_meta: EOT token        = 128009 '<|eot_id|>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOG token        = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
ggml_vulkan: Compiling shaders................................................Done!
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/33 layers to GPU
llm_load_tensors:   CPU_Mapped model buffer size =  3024.38 MiB
...................................................................................
llama_new_context_with_model: n_seq_max     = 1
llama_new_context_with_model: n_ctx         = 4096
llama_new_context_with_model: n_ctx_per_seq = 4096
llama_new_context_with_model: n_batch       = 2048
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 0
llama_new_context_with_model: freq_base     = 500000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (8192) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 32, can_shift = 1
llama_kv_cache_init:        CPU KV buffer size =   512.00 MiB
llama_new_context_with_model: KV self size  =  512.00 MiB, K (f16):  256.00 MiB, V (f16):  256.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.49 MiB
llama_new_context_with_model:    Vulkan0 compute buffer size =   669.48 MiB
llama_new_context_with_model: Vulkan_Host compute buffer size =    16.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 356 (with bs=512), 1 (with bs=1)
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
Illegal instruction (core dumped)

I am amazed I got this far. I'll probably spend some time later trying to figure out what's wrong. I think that the model may just be too big for a K80 and might look for something smaller.
 
I actually have more fun poking around in FreeBSD base system than the AI stuff. Because for the most part that is all I learn at school. You can literally build the complete FreeBSD OS kernel - yes you read that right. If that is not cooler than AI, I don't know what is
 
As noted before, CUDA on FreeBSD requires Linux (by default, still Centos7-based) binaries that can run on Linuxulator.
And if you're using legacy x11/nvidia-driver-470, you need to install corresponding x11/linux-nvidia-libs-470.
 
Success!

Turns out I had to pass a flag to tell it to offload 'stuff' to the GPU.

Code: 
llama-server --host 0.0.0.0 -m Meta-Llama-3-8B-Instruct.Q2_K.gguf --chat-template gemma -ngl 33

Where did I get that '33' from? From this line:

Code: 
llm_load_tensors: offloaded 0/33 layers to GPU

Once I passed the flag in, I could see it make use of the GPU:

Code: 
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 33/33 layers to GPU

The server is up and running:

Code: 
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 4096
main: model loaded
main: chat template, chat_template: gemma, example_format: '<start_of_turn>user
You are a helpful assistant

Hello<end_of_turn>
<start_of_turn>model
Hi there<end_of_turn>
<start_of_turn>user
How are you?<end_of_turn>
<start_of_turn>model
'
main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle

I can then start up my browser, connect and start the conversation with the AI model (I can hear the server fans spin up dramatically in the background as it responds).

llama-cpp-first-run.png


It's a little slow, but not unbearable. Completely expected given the age of the hardware.

rcbsdpge said:
I actually have more fun poking around in FreeBSD base system than the AI stuff. Because for the most part that is all I learn at school. You can literally build the complete FreeBSD OS kernel - yes you read that right. If that is not cooler than AI, I don't know what is
Click to expand...

That's cool. I've had my fun compiling the FreeBSD kernel as well. Today (at least to me), getting all this hardware and software to run on FreeBSD is super fun.
 
tuaris said:
That's cool. I've had my fun compiling the FreeBSD kernel as well. Today (at least to me), getting all this hardware and software to run on FreeBSD is super fun.
Click to expand...
This as well. What I like about FreeBSD base system is you get to learn the intrinsics that you would not even get on Linux distributions.

But I meant more like building from the FreeBSD kernel as a base BSD kernel similar to the Linux kernel and packaged Linux distributions. You could build your own BSD distro and tune it to what you like. My student club does side projects for fun. I just built a kernel builder for the Linux kernel and added a framework to build the FreeBSD kernel as well. It’s a wip but I want to build for AI frameworks on FreeBSD like this use case.
 
You must log in or register to reply here.
Back
Top