mirrors/ollama-for-amd
1
0
Fork 0
mirror of https://github.com/likelovewant/ollama-for-amd.git synced 2025-12-25 07:58:01 +00:00
Code Issues Packages Projects Releases Wiki Activity

cuda: skip large batches

Browse Source 
cuda panics on batches larger than 1024 so skip those and fallback to
cpu
This commit is contained in:
Michael Yang
2025-11-18 11:13:37 -08:00
committed by Michael Yang
parent 92981ae3f2
commit 0796d79d19
9 changed files with 35 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
llama/patches/0032-vulkan-Fuse-rope-set_rows-16769.patch
View File

@@ -31,7 +31,7 @@ Add new backend tests.
6 files changed, 371 insertions(+), 117 deletions(-)
diff --git a/ggml/src/ggml-vulkan/ggml-vulkan.cpp b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
index b2855b078..aaf4334b5 100644
index 63a762ec2..db92a7901 100644
--- a/ggml/src/ggml-vulkan/ggml-vulkan.cpp
+++ b/ggml/src/ggml-vulkan/ggml-vulkan.cpp
@@ -458,6 +458,11 @@ static topk_moe_mode ggml_vk_num_additional_ops_to_topk_moe_mode(uint32_t num) {