mirrors/ollama-for-amd
1
0
Fork 0
mirror of https://github.com/likelovewant/ollama-for-amd.git synced 2025-12-22 06:43:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
58b9ec1f6b9c85369349e971cb2e17a016ce35dd
ollama-for-amd/sample/transforms_test.go
Parth Sareen 0682dae027 sample: improve ollama engine sampler performance (#9374) 
This change bring in various interface cleanups along with greatly improving the performance of the sampler.

Tested with llama3.2 on local machine.
Improves performance from ~ 70 tokens/s -> 135 tokens/s with topK(40) enabled.
Without topK performance is ~ 110 tokens/s
2025-03-07 12:37:48 -08:00

4.0 KiB
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink