llm: New memory management

Browse Source

This changes the memory allocation strategy from upfront estimation to
tracking actual allocations done by the engine and reacting to that. The
goal is avoid issues caused by both under-estimation (crashing) and
over-estimation (low performance due to under-utilized GPUs).

It is currently opt-in and can be enabled for models running on the
Ollama engine by setting OLLAMA_NEW_ESTIMATES=1. Behavior in other
cases is unchanged and will continue to use the existing estimates.

...

This commit is contained in:

Jesse Gross

2025-05-29 12:21:48 -07:00

committed by Jesse Gross

parent ef7d26ba2c

commit d5a0d8d904

26 changed files with 1860 additions and 900 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

1049

llm/server.go

File diff suppressed because it is too large Load Diff