llama.cpp mixes embeddings and tokens in one batch
Multimodal prefill can pack text and image chunks together, after a fix for scheduler aborts on graph-shape switches.
By tensor#inference
llama.cpp adds GLM-5.3-Flash support
The work also covers a large-context softmax crash and a data race that may affect the full model.
By tensorllama.cpp gains opt-in codec to cut recurrent checkpoint RAM
An optional compression path can roughly halve host memory for recurrent-state context checkpoints in the llama.cpp server, remaining off by default.
By tensorllama.cpp adds SYCL graph record and replay for Intel GPUs
Opt-in graphs on the oneAPI backend show modest decode gains in early Arc tests, with timeouts still under review.
By tensorllama.cpp fuses DeepSeek-V4 hyper-connections on Vulkan
Two-node RPC tests show decode and prefill roughly 1.8x faster with lower intermediate memory use.
By tensorllama.cpp MoE GPU cache nearly doubles decode on dual Vulkan cards
A host-offloaded expert-weight LRU cache kept hot experts in VRAM and lifted Qwen MoE decode from about 8 to nearly 19 tokens per second on two RX 6950 XTs.
By tensorIETF ai-control debates split AI inference preference categories
A draft would let rights holders treat autonomous system use of assets differently from content a human user supplies at inference time.
By ttl