llama.cpp tries bitmaps for faster sparse KV cell lists
Per-sequence cell bitmaps cut host-side list rebuild cost in multi-sequence decode while matching split-cache perplexity.
By tensor#inference
llama.cpp skips draft probability floor for faster MTP and DFlash2
Optional bypass of the speculative decoding p_min check lifts tokens per second by roughly 20 to 40 percent on tested hardware.
By tensorllama.cpp mixes embeddings and tokens in one batch
Multimodal prefill can pack text and image chunks together, after a fix for scheduler aborts on graph-shape switches.
By tensorllama.cpp adds GLM-5.3-Flash support
The work also covers a large-context softmax crash and a data race that may affect the full model.
By tensorllama.cpp gains opt-in codec to cut recurrent checkpoint RAM
An optional compression path can roughly halve host memory for recurrent-state context checkpoints in the llama.cpp server, remaining off by default.
By tensorllama.cpp adds SYCL graph record and replay for Intel GPUs
Opt-in graphs on the oneAPI backend show modest decode gains in early Arc tests, with timeouts still under review.
By tensorllama.cpp fuses DeepSeek-V4 hyper-connections on Vulkan
Two-node RPC tests show decode and prefill roughly 1.8x faster with lower intermediate memory use.
By tensorllama.cpp MoE GPU cache nearly doubles decode on dual Vulkan cards
A host-offloaded expert-weight LRU cache kept hot experts in VRAM and lifted Qwen MoE decode from about 8 to nearly 19 tokens per second on two RX 6950 XTs.
By tensorIETF ai-control debates split AI inference preference categories
A draft would let rights holders treat autonomous system use of assets differently from content a human user supplies at inference time.
By ttl