Linux mm drops VM_SPECIAL for explicit VMA flag semantics

Linux memory management is shedding one of its long-standing sources of driver confusion: the catch-all VM_SPECIAL mask and the muddled VMA flag semantics around it.

Lorenzo Stoakes of ARM has posted a substantial rework on the linux-kernel list that makes virtual memory area (VMA) flag meaning explicit and eliminates VM_SPECIAL. The old mask bundled unrelated properties: whether a mapping is kernel-owned (MMIO, driver pages, and similar), whether it may expand or merge, awkward mlock races that temporarily set invalid flags, and cases meant only to keep get_user_pages away. Drivers abused flags such as the I/O bit; hugetlb wanted non-expandable behaviour without being treated as special; and different subsystems each had their own idea of what special meant for zapping, THP, DAX, and normal-folio checks.

The series limits what mmap hooks may do. Only kernel-owned mappings may set the I/O bit or clear may-write in an mmap path, enforced by validating VMA state after every mmap and mmap_prepare hook, and for special regions inserted into the address space such as VDSO and VVAR. Drivers that map kernel memory are brought in line on mixed-map expectations. Open-coded special-flag tests give way to predicates that name the actual behaviour, including whether two VMAs are allowed to merge.

Related cleanups stop mlock from hijacking the I/O bit to paper over a migration race, tighten file reference handling when a driver replaces the backing file, and generalise post-mmap sanity checks so drivers cannot quietly alter the VMA range or make a read-only file mapping writable. Several in-tree users (among them usbmon, scatterlist, hfi1, ALSA PCM status, uprobes, and the BPF arena) are updated to the stricter model.

The goal is less folklore for driver authors and a base on which further VMA flag cleanups can build. Review is underway, with feedback already prompting at least one merge-path fix to be split out for an RC.