freenode
Databases & Infrastructure

REPACK CONCURRENTLY can silently drop TOAST updates

A race while building the initial snapshot lets a toast rewrite leave committed column values behind with no error.

REPACK (CONCURRENTLY) in PostgreSQL can silently discard committed updates to TOASTed columns when the toast relation is rewritten while the operation is starting up. There is no error, and heap and index checks still report a clean table, so the loss is easy to miss.

Thom Brown found the failure under stress testing. At the start of a concurrent repack, logical decoding records the toast table's current storage identity and then waits for running transactions to finish before the main backend locks that toast table. In that gap the toast relation can still be rewritten (for example with VACUUM FULL or CLUSTER) because the parent table is held only with ShareUpdateExclusiveLock. Decoding then follows the old storage while later updates land on the new one, and the finished repack keeps the stale values.

Manu confirmed the race on current master without debug hooks. A long-running transaction is enough to hold the window open; five of five runs lost the updates once someone rewrote the toast table in the middle and then updated the toasted rows. Maintenance scripts that vacuum large toast tables hard are a realistic way to hit the same pattern in production.

Holding the toast lock from the first step deadlocks: a waiter for AccessExclusiveLock takes an XID before blocking, and the decoding setup waits for all XIDs. Shihao Zhong's fix instead re-checks the toast storage identity after the snapshot is built and the toast table is locked. If it no longer matches what the worker saw, the worker is discarded and a new one is started before any copy begins. Retries under continuous toast rewrites kept the data correct and did not deadlock; an undisturbed run paid nothing extra.

Melanie Plageman asked whether the issue should be tracked as an open item for the PostgreSQL 19 cycle.