The symptom

You have a table that stores user state. Something happens — an event arrives — and your code does the obvious thing:

BEGIN;
DELETE FROM user_state WHERE user_id = $1;
INSERT INTO user_state (user_id, state, score, updated_at) VALUES ($1, $2, $3, now());
COMMIT;

It’s clean. It’s idempotent. It doesn’t require you to think about which columns changed. For a long time it works fine.

Then event volume grows, and PostgreSQL’s CPU starts climbing in a way that nothing in your query plans explains. Your queries are still index scans. Your pg_stat_statements top entries look reasonable. But CPU is pinned, replication lag creeps up, and every so often the whole database gets slow for a few minutes and then recovers on its own.

If you look at pg_stat_activity during one of those windows, you’ll usually find the culprit sitting right there:

autovacuum: VACUUM public.user_state

That’s the story. But “autovacuum is expensive” is not an explanation — it’s a restatement of the symptom. To actually fix this you need to understand what a DELETE followed by an INSERT does to the physical 8 KB pages on disk, and why that specific shape of write is close to the worst case for PostgreSQL’s storage engine.

Let’s go all the way down.

Part 1: PostgreSQL never modifies a row in place

This is the single fact that everything else follows from.

PostgreSQL implements MVCC (Multi-Version Concurrency Control) by keeping multiple physical versions of every row and letting each transaction see the version appropriate to its snapshot. There is no undo log (unlike Oracle or MySQL’s InnoDB, which keep old versions in a separate rollback segment). In PostgreSQL, old row versions live in the table itself, mixed in with the live ones.

Concretely:

• INSERT writes a new tuple and stamps it with xmin = the inserting transaction ID.
• DELETE does not remove anything. It stamps the existing tuple with xmax = the deleting transaction ID. The bytes stay exactly where they were.
• UPDATE is internally a DELETE + INSERT: it stamps xmax on the old tuple and writes a brand-new tuple somewhere.

So a “deleted” row is still fully present on disk, occupying its bytes, until some later process comes along and physically reclaims it. That process is vacuum.

Here’s this happening for real. A 100-row table, then delete 50 rows and insert 50 new ones:

CREATE EXTENSION pageinspect;
CREATE TABLE demo (id int PRIMARY KEY, state text);
INSERT INTO demo SELECT g, 'active' FROM generate_series(1,100) g;
-- delete half, re-insert half (the pattern from our app)
DELETE FROM demo WHERE id <= 50;
INSERT INTO demo SELECT g, 'active' FROM generate_series(1,50) g;
SELECT lp, lp_flags, lp_len, t_xmin, t_xmax
FROM heap_page_items(get_raw_page('demo', 0))
WHERE lp BETWEEN 48 AND 54;

 lp | lp_flags | lp_len | t_xmin | t_xmax
----+----------+--------+--------+--------
 48 |        1 |     35 |    801 |    802   <- deleted, still on the page
 49 |        1 |     35 |    801 |    802   <- deleted, still on the page
 50 |        1 |     35 |    801 |    802   <- deleted, still on the page
 51 |        1 |     35 |    801 |      0   <- live
 52 |        1 |     35 |    801 |      0   <- live
 53 |        1 |     35 |    801 |      0   <- live
 54 |        1 |     35 |    801 |      0   <- live

The deleted tuples have t_xmax = 802 (the transaction that deleted them) and are still LP_NORMAL (lp_flags = 1). They occupy real space. The table now has 150 line pointers where the logical row count is 100.

SELECT tuple_count, dead_tuple_count, dead_tuple_percent, free_percent
FROM pgstattuple('demo');

 tuple_count | dead_tuple_count | dead_tuple_percent | free_percent
-------------+------------------+--------------------+--------------
         100 |               50 |              21.36 |        19.09

21% of that table is garbage. And that’s after one round.

Part 2: Anatomy of an 8 KB page

PostgreSQL stores everything in fixed-size blocks — 8 KB by default (SHOW block_size). Understanding the layout of one page is the key to understanding why churn is expensive.

Two structures grow toward each other from opposite ends: the line pointer array from the top, the tuple data from the bottom. The page is full when they meet.

The line pointer (also called an item identifier, ItemIdData) is a tiny 4-byte struct that says “the tuple at slot N lives at byte offset X and is Y bytes long.” It has four possible states, and you’ll see all of them in this article:

The tuple header (HeapTupleHeader, 23 bytes padded to 24) is where MVCC lives:

That’s 24 bytes of overhead per row version, plus 4 bytes for the line pointer. For a table storing a user ID, a short state string, and a timestamp, the MVCC bookkeeping can easily be 40% of the row’s on-disk size.

Now here’s the critical bit for our problem:

A page’s free space is only usable for a new tuple if it is contiguous and the page has a free line pointer or room to add one. Dead tuples fragment the page. Only vacuum can defragment it.

Part 3: Why DELETE + INSERT is the worst shape

Compare what the two write patterns do to a page.

A DELETE + INSERT can never be a HOT update. HOT (Heap-Only Tuple) is an optimization that only applies to UPDATE, and only when two conditions hold:

1. No indexed column changed.
2. The new version fits on the same page as the old one.

When both hold, PostgreSQL writes the new version onto the same page and links it via t_ctid, leaving the index entries alone. That’s an enormous saving: index maintenance is usually the dominant cost of a write, because B-tree inserts are random-access, and every index on the table pays it.

By splitting your logical update into two statements, you’ve told PostgreSQL: this is a brand-new row that has nothing to do with the old one. It has no choice but to write full index entries into every index, and to leave a dead tuple behind that vacuum must chase down.

You can see HOT working with pageinspect. In a table where updates are HOT-eligible, the page fills up with redirect pointers:

SELECT lp, lp_flags, lp_len, t_ctid
FROM heap_page_items(get_raw_page('user_state_hot', 0)) LIMIT 5;

 lp | lp_flags | lp_len | t_ctid
----+----------+--------+--------
  1 |        2 |      0 |          <- LP_REDIRECT
  2 |        2 |      0 |          <- LP_REDIRECT
  3 |        2 |      0 |          <- LP_REDIRECT
  4 |        2 |      0 |          <- LP_REDIRECT
  5 |        2 |      0 |          <- LP_REDIRECT

Those redirect pointers are the physical trace of updates that cost zero index writes.

Part 4: What vacuum actually does

Vacuum is not one operation. A standard (non-full) VACUUM of a table with indexes runs in phases, and understanding the phases explains where the CPU goes.

Phase 1 — Heap scan

Vacuum scans the table’s pages (skipping ones the visibility map marks all-visible, which is the main reason vacuum on an append-only table is cheap and vacuum on a churned table is not). For each page it:

• Prunes HOT chains and marks reclaimable tuples.
• Collects the TIDs (page number, line pointer) of dead tuples into memory.
• Sets LP_DEAD on their line pointers.

The dead TID list is bounded by maintenance_work_mem (or autovacuum_work_mem). In PostgreSQL 17+ this list is stored in an adaptive radix tree (TidStore) rather than a flat array, which cut memory usage dramatically and removed the old 1 GB cap — a genuinely large improvement for high-churn tables.

Phase 2 — Index vacuum

This is the expensive part. For every index on the table, vacuum performs a full scan (a btbulkdelete) and removes every index entry pointing at a dead TID.

Note what that means: the cost is per index, and it’s a full index scan, not a lookup. A table with five indexes pays five full index scans per vacuum cycle. This is why “just add an index” decisions compound badly on high-churn tables.

Phase 3 — Heap vacuum

Vacuum revisits the pages and turns LP_DEAD line pointers into LP_UNUSED, then defragments the page: it slides the remaining tuples together so the free space becomes one contiguous block. Now that space is genuinely reusable.

If the dead-TID list overflowed maintenance_work_mem in phase 1, vacuum has to loop: scan more heap → vacuum all indexes again → vacuum heap again. A single vacuum of a big churned table can do the full index pass multiple times. This is the classic “why has autovacuum been running for six hours” answer.

Phase 4 — Freezing

Separately, vacuum replaces old xmin values with a frozen marker so transaction IDs can wrap around safely. Anti-wraparound vacuums are non-cancelable and will run whether you like it or not. PostgreSQL 18 added eager freezing (vacuum_max_eager_freeze_failure_rate, default 0.03) so that normal vacuums opportunistically freeze pages, spreading this cost out instead of saving it up for one giant emergency vacuum.

Phase 5 — Truncation

If there are empty pages at the end of the table, vacuum tries to return them to the OS. This requires an ACCESS EXCLUSIVE lock (briefly). Note the word end: vacuum almost never shrinks a bloated table, because the free space is scattered throughout, not conveniently at the tail. It makes the space reusable by PostgreSQL, not available to your filesystem.

Here’s the before/after on our demo table:

VACUUM demo;
SELECT lp_flags, count(*) 
FROM heap_page_items(get_raw_page('demo',0)) GROUP BY 1;

Free space on the page went from 19.09% → 43.51%. That’s the work vacuum did. Now multiply it by every page you churned.

Part 5: The amplification math

Here is the part people underestimate. One logical “user state changed” event does not cause one write. It causes, at minimum:

Steps 7–13 are invisible in your application metrics. Your app measures step 1–6 latency and reports “the insert took 2 ms.” The other two-thirds of the cost lands on a background worker, asynchronously, in a burst, on the same CPU.

That’s the answer to “why does CPU go up but my queries look fine.”

Point 5b: the full-page-image trap

After each checkpoint, the first write to any page triggers a full 8 KB page image written into WAL (full_page_writes, on by default, and required for crash safety on most filesystems). If your churn touches many distinct pages, you pay a lot of FPIs. If your churn is concentrated on a few pages, you pay far fewer. Delete+insert scatters writes across pages — the new tuple often lands in a different page from the one you dirtied with the xmax write. Two dirty pages instead of one, two FPI candidates instead of one.

Point 5c: the read-side cost nobody mentions

Dead tuples also make reads slower and CPU-hungrier:

• An index scan follows an index entry to the heap, reads the tuple, checks visibility, discovers it’s dead, and discards it. That’s a wasted buffer access plus a visibility check. You can see this directly: EXPLAIN (ANALYZE, BUFFERS) will show a rising Buffers: shared hit count for a query returning the same number of rows.
• Bloated tables have worse cache locality. The same 100k live rows spread over 2× the pages means 2× the shared buffers to hold them, which evicts other things.
• On a read, PostgreSQL may perform opportunistic HOT pruning on the page — real CPU work done inside your SELECT.

Part 6: Measurements

I ran a controlled comparison: 100,000 rows, ten rounds of modifying 20,000 of them, with an explicit VACUUM after each round so we’re looking at steady state, not a runaway. Three variants:

Read those rows carefully, because the middle one is the trap.

Switching from **DELETE+**INSERT to UPDATE bought essentially nothing on its own. The second variant still generated more WAL than the first. Why? Because the UPDATE touched updated_at, and updated_at had an index on it. An UPDATE that modifies any indexed column is not HOT-eligible, so it does exactly what delete+insert does: new heap tuple, new entries in every index, dead tuple left behind.

The third variant — same UPDATE, but with no index on the mutated columns and fillfactor lowered to 80 — generated less than half the WAL and made 73% of its updates HOT.

The cost of that win is visible too: the HOT table starts at 1042 heap pages instead of 834, because fillfactor=80 deliberately leaves 20% of every page empty. That empty space is what new row versions land in. You are trading disk space for CPU and I/O, which on modern hardware is almost always the right trade.

(These are synthetic single-client numbers on PostgreSQL 16 — treat the ratios as directional, not as a benchmark of your workload.)

Part 7: Fixing it, in order of leverage

7.1 Don’t write when nothing changed

The cheapest write is the one that doesn’t happen. A shocking amount of churn in “collect user state” systems is rewriting a row to the exact same value because an event arrived that didn’t actually change anything.

INSERT INTO user_state (user_id, state, score, updated_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (user_id) DO UPDATE
SET state      = EXCLUDED.state,
    score      = EXCLUDED.score,
    updated_at = now()
WHERE user_state.state  IS DISTINCT FROM EXCLUDED.state
   OR user_state.score  IS DISTINCT FROM EXCLUDED.score;

That trailing WHERE on the DO UPDATE clause is the whole trick: if nothing changed, PostgreSQL writes nothing — no new tuple, no dead tuple, no index entry, no WAL. In event-driven systems where the same state gets recomputed repeatedly, this alone often removes 50–90% of write volume.

Measure how much you’d save before you build anything else:

-- how many of your writes are no-ops?
SELECT count(*) FILTER (WHERE state = $2 AND score = $3) AS would_be_noop,
       count(*) AS total
FROM user_state WHERE user_id = ANY($1);

7.2 Make your updates HOT-eligible

Two requirements, both actionable:

(a) Don’t index the columns that change. Audit every index against your write path:

SELECT i.relname AS index_name,
       array_agg(a.attname ORDER BY x.n) AS indexed_columns,
       s.idx_scan AS times_used
FROM pg_index ix
JOIN pg_class i ON i.oid = ix.indexrelid
JOIN pg_class t ON t.oid = ix.indrelid
JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS x(attnum, n) ON true
JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = x.attnum
LEFT JOIN pg_stat_user_indexes s ON s.indexrelid = i.oid
WHERE t.relname = 'user_state'
GROUP BY i.relname, s.idx_scan
ORDER BY s.idx_scan NULLS FIRST;

An index on updated_at is the single most common HOT-killer in exactly this kind of table. Ask whether you truly need it, or whether a partial index, a coarser index (date_trunc(‘hour’, updated_at) doesn’t help — but a separate low-churn audit table might), or no index at all would do.

Unused indexes are pure loss: they cost writes and vacuum passes and return nothing. Drop them.

(b) Leave room on the page.

ALTER TABLE user_state SET (fillfactor = 80);
-- fillfactor only applies to newly written pages;
-- rewrite the table to apply it to existing data:
VACUUM FULL user_state;   -- takes ACCESS EXCLUSIVE lock
-- or, online:
-- pg_repack -t user_state

For a heavily-updated table, 70–85 is a reasonable range. Below ~70 you’re wasting a lot of buffer cache on empty space.

Then confirm it’s working:

SELECT relname,
       n_tup_upd,
       n_tup_hot_upd,
       round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd,0), 1) AS hot_pct
FROM pg_stat_user_tables
WHERE relname = 'user_state';

If hot_pct is near zero on an update-heavy table, that is your bug. Aim for 80%+.

7.3 Tune autovacuum for this table, not globally

The defaults are tuned for a small table on a small machine. autovacuum_vacuum_scale_factor = 0.2 means vacuum waits until 20% of the table is dead before acting. On a 500-million-row table that’s 100 million dead tuples before anything happens — and then one enormous, disruptive vacuum.

You want the opposite: vacuum more often, doing less each time.

ALTER TABLE user_state SET (
  autovacuum_vacuum_scale_factor  = 0.01,   -- act at 1% dead, not 20%
  autovacuum_vacuum_threshold     = 1000,
  autovacuum_analyze_scale_factor = 0.02,
  autovacuum_vacuum_cost_delay    = 0,      -- don't throttle this table
  autovacuum_vacuum_cost_limit    = 3000
);

And globally, if your hardware is modern (NVMe, many cores), the shipped throttle is far too conservative:

autovacuum_max_workers      = 6          # default 3
autovacuum_naptime          = 15s        # default 1min
autovacuum_vacuum_cost_delay = 2ms       # default 2ms; consider 0 on fast storage
vacuum_cost_limit           = 2000       # default 200
maintenance_work_mem        = 2GB        # gives vacuum a big dead-TID store
autovacuum_work_mem         = 1GB

The cost-delay mechanism is the one most often left at a value that makes vacuum unable to keep up: vacuum accumulates “cost” (1 per buffer hit, 2 per miss, 20 per dirtied page) and sleeps cost_delay whenever it exceeds cost_limit. With the defaults, vacuum is deliberately slowed to something like a few MB/s. On a table generating hundreds of MB of dead tuples per minute, it will never catch up, and dead tuples accumulate forever.

PostgreSQL 18 also added autovacuum_vacuum_max_threshold (default 100,000,000), which caps the scale-factor calculation so very large tables can’t defer vacuum indefinitely. It helps, but a per-table scale factor is still better.

7.4 Reconsider the data model

If your state genuinely is “replace the whole thing on every event,” ask whether it belongs in a row-per-user table at all:

• Collapse many rows into one. If you’re storing N state rows per user and replacing them all, one row with a jsonb column can turn N deletes + N inserts into one update — and if jsonb is under the TOAST threshold and no indexed column changes, that update can be HOT. (Careful: a large jsonb that gets TOASTed is a whole separate write amplification problem. This works when the document is small, roughly under 2 KB.)
• Append-only + a materialized “current” view. Write events to a partitioned append-only table (no updates, no deletes, so almost no vacuum work — just insert-triggered vacuums for freezing), and maintain the current state separately, or compute it with a DISTINCT ON query over a well-indexed event log. Append-only tables are the cheapest thing PostgreSQL does.
• Move the hottest state out of PostgreSQL. If a piece of state changes many times per second per user and you only ever need the latest value, Redis or a dedicated KV store is the honest answer. Not everything needs to be transactional and durable.

Part 8: The hard question — updating billions of rows

This is the case where all the usual advice breaks, so it deserves its own treatment.

Why the naive approach fails, precisely

UPDATE huge_table SET status = 'migrated' WHERE status = 'pending';
-- 2,000,000,000 rows

Four separate disasters:

1. Table size doubles. Every updated row writes a new version while the old one stays. A 2 TB table becomes 4 TB mid-statement. Then vacuum makes the space reusable but does not return it to the OS. You now permanently need 4 TB of disk for 2 TB of data.

2. It generates terabytes of WAL in one burst. Replicas fall behind, archiving falls behind, and if you’re on a cloud provider you may fill the WAL volume and take the primary down.

3. It holds a single transaction open for hours — and this is the one that hurts most. Vacuum can only remove tuples older than the oldest running transaction’s snapshot (the xmin horizon). A long transaction pins that horizon. So while your giant UPDATE runs, vacuum cannot clean up dead tuples anywhere in the entire database — not just this table. Every other table quietly bloats. This is how one migration takes down a whole cluster.

4. If it fails at 95%, all the work is thrown away — but the dead tuples it created are not. You get the bloat with none of the benefit.

You can watch the horizon problem happen:

SELECT pid, age(backend_xid) AS xid_age, state,
       now() - xact_start AS duration, left(query, 60)
FROM pg_stat_activity
WHERE backend_xid IS NOT NULL
ORDER BY xact_start
LIMIT 5;

Also check for the other two horizon-pinners people forget: replication slots and prepared transactions.

SELECT slot_name, active, age(xmin) AS xmin_age,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;
SELECT gid, prepared, age(transaction) FROM pg_prepared_xacts;

An inactive replication slot will pin the xmin horizon forever and no amount of vacuum tuning will save you.

Strategy A: Batched backfill with keyset pagination

The workhorse. Small transactions, each one committing so the xmin horizon advances and vacuum can keep pace.

-- Track progress in a real table so the job is resumable
CREATE TABLE IF NOT EXISTS backfill_progress (
  job_name text PRIMARY KEY,
  last_id  bigint NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now()
);

-- One batch. Run this in a loop from your application or a scheduler.
WITH bounds AS (
  SELECT last_id FROM backfill_progress WHERE job_name = 'status_migration'
),
batch AS (
  SELECT id FROM huge_table
  WHERE id > (SELECT last_id FROM bounds)
  ORDER BY id
  LIMIT 5000
  FOR UPDATE SKIP LOCKED
),
upd AS (
  UPDATE huge_table t
  SET status = 'migrated'
  FROM batch b
  WHERE t.id = b.id
    AND t.status IS DISTINCT FROM 'migrated'   -- skip no-op writes
  RETURNING t.id
)
UPDATE backfill_progress
SET last_id = COALESCE((SELECT max(id) FROM batch), last_id),
    updated_at = now()
WHERE job_name = 'status_migration'
RETURNING last_id, (SELECT count(*) FROM upd) AS rows_updated;

Rules that make the difference between this working and this being a slow disaster:

• Keyset pagination, never OFFSET. OFFSET 10000000 re-scans ten million rows every batch. WHERE id > $last uses the index.
• Batch size 1,000–10,000 rows. Big enough to amortize overhead, small enough that each transaction finishes in well under a second.
• AND col IS DISTINCT FROM $new so re-running a batch is free and no-ops don’t write.
• Throttle deliberately. Sleep 50–200 ms between batches. You are racing autovacuum; give it room to win. Adaptive throttling — sleep longer when replication lag or dead-tuple count rises — is worth the extra code:

-- feed this into your loop's sleep decision
SELECT (SELECT n_dead_tup FROM pg_stat_user_tables WHERE relname='huge_table') AS dead,
       (SELECT coalesce(max(pg_wal_lsn_diff(sent_lsn, replay_lsn)), 0)
        FROM pg_stat_replication) AS replica_lag_bytes;

• Run it during a low-traffic window if you have one, but design it to be safe at any time.

Expected duration is the thing to be honest about: 2 billion rows at 5,000 rows/batch with a 100 ms sleep is 400,000 batches ≈ 11+ hours minimum, realistically a few days. That’s fine. Plan for days, make it resumable, make it observable.

Strategy B: Partitioning + partition swap

If your data is naturally partitionable (by time, tenant, hash of user ID), this changes the game completely, because you can rebuild a partition offline and swap it in atomically.

-- Assume: huge_table PARTITION BY RANGE (created_at)
--         with partition huge_table_2026_03
-- 1. Build the new version at full speed, outside the live table.
CREATE TABLE huge_table_2026_03_new (LIKE huge_table INCLUDING ALL);
INSERT INTO huge_table_2026_03_new
SELECT id, user_id, 'migrated'::text AS status, payload, created_at
FROM huge_table_2026_03;
-- 2. Indexes AFTER the load - far faster than maintaining them during it.
--    No CONCURRENTLY needed: nothing is reading this table yet.
CREATE INDEX ON huge_table_2026_03_new (user_id);
ANALYZE huge_table_2026_03_new;
-- 3. Atomic swap: milliseconds of ACCESS EXCLUSIVE lock, not hours.
BEGIN;
  ALTER TABLE huge_table DETACH PARTITION huge_table_2026_03;
  ALTER TABLE huge_table ATTACH PARTITION huge_table_2026_03_new
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');
COMMIT;
-- 4. Keep the old one around until you're confident, then drop it.
-- DROP TABLE huge_table_2026_03;

Why this is so much better:

• The bulk INSERT into an empty, unindexed table is the fastest write path PostgreSQL has. No dead tuples are created at all.
• Zero vacuum debt: you never made garbage, so there’s nothing to collect.
• The lock window is milliseconds.
• Rollback is trivial: swap the old partition back.
• You can do partitions one at a time, over weeks, with a rollback point after each.

There’s a caveat worth knowing: ATTACH PARTITION validates that existing rows satisfy the partition constraint, which requires a scan — unless you add a matching CHECK constraint to the new table first, in which case PostgreSQL skips the scan.

ALTER TABLE huge_table_2026_03_new
  ADD CONSTRAINT part_check
  CHECK (created_at >= '2026-03-01' AND created_at < '2026-04-01') NOT VALID;
ALTER TABLE huge_table_2026_03_new VALIDATE CONSTRAINT part_check;
-- now ATTACH is instant

If you have billions of rows and no partitioning, partitioning is probably the actual project — not the update itself. Everything downstream gets easier: DROP PARTITION replaces mass DELETE (instant, and creates zero dead tuples), vacuum works per-partition in parallel, and index sizes stay manageable.

Strategy C: Rebuild and swap the whole table

When the change is uniform and you can’t partition, build a new table and cut over.

For a bloated table with no schema change, pg_repack does this online: it builds a fresh copy, captures concurrent changes via triggers, replays them, and swaps — taking the exclusive lock only for the final instant.

pg_repack -h host -d mydb -t huge_table --jobs 4

For a schema/data change, the same idea by hand with logical replication or triggers; tools like pgroll and Percona’s approach automate it. The pattern is always: new table, bulk load, keep in sync, atomic rename.

Budget for double the disk space. That’s the price of every swap-based strategy.

Strategy D: Don’t update at all

The strategy people skip past. Ask whether the update is necessary:

• Add a new column with a default instead of rewriting values — since PostgreSQL 11, adding a column with a non-volatile default is a metadata-only change that rewrites nothing, no matter how big the table.
• Write a view that computes the new value from the old one, and migrate readers to it.
• Use a generated column if the new value is derivable.
• Let it happen lazily: update rows as they’re naturally touched by the application, with a background job mopping up the long tail over months.

A billion-row update you don’t have to run is strictly better than one you run well.

Part 9: What to actually watch

Four queries. Put them on a dashboard.

1. Is vacuum keeping up?

SELECT relname,
       n_live_tup,
       n_dead_tup,
       round(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_pct,
       last_autovacuum,
       autovacuum_count,
       n_tup_upd,
       n_tup_hot_upd,
       round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd, 0), 1) AS hot_pct
FROM pg_stat_user_tables
WHERE n_dead_tup > 10000
ORDER BY n_dead_tup DESC
LIMIT 20;

Sustained dead_pct above ~10% on a busy table means vacuum is losing. hot_pct near zero on an update-heavy table means you’re paying full index cost on every write.

2. What is vacuum doing right now?

SELECT p.pid, p.relid::regclass AS table, p.phase,
       pg_size_pretty(p.heap_blks_total * 8192::bigint) AS total,
       round(100.0 * p.heap_blks_scanned / NULLIF(p.heap_blks_total,0), 1) AS pct_scanned,
       p.index_vacuum_count,
       now() - a.xact_start AS running_for
FROM pg_stat_progress_vacuum p
JOIN pg_stat_activity a USING (pid);

index_vacuum_count > 1 is the smoking gun for maintenance_work_mem being too small — vacuum had to do the index pass more than once.

3. Is anything pinning the xmin horizon?

SELECT 'backend' AS src, pid::text AS id, age(backend_xmin) AS xmin_age
FROM pg_stat_activity WHERE backend_xmin IS NOT NULL
UNION ALL
SELECT 'slot', slot_name, age(xmin) FROM pg_replication_slots WHERE xmin IS NOT NULL
UNION ALL
SELECT 'prepared_xact', gid, age(transaction) FROM pg_prepared_xacts
ORDER BY xmin_age DESC NULLS LAST
LIMIT 10;

Anything with a large age here is preventing vacuum from cleaning up database-wide.

4. How bloated is it, really?

CREATE EXTENSION IF NOT EXISTS pgstattuple;
-- accurate but does a full scan; use the sampling version on huge tables
SELECT * FROM pgstattuple('user_state');
SELECT * FROM pgstattuple_approx('huge_table');
SELECT * FROM pgstatindex('user_state_user_id_idx');

For indexes, avg_leaf_density below ~50 means the index is worth rebuilding:

REINDEX INDEX CONCURRENTLY user_state_user_id_idx;

The checklist

If you’re in the situation this article opened with, do these in order:

1. Measure first. dead_pct, hot_pct, pg_stat_progress_vacuum, xmin horizon. Don’t tune blind.
2. Kill no-op writes with ON CONFLICT DO UPDATE … WHERE … IS DISTINCT FROM. Usually the biggest single win, and the cheapest to ship.
3. Convert delete+insert to UPDATE — but only after step 4, or it buys you nothing.
4. Make updates HOT-eligible. Drop or move indexes on frequently-changed columns (especially updated_at). Set fillfactor to 80. Verify hot_pct climbs above 80%.
5. Tune autovacuum per-table: scale factor down to 0.01, cost delay to 0, cost limit way up, maintenance_work_mem to gigabytes.
6. Drop unused indexes. Every one is a write tax and an extra full scan per vacuum.
7. Partition if the table is large and the data has a natural key. This makes every future problem easier.
8. For mass updates: batched keyset backfill, or partition rebuild-and-swap. Never one giant statement.

The underlying idea is simple enough to hold in your head: PostgreSQL is a copy-on-write storage engine that trades write amplification for concurrency. Delete+insert opts into the maximum possible amplification — no HOT, full index maintenance, guaranteed dead tuples, guaranteed vacuum work. Understanding what happens on the 8 KB page is what turns “the database is slow” into a list of specific, fixable things.

If this was useful, the two commands worth internalizing are pageinspect’s heap_page_items() and pgstattuple(). Point them at your hottest table. What you find will probably explain your CPU graph.

References

• PostgreSQL 18: Routine Vacuuming
• PostgreSQL 18: Vacuuming configuration parameters
• PostgreSQL 18 Released
• PostgreSQL 18 Vacuuming Improvements Explained (Microsoft)
• autovacuum_vacuum_max_threshold in PostgreSQL 18 (dbi services)
• Faster VACUUM with Adaptive Radix Trees in PostgreSQL 17 (pganalyze)
• VACUUM at the Page Level (boringSQL)

There is a specific kind of bug that only shows up in production, only in October, and only for users in one country. It gets closed as “not reproducible” twice before someone finally sits down with a coffee and reads the column definition.

It is almost always a datetime bug.

The frustrating part is that both Java and PostgreSQL have excellent date/time support. java.time is one of the best-designed APIs in the JDK. PostgreSQL’s temporal types are precise and well documented. The bugs happen at the seam — in the assumptions each side makes about what the other means, and in the two or three words in the type names that mean the opposite of what you’d guess.

This guide walks through both models, then spends real time on the part that actually causes outages: time zones.

Every code output and query result below was executed — Java 21 and PostgreSQL 16 — not written from memory.

Part 1: The Java Side

Ask three questions, get one type

Every java.time class is an answer to some combination of three questions:

1. Do I know the date?
2. Do I know the time of day?
3. Do I know which point on the global timeline this is?

That last one is the important one, and it’s the one people skip. Here’s the map:

The single most useful sentence in this article: LocalDateTime is not a point in time. 2026-08-15T12:00 is twelve o’clock somewhere, and until you say where, it refers to a 26-hour-wide smear across the planet. If you have an Instant and you call .toLocalDateTime(), you have thrown away information and the compiler will not warn you.

Instant is the boring, correct default

Instant now = Instant.now();     // 2026-08-15T10:00:00.123456789Z

An Instant is nanoseconds since the epoch. No zone, no offset, no calendar. When it prints with a Z, that isn’t a stored time zone — it’s the ISO-8601 way of rendering an epoch value.

Anything that happened — a login, a payment, a row insert — is an Instant. Reach for something else only when you have a reason.

ZoneId vs ZoneOffset: not the same thing

This distinction matters more than any other in this article.

ZoneId.of("Europe/London")  // -> java.time.ZoneRegion
ZoneId.of("+01:00")         // -> java.time.ZoneOffset

A ZoneOffset is a fixed number of hours and minutes from UTC. +01:00. That’s the whole thing. It’s a fact about one moment.

A ZoneId (specifically a ZoneRegion, like Europe/London) is a rulebook. It knows that London was +00:00 in January, +01:00 in August, and it knows the exact instants at which it flips — going back decades and forward as far as legislators have declared.

ZoneOffset is a value. ZoneId is a function from instant to offset. Confusing them is the root cause of most scheduling bugs.

OffsetDateTime vs ZonedDateTime

Both identify a real instant. The difference is what survives.

ZonedDateTime nyc = ZonedDateTime.of(2026, 7, 4, 12, 0, 0, 0,
                                     ZoneId.of("America/New_York"));
nyc                    // 2026-07-04T12:00-04:00[America/New_York]
nyc.toOffsetDateTime() // 2026-07-04T12:00-04:00
nyc.toInstant()        // 2026-07-04T16:00:00Z

All three describe the same moment. But the ZonedDateTime still knows it’s in New York, so if you add six months to it, it will correctly land on -05:00. The OffsetDateTime has been flattened — it remembers that the offset was -04:00, not why. Add six months and you get a wrong wall-clock time.

Rule of thumb: ZonedDateTime for calculating, OffsetDateTime for transporting, Instant for storing.

Duration vs Period

LocalDate.of(2026, 1, 31).plusMonths(1)  // 2026-02-28

Duration is exact elapsed time — seconds and nanos. Period is calendar time — years, months, days. “One month” is not a fixed number of seconds, and plusMonths clamps rather than overflowing. Both behaviors are correct; mixing them up is not.

This distinction resurfaces the moment DST enters the picture, and we’ll come back to it.

A word on the old API

java.util.Date is a mislabeled Instant (it has no time zone despite what toString() implies). Calendar is mutable and lenient. java.sql.Timestamp extends java.util.Date and adds nanoseconds while breaking the equals contract.

If you’re on a legacy codebase, convert at the boundary and never let them into your domain:

Instant i = oldDate.toInstant();
Date    d = Date.from(instant);

Part 2: The PostgreSQL Side

timestamptz does not store a time zone

This is the single biggest source of confusion, and it comes straight from the type name. TIMESTAMP WITH TIME ZONE is an 8-byte value. So is TIMESTAMP WITHOUT TIME ZONE. There is no room in there for a zone, and PostgreSQL doesn’t try:

SELECT pg_column_size('2026-08-15 12:00:00'::timestamp)     AS ts_bytes,     
       --  8
       pg_column_size('2026-08-15 12:00:00+02'::timestamptz) AS tstz_bytes,  
       --  8
       pg_column_size('12:00:00+02'::timetz)                 AS timetz_bytes; 
       -- 12

What timestamptz actually does is convert to UTC on write and convert back on read. The offset you supplied on input is used and then discarded. From the official docs:

In either case, the value is stored internally as UTC, and the originally stated or assumed time zone is not retained.

A better mental model: timestamptz is Instant. timestamp is LocalDateTime. The names lie; the semantics don’t.

The demo that makes it click

Insert the same two literals into both column types, with the session zone set to London:

SET TimeZone = 'Europe/London';
CREATE TABLE t (id int, ts timestamp, tstz timestamptz);
INSERT INTO t VALUES (1, '2026-08-15 12:00:00',    '2026-08-15 12:00:00');
INSERT INTO t VALUES (2, '2026-08-15 12:00:00+00', '2026-08-15 12:00:00+00');
SELECT * FROM t;

 id |         ts          |          tstz
----+---------------------+------------------------
  1 | 2026-08-15 12:00:00 | 2026-08-15 12:00:00+01
  2 | 2026-08-15 12:00:00 | 2026-08-15 13:00:00+01

Two things happened. Row 1’s tstz had no offset, so PostgreSQL assumed the session zone. Row 2’s explicit +00 was honoured and re-rendered as 13:00+01. Meanwhile the ts column ignored the offset entirely and stored the digits.

Now reconnect from New York and run the identical SELECT:

 id |         ts          |          tstz
----+---------------------+------------------------
  1 | 2026-08-15 12:00:00 | 2026-08-15 07:00:00-04
  2 | 2026-08-15 12:00:00 | 2026-08-15 08:00:00-04

The tstz column re-renders per client. The ts column is frozen. Neither row changed on disk.

That last point deserves emphasis, because it’s where the panic starts: when a developer sees a different value than QA saw, the instinct is “the data is corrupted.” It isn’t. Two clients with different TimeZone settings are looking at the same UTC value through different windows.

AT TIME ZONE runs both ways

AT TIME ZONE is the one operator you need, and its behavior flips depending on the input type:

-- timestamptz -> timestamp: "what does the clock read there?"
SELECT '2026-08-15 12:00:00+00'::timestamptz AT TIME ZONE 'Asia/Tokyo';
-- 2026-08-15 21:00:00   (type: timestamp without time zone)

-- timestamp -> timestamptz: "this wall clock reading WAS in Tokyo"
SELECT '2026-08-15 12:00:00'::timestamp AT TIME ZONE 'Asia/Tokyo';
-- 2026-08-15 03:00:00+00   (type: timestamp with time zone)

One removes zone context, the other attaches it. Read it as “interpret this value in that zone.” The function form timezone(‘Asia/Tokyo’, value) is identical.

The rest of the family

• date — 4 bytes, no zone, no ambiguity. Great for things that are genuinely dates.
• time — time of day, no zone.
• timetz — 12 bytes, and PostgreSQL itself recommends against it. A time of day with an offset but no date can’t resolve DST, so it’s not meaningfully anchored. It won’t even accept a zone name:

SELECT '12:00:00 Europe/London'::timetz; -- ERROR: invalid input syntax for type time with time zone

• interval — 16 bytes storing months, days, and microseconds separately, which is exactly why it can be DST-aware.

now() vs localtimestamp

SELECT pg_typeof(now()),            -- timestamp with time zone
       pg_typeof(current_timestamp), -- timestamp with time zone
       pg_typeof(localtimestamp),    -- timestamp without time zone
       pg_typeof(current_date);      -- date

now() and current_timestamp are timestamptz and are safe. localtimestamp and current_date are evaluated in the session zone, which means a DEFAULT current_date column produces different answers for different clients. Be deliberate.

So which do you pick?

The short version: use timestamptz for essentially everything. Reach for timestamp only when you genuinely mean a floating wall-clock time that must not shift — “the store opens at 09:00 local, whatever local means at that branch” — and store the zone ID beside it.

Part 3: Time Zones, Properly

This is where the real complexity lives, so let’s slow down.

Three things called “time zone”

1. An offset — +01:00. A number.
2. An IANA zone ID — Europe/London. A named set of historical and future rules. PostgreSQL ships ~499 of these in pg_timezone_names; Java reads the same tzdata.
3. An abbreviation — EST, BST, IST. Ambiguous, unversioned, and a trap.

That third one is genuinely dangerous:

SELECT '2026-08-15 12:00:00 EST'::timestamptz,               -- 2026-08-15 17:00:00+00
       '2026-08-15 12:00:00 America/New_York'::timestamptz;  -- 2026-08-15 16:00:00+00

One hour apart. EST is a fixed -05:00 in PostgreSQL — it does not mean “Eastern time,” it means “Eastern Standard Time, always.” In August, New York is on -04:00. If you accept zone abbreviations from user input, you have already shipped a bug. (IST is worse — it can mean India, Ireland, or Israel depending on who you ask.)

Always use full IANA names.

Offsets aren’t zones either. Nepal is +05:45; the Chatham Islands are +12:45:

ZonedDateTime.of(2026,8,15,12,0,0,0, ZoneId.of("Asia/Kathmandu"))
// 2026-08-15T12:00+05:45[Asia/Kathmandu]

If your API contract says “offset in whole hours,” it’s wrong.

Where PostgreSQL’s session zone comes from

The TimeZone GUC is a per-connection setting. Its value is decided by, in order: an explicit SET TimeZone, the connection’s startup parameters, the postgresql.conf value, and finally the server’s OS zone.

That last fallback is why the same query returns different results on a developer laptop and in CI. And with connection pooling, SET TimeZone on a borrowed connection leaks to whoever gets it next.

Pin it. Set timezone = ‘UTC’ in postgresql.conf, and add it to the JDBC URL so it survives pool reuse:

jdbc:postgresql://host:5432/db?options=-c%20TimeZone%3DUTC

Then make the client responsible for rendering. Formatting a timestamp for a human is a presentation concern, not a database one.

The spring-forward gap

On 29 March 2026, London jumps from 01:00 to 02:00. The local times between them do not exist.

Ask Java for one anyway:

LocalDateTime gap = LocalDateTime.of(2026, 3, 29, 1, 30);
gap.atZone(ZoneId.of("Europe/London"));
// 2026-03-29T02:30+01:00[Europe/London]
ZoneId.of("Europe/London").getRules().getTransition(gap);
// Transition[Gap at 2026-03-29T01:00Z to +01:00]

Java doesn’t throw. It silently pushes the value forward by the length of the gap. PostgreSQL does the same thing:

SET TimeZone='Europe/London';
SELECT '2026-03-29 01:30:00'::timestamptz;
-- 2026-03-29 02:30:00+01

Both are “correct” — but neither told you. If a user sets a 01:30 alarm on that date, you should be validating with getTransition() and telling them, not letting the platform quietly reschedule.

The fall-back overlap — where Java and PostgreSQL disagree

On 25 October 2026, London goes from 02:00 back to 01:00. Local times between 01:00 and 02:00 happen twice, one hour apart.

Java’s rules expose both:

ZoneId london = ZoneId.of("Europe/London");
LocalDateTime overlap = LocalDateTime.of(2026, 10, 25, 1, 30);
london.getRules().getValidOffsets(overlap);
// [+01:00, Z]
ZonedDateTime earlier = overlap.atZone(london);
// 2026-10-25T01:30+01:00[Europe/London]  instant = 2026-10-25T00:30:00Z
ZonedDateTime later = earlier.withLaterOffsetAtOverlap();
// 2026-10-25T01:30Z[Europe/London]       instant = 2026-10-25T01:30:00Z

Java’s default for an ambiguous local time is the earlier offset — 00:30Z.

Now ask PostgreSQL the same question:

SELECT '2026-10-25 01:30:00'::timestamp AT TIME ZONE 'Europe/London';
-- 2026-10-25 01:30:00+00

PostgreSQL picked 01:30Z — the later one.

Same input, same tzdata, one hour apart. If your application resolves an ambiguous local time in Java and your reporting job resolves it in SQL, they will disagree for one hour, twice a year, in every DST zone.

There is no clever fix. The lesson is: never let a LocalDateTime cross the boundary and get resolved on the other side. Resolve to an Instant in exactly one place, with explicit overlap handling, and send the instant.

DST arithmetic: plusDays is not plusHours(24)

ZonedDateTime start = ZonedDateTime.of(2026, 3, 28, 12, 0, 0, 0, london);
// 2026-03-28T12:00Z[Europe/London]
start.plusDays(1);                 // 2026-03-29T12:00+01:00  (23 real hours)
start.plus(Duration.ofDays(1));    // 2026-03-29T13:00+01:00  (24 real hours)
start.plusHours(24);               // 2026-03-29T13:00+01:00

plusDays preserves the wall clock. Duration preserves elapsed time. PostgreSQL draws the same line, using the same value:

SET TimeZone='Europe/London';
SELECT '2026-03-28 12:00:00+00'::timestamptz + interval '1 day',   -- 2026-03-29 12:00:00+01
       '2026-03-28 12:00:00+00'::timestamptz + interval '24 hours'; -- 2026-03-29 13:00:00+01

This is one place the two platforms agree beautifully — interval ‘1 day’ and plusDays(1) mean the same thing, and interval ‘24 hours’ and Duration.ofDays(1) mean the same thing. “Every day at 9am” is plusDays / interval ‘1 day’. “Every 24 hours” is Duration / interval ‘24 hours’. Pick the one the product actually means.

Future events: an instant is not enough

ZoneId ny = ZoneId.of("America/New_York");
ZonedDateTime.of(2027, 1, 15, 9, 0, 0, 0, ny).withZoneSameInstant(UTC); // 2027-01-15T14:00Z
ZonedDateTime.of(2027, 7, 15, 9, 0, 0, 0, ny).withZoneSameInstant(UTC); // 2027-07-15T13:00Z

“9am in New York” is a different UTC instant depending on the date. So if a user schedules a recurring 9am standup and a government later changes DST rules — which happens several times a year somewhere — the stored instant becomes wrong.

For past events, store the instant. That’s history; it can’t change.

For future user-scheduled events, store three things:

CREATE TABLE meetings (
  id            bigserial PRIMARY KEY,
  local_start   timestamp   NOT NULL,  -- 2027-01-15 09:00
  zone_id       text        NOT NULL,  -- 'America/New_York'
  computed_utc  timestamptz NOT NULL   -- cache; recompute on tzdata update
);

The first two are the truth. The third is a derived index for querying, refreshed when tzdata changes.

tzdata is a moving target

Java bundles a copy of the IANA database (2025b on the JDK I ran this on); PostgreSQL bundles its own; your OS has a third. They drift. When Java and PostgreSQL are on different tzdata versions, they will resolve recently-changed zones differently.

Check what you have:

ZoneRulesProvider.getVersions("Europe/London").lastKey();  // 2025b

SELECT name, utc_offset, is_dst FROM pg_timezone_names WHERE name = 'Europe/London';

Pin tzdata in your base images, and treat a tzdata bump as a deployment that requires recomputing cached future instants.

Part 4: Wiring Java to PostgreSQL

The mapping table

Per the pgJDBC documentation:

Note the omission, in the driver’s own words:

ZonedDateTime, Instant and OffsetTime / TIME WITH TIME ZONE are not supported.

So don’t hand the driver an Instant and hope — depending on the driver version you’ll either get a Can’t infer the SQL type to use for an instance of java.time.Instant exception or undocumented behavior. Convert explicitly at the boundary instead:

// write
ps.setObject(1, instant.atOffset(ZoneOffset.UTC));
// read
Instant i = rs.getObject("occurred_at", OffsetDateTime.class).toInstant();

Note that reading a timestamptz gives you an OffsetDateTime normalized to UTC — the driver’s offset is always Z, because that’s what the backend holds. The offset carries no information. Convert straight to Instant and don’t be tempted to read meaning into it.

Avoid setTimestamp / getTimestamp

The java.sql.Timestamp overloads convert using the JVM default time zone unless you pass an explicit Calendar. That makes your persistence layer’s correctness depend on the container’s TZ environment variable. Use the java.time setObject/getObject methods instead — they’re unambiguous.

If you must keep the JVM zone from mattering at all, set it explicitly at startup:

-Duser.timezone=UTC

Belt and braces, not a substitute for correct types.

Precision: Java has nanos, PostgreSQL has micros

Instant.parse("2026-08-15T10:00:00.123456789Z")           // .123456789
    .truncatedTo(ChronoUnit.MICROS)                        // .123456

PostgreSQL stores microseconds. Those last three digits are gone on the round trip, and PostgreSQL may round rather than truncate:

SELECT '2026-08-15 10:00:00.123456789'::timestamptz;  -- 10:00:00.123457

If you have an equality test or an optimistic-lock check comparing an in-memory Instant to one that’s been through the database, truncate to micros before you persist so both sides agree.

Don’t wrap the column in a function

Because timestamptz is stored as UTC, “give me everything that happened on 1 Feb in New York” tempts you into this:

EXPLAIN SELECT count(*) FROM events
WHERE (occurred_at AT TIME ZONE 'America/New_York')::date = DATE '2026-02-01';
--  Seq Scan on events

Index gone. Compute the boundary instants instead and do a half-open range scan:

SELECT (DATE '2026-02-01' + time '00:00') AT TIME ZONE 'America/New_York',  -- 2026-02-01 05:00+00
       (DATE '2026-02-02' + time '00:00') AT TIME ZONE 'America/New_York';  -- 2026-02-02 05:00+00
EXPLAIN SELECT count(*) FROM events
WHERE occurred_at >= '2026-02-01 05:00:00+00'
  AND occurred_at <  '2026-02-02 05:00:00+00';
--  Index Only Scan using events_occurred_at_idx

Note that this “day” is 24 hours only because February has no DST transition. On a transition day it’s 23 or 25 — which the boundary-instant approach handles correctly and a naive + interval ‘24 hours’ does not. Half-open ranges (>= and <) also dodge the microsecond-boundary problem that BETWEEN creates.

Part 5: JPA and Hibernate

Hibernate 6 made zoned types a first-class concern via TimeZoneStorageType:

For PostgreSQL, the practical consequence is settled by the database, not the ORM: timestamptz discards the offset, so under DEFAULT your OffsetDateTime round-trips as UTC. Which is fine — as long as you expected it.

Sensible defaults:

spring:
  jpa:
    properties:
      hibernate:
        jdbc:
          time_zone: UTC
        timezone:
          default_storage: NORMALIZE_UTC

And in the entity, be explicit about intent:

@Entity
public class Booking {
    
    // A moment. Maps to timestamptz.
    @Column(nullable = false)
    private Instant createdAt;
    
    // A date. Not a moment.
    @Column(nullable = false)
    private LocalDate serviceDate;
    
    // A wall-clock time that must not shift. Maps to timestamp.
    @Column(nullable = false)
    private LocalDateTime localStart;
    
    @Column(nullable = false)
    private String zoneId;   // 'Europe/London'
    
    public ZonedDateTime start() {
        return localStart.atZone(ZoneId.of(zoneId));
    }
}

If you genuinely need the user’s original offset preserved, @TimeZoneStorage(TimeZoneStorageType.COLUMN) with @TimeZoneColumn writes it to a companion column. Use it when the offset is data — an audit trail of what the user’s device claimed — not as a default.

Part 6: Testing This Stuff

Datetime bugs are only cheap if you catch them before the clocks change. Three habits do most of the work.

1. Inject a Clock

Instant.now() scattered through business logic is untestable. Inject java.time.Clock and you can move time at will:

@Bean Clock clock() { return Clock.systemUTC(); }
// in a test
Clock fixed = Clock.fixed(Instant.parse("2026-10-25T00:30:00Z"), ZoneOffset.UTC);

2. Run CI in a hostile time zone

If your JVM, your database, and your CI runner are all UTC, every zone bug hides until production. Force one of them somewhere awkward:

-Duser.timezone=Pacific/Chatham

+12:45 and it observes DST. If your test suite survives Chatham, it will survive most things. Do the same for the PostgreSQL container:

@Container
static PostgreSQLContainer<?> pg = new PostgreSQLContainer<>("postgres:16")
        .withEnv("TZ", "Asia/Kathmandu")
        .withUrlParam("options", "-c TimeZone=Asia/Kathmandu");

The point isn’t that production runs there — it’s that a test which only passes in UTC is a test that has proven nothing about zone handling.

3. Write the four transition tests

For any code that converts between local and instant, pin these explicitly:

static final ZoneId LONDON = ZoneId.of("Europe/London");
@Test void gapTimesAreRejectedNotSilentlyShifted() {
    LocalDateTime gap = LocalDateTime.of(2026, 3, 29, 1, 30);
    assertThat(LONDON.getRules().getValidOffsets(gap)).isEmpty();
    assertThatThrownBy(() -> scheduler.schedule(gap, LONDON))
        .isInstanceOf(NonExistentLocalTimeException.class);
}
@Test void overlapTimesResolveToTheDocumentedChoice() {
    LocalDateTime overlap = LocalDateTime.of(2026, 10, 25, 1, 30);
    assertThat(LONDON.getRules().getValidOffsets(overlap)).hasSize(2);
    assertThat(scheduler.resolve(overlap, LONDON))
        .isEqualTo(Instant.parse("2026-10-25T00:30:00Z")); // earlier, by policy
}
@Test void dailyRecurrenceKeepsWallClockAcrossDst() {
    ZonedDateTime start = ZonedDateTime.of(2026, 3, 28, 9, 0, 0, 0, LONDON);
    assertThat(recurrence.next(start).toLocalTime()).isEqualTo(LocalTime.of(9, 0));
}
@Test void instantSurvivesTheDatabaseRoundTrip() {
    Instant original = Instant.now().truncatedTo(ChronoUnit.MICROS);
    repository.save(new Event(original));
    assertThat(repository.findLatest().occurredAt()).isEqualTo(original);
}

The fourth one catches more real bugs than you’d expect — it’s the one that fails when someone quietly changes a column from timestamptz to timestamp.

The Short Version

If you skim one section, make it this one.

Schema

• timestamptz by default. It’s the same 8 bytes and it’s unambiguous.
• timestamp only for wall-clock times that must not shift — and store the zone ID beside it.
• date for dates. timetz for nothing.
• Pin timezone = ‘UTC’ in postgresql.conf and in the JDBC URL.

Java

• Instant for things that happened. LocalDate for dates. ZonedDateTime for scheduling. OffsetDateTime at the JDBC boundary.
• LocalDateTime is not a point in time. Treat every use as a question to answer, not an answer.
• Never accept zone abbreviations. Full IANA names only.
• plusDays ≠ plus(Duration.ofDays(1)). Know which one your product means.

The seam

• pgJDBC speaks LocalDate/LocalTime/LocalDateTime/OffsetDateTime. Convert Instant explicitly.
• Avoid setTimestamp/getTimestamp — they use the JVM default zone.
• Truncate to microseconds before persisting.
• Don’t wrap indexed timestamp columns in AT TIME ZONE; compute boundary instants instead.

And the one that bites hardest

Java resolves an ambiguous local time to the earlier offset. PostgreSQL picks the later one. Resolve ambiguity in exactly one place, deliberately, and pass instants across the boundary — never local times.

Store the instant, render the local time, and keep the zone rules in exactly one layer. Most datetime bugs are just that rule, broken somewhere.

All examples verified on OpenJDK 21.0.10 (tzdata 2025b) and PostgreSQL 16.13.

Sources

• PostgreSQL: Date/Time Types
• pgJDBC: Using Java 8 Date and Time classes
• Hibernate TimeZoneStorageType Javadoc
• Hibernate 6 migration guide
• Thorben Janssen: TimezoneStorageType

When a migration hangs, a query slows to a crawl, or a deploy just sits there, most people reach for the same first move: kill it and try again. That works about as often as it doesn’t. Postgres already knows what’s wrong — it’s just keeping the answer in a handful of system catalogs and stats views that almost nobody looks at until things are already on fire.

This is a walkthrough of the tables I actually check, in the order I check them, when something breaks — with a running example: a migration that adds a NOT NULL column and backfills it on a 20-million-row orders table, which has been “running” for 25 minutes with no sign of finishing.

Start with: is it even running?

Before touching statistics or query plans, find out if the process is alive, blocked, or just slow.

SELECT pid, state, wait_event_type, wait_event, query, query_start
FROM pg_stat_activity
WHERE state != 'idle';

pg_stat_activity is a live snapshot of every connection to the database. The columns that matter most here are wait_event_type and wait_event. If a process shows wait_event_type = ‘Lock’, it isn’t broken — it’s queued behind something else. If it shows Client or IO, that’s a different problem entirely (waiting on the client, or waiting on disk).

For the orders migration, this comes back with:

pid  | state  | wait_event_type | wait_event | query
-----+--------+-----------------+------------+----------------------------------
8842 | active | Lock            | relation   | ALTER TABLE orders ADD COLUMN ...

wait_event_type = Lock tells us immediately: the migration isn’t slow, it’s stuck behind something else that holds a conflicting lock on orders.

If you see a lock wait, the next stop is obvious:

SELECT blocked_locks.pid AS blocked_pid,
       blocking_locks.pid AS blocking_pid,
       blocked_activity.query AS blocked_query,
       blocking_activity.query AS blocking_query
FROM pg_locks blocked_locks
JOIN pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
JOIN pg_locks blocking_locks
  ON blocking_locks.locktype = blocked_locks.locktype
  AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
  AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
  AND blocking_locks.pid != blocked_locks.pid
JOIN pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;

This join is the single most useful query in Postgres debugging. It tells you exactly which session is blocking which, and what each of them is running. Most “stuck migration” incidents end here: some other transaction — often a long-running read, an ORM that forgot to close a connection, or a previous deploy that never finished — is sitting on a lock the migration needs.

Running it for the orders case turns up the problem:

blocked_pid | blocking_pid | blocked_query                   | blocking_query
------------+--------------+---------------------------------+---------------------------------------------
 8842       | 6110         | ALTER TABLE orders ADD COLUMN.. | SELECT * FROM orders WHERE ... (idle in txn)

PID 6110 is an analytics job that opened a transaction, ran a read, and never committed — “idle in transaction” — leaving an ACCESS SHARE lock on orders that blocks the migration’s ACCESS EXCLUSIVE lock. The fix here isn’t in the migration at all: it’s killing or fixing PID 6110 (SELECT pg_terminate_backend(6110); if you’re sure it’s safe to end it).

If it’s a long-running operation, check its actual progress

Postgres exposes live progress for the operations that most commonly make migrations feel frozen:

SELECT * FROM pg_stat_progress_create_index;
SELECT * FROM pg_stat_progress_vacuum;
SELECT * FROM pg_stat_progress_analyze;

If you’re running CREATE INDEX CONCURRENTLY as part of a migration, pg_stat_progress_create_index shows you the current phase (building index, index validation, etc.) and how many tuples it has scanned versus the total. This turns “is this hung or just big?” from a guess into a fact.

Say the next step of the migration adds an index on the new column instead of blocking on a lock — the progress view might show:

pid  | phase                         | tuples_total | tuples_done
-----+-------------------------------+--------------+-------------
9021 | building index: scanning table| 20450112     | 14300556

That’s roughly 70% through a single scan phase, moving steadily — not stuck, just genuinely working through 20 million rows. Compare that against a snapshot taken a minute later: if tuples_done hasn’t moved, now you know it actually is stalled, and it’s worth going back to pg_locks.

If nothing’s blocked, suspect the query plan

Not every broken migration is a lock problem. Sometimes the migration runs, but a backfill step or an added index makes the planner choose a catastrophically bad plan. That’s a statistics problem, and it lives in two places.

pg_stat_user_tables tells you whether Postgres has been keeping its stats up to date on the table in question:

SELECT relname, n_live_tup, n_dead_tup, n_mod_since_analyze,
       last_vacuum, last_autovacuum, last_analyze, last_autoanalyze
FROM pg_stat_user_tables
WHERE relname = 'your_table';

If n_mod_since_analyze is large relative to n_live_tup, or last_autoanalyze is suspiciously old, the planner is working off outdated information. This happens constantly during migrations, because a migration that inserts, updates, or deletes a large chunk of a table changes its shape faster than autovacuum’s default thresholds react to.

For orders, after the backfill step runs:

relname | n_live_tup | n_dead_tup | n_mod_since_analyze | last_autoanalyze
--------+------------+------------+---------------------+-------------------
orders  | 20450112   | 3800000    | 18900000            | 2026-07-28 03:14

18.9 million modified rows since the last analyze, against 20 million live rows — nearly the entire table has changed and autovacuum hasn’t caught up yet (last_autoanalyze is five days old). Any query the migration runs after this point is planning against numbers that are effectively meaningless.

pg_stats (the readable view over the raw pg_statistic catalog) shows what the planner actually believes about each column:

SELECT attname, n_distinct, correlation, null_frac, most_common_vals
FROM pg_stats
WHERE tablename = 'your_table';

n_distinct and correlation drive index-vs-sequential-scan decisions. If a migration adds a new index and the planner still refuses to use it, stale stats here are the usual suspect.

On the new orders.status column right after the backfill, before an ANALYZE:

 attname | n_distinct | correlation | null_frac
---------+------------+-------------+-----------
 status  | 1          | 1           | 1.0

The planner still believes every row is null (null_frac = 1.0) because that was true before the backfill ran. Any query filtering on status will get a wildly wrong row estimate and likely pick a sequential scan or a bad join order. The fix is almost always to run ANALYZE your_table; — either before the migration, or as an explicit step inside it, rather than waiting for autovacuum to get around to it. After ANALYZE orders;, the same query returns something like n_distinct = 4, null_frac = 0.0 — matching reality, and the planner’s decisions follow accordingly.

pg_class holds the planner’s cached size estimates directly:

SELECT relname, reltuples, relpages
FROM pg_class
WHERE relname = 'your_table';

reltuples is an estimate, not a live count — it’s only refreshed by ANALYZE or VACUUM. If this number is wildly off from the real row count, every cost estimate downstream is wrong too. For orders, reltuples might read 20450112 while a SELECT count(*) returns closer to 24000000 — a four-million-row gap that skews every cost estimate the planner builds on top of it.

If you need the exact query and its cost, not just the table

pg_stat_statements (an extension, so it needs to be enabled) aggregates every distinct query the server has run, with real timing:

SELECT query, calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;

This is the difference between “the migration is slow” and “this specific UPDATE inside the migration is slow, and it’s been called 40,000 times.” For the orders backfill, this might show:

query                                        | calls | mean_exec_time | rows
---------------------------------------------+-------+----------------+--------
 UPDATE orders SET status = $1 WHERE id = $2 | 40000 | 812.4 ms       | 40000

40,000 single-row updates averaging 800ms each explains a lot on its own — that’s batching the backfill row-by-row instead of in bulk, and it would still be slow even with perfect statistics. If it’s enabled in your environment, it’s usually the fastest way to go from vague suspicion to a concrete line of SQL to fix — in this case, rewriting the backfill as a single set-based UPDATE … WHERE status IS NULL in batches, rather than one row at a time.

A rough order of operations

Put together, a sensible debugging sequence looks like this: check pg_stat_activity to see if the process is alive and what it’s waiting on; if it’s waiting on a lock, join into pg_locks to find the blocker; if it’s a long index build or vacuum, check the matching pg_stat_progress_* view for real progress; if nothing is blocked but it’s still slow, check pg_stat_user_tables and pg_stats for stale statistics, and pg_class for stale row estimates; and if pg_stat_statements is available, use it to find the exact query eating the time.

None of these tables are exotic. They ship with every Postgres install (aside from pg_stat_statements, which just needs CREATE EXTENSION). The only real skill here is knowing which one answers which question — and reaching for pg_stat_activity and pg_locks before assuming it’s a statistics problem, since lock contention is a far more common cause of a “broken” migration than a bad query plan.