<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Query-Optimization on Vitalii Sukach</title><link>https://sukachvitally.github.io/tags/query-optimization/</link><description>Recent content in Query-Optimization on Vitalii Sukach</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Sun, 02 Aug 2026 08:37:40 +0000</lastBuildDate><atom:link href="https://sukachvitally.github.io/tags/query-optimization/index.xml" rel="self" type="application/rss+xml"/><item><title>Debugging Postgres: The System Tables That Actually Tell You What’s Wrong</title><link>https://sukachvitally.github.io/posts/debugging-postgres-the-system-tables-that-actually-tell-you-whats-wrong/</link><pubDate>Sun, 02 Aug 2026 08:37:40 +0000</pubDate><guid>https://sukachvitally.github.io/posts/debugging-postgres-the-system-tables-that-actually-tell-you-whats-wrong/</guid><description>&lt;p>&lt;img loading="lazy" src="https://sukachvitally.github.io/posts/debugging-postgres-the-system-tables-that-actually-tell-you-whats-wrong/image-01.png">&lt;/p>
&lt;p>When a migration hangs, a query slows to a crawl, or a deploy just sits there, most people reach for the same first move: kill it and try again. That works about as often as it doesn’t. Postgres already knows what’s wrong — it’s just keeping the answer in a handful of system catalogs and stats views that almost nobody looks at until things are already on fire.&lt;/p>
&lt;p>This is a walkthrough of the tables I actually check, in the order I check them, when something breaks — with a running example: a migration that adds a NOT NULL column and backfills it on a 20-million-row orders table, which has been &amp;ldquo;running&amp;rdquo; for 25 minutes with no sign of finishing.&lt;/p></description><content:encoded><![CDATA[<p><img loading="lazy" src="https://sukachvitally.github.io/posts/debugging-postgres-the-system-tables-that-actually-tell-you-whats-wrong/image-01.png"></p>
<p>When a migration hangs, a query slows to a crawl, or a deploy just sits there, most people reach for the same first move: kill it and try again. That works about as often as it doesn’t. Postgres already knows what’s wrong — it’s just keeping the answer in a handful of system catalogs and stats views that almost nobody looks at until things are already on fire.</p>
<p>This is a walkthrough of the tables I actually check, in the order I check them, when something breaks — with a running example: a migration that adds a NOT NULL column and backfills it on a 20-million-row orders table, which has been &ldquo;running&rdquo; for 25 minutes with no sign of finishing.</p>
<h3 id="start-with-is-it-evenrunning">Start with: is it even running?</h3>
<p>Before touching statistics or query plans, find out if the process is alive, blocked, or just slow.</p>
<pre tabindex="0"><code>SELECT pid, state, wait_event_type, wait_event, query, query_start
FROM pg_stat_activity
WHERE state != &#39;idle&#39;;
</code></pre><p>pg_stat_activity is a live snapshot of every connection to the database. The columns that matter most here are wait_event_type and wait_event. If a process shows wait_event_type = &lsquo;Lock&rsquo;, it isn&rsquo;t broken — it&rsquo;s queued behind something else. If it shows Client or IO, that&rsquo;s a different problem entirely (waiting on the client, or waiting on disk).</p>
<p>For the orders migration, this comes back with:</p>
<pre tabindex="0"><code>pid  | state  | wait_event_type | wait_event | query
-----+--------+-----------------+------------+----------------------------------
8842 | active | Lock            | relation   | ALTER TABLE orders ADD COLUMN ...
</code></pre><p>wait_event_type = Lock tells us immediately: the migration isn&rsquo;t slow, it&rsquo;s stuck behind something else that holds a conflicting lock on orders.</p>
<p>If you see a lock wait, the next stop is obvious:</p>
<pre tabindex="0"><code>SELECT blocked_locks.pid AS blocked_pid,
       blocking_locks.pid AS blocking_pid,
       blocked_activity.query AS blocked_query,
       blocking_activity.query AS blocking_query
FROM pg_locks blocked_locks
JOIN pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
JOIN pg_locks blocking_locks
  ON blocking_locks.locktype = blocked_locks.locktype
  AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
  AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
  AND blocking_locks.pid != blocked_locks.pid
JOIN pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;
</code></pre><p>This join is the single most useful query in Postgres debugging. It tells you exactly which session is blocking which, and what each of them is running. Most “stuck migration” incidents end here: some other transaction — often a long-running read, an ORM that forgot to close a connection, or a previous deploy that never finished — is sitting on a lock the migration needs.</p>
<p>Running it for the orders case turns up the problem:</p>
<pre tabindex="0"><code>blocked_pid | blocking_pid | blocked_query                   | blocking_query
------------+--------------+---------------------------------+---------------------------------------------
 8842       | 6110         | ALTER TABLE orders ADD COLUMN.. | SELECT * FROM orders WHERE ... (idle in txn)
</code></pre><p>PID 6110 is an analytics job that opened a transaction, ran a read, and never committed — “idle in transaction” — leaving an ACCESS SHARE lock on orders that blocks the migration&rsquo;s ACCESS EXCLUSIVE lock. The fix here isn&rsquo;t in the migration at all: it&rsquo;s killing or fixing PID 6110 (SELECT pg_terminate_backend(6110); if you&rsquo;re sure it&rsquo;s safe to end it).</p>
<h3 id="if-its-a-long-running-operation-check-its-actualprogress">If it’s a long-running operation, check its actual progress</h3>
<p>Postgres exposes live progress for the operations that most commonly make migrations feel frozen:</p>
<pre tabindex="0"><code>SELECT * FROM pg_stat_progress_create_index;
SELECT * FROM pg_stat_progress_vacuum;
SELECT * FROM pg_stat_progress_analyze;
</code></pre><p>If you’re running CREATE INDEX CONCURRENTLY as part of a migration, pg_stat_progress_create_index shows you the current phase (building index, index validation, etc.) and how many tuples it has scanned versus the total. This turns &ldquo;is this hung or just big?&rdquo; from a guess into a fact.</p>
<p>Say the next step of the migration adds an index on the new column instead of blocking on a lock — the progress view might show:</p>
<pre tabindex="0"><code>pid  | phase                         | tuples_total | tuples_done
-----+-------------------------------+--------------+-------------
9021 | building index: scanning table| 20450112     | 14300556
</code></pre><p>That’s roughly 70% through a single scan phase, moving steadily — not stuck, just genuinely working through 20 million rows. Compare that against a snapshot taken a minute later: if tuples_done hasn&rsquo;t moved, now you know it actually is stalled, and it&rsquo;s worth going back to pg_locks.</p>
<h3 id="if-nothings-blocked-suspect-the-queryplan">If nothing’s blocked, suspect the query plan</h3>
<p>Not every broken migration is a lock problem. Sometimes the migration runs, but a backfill step or an added index makes the planner choose a catastrophically bad plan. That’s a statistics problem, and it lives in two places.</p>
<p><strong>pg_stat_user_tables</strong> tells you whether Postgres has been keeping its stats up to date on the table in question:</p>
<pre tabindex="0"><code>SELECT relname, n_live_tup, n_dead_tup, n_mod_since_analyze,
       last_vacuum, last_autovacuum, last_analyze, last_autoanalyze
FROM pg_stat_user_tables
WHERE relname = &#39;your_table&#39;;
</code></pre><p>If n_mod_since_analyze is large relative to n_live_tup, or last_autoanalyze is suspiciously old, the planner is working off outdated information. This happens constantly during migrations, because a migration that inserts, updates, or deletes a large chunk of a table changes its shape faster than autovacuum&rsquo;s default thresholds react to.</p>
<p>For orders, after the backfill step runs:</p>
<pre tabindex="0"><code>relname | n_live_tup | n_dead_tup | n_mod_since_analyze | last_autoanalyze
--------+------------+------------+---------------------+-------------------
orders  | 20450112   | 3800000    | 18900000            | 2026-07-28 03:14
</code></pre><p>18.9 million modified rows since the last analyze, against 20 million live rows — nearly the entire table has changed and autovacuum hasn’t caught up yet (last_autoanalyze is five days old). Any query the migration runs after this point is planning against numbers that are effectively meaningless.</p>
<p><strong>pg_stats</strong> (the readable view over the raw pg_statistic catalog) shows what the planner actually believes about each column:</p>
<pre tabindex="0"><code>SELECT attname, n_distinct, correlation, null_frac, most_common_vals
FROM pg_stats
WHERE tablename = &#39;your_table&#39;;
</code></pre><p>n_distinct and correlation drive index-vs-sequential-scan decisions. If a migration adds a new index and the planner still refuses to use it, stale stats here are the usual suspect.</p>
<p>On the new orders.status column right after the backfill, before an ANALYZE:</p>
<pre tabindex="0"><code> attname | n_distinct | correlation | null_frac
---------+------------+-------------+-----------
 status  | 1          | 1           | 1.0
</code></pre><p>The planner still believes every row is null (null_frac = 1.0) because that was true before the backfill ran. Any query filtering on status will get a wildly wrong row estimate and likely pick a sequential scan or a bad join order. The fix is almost always to run ANALYZE your_table; — either before the migration, or as an explicit step inside it, rather than waiting for autovacuum to get around to it. After ANALYZE orders;, the same query returns something like n_distinct = 4, null_frac = 0.0 — matching reality, and the planner&rsquo;s decisions follow accordingly.</p>
<p><strong>pg_class</strong> holds the planner&rsquo;s cached size estimates directly:</p>
<pre tabindex="0"><code>SELECT relname, reltuples, relpages
FROM pg_class
WHERE relname = &#39;your_table&#39;;
</code></pre><p>reltuples is an estimate, not a live count — it&rsquo;s only refreshed by ANALYZE or VACUUM. If this number is wildly off from the real row count, every cost estimate downstream is wrong too. For orders, reltuples might read 20450112 while a SELECT count(*) returns closer to 24000000 — a four-million-row gap that skews every cost estimate the planner builds on top of it.</p>
<h3 id="if-you-need-the-exact-query-and-its-cost-not-just-thetable">If you need the exact query and its cost, not just the table</h3>
<p>pg_stat_statements (an extension, so it needs to be enabled) aggregates every distinct query the server has run, with real timing:</p>
<pre tabindex="0"><code>SELECT query, calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;
</code></pre><p>This is the difference between “the migration is slow” and “this specific UPDATE inside the migration is slow, and it’s been called 40,000 times.” For the orders backfill, this might show:</p>
<pre tabindex="0"><code>query                                        | calls | mean_exec_time | rows
---------------------------------------------+-------+----------------+--------
 UPDATE orders SET status = $1 WHERE id = $2 | 40000 | 812.4 ms       | 40000
</code></pre><p>40,000 single-row updates averaging 800ms each explains a lot on its own — that’s batching the backfill row-by-row instead of in bulk, and it would still be slow even with perfect statistics. If it’s enabled in your environment, it’s usually the fastest way to go from vague suspicion to a concrete line of SQL to fix — in this case, rewriting the backfill as a single set-based UPDATE &hellip; WHERE status IS NULL in batches, rather than one row at a time.</p>
<h3 id="a-rough-order-of-operations">A rough order of operations</h3>
<p>Put together, a sensible debugging sequence looks like this: check pg_stat_activity to see if the process is alive and what it&rsquo;s waiting on; if it&rsquo;s waiting on a lock, join into pg_locks to find the blocker; if it&rsquo;s a long index build or vacuum, check the matching pg_stat_progress_* view for real progress; if nothing is blocked but it&rsquo;s still slow, check pg_stat_user_tables and pg_stats for stale statistics, and pg_class for stale row estimates; and if pg_stat_statements is available, use it to find the exact query eating the time.</p>
<p>None of these tables are exotic. They ship with every Postgres install (aside from pg_stat_statements, which just needs CREATE EXTENSION). The only real skill here is knowing which one answers which question — and reaching for pg_stat_activity and pg_locks before assuming it&rsquo;s a statistics problem, since lock contention is a far more common cause of a &ldquo;broken&rdquo; migration than a bad query plan.</p>
]]></content:encoded></item></channel></rss>