[{"content":"Chunks, vectors, a bit of geometry, and no magic. Everything runs on your laptop.\nAsk an LLM “When does the nightly settlement batch run at Harbourline Payments?” and one of two things happens.\nThe polite model says it doesn’t know. The confident model says “Usually around midnight!” with the energy of a student who didn’t read the book but really wants a good mark.\nBoth are fair, because Harbourline Payments doesn’t exist. I made it up. It’s a fictional company with a fictional engineering handbook: founders, services, deploy rules, on-call pay, and even a famous incident called “the Double Tap”. No model has seen this text during training.\nThat’s exactly why it’s a good test. If the model answers “02:30 UK time” correctly, it didn’t remember it. It found it. That’s RAG.\nIn this article we’ll build a small RAG service in Java with Spring Boot, LangChain4j, PostgreSQL with pgvector, and Ollama. No cloud, no API keys, no bill at the end of the month.\nThe full code is on GitHub: SukachVitally/rag-demo.\nWhat is RAG, in one exam RAG stands for Retrieval-Augmented Generation. Long name, simple idea.\nThink about two exams:\nClosed-book exam. The student answers from memory. This is a normal LLM. It knows a lot, but only what it learned during training. It knows nothing about your company’s wiki, last week’s incident, or your internal rules. If it doesn’t know, it may still try to sound smart. Open-book exam. Before answering, the student finds the right pages in the book and reads them. This is RAG. So RAG doesn’t make the model smarter. It gives the model a cheat sheet for each question: “Here are 5 pieces of text that look relevant. Answer using only these.”\nWhy not just put the whole handbook into the prompt? For one small file you can. But real knowledge bases have thousands of documents. They don’t fit into the context window, and even when they do, you pay for every token on every question. RAG sends only the few pieces that matter.\nThe big idea The whole thing is two flows.\nIngest (once per document):\nTake the text and split it into small pieces called chunks. Ask an embedding model to turn each chunk into a vector: a long list of numbers. Save each chunk together with its vector in the database. Ask (every question):\nAsk the same embedding model to turn the question into a vector. Find the chunks whose vectors are closest to the question vector. Send the question and those chunks to a chat model and ask it to answer using only these chunks. That’s it. The rest of this article is details.\nNotice that there are two different models:\nThe embedding model is small and fast. The chat model is the big one. Both run locally in Ollama.\nWait, what is this vector? This is the most interesting part, so let’s slow down a little.\nAn embedding model reads a piece of text and returns a fixed-size array of numbers. nomic-embed-text returns 768 numbers for any text, whether it\u0026rsquo;s one word or a full paragraph:\n\u0026#34;settlement batch runs at 02:30\u0026#34; → [-0.064, 0.142, 0.009, ... 765 more ...] The model is trained so that texts with similar meaning get vectors that point in a similar direction. “When does the nightly batch run?” and “settlement-service runs the nightly batch at 02:30” share almost no grammar, but their meaning is close, so their vectors are close too.\nNow the fun part. It’s natural to think that each number means something: dimension 17 is “money”, dimension 302 is “time of day”, dimension 555 is “how angry the on-call engineer is”. It doesn’t work like that.\nNo single dimension has a physical meaning. Nobody designed these axes. The model learned the layout by itself during training. Meaning is spread across all 768 numbers at once, a bit like a smell is spread across many different molecules. You can’t point at one number and say “this is the time part”. Only the vector as a whole means something, and only when you compare it with other vectors.\nHere’s a 2D sketch of what that looks like. The real space has 768 dimensions, and I have only two axes on my screen:\nThe question vector points almost in the same direction as the “settlement batch runs at 02:30” chunk. The sourdough recipe chunk points somewhere else entirely. So for a computer, “similar meaning” becomes simple geometry: a small angle between two vectors.\nThe usual way to measure this is cosine similarity, the cosine of that angle:\n1: same direction, very similar 0: unrelated -1: opposite direction LangChain4j turns it into a friendlier score from 0 to 1: (1 + cos θ) / 2. That\u0026rsquo;s the score you\u0026rsquo;ll see in the API response later.\nOne important rule: vectors from different embedding models are not compatible. Each model builds its own space with its own layout. If you change the embedding model, you have to re-embed all your documents. It’s like changing the map projection: the old coordinates don’t match the new map.\nThe stack Java 25 and Spring Boot 4.1: the usual REST app. LangChain4j 1.22: the Java library that does the AI work. Ollama: runs both models locally. PostgreSQL 17 + pgvector: stores chunks and vectors, and searches by similarity. Testcontainers: for functional tests with a real database. Run the infrastructure with one command:\nSource: docker-compose.yaml\nservices: db: image: pgvector/pgvector:pg17 environment: POSTGRES_DB: rag POSTGRES_USER: rag POSTGRES_PASSWORD: rag ports: [\u0026#34;5432:5432\u0026#34;] ollama: image: ollama/ollama ports: [\u0026#34;11434:11434\u0026#34;] volumes: - ollama-models:/root/.ollama # keeps downloaded models between restarts ollama-init: image: ollama/ollama depends_on: [ollama] environment: OLLAMA_HOST: ollama:11434 entrypoint: [\u0026#34;sh\u0026#34;, \u0026#34;-c\u0026#34;, \u0026#34;sleep 5 \u0026amp;\u0026amp; ollama pull nomic-embed-text \u0026amp;\u0026amp; ollama pull llama3.1:8b\u0026#34;] volumes: ollama-models: The ollama-init container is a one-off job that downloads the models. The first time it pulls a few GB, so it\u0026rsquo;s a good moment to make some tea.\nLangChain4j: AI LEGO for Java LangChain4j is to AI roughly what Spring Data is to databases. It gives you small interfaces with many implementations, so your code doesn’t depend on a specific vendor.\nThe demo uses four building blocks, plus one helper that glues three of them together:\nDocumentSplitter cuts text into chunks. EmbeddingModel turns text into a vector. Here it\u0026rsquo;s OllamaEmbeddingModel. EmbeddingStore\u0026lt;TextSegment\u0026gt; saves vectors and finds the nearest ones. Here it\u0026rsquo;s PgVectorEmbeddingStore. ChatModel sends a prompt and gets an answer. Here it\u0026rsquo;s OllamaChatModel. EmbeddingStoreIngestor runs split → embed → store in one call. The nice part is the bottom of that picture. Want OpenAI instead of Ollama? Change one bean. Want Qdrant instead of Postgres? Change one bean. The services don’t care.\nDependencies Source: build.gradle\ndependencies { implementation \u0026#39;org.springframework.boot:spring-boot-starter-jdbc\u0026#39; implementation \u0026#39;org.springframework.boot:spring-boot-starter-webmvc\u0026#39; implementation \u0026#39;dev.langchain4j:langchain4j:1.22.0\u0026#39; implementation \u0026#39;dev.langchain4j:langchain4j-ollama:1.22.0\u0026#39; implementation \u0026#39;dev.langchain4j:langchain4j-pgvector:1.22.0-beta32\u0026#39; runtimeOnly \u0026#39;org.postgresql:postgresql\u0026#39; } Three LangChain4j modules: the core, the Ollama integration and the pgvector store. That’s all.\nWiring the beans Source: LangChain4jConfig.java\n@Configuration public class LangChain4jConfig { public static final String CHUNK_TABLE = \u0026#34;chunk_embedding\u0026#34;; @Bean EmbeddingModel embeddingModel(OllamaClientProperty property) { return OllamaEmbeddingModel.builder() .baseUrl(property.baseUrl()) .modelName(property.embedModel()) .build(); } @Bean ChatModel chatModel(OllamaClientProperty property) { return OllamaChatModel.builder() .baseUrl(property.baseUrl()) .modelName(property.chatModel()) .build(); } @Bean DocumentSplitter documentSplitter(ChunkProperty property) { return DocumentSplitters.recursive(property.size(), property.overlap()); } @Bean EmbeddingStore\u0026lt;TextSegment\u0026gt; embeddingStore(DataSource dataSource) { return PgVectorEmbeddingStore.datasourceBuilder() .datasource(dataSource) .table(CHUNK_TABLE) .createTable(false) .skipCreateVectorExtension(true) .metadataStorageConfig(DefaultMetadataStorageConfig.builder() .storageMode(MetadataStorageMode.COMBINED_JSONB) .columnDefinitions(List.of(\u0026#34;metadata JSONB NULL\u0026#34;)) .build()) .build(); } } The embedding store reuses Spring’s DataSource, so there\u0026rsquo;s one connection pool and one config. I also told it not to create the table (createTable(false)) because I prefer to own the schema myself:\nSource: schema.sql\nCREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE IF NOT EXISTS chunk_embedding ( embedding_id UUID PRIMARY KEY, embedding vector(768) NOT NULL, text TEXT NULL, metadata JSONB NULL ); CREATE INDEX IF NOT EXISTS chunk_embedding_hnsw ON chunk_embedding USING hnsw (embedding vector_cosine_ops); Two things to notice:\nvector(768) is the pgvector column type. 768 must match the embedding model\u0026rsquo;s output. If you switch to a model with 1024 dimensions, this column has to change too. The HNSW index with vector_cosine_ops makes nearest-neighbour search fast. Without it, Postgres compares the question with every row. That\u0026rsquo;s fine for 9 chunks, but not for 9 million. HNSW is an approximate search: it trades a tiny bit of accuracy for a lot of speed. Everything is configurable in application.yaml:\napp: olama-client: base-url: http://localhost:11434 embed-model: nomic-embed-text chat-model: llama3.1:8b chunk: size: 1000 overlap: 150 Part 1: Ingest, or feeding the database Here’s what happens to a document on the way in:\nWhy chunks? Why not embed the whole document as one vector? Because one vector for a whole handbook would be an average of everything: founders, deploy rules, on-call pay, the sourdough recipe someone accidentally uploaded. It would be “kind of about everything”, and that means it’s close to nothing.\nSmall chunks keep the meaning sharp. One chunk is about the settlement batch, another is about deploy windows. Now the search can find the right one.\nThe demo uses DocumentSplitters.recursive(1000, 150):\n1000: the maximum chunk size in characters. 150: overlap. Each chunk repeats about 150 characters from the end of the previous one. So if an important sentence lands on a border, it’s still complete in at least one chunk. Recursive means it tries to cut nicely. First by paragraphs. If a paragraph is too big, then by lines, then by sentences, then by words. It cuts the text like a polite butcher, not with a chainsaw. The ingest service Source: IngestService.java\n@Service @RequiredArgsConstructor public class IngestService { public static final String SOURCE = \u0026#34;source\u0026#34;; private final DocumentSplitter splitter; private final EmbeddingModel embeddingModel; private final EmbeddingStore\u0026lt;TextSegment\u0026gt; store; public int ingest(String source, String text) { if (text.isBlank()) return 0; var chunks = new AtomicInteger(); EmbeddingStoreIngestor.builder() .documentSplitter(splitter) .textSegmentTransformer(segment -\u0026gt; { chunks.incrementAndGet(); return segment; }) .embeddingModel(embeddingModel) .embeddingStore(store) .build() .ingest(Document.from(text, Metadata.from(SOURCE, source))); return chunks.get(); } } EmbeddingStoreIngestor does all three steps for us: split, embed, store. The textSegmentTransformer is a hook that runs for every chunk. Here I only use it to count chunks for the API response, but it\u0026rsquo;s also the right place to clean text or add extra metadata.\nThe source metadata (the file name) travels with every chunk into the metadata JSONB column. Later it lets the answer say where the information came from.\nThe controller is a plain multipart upload:\nSource: RagController.java\n@PostMapping(value = \u0026#34;/ingest\u0026#34;, consumes = MediaType.MULTIPART_FORM_DATA_VALUE) public Map\u0026lt;String, Object\u0026gt; upload(@RequestParam MultipartFile file) throws IOException { String text = new String(file.getBytes(), StandardCharsets.UTF_8); return Map.of(\u0026#34;chunks\u0026#34;, ingest.ingest(file.getOriginalFilename(), text)); } Let’s feed it the Harbourline handbook:\ncurl -F file=@docs/notes.md http://localhost:8080/api/ingest { \u0026#34;chunks\u0026#34;: 9 } Nine chunks, nine vectors, nine rows. The database now knows about a company that doesn’t exist.\nPart 2: Ask, or the open-book exam Source: RagService.java\n@Service @RequiredArgsConstructor public class RagService { private final EmbeddingModel embeddingModel; private final EmbeddingStore\u0026lt;TextSegment\u0026gt; store; private final ChatModel chatModel; private static final String SYSTEM = \u0026#34;\u0026#34;\u0026#34; Answer using ONLY the provided context. If the answer is not in the context, say you don\u0026#39;t know. Cite sources as [1], [2]...\u0026#34;\u0026#34;\u0026#34;; public record Answer(String answer, List\u0026lt;Hit\u0026gt; sources) {} public Answer ask(String question) { var q = embeddingModel.embed(question).content(); var hits = store.search(EmbeddingSearchRequest.builder().queryEmbedding(q).maxResults(5).build()) .matches().stream() .map(m -\u0026gt; new Hit(m.embedded().metadata().getString(IngestService.SOURCE), m.embedded().text(), m.score())) .toList(); var ctx = new StringBuilder(); for (int i = 0; i \u0026lt; hits.size(); i++) ctx.append(\u0026#34;[\u0026#34;).append(i + 1).append(\u0026#34;] (\u0026#34;).append(hits.get(i).source()).append(\u0026#34;)\\n\u0026#34;) .append(hits.get(i).content()).append(\u0026#34;\\n\\n\u0026#34;); String prompt = \u0026#34;Context:\\n\u0026#34; + ctx + \u0026#34;\\nQuestion: \u0026#34; + question; return new Answer(chatModel.chat(SystemMessage.from(SYSTEM), UserMessage.from(prompt)).aiMessage().text(), hits); } } This is the heart of the project, and it follows the plan exactly:\nEmbed the question with the same model we used for the chunks. Same model means same space, so the distances make sense. Search the store for the 5 nearest vectors. Under the hood, pgvector uses the HNSW index and cosine distance. Build the context: number each chunk and add its source. Ask the chat model with a strict system message: use only the context, say “I don’t know” if the answer isn’t there, and cite sources. This is roughly what llama actually receives:\nSYSTEM: Answer using ONLY the provided context. If the answer is not in the context, say you don\u0026#39;t know. Cite sources as [1], [2]... USER: Context: [1] (notes.md) ... settlement-service — runs the nightly settlement batch at 02:30 UK time ... [2] (notes.md) ... Question: When does the nightly settlement batch run? The chat model never sees the database, the vectors or the other chunks. It sees a small, focused cheat sheet and a question. Its job is now easy: read and summarise.\nLet’s try it:\ncurl -H \u0026#39;Content-Type: application/json\u0026#39; \\ -d \u0026#39;{\u0026#34;question\u0026#34;: \u0026#34;When does the nightly settlement batch run?\u0026#34;}\u0026#39; \\ http://localhost:8080/api/ask { \u0026#34;answer\u0026#34;: \u0026#34;According to the context in [3] (notes.md), the nightly settlement batch runs at 02:30 UK time.\u0026#34;, \u0026#34;sources\u0026#34;: [ { \u0026#34;source\u0026#34;: \u0026#34;notes.md\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;...\u0026#34;, \u0026#34;score\u0026#34;: 0.79 } ] } 02:30 UK time. Correct, and with a source. Open-book exam passed. 🎓\nThe sources list matters as much as the answer. You can show it to users, so they can check where the answer came from instead of trusting the model blindly. Trust, but verify, especially with an 8B model.\nThings to keep in mind The demo is small on purpose. Before you take it to production, think about these:\nChunk size is a trade-off. Too small, and a chunk loses context (“it runs at 02:30”, but what is “it”?). Too big, and the meaning gets blurry again. 1000/150 is a reasonable start, not a law. The top 5 are always returned, even when nothing is relevant. Ask about pizza, and you’ll still get the 5 “least irrelevant” chunks. The strict system prompt helps the model say “I don’t know”. In a real system I’d also add a minimum score (EmbeddingSearchRequest supports minScore). Changing the embedding model means re-ingesting everything. Different model, different space, different vector(N) column. Re-uploading the same file creates duplicates. The demo only appends. A real system needs to update or delete chunks by source. Garbage in, garbage out. RAG is only as good as the documents you give it. If the wiki is wrong, the answer will be wrong, but now with a citation. Summary RAG sounds like a fancy AI term, but in code it’s a short pipeline:\nIngest: split text into chunks → embed each chunk into a vector → store the text and vector in pgvector. Ask: embed the question → find the closest chunk vectors → give the question and those chunks to the chat model → answer with sources. The vectors are the only “magic” part. They are 768 numbers with no human meaning in any single dimension, but their direction captures meaning. Similar text means a small angle, and a small angle is something a database can search.\nLangChain4j turns each step into one small interface, so the whole service is a handful of Spring beans. You can swap Ollama for OpenAI, or pgvector for Qdrant, without touching the business code.\nAnd now a local 8B model knows when Harbourline’s settlement batch runs. Harbourline still doesn’t exist, but at least its batch is on time. ⏰\nThe full code is on GitHub: SukachVitally/rag-demo. Clone it, run docker compose up -d, upload your own notes and ask them questions.\n","permalink":"https://sukachvitally.github.io/posts/rag-in-java-with-langchain4j-teaching-a-local-llm-about-a-company-that-doesnt-exist/","summary":"\u003cp\u003e\u003cem\u003eChunks, vectors, a bit of geometry, and no magic. Everything runs on your laptop.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/rag-in-java-with-langchain4j-teaching-a-local-llm-about-a-company-that-doesnt-exist/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eAsk an LLM “When does the nightly settlement batch run at Harbourline Payments?” and one of two things happens.\u003c/p\u003e\n\u003cp\u003eThe polite model says it doesn’t know. The confident model says “Usually around midnight!” with the energy of a student who didn’t read the book but really wants a good mark.\u003c/p\u003e\n\u003cp\u003eBoth are fair, because Harbourline Payments doesn’t exist. I made it up. It’s a fictional company with a fictional engineering handbook: founders, services, deploy rules, on-call pay, and even a famous incident called “the Double Tap”. No model has seen this text during training.\u003c/p\u003e","title":"RAG in Java with LangChain4j: Teaching a Local LLM About a Company That Doesn’t Exist"},{"content":"Four annotated methods, one security config, and no JSON-RPC code of your own.\nIn my previous article I built an MCP server from scratch in Python. I wrote the JSON-RPC 2.0 message handling, declared tool schemas by hand and wired up the STDIO transport myself. It’s a good way to learn how the protocol works on the wire.\nPython isn’t the only option, though. Most of the systems I work on are Java and Spring, and that’s where the data lives. So this time I built the same kind of server in Java.\nIt turns out you don’t need to implement JSON-RPC yourself. Spring AI has a Boot starter for MCP servers. It handles the protocol, the transport, schema generation and tool registration. You write ordinary Spring beans and add a few annotations.\nThe full code is on GitHub: SukachVitally/mcp-demo.\nWhy bother: the server is the boundary The business logic here is deliberately boring. It’s a task list: you create tasks, move them through statuses, and finish or cancel them. Swap in your own domain; the MCP part stays the same.\nThe point is how Claude Code reaches the data:\nClaude Code ──HTTP + API key──▶ MCP server (Spring) ──JDBC──▶ PostgreSQL Claude never sees database credentials. The server holds the connection. Claude holds only an API key that works for the MCP endpoint and nothing else. Claude can only do what you expose. There’s no SQL console. If you don’t write a delete_task tool, tasks can\u0026rsquo;t be deleted, however the prompt is phrased. Your validation and transactions still apply. Every call goes through the same service layer as the rest of your application. Giving an agent a database URL and a password is quick, but you can’t take it back. An MCP server gives the agent a small, typed and auditable API instead.\nWhat we’re building A PostgreSQL table of tasks. Each task has a name and one of four statuses: NEW, IN_PROGRESS, DONE or CANCELLED. The server exposes four tools:\nStack: Java 25, Spring Boot 4, Spring AI 2.0 MCP server, PostgreSQL 18 with Liquibase, and Spring Security.\nStep 1: Dependencies Source: build.gradle\ndependencies { implementation \u0026#39;org.springframework.boot:spring-boot-starter-data-jpa\u0026#39; implementation \u0026#39;org.springframework.boot:spring-boot-starter-liquibase\u0026#39; implementation \u0026#39;org.springframework.boot:spring-boot-starter-webmvc\u0026#39; implementation \u0026#39;org.springframework.boot:spring-boot-starter-security\u0026#39; implementation \u0026#39;org.springframework.boot:spring-boot-starter-validation\u0026#39; implementation platform(\u0026#39;org.springframework.ai:spring-ai-bom:2.0.1\u0026#39;) implementation \u0026#39;org.springframework.ai:spring-ai-starter-mcp-server-webmvc\u0026#39; implementation \u0026#39;org.springaicommunity:mcp-server-security-spring-boot:0.1.14\u0026#39; runtimeOnly \u0026#39;org.postgresql:postgresql\u0026#39; } Two lines do the MCP work:\nspring-ai-starter-mcp-server-webmvc is the MCP server over HTTP on Spring MVC. There\u0026rsquo;s a -webflux variant for reactive apps, and a plain spring-ai-starter-mcp-server for stdio. mcp-server-security-spring-boot from the spring-ai-community/mcp-security project adds API-key and OAuth2 support on top of Spring Security. We\u0026rsquo;ll come back to it. Step 2: Configure the server Source: application.yaml\nspring: ai: mcp: server: protocol: STATELESS type: sync name: \u0026#34;Task MCP demo\u0026#34; instructions: \u0026gt;- Manages a simple task list. Create tasks with add_task, look them up with get_task or get_task_list, and move them through the statuses NEW, IN_PROGRESS, DONE and CANCELLED with update_task_status. streamable-http: mcp-endpoint: /mcp What these settings mean:\nprotocol: STATELESS: plain HTTP request and response, with no session kept between calls. Our tools never need to push anything back to the client, so we don\u0026rsquo;t need sessions, and the server scales horizontally like any other REST service. Use STREAMABLE if you need server-to-client messages. The older SSE option is deprecated since Spring AI 2.0. type: sync: tools are ordinary blocking methods. With async, they return Reactor Mono/Flux types instead. instructions: sent to the client when it connects. It\u0026rsquo;s the server\u0026rsquo;s short explanation to the model of how the tools fit together. Treat it as part of the prompt. mcp-endpoint: /mcp: the URL that Claude Code will call. Step 3: Business logic (keep it boring) Nothing here knows about MCP. It’s a normal JPA entity, repository and service:\nSource: TaskService.java\n@Service @Validated @RequiredArgsConstructor @Transactional(readOnly = true) public class TaskService { public static final int NAME_MAX_LEN GTH = 255; private final TaskRepository repository; private final TaskMapper mapper; @Transactional public TaskDto create(@NotBlank @Size(max = NAME_MAX_LENGTH) String name) { var entity = new TaskEntity(); entity.setName(name); entity.setStatus(TaskStatus.NEW); return mapper.toDto(repository.save(entity)); } @Transactional public TaskDto updateStatus(Long id, @NotNull TaskStatus status) { var entity = findEntity(id); entity.setStatus(status); return mapper.toDto(entity); } // get(id), getList(), findEntity(id) throws TaskNotFoundException ... } The service returns a small record, not the entity:\nSource: TaskDto.java\npublic record TaskDto(Long id, String name, TaskStatus status) {} That matters more than it looks. Whatever a tool returns is serialized and handed to the model, so the DTO decides exactly which fields leave your system.\nStep 4: TaskMcpTool, the actual MCP server This is the whole MCP layer:\nSource: TaskMcpTool.java\n@Component @RequiredArgsConstructor public class TaskMcpTool { private final TaskService service; @McpTool(name = \u0026#34;add_task\u0026#34;, description = \u0026#34;Create a new task with status NEW and return it\u0026#34;) public TaskDto add(@McpToolParam(description = \u0026#34;Task name, 1-255 characters\u0026#34;) String name) { return service.create(name); } @McpTool(name = \u0026#34;get_task\u0026#34;, description = \u0026#34;Get a task by its id. Fails if the task does not exist\u0026#34;) public TaskDto get(@McpToolParam(description = \u0026#34;Task id\u0026#34;) Long id) { return service.get(id); } @McpTool(name = \u0026#34;update_task_status\u0026#34;, description = \u0026#34;Change the status of a task and return the updated task. Fails if the task does not exist\u0026#34;) public TaskDto updateStatus(@McpToolParam(description = \u0026#34;Task id\u0026#34;) Long id, @McpToolParam(description = \u0026#34;New task status\u0026#34;) TaskStatus status) { return service.updateStatus(id, status); } @McpTool(name = \u0026#34;get_task_list\u0026#34;, description = \u0026#34;List all tasks\u0026#34;) public List\u0026lt;TaskDto\u0026gt; getList() { return service.getList(); } } Only a thin adapter: each method delegates to the service. Most of the work happens in the annotations.\nHow the tools get registered The class is a plain @Component. When the application starts, Spring AI\u0026rsquo;s annotation scanner finds every bean method annotated with @McpTool and registers it as an MCP tool. You don\u0026rsquo;t need a registry, a ToolCallback list or any manual wiring. To add a tool, write another annotated method.\n@McpTool: the name and description the model sees name is the tool id the model calls. I use snake_case verbs (add_task, get_task_list) because that\u0026rsquo;s the convention across MCP servers, and models pick such names up reliably. Without a name, the Java method name is used, and add on its own tells the model very little. description is the most important string in the class. It\u0026rsquo;s the only thing the model reads when deciding which tool to call and what to expect back. Say what the tool does, what it returns and when it fails (\u0026ldquo;Fails if the task does not exist\u0026rdquo;). A vague description gives you a model that guesses. @McpToolParam: the schema is generated for you Each parameter gets a description, and Spring AI builds a JSON Schema from the Java type. Here’s what the server actually returns for update_task_status when Claude Code asks for tools/list:\n{ \u0026#34;name\u0026#34;: \u0026#34;update_task_status\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Change the status of a task and return the updated task. Fails if the task does not exist\u0026#34;, \u0026#34;inputSchema\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;id\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;integer\u0026#34;, \u0026#34;format\u0026#34;: \u0026#34;int64\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Task id\u0026#34; }, \u0026#34;status\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;enum\u0026#34;: [\u0026#34;NEW\u0026#34;, \u0026#34;IN_PROGRESS\u0026#34;, \u0026#34;DONE\u0026#34;, \u0026#34;CANCELLED\u0026#34;], \u0026#34;description\u0026#34;: \u0026#34;New task status\u0026#34; } }, \u0026#34;required\u0026#34;: [\u0026#34;id\u0026#34;, \u0026#34;status\u0026#34;] } } Things to notice:\nLong became integer with int64 format. You don\u0026rsquo;t write any schema by hand. Parameters are required by default (required = true). Set @McpToolParam(required = false) for optional ones. The TaskStatus enum became a JSON enum. The model sees the four allowed values up front, so it can\u0026rsquo;t invent a status like DELETED. If it tries, the call is rejected by schema validation before your method runs: Tool (update_task_status) input validation failed: ... /status: does not have a value in the enumeration [\u0026#34;NEW\u0026#34;, \u0026#34;IN_PROGRESS\u0026#34;, \u0026#34;DONE\u0026#34;, \u0026#34;CANCELLED\u0026#34;] Use enums for anything with a fixed set of values. You get validation and documentation in one place.\nReturn values and errors You return a normal Java object, and Spring AI serializes it to JSON as the tool result. TaskDto and List\u0026lt;TaskDto\u0026gt; just work.\nExceptions work too. When TaskService throws TaskNotFoundException, the client doesn\u0026rsquo;t get a stack trace or a broken JSON-RPC response. It gets a tool result with isError: true and the exception message as the text:\n{ \u0026#34;content\u0026#34;: [{ \u0026#34;type\u0026#34;: \u0026#34;text\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;Task with id -1 is not found\u0026#34; }], \u0026#34;isError\u0026#34;: true } That’s the behaviour you want: the model reads the message, understands what went wrong and can recover, for example by calling get_task_list to find the right id. Write exception messages for the model to read, not just for your logs.\nThe same applies to Bean Validation. A blank name fails @NotBlank in the service and comes back to the model as an error result, not a 500.\nA tip: add behaviour hints If you look at the full tools/list response, every tool reports \u0026ldquo;readOnlyHint\u0026rdquo;: false, \u0026ldquo;destructiveHint\u0026rdquo;: true, including get_task_list. Those are the defaults. Clients such as Claude Code can use these hints to decide how cautious to be, so it\u0026rsquo;s worth being honest about read-only tools:\n@McpTool(name = \u0026#34;get_task_list\u0026#34;, description = \u0026#34;List all tasks\u0026#34;, annotations = @McpTool.McpAnnotations(readOnlyHint = true, destructiveHint = false)) These are hints, not enforcement. What actually protects your data is the fact that the tool only reads.\nStep 5: SecurityConfig, briefly By default, an HTTP MCP endpoint is an unauthenticated JSON-RPC endpoint. Anyone who can reach /mcp can call your tools. The mcp-security library closes it with a few lines:\nSource: SecurityConfig.java\n@Configuration @EnableWebSecurity class SecurityConfig { @Bean SecurityFilterChain securityFilterChain(HttpSecurity http, ApiKeyEntityRepository\u0026lt;ApiKeyEntityImpl\u0026gt; apiKeyRepository) { return http .authorizeHttpRequests(auth -\u0026gt; auth.anyRequest().authenticated()) .with(mcpServerApiKey(), apiKey -\u0026gt; apiKey.apiKeyRepository(apiKeyRepository)) .build(); } @Bean ApiKeyEntityRepository\u0026lt;ApiKeyEntityImpl\u0026gt; apiKeyRepository(@Value(\u0026#34;${app.mcp.api-key.secret}\u0026#34;) String secret) { var key = ApiKeyEntityImpl.builder() .name(\u0026#34;demo key\u0026#34;) .id(\u0026#34;demo\u0026#34;) .secret(secret) .build(); return new InMemoryApiKeyEntityRepository\u0026lt;\u0026gt;(List.of(key)); } } mcpServerApiKey() plugs API-key authentication into the normal Spring Security filter chain. The client sends an X-API-key header in the form \u0026lt;id\u0026gt;.\u0026lt;secret\u0026gt;, so demo.\u0026lt;your secret\u0026gt; here. Requests without it get 401. Step 6: Run it and connect Claude Code Start PostgreSQL and the app:\nSource: docker-compose.yaml,.env.tpl, README\ncp .env.tpl .env # sets MCP_API_KEY_SECRET docker compose up -d ./gradlew bootRun Register the server with Claude Code:\nclaude mcp add --transport http todos http://localhost:8080/mcp \\ --header \u0026#34;X-API-key: demo.mcp-demo-test-secret\u0026#34; Then ask in plain language:\n“Add todo with name test” becomes an add_task call. \u0026ldquo;Update task with id 1 to DONE\u0026rdquo; becomes update_task_status with a valid enum value. Claude never saw a JDBC URL, a username or a password.\nTo inspect the server without Claude, use the MCP Inspector (npx @modelcontextprotocol/inspector). Choose the Streamable HTTP transport and add the same header.\nTesting Because the server speaks standard MCP, the tests can use the official Java MCP client against a real server started by @SpringBootTest, with PostgreSQL in Testcontainers:\nSource: TaskMcpToolTest.java\nvar transport = HttpClientStreamableHttpTransport .builder(\u0026#34;http://localhost:\u0026#34; + port) .httpRequestCustomizer((builder, method, endpoint, body, context) -\u0026gt; builder.header(\u0026#34;X-API-key\u0026#34;, \u0026#34;demo.\u0026#34; + API_KEY_SECRET)) .build(); client = McpClient.sync(transport).build(); client.initialize(); var result = client.callTool(McpSchema.CallToolRequest.builder(\u0026#34;get_task\u0026#34;) .arguments(Map.of(\u0026#34;id\u0026#34;, -1L)).build()); assertThat(result.isError()).isTrue(); This covers the whole chain in one test: authentication, schema, serialization, error mapping and the database.\nTakeaways You don’t write JSON-RPC in Java. spring-ai-starter-mcp-server-webmvc handles the protocol, the transport and tool discovery. An MCP tool is an annotated Spring bean method. @McpTool names and describes it, and @McpToolParam describes its inputs. The JSON Schema is generated from your Java types. Descriptions and error messages are prompts. The model reads them, so write them for the model. Use enums and DTOs to control what the model can send in and what it gets back. Secure the endpoint. API keys via mcp-security take ten lines, and OAuth2 is available when you need it. The MCP server is the security boundary. Claude Code works with your data through the operations you choose, never with your credentials. The Python version showed how MCP works on the wire. The Spring version shows how little code you need once a framework handles it. Take the repo, replace tasks with your own domain, and give your agent an API instead of a password.\nLinks\nDemo code: github.com/SukachVitally/mcp-demo Spring AI MCP Server Boot Starter: docs.spring.io MCP security for Spring: github.com/spring-ai-community/mcp-security Previous article, MCP in Python from scratch: MCP: The Protocol That Finally Taught AI to Stop Being Useless ","permalink":"https://sukachvitally.github.io/posts/building-an-mcp-server-in-java-with-spring-ai-let-claude-code-use-your-database-without-seeing-a/","summary":"\u003cp\u003e\u003cem\u003eFour annotated methods, one security config, and no JSON-RPC code of your own.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/building-an-mcp-server-in-java-with-spring-ai-let-claude-code-use-your-database-without-seeing-a/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eIn my \u003ca href=\"https://medium.com/@sukach.v.v/mcp-the-protocol-that-finally-taught-ai-to-stop-being-useless-759de3745b79\"\u003eprevious article\u003c/a\u003e I built an MCP server from scratch in Python. I wrote the JSON-RPC 2.0 message handling, declared tool schemas by hand and wired up the STDIO transport myself. It’s a good way to learn how the protocol works on the wire.\u003c/p\u003e\n\u003cp\u003ePython isn’t the only option, though. Most of the systems I work on are Java and Spring, and that’s where the data lives. So this time I built the same kind of server in Java.\u003c/p\u003e","title":"Building an MCP Server in Java with Spring AI: Let Claude Code Use Your Database Without Seeing a Password"},{"content":"What actually happens to your data between two threads — and the small set of tools that let you control it Ask ten Java developers what “the Java Memory Model” means and you’ll get two different answers.\nHalf of them will draw a box called Heap and another called Stack. The other half will start talking about volatile and happens-before.\nBoth groups are talking about something real, but only the second group is talking about the actual Java Memory Model (JMM). The first group is describing the runtime data areas — where objects physically live.\nYou need both to write correct concurrent code, so let’s go through them in order. Memory layout first, because it’s the easier one, then the JMM proper.\nPart 1: Where your data lives When the JVM starts, it carves the process memory into a few distinct regions. Each has an owner and a lifetime.\nHeap — every object and array you ever create lives here. It’s shared, which is exactly why concurrency is hard. Garbage collection happens here. Modern collectors split it further (young generation, old generation, or regions in G1/ZGC), but that’s a GC detail, not a memory model one.\nMetaspace — class metadata, bytecode, and static fields. It sits in native memory since Java 8, which is why PermGen errors disappeared and Metaspace OOM errors appeared instead.\nThread stack — each thread gets its own. Every method call pushes a frame containing local variables, parameters, and the return address. Frame pops when the method returns. This is where StackOverflowError comes from. Nothing here is shared, so locals are never a concurrency problem — unless the value they point to is a shared object.\nPC register — the address of the instruction the thread is currently executing. One per thread.\nNative method stack — the same idea, but for native (JNI) calls.\nOne thing worth burning in:\nOrder order = new Order(); The variable order is a reference sitting on the stack. The Order object itself sits on the heap. Two threads can hold two separate stack references pointing to the same heap object. That\u0026rsquo;s the whole source of the trouble.\nPart 2: The actual Java Memory Model The runtime areas tell you where data sits. They tell you nothing about when one thread sees a write made by another.\nThat question is what the JMM answers. It’s a specification (JSR-133, in Chapter 17 of the Java Language Specification) that defines exactly what guarantees you get about reads and writes across threads.\nThe problem it exists to solve Modern CPUs are fast because they cheat, aggressively and correctly. Two things they do:\n1. Caching. Each core has its own L1/L2 cache. When a thread writes to a field, that write may sit in the core’s cache or store buffer for a while before it becomes visible to other cores.\n2. Reordering. The compiler, the JIT, and the CPU are all allowed to reorder instructions, as long as the result looks the same for a single thread. That last clause is the trap — nobody promised anything about how it looks to other threads.\nSo the JMM works with an abstraction:\nThread A and Thread B never see each other’s working memory. Everything they share has to travel through main memory — and the JMM decides when it does.\nEvery thread has a conceptual working memory (registers, cache, store buffers) holding copies of the variables it touches. Threads never talk to each other directly — they communicate only through main memory. The JMM specifies when a value must be flushed to main memory and when it must be re-read from it.\nHere’s the classic demonstration:\npublic class StopFlag { private boolean running = true; // no volatile public void run() { while (running) { // do work } System.out.println(\u0026#34;stopped\u0026#34;); } public void stop() { running = false; } } Thread A runs run(). Thread B calls stop(). Very often, thread A never stops.\nNothing is broken. The JIT sees that running is never modified inside the loop, so it\u0026rsquo;s free to hoist the read out:\nboolean r = running; while (r) { /* work */ } // now an infinite loop Without a memory model rule forcing it, the JVM has no obligation to ever re-read that field. Your bug is not a race in the “two writes collided” sense — it’s a visibility failure.\nThe three guarantees Every concurrency problem in Java falls into one of these buckets.\ncount++ is the textbook atomicity failure — it\u0026rsquo;s three operations (read, add, write), and two threads can interleave in the middle. Two increments, one result.\nThe base components of the JMM Here’s the toolbox. It’s smaller than most people expect.\n1. happens-before — the rule that underpins everything Everything else is a special case of this one relation. If action A happens-before action B, then:\neverything A wrote is visible to B, and A is guaranteed to appear to occur before B. It is not just “A ran earlier in wall-clock time.” Two actions can be ordered in time and still have no happens-before relationship, which means B may see stale data.\nThe rules the JLS gives you:\nProgram order — within one thread, each statement happens-before the ones written after it. Monitor lock — unlocking a monitor happens-before any later lock of the same monitor. volatile — a write to a volatile field happens-before every later read of that field. Thread start — Thread.start() happens-before anything in the started thread. Thread join — everything in a thread happens-before another thread’s successful join() on it. Interruption — a call to interrupt() happens-before the interrupted thread detects it. Constructor / finalizer — the end of a constructor happens-before the start of that object’s finalizer. Transitivity — if A hb B and B hb C, then A hb C. That last one does most of the practical work. It’s why writing a plain field and then a volatile flag safely publishes the plain field.\n2. volatile — visibility and ordering, no atomicity private volatile boolean running = true; This single keyword fixes the StopFlag example. It gives you:\nVisibility — a write goes to main memory immediately; a read comes from main memory, never a cache. Ordering — the JVM inserts memory barriers so reads and writes can’t be reordered across the volatile access. What it does not give you is atomicity:\nprivate volatile int count = 0; public void increment() { count++; // STILL BROKEN } Use volatile for flags and for single-writer state. For compound operations, use something else.\n3. synchronized — all three guarantees private int count = 0; public synchronized void increment() { count++; // correct } public synchronized int get() { return count; // also needs to be synchronized } Entering the monitor gives mutual exclusion (atomicity). Exiting flushes to main memory, entering invalidates the local copy (visibility). Barriers around the block prevent reordering out of it.\nThe most common mistake here is synchronizing the writer but not the reader. Both sides need the same lock, or there’s no happens-before edge between them.\nReentrantLock gives you the same memory guarantees, plus tryLock, timeouts, fairness, and multiple conditions.\n4. final — safe publication for free public class Config { private final String url; public Config(String url) { this.url = url; // guaranteed visible after construction } } The JMM guarantees that any thread that sees a properly constructed object sees the correct values of its final fields — no synchronization required. This is why immutable objects are the easiest possible concurrency strategy.\nThe catch is “properly constructed.” If the constructor leaks this (registering a listener, starting a thread), the guarantee is void.\n5. Atomics and VarHandle java.util.concurrent.atomic gives you lock-free compound operations built on CPU compare-and-swap instructions:\nprivate final AtomicInteger count = new AtomicInteger(); public void increment() { count.incrementAndGet(); // atomic, visible, ordered } Under contention LongAdder usually beats AtomicLong — it spreads writes across cells and sums them on read, avoiding cache-line contention. VarHandle (Java 9+) is the modern, supported way to reach for finer-grained modes like getAcquire / setRelease when you know exactly what you\u0026rsquo;re doing.\nThree bugs this explains Unsafe publication. Building an object and handing the reference to another thread without a happens-before edge. The other thread can see a non-null reference to a half-initialized object — fields still at their defaults. Publish through a volatile field, a lock, a final field, or a concurrent collection.\nDouble-checked locking without volatile. The infamous one:\nprivate static volatile Singleton instance; // volatile is mandatory public static Singleton getInstance() { if (instance == null) { synchronized (Singleton.class) { if (instance == null) { instance = new Singleton(); } } } return instance; } Drop the volatile and it breaks, because instance = new Singleton() is three steps — allocate, construct, assign — and the last two can be reordered. Another thread can grab a reference to an object whose constructor hasn\u0026rsquo;t finished.\nCheck-then-act. if (!map.containsKey(k)) map.put(k, v); is two atomic operations, not one atomic operation. Use putIfAbsent or computeIfAbsent.\nThe practical cheat sheet Shared and mutable? It needs synchronization. Every time. One thread writes a flag, others read it? volatile. Compound operation on one variable? An atomic class. Compound operation across several variables? synchronized or ReentrantLock. Can you make it immutable instead? Do that. final fields, no setters, no shared mutation, no memory model to reason about. Reading is also access. Unsynchronized reads of synchronized state are still broken. “It works on my machine” proves nothing. x86 has a strong memory model that hides many of these bugs. ARM (Apple Silicon, Graviton) doesn’t. Code that ran fine for years on Intel servers can fail the week you move it to ARM instances. Wrapping up The runtime areas — heap, stacks, metaspace — tell you where your data sits. The Java Memory Model tells you when one thread’s writes become visible to another, and it comes down to a handful of pieces: happens-before as the underlying rule, and volatile, synchronized, final, and the atomics as the tools that create it.\nYou don’t need to memorize the specification. You need one habit: whenever two threads touch the same data, ask what creates the happens-before edge between them. If you can’t name it, it isn’t there.\nIf this was useful, the natural next step is Brian Goetz’s Java Concurrency in Practice — still the best treatment of this material, twenty years on.\n","permalink":"https://sukachvitally.github.io/posts/the-java-memory-model-explained-without-the-headache/","summary":"\u003ch4 id=\"what-actually-happens-to-your-data-between-two-threadsand-the-small-set-of-tools-that-let-you-controlit\"\u003eWhat actually happens to your data between two threads — and the small set of tools that let you control it\u003c/h4\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/the-java-memory-model-explained-without-the-headache/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eAsk ten Java developers what “the Java Memory Model” means and you’ll get two different answers.\u003c/p\u003e\n\u003cp\u003eHalf of them will draw a box called \u003cem\u003eHeap\u003c/em\u003e and another called \u003cem\u003eStack\u003c/em\u003e. The other half will start talking about volatile and happens-before.\u003c/p\u003e\n\u003cp\u003eBoth groups are talking about something real, but only the second group is talking about the actual \u003cstrong\u003eJava Memory Model (JMM)\u003c/strong\u003e. The first group is describing the \u003cstrong\u003eruntime data areas\u003c/strong\u003e — where objects physically live.\u003c/p\u003e","title":"The Java Memory Model, Explained Without the Headache"},{"content":"A deep dive into MVCC, heap pages, vacuum, and what to do when you have billions of rows to update.\nThe symptom You have a table that stores user state. Something happens — an event arrives — and your code does the obvious thing:\nBEGIN; DELETE FROM user_state WHERE user_id = $1; INSERT INTO user_state (user_id, state, score, updated_at) VALUES ($1, $2, $3, now()); COMMIT; It’s clean. It’s idempotent. It doesn’t require you to think about which columns changed. For a long time it works fine.\nThen event volume grows, and PostgreSQL’s CPU starts climbing in a way that nothing in your query plans explains. Your queries are still index scans. Your pg_stat_statements top entries look reasonable. But CPU is pinned, replication lag creeps up, and every so often the whole database gets slow for a few minutes and then recovers on its own.\nIf you look at pg_stat_activity during one of those windows, you\u0026rsquo;ll usually find the culprit sitting right there:\nautovacuum: VACUUM public.user_state That’s the story. But “autovacuum is expensive” is not an explanation — it’s a restatement of the symptom. To actually fix this you need to understand what a DELETE followed by an INSERT does to the physical 8 KB pages on disk, and why that specific shape of write is close to the worst case for PostgreSQL\u0026rsquo;s storage engine.\nLet’s go all the way down.\nPart 1: PostgreSQL never modifies a row in place This is the single fact that everything else follows from.\nPostgreSQL implements MVCC (Multi-Version Concurrency Control) by keeping multiple physical versions of every row and letting each transaction see the version appropriate to its snapshot. There is no undo log (unlike Oracle or MySQL’s InnoDB, which keep old versions in a separate rollback segment). In PostgreSQL, old row versions live in the table itself, mixed in with the live ones.\nConcretely:\nINSERT writes a new tuple and stamps it with xmin = the inserting transaction ID. DELETE does not remove anything. It stamps the existing tuple with xmax = the deleting transaction ID. The bytes stay exactly where they were. UPDATE is internally a DELETE + INSERT: it stamps xmax on the old tuple and writes a brand-new tuple somewhere. So a “deleted” row is still fully present on disk, occupying its bytes, until some later process comes along and physically reclaims it. That process is vacuum.\nHere’s this happening for real. A 100-row table, then delete 50 rows and insert 50 new ones:\nCREATE EXTENSION pageinspect; CREATE TABLE demo (id int PRIMARY KEY, state text); INSERT INTO demo SELECT g, \u0026#39;active\u0026#39; FROM generate_series(1,100) g; -- delete half, re-insert half (the pattern from our app) DELETE FROM demo WHERE id \u0026lt;= 50; INSERT INTO demo SELECT g, \u0026#39;active\u0026#39; FROM generate_series(1,50) g; SELECT lp, lp_flags, lp_len, t_xmin, t_xmax FROM heap_page_items(get_raw_page(\u0026#39;demo\u0026#39;, 0)) WHERE lp BETWEEN 48 AND 54; lp | lp_flags | lp_len | t_xmin | t_xmax ----+----------+--------+--------+-------- 48 | 1 | 35 | 801 | 802 \u0026lt;- deleted, still on the page 49 | 1 | 35 | 801 | 802 \u0026lt;- deleted, still on the page 50 | 1 | 35 | 801 | 802 \u0026lt;- deleted, still on the page 51 | 1 | 35 | 801 | 0 \u0026lt;- live 52 | 1 | 35 | 801 | 0 \u0026lt;- live 53 | 1 | 35 | 801 | 0 \u0026lt;- live 54 | 1 | 35 | 801 | 0 \u0026lt;- live The deleted tuples have t_xmax = 802 (the transaction that deleted them) and are still LP_NORMAL (lp_flags = 1). They occupy real space. The table now has 150 line pointers where the logical row count is 100.\nSELECT tuple_count, dead_tuple_count, dead_tuple_percent, free_percent FROM pgstattuple(\u0026#39;demo\u0026#39;); tuple_count | dead_tuple_count | dead_tuple_percent | free_percent -------------+------------------+--------------------+-------------- 100 | 50 | 21.36 | 19.09 21% of that table is garbage. And that’s after one round.\nPart 2: Anatomy of an 8 KB page PostgreSQL stores everything in fixed-size blocks — 8 KB by default (SHOW block_size). Understanding the layout of one page is the key to understanding why churn is expensive.\nTwo structures grow toward each other from opposite ends: the line pointer array from the top, the tuple data from the bottom. The page is full when they meet.\nThe line pointer (also called an item identifier, ItemIdData) is a tiny 4-byte struct that says \u0026ldquo;the tuple at slot N lives at byte offset X and is Y bytes long.\u0026rdquo; It has four possible states, and you\u0026rsquo;ll see all of them in this article:\nThe tuple header (HeapTupleHeader, 23 bytes padded to 24) is where MVCC lives:\nThat’s 24 bytes of overhead per row version, plus 4 bytes for the line pointer. For a table storing a user ID, a short state string, and a timestamp, the MVCC bookkeeping can easily be 40% of the row’s on-disk size.\nNow here’s the critical bit for our problem:\nA page’s free space is only usable for a new tuple if it is contiguous and the page has a free line pointer or room to add one. Dead tuples fragment the page. Only vacuum can defragment it.\nPart 3: Why DELETE + INSERT is the worst shape Compare what the two write patterns do to a page.\nA DELETE + INSERT can never be a HOT update. HOT (Heap-Only Tuple) is an optimization that only applies to UPDATE, and only when two conditions hold:\nNo indexed column changed. The new version fits on the same page as the old one. When both hold, PostgreSQL writes the new version onto the same page and links it via t_ctid, leaving the index entries alone. That\u0026rsquo;s an enormous saving: index maintenance is usually the dominant cost of a write, because B-tree inserts are random-access, and every index on the table pays it.\nBy splitting your logical update into two statements, you’ve told PostgreSQL: this is a brand-new row that has nothing to do with the old one. It has no choice but to write full index entries into every index, and to leave a dead tuple behind that vacuum must chase down.\nYou can see HOT working with pageinspect. In a table where updates are HOT-eligible, the page fills up with redirect pointers:\nSELECT lp, lp_flags, lp_len, t_ctid FROM heap_page_items(get_raw_page(\u0026#39;user_state_hot\u0026#39;, 0)) LIMIT 5; lp | lp_flags | lp_len | t_ctid ----+----------+--------+-------- 1 | 2 | 0 | \u0026lt;- LP_REDIRECT 2 | 2 | 0 | \u0026lt;- LP_REDIRECT 3 | 2 | 0 | \u0026lt;- LP_REDIRECT 4 | 2 | 0 | \u0026lt;- LP_REDIRECT 5 | 2 | 0 | \u0026lt;- LP_REDIRECT Those redirect pointers are the physical trace of updates that cost zero index writes.\nPart 4: What vacuum actually does Vacuum is not one operation. A standard (non-full) VACUUM of a table with indexes runs in phases, and understanding the phases explains where the CPU goes.\nPhase 1 — Heap scan Vacuum scans the table’s pages (skipping ones the visibility map marks all-visible, which is the main reason vacuum on an append-only table is cheap and vacuum on a churned table is not). For each page it:\nPrunes HOT chains and marks reclaimable tuples. Collects the TIDs (page number, line pointer) of dead tuples into memory. Sets LP_DEAD on their line pointers. The dead TID list is bounded by maintenance_work_mem (or autovacuum_work_mem). In PostgreSQL 17+ this list is stored in an adaptive radix tree (TidStore) rather than a flat array, which cut memory usage dramatically and removed the old 1 GB cap — a genuinely large improvement for high-churn tables.\nPhase 2 — Index vacuum This is the expensive part. For every index on the table, vacuum performs a full scan (a btbulkdelete) and removes every index entry pointing at a dead TID.\nNote what that means: the cost is per index, and it’s a full index scan, not a lookup. A table with five indexes pays five full index scans per vacuum cycle. This is why “just add an index” decisions compound badly on high-churn tables.\nPhase 3 — Heap vacuum Vacuum revisits the pages and turns LP_DEAD line pointers into LP_UNUSED, then defragments the page: it slides the remaining tuples together so the free space becomes one contiguous block. Now that space is genuinely reusable.\nIf the dead-TID list overflowed maintenance_work_mem in phase 1, vacuum has to loop: scan more heap → vacuum all indexes again → vacuum heap again. A single vacuum of a big churned table can do the full index pass multiple times. This is the classic \u0026ldquo;why has autovacuum been running for six hours\u0026rdquo; answer.\nPhase 4 — Freezing Separately, vacuum replaces old xmin values with a frozen marker so transaction IDs can wrap around safely. Anti-wraparound vacuums are non-cancelable and will run whether you like it or not. PostgreSQL 18 added eager freezing (vacuum_max_eager_freeze_failure_rate, default 0.03) so that normal vacuums opportunistically freeze pages, spreading this cost out instead of saving it up for one giant emergency vacuum.\nPhase 5 — Truncation If there are empty pages at the end of the table, vacuum tries to return them to the OS. This requires an ACCESS EXCLUSIVE lock (briefly). Note the word end: vacuum almost never shrinks a bloated table, because the free space is scattered throughout, not conveniently at the tail. It makes the space reusable by PostgreSQL, not available to your filesystem.\nHere’s the before/after on our demo table:\nVACUUM demo; SELECT lp_flags, count(*) FROM heap_page_items(get_raw_page(\u0026#39;demo\u0026#39;,0)) GROUP BY 1; Free space on the page went from 19.09% → 43.51%. That’s the work vacuum did. Now multiply it by every page you churned.\nPart 5: The amplification math Here is the part people underestimate. One logical “user state changed” event does not cause one write. It causes, at minimum:\nSteps 7–13 are invisible in your application metrics. Your app measures step 1–6 latency and reports “the insert took 2 ms.” The other two-thirds of the cost lands on a background worker, asynchronously, in a burst, on the same CPU.\nThat’s the answer to “why does CPU go up but my queries look fine.”\nPoint 5b: the full-page-image trap After each checkpoint, the first write to any page triggers a full 8 KB page image written into WAL (full_page_writes, on by default, and required for crash safety on most filesystems). If your churn touches many distinct pages, you pay a lot of FPIs. If your churn is concentrated on a few pages, you pay far fewer. Delete+insert scatters writes across pages — the new tuple often lands in a different page from the one you dirtied with the xmax write. Two dirty pages instead of one, two FPI candidates instead of one.\nPoint 5c: the read-side cost nobody mentions Dead tuples also make reads slower and CPU-hungrier:\nAn index scan follows an index entry to the heap, reads the tuple, checks visibility, discovers it’s dead, and discards it. That’s a wasted buffer access plus a visibility check. You can see this directly: EXPLAIN (ANALYZE, BUFFERS) will show a rising Buffers: shared hit count for a query returning the same number of rows. Bloated tables have worse cache locality. The same 100k live rows spread over 2× the pages means 2× the shared buffers to hold them, which evicts other things. On a read, PostgreSQL may perform opportunistic HOT pruning on the page — real CPU work done inside your SELECT. Part 6: Measurements I ran a controlled comparison: 100,000 rows, ten rounds of modifying 20,000 of them, with an explicit VACUUM after each round so we\u0026rsquo;re looking at steady state, not a runaway. Three variants:\nRead those rows carefully, because the middle one is the trap.\nSwitching from **DELETE+**INSERT to UPDATE bought essentially nothing on its own. The second variant still generated more WAL than the first. Why? Because the UPDATE touched updated_at, and updated_at had an index on it. An UPDATE that modifies any indexed column is not HOT-eligible, so it does exactly what delete+insert does: new heap tuple, new entries in every index, dead tuple left behind.\nThe third variant — same UPDATE, but with no index on the mutated columns and fillfactor lowered to 80 — generated less than half the WAL and made 73% of its updates HOT.\nThe cost of that win is visible too: the HOT table starts at 1042 heap pages instead of 834, because fillfactor=80 deliberately leaves 20% of every page empty. That empty space is what new row versions land in. You are trading disk space for CPU and I/O, which on modern hardware is almost always the right trade.\n(These are synthetic single-client numbers on PostgreSQL 16 — treat the ratios as directional, not as a benchmark of your workload.)\nPart 7: Fixing it, in order of leverage 7.1 Don’t write when nothing changed The cheapest write is the one that doesn’t happen. A shocking amount of churn in “collect user state” systems is rewriting a row to the exact same value because an event arrived that didn’t actually change anything.\nINSERT INTO user_state (user_id, state, score, updated_at) VALUES ($1, $2, $3, now()) ON CONFLICT (user_id) DO UPDATE SET state = EXCLUDED.state, score = EXCLUDED.score, updated_at = now() WHERE user_state.state IS DISTINCT FROM EXCLUDED.state OR user_state.score IS DISTINCT FROM EXCLUDED.score; That trailing WHERE on the DO UPDATE clause is the whole trick: if nothing changed, PostgreSQL writes nothing — no new tuple, no dead tuple, no index entry, no WAL. In event-driven systems where the same state gets recomputed repeatedly, this alone often removes 50–90% of write volume.\nMeasure how much you’d save before you build anything else:\n-- how many of your writes are no-ops? SELECT count(*) FILTER (WHERE state = $2 AND score = $3) AS would_be_noop, count(*) AS total FROM user_state WHERE user_id = ANY($1); 7.2 Make your updates HOT-eligible Two requirements, both actionable:\n(a) Don’t index the columns that change. Audit every index against your write path:\nSELECT i.relname AS index_name, array_agg(a.attname ORDER BY x.n) AS indexed_columns, s.idx_scan AS times_used FROM pg_index ix JOIN pg_class i ON i.oid = ix.indexrelid JOIN pg_class t ON t.oid = ix.indrelid JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS x(attnum, n) ON true JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = x.attnum LEFT JOIN pg_stat_user_indexes s ON s.indexrelid = i.oid WHERE t.relname = \u0026#39;user_state\u0026#39; GROUP BY i.relname, s.idx_scan ORDER BY s.idx_scan NULLS FIRST; An index on updated_at is the single most common HOT-killer in exactly this kind of table. Ask whether you truly need it, or whether a partial index, a coarser index (date_trunc(\u0026lsquo;hour\u0026rsquo;, updated_at) doesn\u0026rsquo;t help — but a separate low-churn audit table might), or no index at all would do.\nUnused indexes are pure loss: they cost writes and vacuum passes and return nothing. Drop them.\n(b) Leave room on the page.\nALTER TABLE user_state SET (fillfactor = 80); -- fillfactor only applies to newly written pages; -- rewrite the table to apply it to existing data: VACUUM FULL user_state; -- takes ACCESS EXCLUSIVE lock -- or, online: -- pg_repack -t user_state For a heavily-updated table, 70–85 is a reasonable range. Below ~70 you’re wasting a lot of buffer cache on empty space.\nThen confirm it’s working:\nSELECT relname, n_tup_upd, n_tup_hot_upd, round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd,0), 1) AS hot_pct FROM pg_stat_user_tables WHERE relname = \u0026#39;user_state\u0026#39;; If hot_pct is near zero on an update-heavy table, that is your bug. Aim for 80%+.\n7.3 Tune autovacuum for this table, not globally The defaults are tuned for a small table on a small machine. autovacuum_vacuum_scale_factor = 0.2 means vacuum waits until 20% of the table is dead before acting. On a 500-million-row table that\u0026rsquo;s 100 million dead tuples before anything happens — and then one enormous, disruptive vacuum.\nYou want the opposite: vacuum more often, doing less each time.\nALTER TABLE user_state SET ( autovacuum_vacuum_scale_factor = 0.01, -- act at 1% dead, not 20% autovacuum_vacuum_threshold = 1000, autovacuum_analyze_scale_factor = 0.02, autovacuum_vacuum_cost_delay = 0, -- don\u0026#39;t throttle this table autovacuum_vacuum_cost_limit = 3000 ); And globally, if your hardware is modern (NVMe, many cores), the shipped throttle is far too conservative:\nautovacuum_max_workers = 6 # default 3 autovacuum_naptime = 15s # default 1min autovacuum_vacuum_cost_delay = 2ms # default 2ms; consider 0 on fast storage vacuum_cost_limit = 2000 # default 200 maintenance_work_mem = 2GB # gives vacuum a big dead-TID store autovacuum_work_mem = 1GB The cost-delay mechanism is the one most often left at a value that makes vacuum unable to keep up: vacuum accumulates “cost” (1 per buffer hit, 2 per miss, 20 per dirtied page) and sleeps cost_delay whenever it exceeds cost_limit. With the defaults, vacuum is deliberately slowed to something like a few MB/s. On a table generating hundreds of MB of dead tuples per minute, it will never catch up, and dead tuples accumulate forever.\nPostgreSQL 18 also added autovacuum_vacuum_max_threshold (default 100,000,000), which caps the scale-factor calculation so very large tables can\u0026rsquo;t defer vacuum indefinitely. It helps, but a per-table scale factor is still better.\n7.4 Reconsider the data model If your state genuinely is “replace the whole thing on every event,” ask whether it belongs in a row-per-user table at all:\nCollapse many rows into one. If you’re storing N state rows per user and replacing them all, one row with a jsonb column can turn N deletes + N inserts into one update — and if jsonb is under the TOAST threshold and no indexed column changes, that update can be HOT. (Careful: a large jsonb that gets TOASTed is a whole separate write amplification problem. This works when the document is small, roughly under 2 KB.) Append-only + a materialized “current” view. Write events to a partitioned append-only table (no updates, no deletes, so almost no vacuum work — just insert-triggered vacuums for freezing), and maintain the current state separately, or compute it with a DISTINCT ON query over a well-indexed event log. Append-only tables are the cheapest thing PostgreSQL does. Move the hottest state out of PostgreSQL. If a piece of state changes many times per second per user and you only ever need the latest value, Redis or a dedicated KV store is the honest answer. Not everything needs to be transactional and durable. Part 8: The hard question — updating billions of rows This is the case where all the usual advice breaks, so it deserves its own treatment.\nWhy the naive approach fails, precisely UPDATE huge_table SET status = \u0026#39;migrated\u0026#39; WHERE status = \u0026#39;pending\u0026#39;; -- 2,000,000,000 rows Four separate disasters:\n1. Table size doubles. Every updated row writes a new version while the old one stays. A 2 TB table becomes 4 TB mid-statement. Then vacuum makes the space reusable but does not return it to the OS. You now permanently need 4 TB of disk for 2 TB of data.\n2. It generates terabytes of WAL in one burst. Replicas fall behind, archiving falls behind, and if you’re on a cloud provider you may fill the WAL volume and take the primary down.\n3. It holds a single transaction open for hours — and this is the one that hurts most. Vacuum can only remove tuples older than the oldest running transaction’s snapshot (the xmin horizon). A long transaction pins that horizon. So while your giant UPDATE runs, vacuum cannot clean up dead tuples anywhere in the entire database — not just this table. Every other table quietly bloats. This is how one migration takes down a whole cluster.\n4. If it fails at 95%, all the work is thrown away — but the dead tuples it created are not. You get the bloat with none of the benefit.\nYou can watch the horizon problem happen:\nSELECT pid, age(backend_xid) AS xid_age, state, now() - xact_start AS duration, left(query, 60) FROM pg_stat_activity WHERE backend_xid IS NOT NULL ORDER BY xact_start LIMIT 5; Also check for the other two horizon-pinners people forget: replication slots and prepared transactions.\nSELECT slot_name, active, age(xmin) AS xmin_age, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal FROM pg_replication_slots; SELECT gid, prepared, age(transaction) FROM pg_prepared_xacts; An inactive replication slot will pin the xmin horizon forever and no amount of vacuum tuning will save you.\nStrategy A: Batched backfill with keyset pagination The workhorse. Small transactions, each one committing so the xmin horizon advances and vacuum can keep pace.\n-- Track progress in a real table so the job is resumable CREATE TABLE IF NOT EXISTS backfill_progress ( job_name text PRIMARY KEY, last_id bigint NOT NULL, updated_at timestamptz NOT NULL DEFAULT now() ); -- One batch. Run this in a loop from your application or a scheduler. WITH bounds AS ( SELECT last_id FROM backfill_progress WHERE job_name = \u0026#39;status_migration\u0026#39; ), batch AS ( SELECT id FROM huge_table WHERE id \u0026gt; (SELECT last_id FROM bounds) ORDER BY id LIMIT 5000 FOR UPDATE SKIP LOCKED ), upd AS ( UPDATE huge_table t SET status = \u0026#39;migrated\u0026#39; FROM batch b WHERE t.id = b.id AND t.status IS DISTINCT FROM \u0026#39;migrated\u0026#39; -- skip no-op writes RETURNING t.id ) UPDATE backfill_progress SET last_id = COALESCE((SELECT max(id) FROM batch), last_id), updated_at = now() WHERE job_name = \u0026#39;status_migration\u0026#39; RETURNING last_id, (SELECT count(*) FROM upd) AS rows_updated; Rules that make the difference between this working and this being a slow disaster:\nKeyset pagination, never OFFSET. OFFSET 10000000 re-scans ten million rows every batch. WHERE id \u0026gt; $last uses the index. Batch size 1,000–10,000 rows. Big enough to amortize overhead, small enough that each transaction finishes in well under a second. AND col IS DISTINCT FROM $new so re-running a batch is free and no-ops don\u0026rsquo;t write. Throttle deliberately. Sleep 50–200 ms between batches. You are racing autovacuum; give it room to win. Adaptive throttling — sleep longer when replication lag or dead-tuple count rises — is worth the extra code: -- feed this into your loop\u0026#39;s sleep decision SELECT (SELECT n_dead_tup FROM pg_stat_user_tables WHERE relname=\u0026#39;huge_table\u0026#39;) AS dead, (SELECT coalesce(max(pg_wal_lsn_diff(sent_lsn, replay_lsn)), 0) FROM pg_stat_replication) AS replica_lag_bytes; Run it during a low-traffic window if you have one, but design it to be safe at any time. Expected duration is the thing to be honest about: 2 billion rows at 5,000 rows/batch with a 100 ms sleep is 400,000 batches ≈ 11+ hours minimum, realistically a few days. That’s fine. Plan for days, make it resumable, make it observable.\nStrategy B: Partitioning + partition swap If your data is naturally partitionable (by time, tenant, hash of user ID), this changes the game completely, because you can rebuild a partition offline and swap it in atomically.\n-- Assume: huge_table PARTITION BY RANGE (created_at) -- with partition huge_table_2026_03 -- 1. Build the new version at full speed, outside the live table. CREATE TABLE huge_table_2026_03_new (LIKE huge_table INCLUDING ALL); INSERT INTO huge_table_2026_03_new SELECT id, user_id, \u0026#39;migrated\u0026#39;::text AS status, payload, created_at FROM huge_table_2026_03; -- 2. Indexes AFTER the load - far faster than maintaining them during it. -- No CONCURRENTLY needed: nothing is reading this table yet. CREATE INDEX ON huge_table_2026_03_new (user_id); ANALYZE huge_table_2026_03_new; -- 3. Atomic swap: milliseconds of ACCESS EXCLUSIVE lock, not hours. BEGIN; ALTER TABLE huge_table DETACH PARTITION huge_table_2026_03; ALTER TABLE huge_table ATTACH PARTITION huge_table_2026_03_new FOR VALUES FROM (\u0026#39;2026-03-01\u0026#39;) TO (\u0026#39;2026-04-01\u0026#39;); COMMIT; -- 4. Keep the old one around until you\u0026#39;re confident, then drop it. -- DROP TABLE huge_table_2026_03; Why this is so much better:\nThe bulk INSERT into an empty, unindexed table is the fastest write path PostgreSQL has. No dead tuples are created at all. Zero vacuum debt: you never made garbage, so there’s nothing to collect. The lock window is milliseconds. Rollback is trivial: swap the old partition back. You can do partitions one at a time, over weeks, with a rollback point after each. There’s a caveat worth knowing: ATTACH PARTITION validates that existing rows satisfy the partition constraint, which requires a scan — unless you add a matching CHECK constraint to the new table first, in which case PostgreSQL skips the scan.\nALTER TABLE huge_table_2026_03_new ADD CONSTRAINT part_check CHECK (created_at \u0026gt;= \u0026#39;2026-03-01\u0026#39; AND created_at \u0026lt; \u0026#39;2026-04-01\u0026#39;) NOT VALID; ALTER TABLE huge_table_2026_03_new VALIDATE CONSTRAINT part_check; -- now ATTACH is instant If you have billions of rows and no partitioning, partitioning is probably the actual project — not the update itself. Everything downstream gets easier: DROP PARTITION replaces mass DELETE (instant, and creates zero dead tuples), vacuum works per-partition in parallel, and index sizes stay manageable.\nStrategy C: Rebuild and swap the whole table When the change is uniform and you can’t partition, build a new table and cut over.\nFor a bloated table with no schema change, pg_repack does this online: it builds a fresh copy, captures concurrent changes via triggers, replays them, and swaps — taking the exclusive lock only for the final instant.\npg_repack -h host -d mydb -t huge_table --jobs 4 For a schema/data change, the same idea by hand with logical replication or triggers; tools like pgroll and Percona\u0026rsquo;s approach automate it. The pattern is always: new table, bulk load, keep in sync, atomic rename.\nBudget for double the disk space. That’s the price of every swap-based strategy.\nStrategy D: Don’t update at all The strategy people skip past. Ask whether the update is necessary:\nAdd a new column with a default instead of rewriting values — since PostgreSQL 11, adding a column with a non-volatile default is a metadata-only change that rewrites nothing, no matter how big the table. Write a view that computes the new value from the old one, and migrate readers to it. Use a generated column if the new value is derivable. Let it happen lazily: update rows as they’re naturally touched by the application, with a background job mopping up the long tail over months. A billion-row update you don’t have to run is strictly better than one you run well.\nPart 9: What to actually watch Four queries. Put them on a dashboard.\n1. Is vacuum keeping up?\nSELECT relname, n_live_tup, n_dead_tup, round(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_pct, last_autovacuum, autovacuum_count, n_tup_upd, n_tup_hot_upd, round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd, 0), 1) AS hot_pct FROM pg_stat_user_tables WHERE n_dead_tup \u0026gt; 10000 ORDER BY n_dead_tup DESC LIMIT 20; Sustained dead_pct above ~10% on a busy table means vacuum is losing. hot_pct near zero on an update-heavy table means you\u0026rsquo;re paying full index cost on every write.\n2. What is vacuum doing right now?\nSELECT p.pid, p.relid::regclass AS table, p.phase, pg_size_pretty(p.heap_blks_total * 8192::bigint) AS total, round(100.0 * p.heap_blks_scanned / NULLIF(p.heap_blks_total,0), 1) AS pct_scanned, p.index_vacuum_count, now() - a.xact_start AS running_for FROM pg_stat_progress_vacuum p JOIN pg_stat_activity a USING (pid); index_vacuum_count \u0026gt; 1 is the smoking gun for maintenance_work_mem being too small — vacuum had to do the index pass more than once.\n3. Is anything pinning the xmin horizon?\nSELECT \u0026#39;backend\u0026#39; AS src, pid::text AS id, age(backend_xmin) AS xmin_age FROM pg_stat_activity WHERE backend_xmin IS NOT NULL UNION ALL SELECT \u0026#39;slot\u0026#39;, slot_name, age(xmin) FROM pg_replication_slots WHERE xmin IS NOT NULL UNION ALL SELECT \u0026#39;prepared_xact\u0026#39;, gid, age(transaction) FROM pg_prepared_xacts ORDER BY xmin_age DESC NULLS LAST LIMIT 10; Anything with a large age here is preventing vacuum from cleaning up database-wide.\n4. How bloated is it, really?\nCREATE EXTENSION IF NOT EXISTS pgstattuple; -- accurate but does a full scan; use the sampling version on huge tables SELECT * FROM pgstattuple(\u0026#39;user_state\u0026#39;); SELECT * FROM pgstattuple_approx(\u0026#39;huge_table\u0026#39;); SELECT * FROM pgstatindex(\u0026#39;user_state_user_id_idx\u0026#39;); For indexes, avg_leaf_density below ~50 means the index is worth rebuilding:\nREINDEX INDEX CONCURRENTLY user_state_user_id_idx; The checklist If you’re in the situation this article opened with, do these in order:\nMeasure first. dead_pct, hot_pct, pg_stat_progress_vacuum, xmin horizon. Don\u0026rsquo;t tune blind. Kill no-op writes with ON CONFLICT DO UPDATE \u0026hellip; WHERE \u0026hellip; IS DISTINCT FROM. Usually the biggest single win, and the cheapest to ship. Convert delete+insert to UPDATE — but only after step 4, or it buys you nothing. Make updates HOT-eligible. Drop or move indexes on frequently-changed columns (especially updated_at). Set fillfactor to 80. Verify hot_pct climbs above 80%. Tune autovacuum per-table: scale factor down to 0.01, cost delay to 0, cost limit way up, maintenance_work_mem to gigabytes. Drop unused indexes. Every one is a write tax and an extra full scan per vacuum. Partition if the table is large and the data has a natural key. This makes every future problem easier. For mass updates: batched keyset backfill, or partition rebuild-and-swap. Never one giant statement. The underlying idea is simple enough to hold in your head: PostgreSQL is a copy-on-write storage engine that trades write amplification for concurrency. Delete+insert opts into the maximum possible amplification — no HOT, full index maintenance, guaranteed dead tuples, guaranteed vacuum work. Understanding what happens on the 8 KB page is what turns “the database is slow” into a list of specific, fixable things.\nIf this was useful, the two commands worth internalizing are pageinspect\u0026rsquo;s heap_page_items() and pgstattuple(). Point them at your hottest table. What you find will probably explain your CPU graph.\nReferences PostgreSQL 18: Routine Vacuuming PostgreSQL 18: Vacuuming configuration parameters PostgreSQL 18 Released PostgreSQL 18 Vacuuming Improvements Explained (Microsoft) autovacuum_vacuum_max_threshold in PostgreSQL 18 (dbi services) Faster VACUUM with Adaptive Radix Trees in PostgreSQL 17 (pganalyze) VACUUM at the Page Level (boringSQL) ","permalink":"https://sukachvitally.github.io/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/","summary":"\u003cp\u003e\u003cem\u003eA deep dive into MVCC, heap pages, vacuum, and what to do when you have billions of rows to update.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-01.png\"\u003e\u003c/p\u003e\n\u003ch3 id=\"the-symptom\"\u003eThe symptom\u003c/h3\u003e\n\u003cp\u003eYou have a table that stores user state. Something happens — an event arrives — and your code does the obvious thing:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eBEGIN;\nDELETE FROM user_state WHERE user_id = $1;\nINSERT INTO user_state (user_id, state, score, updated_at) VALUES ($1, $2, $3, now());\nCOMMIT;\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eIt’s clean. It’s idempotent. It doesn’t require you to think about which columns changed. For a long time it works fine.\u003c/p\u003e","title":"Your DELETE + INSERT Is Burning PostgreSQL’s CPU. Here’s What’s Actually Happening on the Page."},{"content":"Everything I wish someone had told me before I stored a meeting time in a timestamp column.\nThere is a specific kind of bug that only shows up in production, only in October, and only for users in one country. It gets closed as “not reproducible” twice before someone finally sits down with a coffee and reads the column definition.\nIt is almost always a datetime bug.\nThe frustrating part is that both Java and PostgreSQL have excellent date/time support. java.time is one of the best-designed APIs in the JDK. PostgreSQL\u0026rsquo;s temporal types are precise and well documented. The bugs happen at the seam — in the assumptions each side makes about what the other means, and in the two or three words in the type names that mean the opposite of what you\u0026rsquo;d guess.\nThis guide walks through both models, then spends real time on the part that actually causes outages: time zones.\nEvery code output and query result below was executed — Java 21 and PostgreSQL 16 — not written from memory.\nPart 1: The Java Side Ask three questions, get one type Every java.time class is an answer to some combination of three questions:\nDo I know the date? Do I know the time of day? Do I know which point on the global timeline this is? That last one is the important one, and it’s the one people skip. Here’s the map:\nThe single most useful sentence in this article: LocalDateTime is not a point in time. 2026-08-15T12:00 is twelve o\u0026rsquo;clock somewhere, and until you say where, it refers to a 26-hour-wide smear across the planet. If you have an Instant and you call .toLocalDateTime(), you have thrown away information and the compiler will not warn you.\nInstant is the boring, correct default Instant now = Instant.now(); // 2026-08-15T10:00:00.123456789Z An Instant is nanoseconds since the epoch. No zone, no offset, no calendar. When it prints with a Z, that isn\u0026rsquo;t a stored time zone — it\u0026rsquo;s the ISO-8601 way of rendering an epoch value.\nAnything that happened — a login, a payment, a row insert — is an Instant. Reach for something else only when you have a reason.\nZoneId vs ZoneOffset: not the same thing This distinction matters more than any other in this article.\nZoneId.of(\u0026#34;Europe/London\u0026#34;) // -\u0026gt; java.time.ZoneRegion ZoneId.of(\u0026#34;+01:00\u0026#34;) // -\u0026gt; java.time.ZoneOffset A ZoneOffset is a fixed number of hours and minutes from UTC. +01:00. That\u0026rsquo;s the whole thing. It\u0026rsquo;s a fact about one moment.\nA ZoneId (specifically a ZoneRegion, like Europe/London) is a rulebook. It knows that London was +00:00 in January, +01:00 in August, and it knows the exact instants at which it flips — going back decades and forward as far as legislators have declared.\nZoneOffset is a value. ZoneId is a function from instant to offset. Confusing them is the root cause of most scheduling bugs.\nOffsetDateTime vs ZonedDateTime Both identify a real instant. The difference is what survives.\nZonedDateTime nyc = ZonedDateTime.of(2026, 7, 4, 12, 0, 0, 0, ZoneId.of(\u0026#34;America/New_York\u0026#34;)); nyc // 2026-07-04T12:00-04:00[America/New_York] nyc.toOffsetDateTime() // 2026-07-04T12:00-04:00 nyc.toInstant() // 2026-07-04T16:00:00Z All three describe the same moment. But the ZonedDateTime still knows it\u0026rsquo;s in New York, so if you add six months to it, it will correctly land on -05:00. The OffsetDateTime has been flattened — it remembers that the offset was -04:00, not why. Add six months and you get a wrong wall-clock time.\nRule of thumb: ZonedDateTime for calculating, OffsetDateTime for transporting, Instant for storing.\nDuration vs Period LocalDate.of(2026, 1, 31).plusMonths(1) // 2026-02-28 Duration is exact elapsed time — seconds and nanos. Period is calendar time — years, months, days. \u0026ldquo;One month\u0026rdquo; is not a fixed number of seconds, and plusMonths clamps rather than overflowing. Both behaviors are correct; mixing them up is not.\nThis distinction resurfaces the moment DST enters the picture, and we’ll come back to it.\nA word on the old API java.util.Date is a mislabeled Instant (it has no time zone despite what toString() implies). Calendar is mutable and lenient. java.sql.Timestamp extends java.util.Date and adds nanoseconds while breaking the equals contract.\nIf you’re on a legacy codebase, convert at the boundary and never let them into your domain:\nInstant i = oldDate.toInstant(); Date d = Date.from(instant); Part 2: The PostgreSQL Side timestamptz does not store a time zone This is the single biggest source of confusion, and it comes straight from the type name. TIMESTAMP WITH TIME ZONE is an 8-byte value. So is TIMESTAMP WITHOUT TIME ZONE. There is no room in there for a zone, and PostgreSQL doesn\u0026rsquo;t try:\nSELECT pg_column_size(\u0026#39;2026-08-15 12:00:00\u0026#39;::timestamp) AS ts_bytes, -- 8 pg_column_size(\u0026#39;2026-08-15 12:00:00+02\u0026#39;::timestamptz) AS tstz_bytes, -- 8 pg_column_size(\u0026#39;12:00:00+02\u0026#39;::timetz) AS timetz_bytes; -- 12 What timestamptz actually does is convert to UTC on write and convert back on read. The offset you supplied on input is used and then discarded. From the official docs:\nIn either case, the value is stored internally as UTC, and the originally stated or assumed time zone is not retained.\nA better mental model: timestamptz is Instant. timestamp is LocalDateTime. The names lie; the semantics don\u0026rsquo;t.\nThe demo that makes it click Insert the same two literals into both column types, with the session zone set to London:\nSET TimeZone = \u0026#39;Europe/London\u0026#39;; CREATE TABLE t (id int, ts timestamp, tstz timestamptz); INSERT INTO t VALUES (1, \u0026#39;2026-08-15 12:00:00\u0026#39;, \u0026#39;2026-08-15 12:00:00\u0026#39;); INSERT INTO t VALUES (2, \u0026#39;2026-08-15 12:00:00+00\u0026#39;, \u0026#39;2026-08-15 12:00:00+00\u0026#39;); SELECT * FROM t; id | ts | tstz ----+---------------------+------------------------ 1 | 2026-08-15 12:00:00 | 2026-08-15 12:00:00+01 2 | 2026-08-15 12:00:00 | 2026-08-15 13:00:00+01 Two things happened. Row 1’s tstz had no offset, so PostgreSQL assumed the session zone. Row 2\u0026rsquo;s explicit +00 was honoured and re-rendered as 13:00+01. Meanwhile the ts column ignored the offset entirely and stored the digits.\nNow reconnect from New York and run the identical SELECT:\nid | ts | tstz ----+---------------------+------------------------ 1 | 2026-08-15 12:00:00 | 2026-08-15 07:00:00-04 2 | 2026-08-15 12:00:00 | 2026-08-15 08:00:00-04 The tstz column re-renders per client. The ts column is frozen. Neither row changed on disk.\nThat last point deserves emphasis, because it’s where the panic starts: when a developer sees a different value than QA saw, the instinct is “the data is corrupted.” It isn’t. Two clients with different TimeZone settings are looking at the same UTC value through different windows.\nAT TIME ZONE runs both ways AT TIME ZONE is the one operator you need, and its behavior flips depending on the input type:\n-- timestamptz -\u0026gt; timestamp: \u0026#34;what does the clock read there?\u0026#34; SELECT \u0026#39;2026-08-15 12:00:00+00\u0026#39;::timestamptz AT TIME ZONE \u0026#39;Asia/Tokyo\u0026#39;; -- 2026-08-15 21:00:00 (type: timestamp without time zone) -- timestamp -\u0026gt; timestamptz: \u0026#34;this wall clock reading WAS in Tokyo\u0026#34; SELECT \u0026#39;2026-08-15 12:00:00\u0026#39;::timestamp AT TIME ZONE \u0026#39;Asia/Tokyo\u0026#39;; -- 2026-08-15 03:00:00+00 (type: timestamp with time zone) One removes zone context, the other attaches it. Read it as “interpret this value in that zone.” The function form timezone(\u0026lsquo;Asia/Tokyo\u0026rsquo;, value) is identical.\nThe rest of the family date — 4 bytes, no zone, no ambiguity. Great for things that are genuinely dates. time — time of day, no zone. timetz — 12 bytes, and PostgreSQL itself recommends against it. A time of day with an offset but no date can\u0026rsquo;t resolve DST, so it\u0026rsquo;s not meaningfully anchored. It won\u0026rsquo;t even accept a zone name: SELECT \u0026#39;12:00:00 Europe/London\u0026#39;::timetz; -- ERROR: invalid input syntax for type time with time zone interval — 16 bytes storing months, days, and microseconds separately, which is exactly why it can be DST-aware. now() vs localtimestamp SELECT pg_typeof(now()), -- timestamp with time zone pg_typeof(current_timestamp), -- timestamp with time zone pg_typeof(localtimestamp), -- timestamp without time zone pg_typeof(current_date); -- date now() and current_timestamp are timestamptz and are safe. localtimestamp and current_date are evaluated in the session zone, which means a DEFAULT current_date column produces different answers for different clients. Be deliberate.\nSo which do you pick? The short version: use timestamptz for essentially everything. Reach for timestamp only when you genuinely mean a floating wall-clock time that must not shift — \u0026ldquo;the store opens at 09:00 local, whatever local means at that branch\u0026rdquo; — and store the zone ID beside it.\nPart 3: Time Zones, Properly This is where the real complexity lives, so let’s slow down.\nThree things called “time zone” An offset — +01:00. A number. An IANA zone ID — Europe/London. A named set of historical and future rules. PostgreSQL ships ~499 of these in pg_timezone_names; Java reads the same tzdata. An abbreviation — EST, BST, IST. Ambiguous, unversioned, and a trap. That third one is genuinely dangerous:\nSELECT \u0026#39;2026-08-15 12:00:00 EST\u0026#39;::timestamptz, -- 2026-08-15 17:00:00+00 \u0026#39;2026-08-15 12:00:00 America/New_York\u0026#39;::timestamptz; -- 2026-08-15 16:00:00+00 One hour apart. EST is a fixed -05:00 in PostgreSQL — it does not mean \u0026ldquo;Eastern time,\u0026rdquo; it means \u0026ldquo;Eastern Standard Time, always.\u0026rdquo; In August, New York is on -04:00. If you accept zone abbreviations from user input, you have already shipped a bug. (IST is worse — it can mean India, Ireland, or Israel depending on who you ask.)\nAlways use full IANA names.\nOffsets aren’t zones either. Nepal is +05:45; the Chatham Islands are +12:45:\nZonedDateTime.of(2026,8,15,12,0,0,0, ZoneId.of(\u0026#34;Asia/Kathmandu\u0026#34;)) // 2026-08-15T12:00+05:45[Asia/Kathmandu] If your API contract says “offset in whole hours,” it’s wrong.\nWhere PostgreSQL’s session zone comes from The TimeZone GUC is a per-connection setting. Its value is decided by, in order: an explicit SET TimeZone, the connection\u0026rsquo;s startup parameters, the postgresql.conf value, and finally the server\u0026rsquo;s OS zone.\nThat last fallback is why the same query returns different results on a developer laptop and in CI. And with connection pooling, SET TimeZone on a borrowed connection leaks to whoever gets it next.\nPin it. Set timezone = \u0026lsquo;UTC\u0026rsquo; in postgresql.conf, and add it to the JDBC URL so it survives pool reuse:\njdbc:postgresql://host:5432/db?options=-c%20TimeZone%3DUTC Then make the client responsible for rendering. Formatting a timestamp for a human is a presentation concern, not a database one.\nThe spring-forward gap On 29 March 2026, London jumps from 01:00 to 02:00. The local times between them do not exist.\nAsk Java for one anyway:\nLocalDateTime gap = LocalDateTime.of(2026, 3, 29, 1, 30); gap.atZone(ZoneId.of(\u0026#34;Europe/London\u0026#34;)); // 2026-03-29T02:30+01:00[Europe/London] ZoneId.of(\u0026#34;Europe/London\u0026#34;).getRules().getTransition(gap); // Transition[Gap at 2026-03-29T01:00Z to +01:00] Java doesn’t throw. It silently pushes the value forward by the length of the gap. PostgreSQL does the same thing:\nSET TimeZone=\u0026#39;Europe/London\u0026#39;; SELECT \u0026#39;2026-03-29 01:30:00\u0026#39;::timestamptz; -- 2026-03-29 02:30:00+01 Both are “correct” — but neither told you. If a user sets a 01:30 alarm on that date, you should be validating with getTransition() and telling them, not letting the platform quietly reschedule.\nThe fall-back overlap — where Java and PostgreSQL disagree On 25 October 2026, London goes from 02:00 back to 01:00. Local times between 01:00 and 02:00 happen twice, one hour apart.\nJava’s rules expose both:\nZoneId london = ZoneId.of(\u0026#34;Europe/London\u0026#34;); LocalDateTime overlap = LocalDateTime.of(2026, 10, 25, 1, 30); london.getRules().getValidOffsets(overlap); // [+01:00, Z] ZonedDateTime earlier = overlap.atZone(london); // 2026-10-25T01:30+01:00[Europe/London] instant = 2026-10-25T00:30:00Z ZonedDateTime later = earlier.withLaterOffsetAtOverlap(); // 2026-10-25T01:30Z[Europe/London] instant = 2026-10-25T01:30:00Z Java’s default for an ambiguous local time is the earlier offset — 00:30Z.\nNow ask PostgreSQL the same question:\nSELECT \u0026#39;2026-10-25 01:30:00\u0026#39;::timestamp AT TIME ZONE \u0026#39;Europe/London\u0026#39;; -- 2026-10-25 01:30:00+00 PostgreSQL picked 01:30Z — the later one.\nSame input, same tzdata, one hour apart. If your application resolves an ambiguous local time in Java and your reporting job resolves it in SQL, they will disagree for one hour, twice a year, in every DST zone.\nThere is no clever fix. The lesson is: never let a LocalDateTime cross the boundary and get resolved on the other side. Resolve to an Instant in exactly one place, with explicit overlap handling, and send the instant.\nDST arithmetic: plusDays is not plusHours(24) ZonedDateTime start = ZonedDateTime.of(2026, 3, 28, 12, 0, 0, 0, london); // 2026-03-28T12:00Z[Europe/London] start.plusDays(1); // 2026-03-29T12:00+01:00 (23 real hours) start.plus(Duration.ofDays(1)); // 2026-03-29T13:00+01:00 (24 real hours) start.plusHours(24); // 2026-03-29T13:00+01:00 plusDays preserves the wall clock. Duration preserves elapsed time. PostgreSQL draws the same line, using the same value:\nSET TimeZone=\u0026#39;Europe/London\u0026#39;; SELECT \u0026#39;2026-03-28 12:00:00+00\u0026#39;::timestamptz + interval \u0026#39;1 day\u0026#39;, -- 2026-03-29 12:00:00+01 \u0026#39;2026-03-28 12:00:00+00\u0026#39;::timestamptz + interval \u0026#39;24 hours\u0026#39;; -- 2026-03-29 13:00:00+01 This is one place the two platforms agree beautifully — interval \u0026lsquo;1 day\u0026rsquo; and plusDays(1) mean the same thing, and interval \u0026lsquo;24 hours\u0026rsquo; and Duration.ofDays(1) mean the same thing. \u0026ldquo;Every day at 9am\u0026rdquo; is plusDays / interval \u0026lsquo;1 day\u0026rsquo;. \u0026ldquo;Every 24 hours\u0026rdquo; is Duration / interval \u0026lsquo;24 hours\u0026rsquo;. Pick the one the product actually means.\nFuture events: an instant is not enough ZoneId ny = ZoneId.of(\u0026#34;America/New_York\u0026#34;); ZonedDateTime.of(2027, 1, 15, 9, 0, 0, 0, ny).withZoneSameInstant(UTC); // 2027-01-15T14:00Z ZonedDateTime.of(2027, 7, 15, 9, 0, 0, 0, ny).withZoneSameInstant(UTC); // 2027-07-15T13:00Z “9am in New York” is a different UTC instant depending on the date. So if a user schedules a recurring 9am standup and a government later changes DST rules — which happens several times a year somewhere — the stored instant becomes wrong.\nFor past events, store the instant. That’s history; it can’t change.\nFor future user-scheduled events, store three things:\nCREATE TABLE meetings ( id bigserial PRIMARY KEY, local_start timestamp NOT NULL, -- 2027-01-15 09:00 zone_id text NOT NULL, -- \u0026#39;America/New_York\u0026#39; computed_utc timestamptz NOT NULL -- cache; recompute on tzdata update ); The first two are the truth. The third is a derived index for querying, refreshed when tzdata changes.\ntzdata is a moving target Java bundles a copy of the IANA database (2025b on the JDK I ran this on); PostgreSQL bundles its own; your OS has a third. They drift. When Java and PostgreSQL are on different tzdata versions, they will resolve recently-changed zones differently.\nCheck what you have:\nZoneRulesProvider.getVersions(\u0026#34;Europe/London\u0026#34;).lastKey(); // 2025b SELECT name, utc_offset, is_dst FROM pg_timezone_names WHERE name = \u0026#39;Europe/London\u0026#39;; Pin tzdata in your base images, and treat a tzdata bump as a deployment that requires recomputing cached future instants.\nPart 4: Wiring Java to PostgreSQL The mapping table Per the pgJDBC documentation:\nNote the omission, in the driver’s own words:\nZonedDateTime, Instant and OffsetTime / TIME WITH TIME ZONE are not supported.\nSo don’t hand the driver an Instant and hope — depending on the driver version you\u0026rsquo;ll either get a Can\u0026rsquo;t infer the SQL type to use for an instance of java.time.Instant exception or undocumented behavior. Convert explicitly at the boundary instead:\n// write ps.setObject(1, instant.atOffset(ZoneOffset.UTC)); // read Instant i = rs.getObject(\u0026#34;occurred_at\u0026#34;, OffsetDateTime.class).toInstant(); Note that reading a timestamptz gives you an OffsetDateTime normalized to UTC — the driver\u0026rsquo;s offset is always Z, because that\u0026rsquo;s what the backend holds. The offset carries no information. Convert straight to Instant and don\u0026rsquo;t be tempted to read meaning into it.\nAvoid setTimestamp / getTimestamp The java.sql.Timestamp overloads convert using the JVM default time zone unless you pass an explicit Calendar. That makes your persistence layer\u0026rsquo;s correctness depend on the container\u0026rsquo;s TZ environment variable. Use the java.time setObject/getObject methods instead — they\u0026rsquo;re unambiguous.\nIf you must keep the JVM zone from mattering at all, set it explicitly at startup:\n-Duser.timezone=UTC Belt and braces, not a substitute for correct types.\nPrecision: Java has nanos, PostgreSQL has micros Instant.parse(\u0026#34;2026-08-15T10:00:00.123456789Z\u0026#34;) // .123456789 .truncatedTo(ChronoUnit.MICROS) // .123456 PostgreSQL stores microseconds. Those last three digits are gone on the round trip, and PostgreSQL may round rather than truncate:\nSELECT \u0026#39;2026-08-15 10:00:00.123456789\u0026#39;::timestamptz; -- 10:00:00.123457 If you have an equality test or an optimistic-lock check comparing an in-memory Instant to one that\u0026rsquo;s been through the database, truncate to micros before you persist so both sides agree.\nDon’t wrap the column in a function Because timestamptz is stored as UTC, \u0026ldquo;give me everything that happened on 1 Feb in New York\u0026rdquo; tempts you into this:\nEXPLAIN SELECT count(*) FROM events WHERE (occurred_at AT TIME ZONE \u0026#39;America/New_York\u0026#39;)::date = DATE \u0026#39;2026-02-01\u0026#39;; -- Seq Scan on events Index gone. Compute the boundary instants instead and do a half-open range scan:\nSELECT (DATE \u0026#39;2026-02-01\u0026#39; + time \u0026#39;00:00\u0026#39;) AT TIME ZONE \u0026#39;America/New_York\u0026#39;, -- 2026-02-01 05:00+00 (DATE \u0026#39;2026-02-02\u0026#39; + time \u0026#39;00:00\u0026#39;) AT TIME ZONE \u0026#39;America/New_York\u0026#39;; -- 2026-02-02 05:00+00 EXPLAIN SELECT count(*) FROM events WHERE occurred_at \u0026gt;= \u0026#39;2026-02-01 05:00:00+00\u0026#39; AND occurred_at \u0026lt; \u0026#39;2026-02-02 05:00:00+00\u0026#39;; -- Index Only Scan using events_occurred_at_idx Note that this “day” is 24 hours only because February has no DST transition. On a transition day it’s 23 or 25 — which the boundary-instant approach handles correctly and a naive + interval \u0026lsquo;24 hours\u0026rsquo; does not. Half-open ranges (\u0026gt;= and \u0026lt;) also dodge the microsecond-boundary problem that BETWEEN creates.\nPart 5: JPA and Hibernate Hibernate 6 made zoned types a first-class concern via TimeZoneStorageType:\nFor PostgreSQL, the practical consequence is settled by the database, not the ORM: timestamptz discards the offset, so under DEFAULT your OffsetDateTime round-trips as UTC. Which is fine — as long as you expected it.\nSensible defaults:\nspring: jpa: properties: hibernate: jdbc: time_zone: UTC timezone: default_storage: NORMALIZE_UTC And in the entity, be explicit about intent:\n@Entity public class Booking { // A moment. Maps to timestamptz. @Column(nullable = false) private Instant createdAt; // A date. Not a moment. @Column(nullable = false) private LocalDate serviceDate; // A wall-clock time that must not shift. Maps to timestamp. @Column(nullable = false) private LocalDateTime localStart; @Column(nullable = false) private String zoneId; // \u0026#39;Europe/London\u0026#39; public ZonedDateTime start() { return localStart.atZone(ZoneId.of(zoneId)); } } If you genuinely need the user’s original offset preserved, @TimeZoneStorage(TimeZoneStorageType.COLUMN) with @TimeZoneColumn writes it to a companion column. Use it when the offset is data — an audit trail of what the user\u0026rsquo;s device claimed — not as a default.\nPart 6: Testing This Stuff Datetime bugs are only cheap if you catch them before the clocks change. Three habits do most of the work.\n1. Inject a Clock Instant.now() scattered through business logic is untestable. Inject java.time.Clock and you can move time at will:\n@Bean Clock clock() { return Clock.systemUTC(); } // in a test Clock fixed = Clock.fixed(Instant.parse(\u0026#34;2026-10-25T00:30:00Z\u0026#34;), ZoneOffset.UTC); 2. Run CI in a hostile time zone If your JVM, your database, and your CI runner are all UTC, every zone bug hides until production. Force one of them somewhere awkward:\n-Duser.timezone=Pacific/Chatham +12:45 and it observes DST. If your test suite survives Chatham, it will survive most things. Do the same for the PostgreSQL container:\n@Container static PostgreSQLContainer\u0026lt;?\u0026gt; pg = new PostgreSQLContainer\u0026lt;\u0026gt;(\u0026#34;postgres:16\u0026#34;) .withEnv(\u0026#34;TZ\u0026#34;, \u0026#34;Asia/Kathmandu\u0026#34;) .withUrlParam(\u0026#34;options\u0026#34;, \u0026#34;-c TimeZone=Asia/Kathmandu\u0026#34;); The point isn’t that production runs there — it’s that a test which only passes in UTC is a test that has proven nothing about zone handling.\n3. Write the four transition tests For any code that converts between local and instant, pin these explicitly:\nstatic final ZoneId LONDON = ZoneId.of(\u0026#34;Europe/London\u0026#34;); @Test void gapTimesAreRejectedNotSilentlyShifted() { LocalDateTime gap = LocalDateTime.of(2026, 3, 29, 1, 30); assertThat(LONDON.getRules().getValidOffsets(gap)).isEmpty(); assertThatThrownBy(() -\u0026gt; scheduler.schedule(gap, LONDON)) .isInstanceOf(NonExistentLocalTimeException.class); } @Test void overlapTimesResolveToTheDocumentedChoice() { LocalDateTime overlap = LocalDateTime.of(2026, 10, 25, 1, 30); assertThat(LONDON.getRules().getValidOffsets(overlap)).hasSize(2); assertThat(scheduler.resolve(overlap, LONDON)) .isEqualTo(Instant.parse(\u0026#34;2026-10-25T00:30:00Z\u0026#34;)); // earlier, by policy } @Test void dailyRecurrenceKeepsWallClockAcrossDst() { ZonedDateTime start = ZonedDateTime.of(2026, 3, 28, 9, 0, 0, 0, LONDON); assertThat(recurrence.next(start).toLocalTime()).isEqualTo(LocalTime.of(9, 0)); } @Test void instantSurvivesTheDatabaseRoundTrip() { Instant original = Instant.now().truncatedTo(ChronoUnit.MICROS); repository.save(new Event(original)); assertThat(repository.findLatest().occurredAt()).isEqualTo(original); } The fourth one catches more real bugs than you’d expect — it’s the one that fails when someone quietly changes a column from timestamptz to timestamp.\nThe Short Version If you skim one section, make it this one.\nSchema\ntimestamptz by default. It\u0026rsquo;s the same 8 bytes and it\u0026rsquo;s unambiguous. timestamp only for wall-clock times that must not shift — and store the zone ID beside it. date for dates. timetz for nothing. Pin timezone = \u0026lsquo;UTC\u0026rsquo; in postgresql.conf and in the JDBC URL. Java\nInstant for things that happened. LocalDate for dates. ZonedDateTime for scheduling. OffsetDateTime at the JDBC boundary. LocalDateTime is not a point in time. Treat every use as a question to answer, not an answer. Never accept zone abbreviations. Full IANA names only. plusDays ≠ plus(Duration.ofDays(1)). Know which one your product means. The seam\npgJDBC speaks LocalDate/LocalTime/LocalDateTime/OffsetDateTime. Convert Instant explicitly. Avoid setTimestamp/getTimestamp — they use the JVM default zone. Truncate to microseconds before persisting. Don’t wrap indexed timestamp columns in AT TIME ZONE; compute boundary instants instead. And the one that bites hardest\nJava resolves an ambiguous local time to the earlier offset. PostgreSQL picks the later one. Resolve ambiguity in exactly one place, deliberately, and pass instants across the boundary — never local times.\nStore the instant, render the local time, and keep the zone rules in exactly one layer. Most datetime bugs are just that rule, broken somewhere.\nAll examples verified on OpenJDK 21.0.10 (tzdata 2025b) and PostgreSQL 16.13.\nSources\nPostgreSQL: Date/Time Types pgJDBC: Using Java 8 Date and Time classes Hibernate TimeZoneStorageType Javadoc Hibernate 6 migration guide Thorben Janssen: TimezoneStorageType ","permalink":"https://sukachvitally.github.io/posts/datetime-handling-in-java-and-postgresql-a-practical-guide/","summary":"\u003cp\u003e\u003cem\u003eEverything I wish someone had told me before I stored a meeting time in a\u003c/em\u003e \u003cem\u003etimestamp column.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/datetime-handling-in-java-and-postgresql-a-practical-guide/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eThere is a specific kind of bug that only shows up in production, only in October, and only for users in one country. It gets closed as “not reproducible” twice before someone finally sits down with a coffee and reads the column definition.\u003c/p\u003e\n\u003cp\u003eIt is almost always a datetime bug.\u003c/p\u003e\n\u003cp\u003eThe frustrating part is that both Java and PostgreSQL have \u003cem\u003eexcellent\u003c/em\u003e date/time support. java.time is one of the best-designed APIs in the JDK. PostgreSQL\u0026rsquo;s temporal types are precise and well documented. The bugs happen at the seam — in the assumptions each side makes about what the other means, and in the two or three words in the type names that mean the opposite of what you\u0026rsquo;d guess.\u003c/p\u003e","title":"DateTime Handling in Java and PostgreSQL: A Practical Guide"},{"content":"\nWhen a migration hangs, a query slows to a crawl, or a deploy just sits there, most people reach for the same first move: kill it and try again. That works about as often as it doesn’t. Postgres already knows what’s wrong — it’s just keeping the answer in a handful of system catalogs and stats views that almost nobody looks at until things are already on fire.\nThis is a walkthrough of the tables I actually check, in the order I check them, when something breaks — with a running example: a migration that adds a NOT NULL column and backfills it on a 20-million-row orders table, which has been \u0026ldquo;running\u0026rdquo; for 25 minutes with no sign of finishing.\nStart with: is it even running? Before touching statistics or query plans, find out if the process is alive, blocked, or just slow.\nSELECT pid, state, wait_event_type, wait_event, query, query_start FROM pg_stat_activity WHERE state != \u0026#39;idle\u0026#39;; pg_stat_activity is a live snapshot of every connection to the database. The columns that matter most here are wait_event_type and wait_event. If a process shows wait_event_type = \u0026lsquo;Lock\u0026rsquo;, it isn\u0026rsquo;t broken — it\u0026rsquo;s queued behind something else. If it shows Client or IO, that\u0026rsquo;s a different problem entirely (waiting on the client, or waiting on disk).\nFor the orders migration, this comes back with:\npid | state | wait_event_type | wait_event | query -----+--------+-----------------+------------+---------------------------------- 8842 | active | Lock | relation | ALTER TABLE orders ADD COLUMN ... wait_event_type = Lock tells us immediately: the migration isn\u0026rsquo;t slow, it\u0026rsquo;s stuck behind something else that holds a conflicting lock on orders.\nIf you see a lock wait, the next stop is obvious:\nSELECT blocked_locks.pid AS blocked_pid, blocking_locks.pid AS blocking_pid, blocked_activity.query AS blocked_query, blocking_activity.query AS blocking_query FROM pg_locks blocked_locks JOIN pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid JOIN pg_locks blocking_locks ON blocking_locks.locktype = blocked_locks.locktype AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation AND blocking_locks.pid != blocked_locks.pid JOIN pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid WHERE NOT blocked_locks.granted; This join is the single most useful query in Postgres debugging. It tells you exactly which session is blocking which, and what each of them is running. Most “stuck migration” incidents end here: some other transaction — often a long-running read, an ORM that forgot to close a connection, or a previous deploy that never finished — is sitting on a lock the migration needs.\nRunning it for the orders case turns up the problem:\nblocked_pid | blocking_pid | blocked_query | blocking_query ------------+--------------+---------------------------------+--------------------------------------------- 8842 | 6110 | ALTER TABLE orders ADD COLUMN.. | SELECT * FROM orders WHERE ... (idle in txn) PID 6110 is an analytics job that opened a transaction, ran a read, and never committed — “idle in transaction” — leaving an ACCESS SHARE lock on orders that blocks the migration\u0026rsquo;s ACCESS EXCLUSIVE lock. The fix here isn\u0026rsquo;t in the migration at all: it\u0026rsquo;s killing or fixing PID 6110 (SELECT pg_terminate_backend(6110); if you\u0026rsquo;re sure it\u0026rsquo;s safe to end it).\nIf it’s a long-running operation, check its actual progress Postgres exposes live progress for the operations that most commonly make migrations feel frozen:\nSELECT * FROM pg_stat_progress_create_index; SELECT * FROM pg_stat_progress_vacuum; SELECT * FROM pg_stat_progress_analyze; If you’re running CREATE INDEX CONCURRENTLY as part of a migration, pg_stat_progress_create_index shows you the current phase (building index, index validation, etc.) and how many tuples it has scanned versus the total. This turns \u0026ldquo;is this hung or just big?\u0026rdquo; from a guess into a fact.\nSay the next step of the migration adds an index on the new column instead of blocking on a lock — the progress view might show:\npid | phase | tuples_total | tuples_done -----+-------------------------------+--------------+------------- 9021 | building index: scanning table| 20450112 | 14300556 That’s roughly 70% through a single scan phase, moving steadily — not stuck, just genuinely working through 20 million rows. Compare that against a snapshot taken a minute later: if tuples_done hasn\u0026rsquo;t moved, now you know it actually is stalled, and it\u0026rsquo;s worth going back to pg_locks.\nIf nothing’s blocked, suspect the query plan Not every broken migration is a lock problem. Sometimes the migration runs, but a backfill step or an added index makes the planner choose a catastrophically bad plan. That’s a statistics problem, and it lives in two places.\npg_stat_user_tables tells you whether Postgres has been keeping its stats up to date on the table in question:\nSELECT relname, n_live_tup, n_dead_tup, n_mod_since_analyze, last_vacuum, last_autovacuum, last_analyze, last_autoanalyze FROM pg_stat_user_tables WHERE relname = \u0026#39;your_table\u0026#39;; If n_mod_since_analyze is large relative to n_live_tup, or last_autoanalyze is suspiciously old, the planner is working off outdated information. This happens constantly during migrations, because a migration that inserts, updates, or deletes a large chunk of a table changes its shape faster than autovacuum\u0026rsquo;s default thresholds react to.\nFor orders, after the backfill step runs:\nrelname | n_live_tup | n_dead_tup | n_mod_since_analyze | last_autoanalyze --------+------------+------------+---------------------+------------------- orders | 20450112 | 3800000 | 18900000 | 2026-07-28 03:14 18.9 million modified rows since the last analyze, against 20 million live rows — nearly the entire table has changed and autovacuum hasn’t caught up yet (last_autoanalyze is five days old). Any query the migration runs after this point is planning against numbers that are effectively meaningless.\npg_stats (the readable view over the raw pg_statistic catalog) shows what the planner actually believes about each column:\nSELECT attname, n_distinct, correlation, null_frac, most_common_vals FROM pg_stats WHERE tablename = \u0026#39;your_table\u0026#39;; n_distinct and correlation drive index-vs-sequential-scan decisions. If a migration adds a new index and the planner still refuses to use it, stale stats here are the usual suspect.\nOn the new orders.status column right after the backfill, before an ANALYZE:\nattname | n_distinct | correlation | null_frac ---------+------------+-------------+----------- status | 1 | 1 | 1.0 The planner still believes every row is null (null_frac = 1.0) because that was true before the backfill ran. Any query filtering on status will get a wildly wrong row estimate and likely pick a sequential scan or a bad join order. The fix is almost always to run ANALYZE your_table; — either before the migration, or as an explicit step inside it, rather than waiting for autovacuum to get around to it. After ANALYZE orders;, the same query returns something like n_distinct = 4, null_frac = 0.0 — matching reality, and the planner\u0026rsquo;s decisions follow accordingly.\npg_class holds the planner\u0026rsquo;s cached size estimates directly:\nSELECT relname, reltuples, relpages FROM pg_class WHERE relname = \u0026#39;your_table\u0026#39;; reltuples is an estimate, not a live count — it\u0026rsquo;s only refreshed by ANALYZE or VACUUM. If this number is wildly off from the real row count, every cost estimate downstream is wrong too. For orders, reltuples might read 20450112 while a SELECT count(*) returns closer to 24000000 — a four-million-row gap that skews every cost estimate the planner builds on top of it.\nIf you need the exact query and its cost, not just the table pg_stat_statements (an extension, so it needs to be enabled) aggregates every distinct query the server has run, with real timing:\nSELECT query, calls, total_exec_time, mean_exec_time, rows FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 20; This is the difference between “the migration is slow” and “this specific UPDATE inside the migration is slow, and it’s been called 40,000 times.” For the orders backfill, this might show:\nquery | calls | mean_exec_time | rows ---------------------------------------------+-------+----------------+-------- UPDATE orders SET status = $1 WHERE id = $2 | 40000 | 812.4 ms | 40000 40,000 single-row updates averaging 800ms each explains a lot on its own — that’s batching the backfill row-by-row instead of in bulk, and it would still be slow even with perfect statistics. If it’s enabled in your environment, it’s usually the fastest way to go from vague suspicion to a concrete line of SQL to fix — in this case, rewriting the backfill as a single set-based UPDATE \u0026hellip; WHERE status IS NULL in batches, rather than one row at a time.\nA rough order of operations Put together, a sensible debugging sequence looks like this: check pg_stat_activity to see if the process is alive and what it\u0026rsquo;s waiting on; if it\u0026rsquo;s waiting on a lock, join into pg_locks to find the blocker; if it\u0026rsquo;s a long index build or vacuum, check the matching pg_stat_progress_* view for real progress; if nothing is blocked but it\u0026rsquo;s still slow, check pg_stat_user_tables and pg_stats for stale statistics, and pg_class for stale row estimates; and if pg_stat_statements is available, use it to find the exact query eating the time.\nNone of these tables are exotic. They ship with every Postgres install (aside from pg_stat_statements, which just needs CREATE EXTENSION). The only real skill here is knowing which one answers which question — and reaching for pg_stat_activity and pg_locks before assuming it\u0026rsquo;s a statistics problem, since lock contention is a far more common cause of a \u0026ldquo;broken\u0026rdquo; migration than a bad query plan.\n","permalink":"https://sukachvitally.github.io/posts/debugging-postgres-the-system-tables-that-actually-tell-you-whats-wrong/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/debugging-postgres-the-system-tables-that-actually-tell-you-whats-wrong/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eWhen a migration hangs, a query slows to a crawl, or a deploy just sits there, most people reach for the same first move: kill it and try again. That works about as often as it doesn’t. Postgres already knows what’s wrong — it’s just keeping the answer in a handful of system catalogs and stats views that almost nobody looks at until things are already on fire.\u003c/p\u003e\n\u003cp\u003eThis is a walkthrough of the tables I actually check, in the order I check them, when something breaks — with a running example: a migration that adds a NOT NULL column and backfills it on a 20-million-row orders table, which has been \u0026ldquo;running\u0026rdquo; for 25 minutes with no sign of finishing.\u003c/p\u003e","title":"Debugging Postgres: The System Tables That Actually Tell You What’s Wrong"},{"content":"Build phases, task graphs, dependency resolution, and what actually happens when the JVM opens your JAR\nYou type gradle build, wait a few seconds, and out comes a JAR. Most Java developers use this pipeline every day without ever looking inside it. But when a build gets slow, a dependency conflict appears, or a NoClassDefFoundError shows up in production at 2 AM, understanding the machinery underneath is the difference between guessing and fixing.\nThis article walks the full journey: Gradle’s three build phases, the Java plugin’s task graph, how dependency resolution actually works stage by stage, what a JAR file really is, and how the JVM reads and executes it.\nPart 1: The Three Phases Every Gradle Build Goes Through Before a single line of your code is compiled, Gradle itself runs through a fixed lifecycle of three phases. This is true for every build, every plugin, every project.\nPhase 1 — Initialization Gradle reads settings.gradle(.kts) and answers one question: which projects are part of this build? For a single-module app that\u0026rsquo;s trivial; for a multi-module monorepo this is where include(\u0026quot;:core\u0026quot;, \u0026ldquo;:api\u0026rdquo;, \u0026ldquo;:app\u0026rdquo;) gets evaluated and a Project object is created for each module.\nPhase 2 — Configuration Now every build.gradle(.kts) script executes top to bottom. This is the phase people misunderstand most. Your build script is code that runs — and it runs here, on every invocation, regardless of which task you asked for. The output of this phase is the task graph: a directed acyclic graph (DAG) of all tasks and their dependencies. Tasks are registered and wired together, but none of their actions execute yet.\nA println at the top level of build.gradle prints during Configuration — even if you run gradle help. Task actions run later. This trips up nearly everyone once.\nPhase 3 — Execution Gradle takes the task you requested, computes the minimal subgraph of tasks needed to produce it, and executes them in dependency order. Tasks whose inputs and outputs haven’t changed since the last run are marked UP-TO-DATE and skipped entirely — this is Gradle\u0026rsquo;s incremental build engine, and it\u0026rsquo;s why the second build is always faster than the first.\nPart 2: The Java Plugin Task Graph Applying the java plugin registers a standard chain of tasks. When you run gradle build, this is the graph that gets executed:\nThe practical shortcuts worth memorizing:\n# fastest feedback: compile only gradle compileJava # package without running tests gradle jar # full build, skip test execution gradle build -x test # nuke build/ and start clean gradle clean build Part 3: How Gradle Dependency Resolution Actually Works This is the part of Gradle most developers treat as magic. You write three lines in a dependencies block, and somehow the right JARs appear on your classpath. Here\u0026rsquo;s the machinery, stage by stage.\nConfigurations: buckets of dependencies Every dependency you declare goes into a named configuration — a bucket with specific rules about where those dependencies are visible:\ndependencies { implementation \u0026#39;org.springframework.boot:spring-boot-starter-web:3.3.1\u0026#39; api \u0026#39;com.fasterxml.jackson.core:jackson-databind:2.17.1\u0026#39; compileOnly \u0026#39;org.projectlombok:lombok:1.18.32\u0026#39; annotationProcessor \u0026#39;org.projectlombok:lombok:1.18.32\u0026#39; runtimeOnly \u0026#39;org.postgresql:postgresql:42.7.3\u0026#39; testImplementation \u0026#39;org.junit.jupiter:junit-jupiter:5.10.2\u0026#39; } The implementation vs api distinction matters in multi-module builds: implementation dependencies don\u0026rsquo;t appear on the compile classpath of modules that depend on yours. That means fewer recompilations when they change, and a cleaner API boundary. Default to implementation; reach for api only when a type from that dependency appears in your public method signatures.\nThe resolution stages Declared dependencies are just coordinates on paper. Resolution is the process of turning them into actual files, and it happens lazily — only when a task first needs a classpath (typically compileJava during Execution). The stages:\nStage 1 — Graph construction. For each declared coordinate (group:name:version), Gradle queries your repositories in the order they\u0026rsquo;re declared and downloads only the metadata first: the POM file or Gradle Module Metadata. That metadata lists the dependency\u0026rsquo;s own dependencies, so Gradle recursively walks outward until the full transitive graph is known. No JARs have been downloaded yet — just descriptors.\nStage 2 — Conflict resolution. Real dependency graphs always contain the same library requested at different versions (your app wants Jackson 2.17, some transitive path wants 2.15). Gradle’s default strategy is highest version wins — unlike Maven, which picks the “nearest” declaration in the tree. This default is usually right, but you can override it:\nconfigurations.all { resolutionStrategy { // hard-pin a version, overriding everything force \u0026#39;com.fasterxml.jackson.core:jackson-databind:2.17.1\u0026#39; // or turn conflicts into build failures failOnVersionConflict() } } Stage 3 — Constraints and rules. Before the graph is finalized, Gradle applies platforms (BOMs), version constraints, exclusions, and dependency substitutions. This is how Spring Boot keeps hundreds of libraries version-aligned with one line:\ndependencies { implementation platform(\u0026#39;org.springframework.boot:spring-boot-dependencies:3.3.1\u0026#39;) // now version numbers can be omitted — the BOM supplies them implementation \u0026#39;org.springframework.boot:spring-boot-starter-web\u0026#39; } Exclusions cut unwanted transitive branches at the declaration site:\nimplementation(\u0026#39;org.springframework.boot:spring-boot-starter-web\u0026#39;) { exclude group: \u0026#39;org.springframework.boot\u0026#39;, module: \u0026#39;spring-boot-starter-tomcat\u0026#39; } Stage 4 — Artifact download. Only now, with the final graph settled, does Gradle download the actual JAR files — and only for modules that survived resolution. Everything lands in the shared cache at ~/.gradle/caches/modules-2, keyed by checksum, shared across all projects on your machine. A dependency is downloaded once, ever, per machine (dynamic versions like 1.+ and snapshots get re-checked periodically).\nStage 5 — Classpath assembly. The resolved files are ordered into a classpath and handed to whichever task requested them: compileClasspath to javac, runtimeClasspath to the test JVM, and so on.\nDebugging resolution Two commands answer 90% of “why is this version on my classpath” questions:\n# full resolved tree per configuration gradle dependencies --configuration runtimeClasspath # trace one library: who pulls it in, which versions were requested, # and why the winner won gradle dependencyInsight --dependency jackson-databind Part 4: What a JAR File Really Is The output of the jar task is nothing exotic — it\u0026rsquo;s a ZIP archive with an agreed-upon internal layout. Rename it to .zip and any archive tool will open it:\nmyapp.jar ├── META-INF/ │ └── MANIFEST.MF ← metadata ├── com/example/App.class ← bytecode, path = package name ├── com/example/util/Helper.class └── application.properties ← resources sit alongside classes The manifest is a plain-text key-value file. The line that matters most for a runnable JAR:\nManifest-Version: 1.0 Main-Class: com.example.App One important caveat: the plain jar task packages only your classes. Your dependencies are not inside. For a self-contained deployable artifact you need a fat JAR — with the Shadow plugin, or Spring Boot\u0026rsquo;s bootJar, which nests dependency JARs and adds its own launcher.\nplugins { id \u0026#39;com.github.johnrengelman.shadow\u0026#39; version \u0026#39;8.1.1\u0026#39; } // gradle shadowJar → one JAR containing everything Part 5: How the JVM Reads Your JAR You run java -jar myapp.jar. Here\u0026rsquo;s the sequence, and it\u0026rsquo;s more interesting than it looks.\nStep 1 — The launcher reads the manifest The java launcher opens the JAR as a ZIP, reads META-INF/MANIFEST.MF, and extracts Main-Class. Note a classic gotcha: with -jar, any -cp argument on the command line is silently ignored. The JAR itself (plus manifest Class-Path entries) becomes the entire application classpath.\nStep 2 — The classloader hierarchy Classes are loaded by a chain of classloaders using parent-first delegation: every load request travels up to the top of the chain first, and only comes back down if no ancestor can satisfy it.\nStep 3 — Lazy loading The JVM does not load every class in the JAR at startup. A class is loaded the first time it’s actively used — instantiated, statically referenced, reflected upon. This is why a missing transitive dependency can let your app boot cleanly and then blow up with NoClassDefFoundError twenty minutes later, the first time a rarely-used code path runs.\nStep 4 — Load, Link, Initialize Per the JVM specification, every class goes through three phases before it’s usable:\nLoading — the class bytes are read from the JAR, parsed, and a Class object is created in metaspace. Linking — three sub-steps: Verification: the bytecode is checked for structural and type safety (stack shapes, valid jumps, access rules). Preparation: static fields are allocated and set to defaults — 0, null, false. Not your values yet. Resolution: symbolic references to other classes, methods, and fields are resolved to concrete ones (often lazily). Initialization — static initializer blocks and static field assignments run, in source order, exactly once, on first active use. This is the point where static { \u0026hellip; } blocks fire. Step 5 — main() and the JIT Once Main-Class is loaded, linked, and initialized, the JVM invokes main(String[] args) on the main thread. Execution starts in the interpreter. As methods get hot — called thousands of times — the JIT compiler (tiered: C1 first for fast, lightly-optimized code, then C2 for aggressively optimized code) compiles them to native machine code. That\u0026rsquo;s why JVM applications famously get faster after warm-up.\nThe Whole Picture src/main/java ──javac──▶ build/classes ──jar──▶ myapp.jar │ java -jar myapp.jar │ read MANIFEST.MF → find Main-Class │ classloader delegation (parent-first) │ Load → Link (verify/prepare/resolve) → Init │ main() runs, classes load lazily, JIT compiles hot paths as you go None of this is knowledge you need daily. But the day your build takes four minutes instead of forty seconds, or two versions of the same library fight on your classpath, or a class mysteriously fails to load in production — this is the map you’ll be reading. Better to have seen it before that day arrives.\n","permalink":"https://sukachvitally.github.io/posts/from-source-to-running-jvm-how-gradle-builds-your-java-application/","summary":"\u003cp\u003eBuild phases, task graphs, dependency resolution, and what actually happens when the JVM opens your JAR\u003c/p\u003e\n\u003cp\u003eYou type gradle build, wait a few seconds, and out comes a JAR. Most Java developers use this pipeline every day without ever looking inside it. But when a build gets slow, a dependency conflict appears, or a NoClassDefFoundError shows up in production at 2 AM, understanding the machinery underneath is the difference between guessing and fixing.\u003c/p\u003e","title":"From Source to Running JVM: How Gradle Builds Your Java Application"},{"content":"How Claude loads specialised instructions on demand — and how you can write your own to transform workflows forever.\nEvery time you open Claude and ask it to create a Word document, build a data analysis pipeline, or write presentation slides — something invisible happens first. Claude scans a directory of skills, finds the most relevant one, reads it, and only then starts working.\nThis system is called Claude Skills. It is one of the most underappreciated features in the entire Claude ecosystem. And once you understand how it works, you will see why it changes everything about how you build AI-powered workflows.\n“Skills are how Claude learns domain-specific workflows without retraining. They are runtime instructions, not model weights.”\nWhat Is a Skill? A skill is a directory containing a SKILL.md file (and optionally supporting assets, scripts, and reference documents). When Claude starts handling a request, it reads the name and description of every available skill. If a description matches the current task, Claude loads the full SKILL.md body and follows its instructions.\nThink of it like a context-sensitive manual. When a carpenter picks up a dovetail saw, they know exactly how to use it — not because they memorised every detail about every tool upfront, but because they learned the right technique for the right moment. Skills work the same way for Claude.\nThe Three-Layer Loading System Skills use progressive disclosure — Claude never loads everything at once. There are three distinct layers, each loaded only as needed:\nThis is elegant engineering. Layer 1 is tiny — just enough for Claude to decide whether a skill applies. Layer 2 gives full instructions once relevant. Layer 3 provides deep reference material only when needed, keeping context usage lean.\nAnatomy of a Skill Directory A skill is a folder with a predictable structure. Here is the full layout:\nOnly SKILL.md is required. Everything else is optional and only loaded when the skill itself explicitly references it.\nInside SKILL.md — The Configuration Fields The file starts with a YAML frontmatter block (between \u0026mdash; delimiters), followed by markdown instructions. Here is every supported field:\nKEY INSIGHTThe description field is the most important part of any skill. Claude decides whether to load your skill based solely on this field — the body never loads unless the description matches. Write it with real triggers, real contexts, and a slight \u0026ldquo;push\u0026rdquo; to combat under-triggering.\nA Minimal Skill — Step by Step Let’s build a real skill from scratch. Imagine you regularly need to convert raw PostgreSQL query results into nicely formatted Markdown reports. Here is how to package that as a skill.\nStep 1 — Create the directory mkdir pg-report-skill cd pg-report-skill touch SKILL.md Step 2 — Write the frontmatter --- name: pg-report description: | Converts raw PostgreSQL query results into formatted Markdown reports with tables, summary stats, and query metadata. Use whenever the user pastes SQL output, mentions \u0026#34;format this query result\u0026#34;, or wants to turn database output into a readable report or document. Also trigger when user asks to summarise a dataset from a .csv or .tsv that looks like a DB export. --- TIP: WRITE THE DESCRIPTION LIKE A TRIGGER RULESETNotice how the description lists multiple phrasings (“pastes SQL output”, “format this query result”, “summarise a dataset”). This breadth is intentional — Claude uses fuzzy matching, so covering several natural phrasings dramatically improves trigger reliability.\nStep 3 — Write the skill body # PostgreSQL Report Formatter ## What this skill does Formats raw PostgreSQL tabular output into clean Markdown reports. ## Required output sections 1. A ## Summary block: row count, column count, date range (if present) 2. A ## Data block: full Markdown table with aligned columns 3. A ## Observations block: 3-5 key insights from the data ## Formatting rules - Numeric columns: right-align, add thousand separators - Date columns: format as YYYY-MM-DD - NULL values: render as `—` (em dash), never blank - Column headers: Title Case ## Example input id | user_name | total_orders | revenue \u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+\u0026mdash;\u0026mdash;\u0026mdash; 1 | alice | 12 | 4500.00 2 | bob | 3 | 890.50\n## Example output ### Summary - **Rows:** 2 | **Columns:** 4 | **Total revenue:** $5,390.50 ### Data | ID | User Name | Total Orders | Revenue | |----|-----------|-------------:|--------:| | 1 | Alice | 12 | $4,500.00 | | 2 | Bob | 3 | $890.50 | Step 4 — Add a reference file (optional) If your skill needs to handle multiple database types — PostgreSQL, MySQL, SQLite — you can create a references/ folder and have the skill body instruct Claude to load the right one:\nIn your SKILL.md body, you then write:\n## Database-specific rules Detect which database produced the output from syntax clues. - If `||` concatenation → read references/sqlite.md - If `AUTO_INCREMENT` → read references/mysql.md - Default → read references/postgres.md Claude will read only the relevant file, keeping the context window lean and responses accurate.\nReal-World Example: The Built-in DOCX Skill Claude ships with several built-in skills. The docx skill is a great example of production-quality configuration. Here is its description field, distilled:\ndescription: | Use this skill whenever the user wants to create, read, edit, or manipulate Word documents (.docx files). Triggers include: any mention of \u0026#39;Word doc\u0026#39;, \u0026#39;word document\u0026#39;, \u0026#39;.docx\u0026#39;, or requests to produce professional documents with formatting like tables of contents, headings, page numbers, or letterheads. Also use when extracting content from .docx files, inserting images, performing find-and-replace, working with tracked changes, or converting content into a polished Word document. Notice the pattern: the description is almost a checklist. Every plausible user phrasing that should trigger the skill is listed. This is intentional — Claude has a natural tendency to under-trigger skills, so descriptions are deliberately thorough.\nWARNING: AVOID GENERIC DESCRIPTIONS\nA description like “Helps with documents” will rarely trigger. Claude needs enough specificity to distinguish your skill from general knowledge. Name the exact task, the exact file types, and the exact user phrases that should activate it.\nThe Progressive Disclosure Principle The most important architectural decision in skill design is knowing what goes where. The three-layer rule is simple:\n1. Description = triggering logic only Never put instructions in the description. It exists solely to help Claude decide whether to load the skill. Keep it focused on “when to use” language, not “how to do”.\n2. SKILL.md body = core workflow Put the main instructions, output format rules, required sections, formatting guidelines, and examples here. Target under 500 lines. If you need more, move deep content into reference files.\n3. References + scripts = deep knowledge Detailed documentation, database-specific rules, platform-specific templates, and executable scripts go in subfolders. The SKILL.md body tells Claude when to read them.\nWriting Descriptions That Actually Trigger The description field is the hardest part to get right. Here is a battle-tested template:\ndescription: | [What the skill does in one sentence]. Use this skill when: [scenario A], [scenario B], [scenario C]. Also trigger when the user [natural phrase], [natural phrase], or mentions [keyword / file type / tool name]. Do NOT use for [explicit exclusion to avoid false positives]. And a concrete example for a “Java Code Review” skill:\ndescription: | Reviews Java source files for correctness, performance, thread safety, and style — producing structured inline feedback. Use whenever the user shares Java code and asks for review, feedback, improvements, or a PR review. Also trigger on phrases like \u0026#34;check my code\u0026#34;, \u0026#34;what\u0026#39;s wrong here\u0026#34;, or \u0026#34;how can I improve this\u0026#34; when the code is Java. Trigger on .java files even without an explicit review request. Do NOT use for Kotlin, Scala, or general architecture questions without code attached. When Skills Use Scripts For deterministic or repetitive transformations — file conversions, data extraction, package generation — skills can include executable scripts. Claude reads the script path from the SKILL.md body, runs it via the bash tool, and uses the output.\nThe built-in docx skill, for example, uses a Python script to write .docxfiles using the python-docx library. The skill body contains step-by-step instructions for calling the script correctly — Claude follows them at runtime.\n## Creating a DOCX file 1. Generate Python code using python-docx from the template at scripts/create_doc.py 2. Run: python scripts/create_doc.py output.docx 3. Copy result to /mnt/user-data/outputs/ 4. Present to user with present_files tool Testing Your Skill Before deploying, write 5–10 test prompts that should trigger your skill, and 3–5 that should not. Run Claude with the skill installed and check whether it loads correctly.\nGood test prompts for a “pg-report” skill:\n# Should trigger: \u0026#34;Here\u0026#39;s my query result, can you format it nicely?\u0026#34; \u0026#34;Turn this SQL output into a report\u0026#34; \u0026#34;I exported this CSV from my database, summarise it\u0026#34; # Should NOT trigger: \u0026#34;How do I write a JOIN query?\u0026#34; \u0026#34;What is normalisation in databases?\u0026#34; PRO TIP: QUANTITATIVE EVALUATIONIf you are building skills at scale, the skill-creator meta-skill includes an automated optimization loop. It splits your test set 60/40 (train/held-out), runs Claude against each case multiple times, proposes description improvements, and iterates up to 5 rounds — outputting the best-performing description with before/after scores.\nPackaging and Sharing Skills A finished skill can be packaged into a .skill file for distribution. Anyone can install it by dropping the file into their skills directory. The package script handles archiving:\npython -m scripts.package_skill path/to/pg-report-skill/ # → outputs pg-report-skill.skill The Bigger Picture Skills are not just a convenience feature. They are the mechanism by which Claude becomes domain-aware without retraining. A legal firm can build a contract-review skill. A data team can build a dashboard-generation skill. A game studio can build a Godot export skill. Each one runs inside the same Claude — just with different runtime instructions.\nThe architecture is deliberately modular. Skills compose. A single request might trigger the pdf-reading skill to extract content, then the docx skill to write the output. Each stays in its lane, and Claude orchestrates.\n“You don’t teach Claude new tricks by changing the model. You teach it by writing better skills.”\nFor solo developers and small teams, this is genuinely powerful. You can codify your best workflows once, as a skill, and they run reliably on every request — without prompting, without repetition, and without hoping Claude happens to “know” your preferred format.\nStart small. Pick one workflow you repeat every week. Write a 30-line SKILL.md. Test it on five prompts. Then iterate. That is all it takes to start building your own personalised AI toolkit.\nRESOURCESThe skills discussed in this article — including docx, pdf, pptx, xlsx, and skill-creator — are built into Claude\u0026rsquo;s computer-use environment. You can explore them at /mnt/skills/public/ in any Claude session with computer use enabled.\n","permalink":"https://sukachvitally.github.io/posts/claude-skills-the-reusable-knowledge-modules-that-make-ai-truly-powerful/","summary":"\u003cp\u003e\u003cem\u003eHow Claude loads specialised instructions on demand — and how you can write your own to transform workflows forever.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/claude-skills-the-reusable-knowledge-modules-that-make-ai-truly-powerful/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eEvery time you open Claude and ask it to create a Word document, build a data analysis pipeline, or write presentation slides — something invisible happens first. Claude scans a directory of \u003cem\u003eskills\u003c/em\u003e, finds the most relevant one, reads it, and only then starts working.\u003c/p\u003e\n\u003cp\u003eThis system is called \u003cstrong\u003eClaude Skills\u003c/strong\u003e. It is one of the most underappreciated features in the entire Claude ecosystem. And once you understand how it works, you will see why it changes everything about how you build AI-powered workflows.\u003c/p\u003e","title":"Claude Skills: The Reusable Knowledge Modules That Make AI Truly Powerful"},{"content":"A beginner’s guide to Model Context Protocol — the thing that lets your AI assistant actually do stuff, not just talk about doing stuff.\nPicture this: you’ve got the most brilliant assistant in the world. Knows everything. Speaks 50 languages. Can explain quantum physics and write poetry at the same time. But here’s the catch — they’re locked in a glass box. They can see your computer screen. They can describe what they’d do if they could reach it. But touch anything? Nope. Read a file? Uh-uh. Check your calendar? Dream on.\nThat was AI assistants until very recently. Incredibly smart, utterly helpless. Like hiring Einstein to organize your sock drawer and then refusing to let him into the room.\nEnter MCP — the Model Context Protocol. The thing that breaks the glass and lets your AI assistant actually do the thing.\nMCP is essentially a USB standard for AI. Before USB, every device needed its own special plug. Now there’s one protocol, and everything just… works.\nBy the end of this article you’ll understand what MCP is, why it exists, how it works, and — most importantly — you’ll build your own working MCP server from scratch and connect it to Claude. Buckle up.\nA Brief History of AI’s Helplessness Problem To understand why MCP is a big deal, we need to appreciate just how spectacularly bad the situation was before it.\nEARLY 2023\nLarge Language Models (LLMs) become genuinely impressive. Everyone rushes to build assistants. The dream: AI that manages your life. The reality: AI that describes your life back to you very eloquently.\nMID 2023\n“Function calling” arrives. Now you can tell the AI: “hey, you can call these functions.” Progress! But every app invents its own format. A plugin for ChatGPT works nowhere else. A LangChain tool chain doesn’t talk to Claude. It’s the Wild West, but with JSON.\nLATE 2023–2024\nDevelopers build increasingly elaborate custom integrations. Everyone reinvents the wheel. Slightly differently. Slightly wrongly. A thousand incompatible “AI tool” standards bloom. This is the Tower of Babel, Silicon Valley edition.\nNOVEMBER 2024\nAnthropic releases the Model Context Protocol. An open standard. One protocol to rule them all. The developer community collectively exhales. Finally.\n2025\nMCP adoption explodes. OpenAI, Google, Microsoft all announce support. Hundreds of MCP servers appear on GitHub. Your AI can now read files, browse the web, query databases, control your computer, send Slack messages, and order pizza. (Well, almost.)\nThe key insight Anthropic had was embarrassingly simple: instead of every AI model learning to speak every tool’s language, make every tool learn one common language. The result is MCP.\nWhat Is MCP, Actually? MCP stands for Model Context Protocol. It’s an open standard (meaning anyone can use it, free of charge) that defines how AI models communicate with external tools and data sources.\nThink of it as a formal language that both sides agree to speak. The AI model is one speaker. Your tool (a file reader, a database, a weather API, your smart fridge) is the other speaker. MCP is the language between them.\nThe one-sentence explanation:\nMCP lets AI models talk to any tool through a single standardized protocol, so you write the connector once and it works everywhere.\nTechnically, MCP is a client-server architecture built on top of JSON-RPC 2.0. Don’t be scared by that. It means messages are passed as readable JSON objects, and there’s a strict format for how requests and responses look. Clean, simple, predictable.\nThe Architecture: Three Amigos MCP has three main characters. Let’s meet them:\nThe Host The application running the AI. Could be Claude Desktop, VS Code with Copilot, or your own app. The host is the conductor — it decides when the AI needs a tool and coordinates everything.\nThe Client Technically a component inside the host. It handles the actual MCP communication — sends requests to servers, gets back results, feeds them to the AI. You don’t usually build this yourself.\nThe Server This is what you build. An MCP server exposes capabilities — tools, resources, and prompts — that the AI can use. Your server could wrap a database, a REST API, the local file system, or anything else. This is where the magic you write lives.\nCore Concepts: Tools, Resources \u0026amp; Prompts MCP servers expose three types of things to AI models. Think of these as different kinds of “capabilities”:\nFor most real-world use cases you’ll mainly deal with Tools. That’s where the action is.\nHow They Talk: The MCP Message Format MCP messages are JSON objects following the JSON-RPC 2.0 specification. Every single message — whether it’s a greeting, a tool call, or an error — follows the same envelope structure. Once you learn the envelope, the rest is just filling in the blanks.\nThe Anatomy of Every Message All JSON-RPC messages share the same top-level fields. Let’s dissect them:\nPhase 1 — The Handshake: initialize Before any tools are called, the host and server must introduce themselves. This is the very first exchange that happens when Claude Desktop spawns your server process. Think of it as two people shaking hands and agreeing on a common language before getting to business.\n1A. HOST → SERVER: INITIALIZE REQUEST\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 0, \u0026#34;method\u0026#34;: \u0026#34;initialize\u0026#34;, \u0026#34;params\u0026#34;: { \u0026#34;protocolVersion\u0026#34;: \u0026#34;2024-11-05\u0026#34;, // MCP spec version the host supports \u0026#34;capabilities\u0026#34;: { \u0026#34;roots\u0026#34;: { \u0026#34;listChanged\u0026#34;: true }, // Host can notify server about root changes \u0026#34;sampling\u0026#34;: {} // Host supports LLM sampling requests }, \u0026#34;clientInfo\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;Claude Desktop\u0026#34;, \u0026#34;version\u0026#34;: \u0026#34;0.7.0\u0026#34; } } } 1B. SERVER → HOST: INITIALIZE RESPONSE\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 0, // Same id — this is the matching reply \u0026#34;result\u0026#34;: { \u0026#34;protocolVersion\u0026#34;: \u0026#34;2024-11-05\u0026#34;, // Server confirms the version \u0026#34;capabilities\u0026#34;: { \u0026#34;tools\u0026#34;: { \u0026#34;listChanged\u0026#34;: false } // Server exposes tools (and won\u0026#39;t hot-reload them) }, \u0026#34;serverInfo\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;my-first-mcp-server\u0026#34;, \u0026#34;version\u0026#34;: \u0026#34;1.0.0\u0026#34; } } } 1C. HOST → SERVER: INITIALIZED NOTIFICATION (NO REPLY EXPECTED)\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;method\u0026#34;: \u0026#34;notifications/initialized\u0026#34; // No \u0026#34;id\u0026#34; field — this is a notification, not a request. // Notifications are fire-and-forget; the server never replies. } After this three-step handshake, the connection is live and both sides know what the other can do. Now the real fun begins.\nPhase 2 — Tool Discovery: tools/list The host immediately asks: “OK, what tools do you have?” This lets the AI know its full menu of capabilities before the user types a single character.\n2A. HOST → SERVER: TOOLS/LIST REQUEST\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 1, \u0026#34;method\u0026#34;: \u0026#34;tools/list\u0026#34; // No \u0026#34;params\u0026#34; needed — just asking for the full list. } 2B. SERVER → HOST: TOOLS/LIST RESPONSE (WITH FULL INPUTSCHEMA)\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 1, \u0026#34;result\u0026#34;: { \u0026#34;tools\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;get_weather\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Get current weather for any city.\u0026#34;, \u0026#34;inputSchema\u0026#34;: { // Standard JSON Schema object \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;city\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;City name, e.g. Belfast\u0026#34; }, \u0026#34;unit\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;enum\u0026#34;: [\u0026#34;celsius\u0026#34;, \u0026#34;fahrenheit\u0026#34;], // Constrained to specific values \u0026#34;default\u0026#34;: \u0026#34;celsius\u0026#34; } }, \u0026#34;required\u0026#34;: [\u0026#34;city\u0026#34;] // \u0026#34;unit\u0026#34; is optional, \u0026#34;city\u0026#34; is not } }, { \u0026#34;name\u0026#34;: \u0026#34;search_database\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Search products by keyword with optional filters.\u0026#34;, \u0026#34;inputSchema\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;query\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34; }, \u0026#34;limit\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;integer\u0026#34;, \u0026#34;minimum\u0026#34;: 1, \u0026#34;maximum\u0026#34;: 100, \u0026#34;default\u0026#34;: 10 }, \u0026#34;tags\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;array\u0026#34;, // Arrays of primitives are fine \u0026#34;items\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34; } } }, \u0026#34;required\u0026#34;: [\u0026#34;query\u0026#34;] } } ] } } The inputSchema is standard JSON Schema. You can use type, enum, minimum, maximum, default, array, nested object — any JSON Schema feature. The AI model reads this schema and uses it to build valid arguments automatically. Good schemas = fewer mistakes by the AI.\nPhase 3 — Tool Execution: tools/call The user says something. The AI decides a tool is needed. It fires a tools/callwith the arguments it constructed from the schema.\n3A. SIMPLE CALL — ONE STRING ARGUMENT\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 2, \u0026#34;method\u0026#34;: \u0026#34;tools/call\u0026#34;, \u0026#34;params\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;get_weather\u0026#34;, \u0026#34;arguments\u0026#34;: { \u0026#34;city\u0026#34;: \u0026#34;Belfast\u0026#34; // Only the required field; \u0026#34;unit\u0026#34; uses its default } } } 3B. COMPLEX CALL — MULTIPLE TYPES INCLUDING ARRAY\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 3, \u0026#34;method\u0026#34;: \u0026#34;tools/call\u0026#34;, \u0026#34;params\u0026#34;: { \u0026#34;name\u0026#34;: \u0026#34;search_database\u0026#34;, \u0026#34;arguments\u0026#34;: { \u0026#34;query\u0026#34;: \u0026#34;wireless headphones\u0026#34;, \u0026#34;limit\u0026#34;: 5, // integer \u0026#34;tags\u0026#34;: [\u0026#34;bluetooth\u0026#34;, \u0026#34;noise-cancelling\u0026#34;] // array } } } 3C. SUCCESS RESPONSE — TEXT CONTENT\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 2, \u0026#34;result\u0026#34;: { \u0026#34;content\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;text\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;Current weather in Belfast: 12°C, wind 18 km/h\u0026#34; } ], \u0026#34;isError\u0026#34;: false // Optional. Explicitly marks success. } } 3D. SUCCESS RESPONSE — MULTIPLE CONTENT BLOCKS\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 3, \u0026#34;result\u0026#34;: { \u0026#34;content\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;text\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;Found 2 results for \u0026#39;wireless headphones\u0026#39;:\u0026#34; }, { \u0026#34;type\u0026#34;: \u0026#34;text\u0026#34;, // Second block — same message, more content \u0026#34;text\u0026#34;: \u0026#34;1. Sony WH-1000XM5 — £279\\n2. Bose QC45 — £249\u0026#34; } // Could also include \u0026#34;image\u0026#34; blocks with base64 data, // or \u0026#34;resource\u0026#34; blocks referencing URIs ] } } Phase 4 — When Things Go Wrong: Error Responses There are two distinct flavours of error in MCP. Don’t mix them up — they mean very different things.\nPROTOCOL ERROR — THE REQUEST ITSELF WAS MALFORMED\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 4, \u0026#34;error\u0026#34;: { // \u0026#34;error\u0026#34; field, NOT \u0026#34;result\u0026#34; \u0026#34;code\u0026#34;: -32602, // Standard JSON-RPC error code \u0026#34;message\u0026#34;: \u0026#34;Invalid params\u0026#34;, \u0026#34;data\u0026#34;: { \u0026#34;details\u0026#34;: \u0026#34;Required field \u0026#39;city\u0026#39; is missing\u0026#34; } } } TOOL-LEVEL ERROR — THE TOOL RAN BUT THE OPERATION FAILED\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 5, \u0026#34;result\u0026#34;: { // Still \u0026#34;result\u0026#34;, NOT \u0026#34;error\u0026#34; ! \u0026#34;content\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;text\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;City \u0026#39;Narnia\u0026#39; not found in geocoding database.\u0026#34; } ], \u0026#34;isError\u0026#34;: true // Flag tells the AI: tool ran, but reported failure } } Key distinction: A protocol error (using the \u0026ldquo;error\u0026rdquo; field) means the communication broke — bad JSON, unknown method, missing required fields. A tool error (using \u0026ldquo;result\u0026rdquo; with \u0026ldquo;isError\u0026rdquo;: true) means the tool ran fine but the operation itself failed — user not found, city doesn\u0026rsquo;t exist, API returned 404. Always use isError: true for business logic failures so the AI can read the message and react intelligently.\nBonus: Resource Reads and Notifications Tools aren’t the only thing MCP can do. Here’s what a resource read looks like — useful when you want to expose file contents or database records the AI can browse:\nRESOURCES/READ REQUEST AND RESPONSE\n// Request { \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 6, \u0026#34;method\u0026#34;: \u0026#34;resources/read\u0026#34;, \u0026#34;params\u0026#34;: { \u0026#34;uri\u0026#34;: \u0026#34;file:///home/user/notes.txt\u0026#34; // Resource URIs identify the data source } } // Response { \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;id\u0026#34;: 6, \u0026#34;result\u0026#34;: { \u0026#34;contents\u0026#34;: [ { \u0026#34;uri\u0026#34;: \u0026#34;file:///home/user/notes.txt\u0026#34;, \u0026#34;mimeType\u0026#34;: \u0026#34;text/plain\u0026#34;, \u0026#34;text\u0026#34;: \u0026#34;Buy milk. Deploy the thing. Don\u0026#39;t forget stand-up.\u0026#34; } ] } } And a server notification — fire-and-forget messages the server sends without being asked (no id, no reply expected):\nSERVER → HOST NOTIFICATION (NO REPLY)\n{ \u0026#34;jsonrpc\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;method\u0026#34;: \u0026#34;notifications/tools/list_changed\u0026#34;, // No \u0026#34;id\u0026#34; — this is a push notification. // The server is telling the host: \u0026#34;Hey, my tool list changed. // Please call tools/list again to refresh.\u0026#34; } Clean and readable. No magic, no wizardry. Just JSON going back and forth. This predictability is exactly what makes MCP powerful — any language, any framework, any tool can implement it.\nTwo Ways to Connect: stdio vs HTTP/SSE MCP servers can communicate via two transport mechanisms:\nstdio (Standard I/O) — The simplest option. Your server reads from stdin and writes to stdout. The host app spawns the server as a subprocess. Perfect for local tools running on the same machine.\nHTTP + SSE (Server-Sent Events) — For remote servers. The client sends POST requests to the server, and the server streams events back. Use this when your MCP server lives on a different machine (a cloud server, a colleague’s computer, etc.).\nStart with stdio. It’s simpler, requires no networking setup, and is perfect for local tooling. You can always switch to HTTP later.\nTutorial: Building server.py Step by Step Enough theory. Let’s write the actual code. Below is a complete, annotated server.py that exposes three tools: a dad joke generator, a task list, and a real live weather lookup. We\u0026rsquo;ll build it section by section so nothing is a mystery.\n📦Install first: pip install mcp — that\u0026rsquo;s the only dependency. Python 3.10+ required.\nPart 1 — Imports \u0026amp; Server Instantiation The MCP SDK ships everything you need in the mcp package. We create a Server instance and tell it our name and version — this is what Claude Desktop sees in the handshake.\nserver.py — Part 1: imports and setup\n# Standard library — no extra installs needed for HTTP calls import asyncio import random import json import urllib.request import urllib.parse import sys # needed for stderr debug output # MCP SDK - install with: pip install mcp from mcp.server import Server from mcp.server.stdio import stdio_server from mcp import types # Create the server. The name/version appear during the MCP handshake. # Claude Desktop displays this name in its connected servers list. app = Server(\u0026#34;my-first-mcp-server\u0026#34;) Part 2 — Declaring Tools with @app.list_tools() This decorator registers the handler that responds to tools/list requests. The AI calls this automatically on startup to discover your full menu of capabilities. The descriptions you write here are read by the AI — write them like you\u0026rsquo;re explaining the tool to a smart colleague, not a compiler.\nserver.py — Part 2: tool declarations\n@app.list_tools() async def list_tools() -\u0026gt; list[types.Tool]: return [ # ── Tool 1: dad_joke ────────────────────────────────────── # No arguments. The inputSchema is an empty object. # The AI will call this whenever the user needs cheering up. types.Tool( name=\u0026#34;dad_joke\u0026#34;, description=\u0026#34;Returns a random, gloriously terrible dad joke. Use when morale is low.\u0026#34;, inputSchema={ \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: {}, # no arguments - jokes are free \u0026#34;required\u0026#34;: [], }, ), # ── Tool 2: get_tasks ───────────────────────────────────── # One required string argument: username. # \u0026#34;required\u0026#34; list enforces that the AI must supply it. types.Tool( name=\u0026#34;get_tasks\u0026#34;, description=\u0026#34;Returns the task list for a given user.\u0026#34;, inputSchema={ \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;username\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;The username whose tasks to fetch, e.g. \u0026#39;alice\u0026#39;\u0026#34;, }, }, \u0026#34;required\u0026#34;: [\u0026#34;username\u0026#34;], }, ), # ── Tool 3: get_weather ─────────────────────────────────── # Calls the real Open-Meteo API. No API key required. # \u0026#34;unit\u0026#34; is optional - has a default value in the handler. types.Tool( name=\u0026#34;get_weather\u0026#34;, description=\u0026#34;Get current weather for any city. Uses Open-Meteo (free, no API key needed).\u0026#34;, inputSchema={ \u0026#34;type\u0026#34;: \u0026#34;object\u0026#34;, \u0026#34;properties\u0026#34;: { \u0026#34;city\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;City name, e.g. Belfast, Tokyo, New York\u0026#34;, }, \u0026#34;unit\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;enum\u0026#34;: [\u0026#34;celsius\u0026#34;, \u0026#34;fahrenheit\u0026#34;], \u0026#34;description\u0026#34;: \u0026#34;Temperature unit. Defaults to celsius.\u0026#34;, }, }, \u0026#34;required\u0026#34;: [\u0026#34;city\u0026#34;], # \u0026#34;unit\u0026#34; is optional - omit from required }, ), ] Part 3 — Handling Tool Calls with @app.call_tool() When the AI decides to invoke a tool, this handler fires. It receives the tool name and the arguments dict the AI built from your schema. Return a list of TextContent objects — even if there\u0026rsquo;s only one. Always return a list; the protocol expects it.\nserver.py — Part 3: tool logic\n@app.call_tool() async def call_tool(name: str, arguments: dict) -\u0026gt; list[types.TextContent]: # ── dad_joke ────────────────────────────────────────────── if name == \u0026#34;dad_joke\u0026#34;: jokes = [ \u0026#34;Why don\u0026#39;t scientists trust atoms? Because they make up everything.\u0026#34;, \u0026#34;I told my wife she was drawing her eyebrows too high. She looked surprised.\u0026#34;, \u0026#34;Why do programmers prefer dark mode? Because light attracts bugs.\u0026#34;, \u0026#34;I asked my dog what two minus two is. He said nothing.\u0026#34;, \u0026#34;Why did the MCP server go to therapy? Too many unhandled exceptions.\u0026#34;, ] return [types.TextContent(type=\u0026#34;text\u0026#34;, text=random.choice(jokes))] # ── get_tasks ───────────────────────────────────────────── # In real life you\u0026#39;d query a DB here. We use a dict for clarity. if name == \u0026#34;get_tasks\u0026#34;: username = arguments.get(\u0026#34;username\u0026#34;, \u0026#34;\u0026#34;).lower() fake_tasks = { \u0026#34;alice\u0026#34;: [\u0026#34;Review PR #42\u0026#34;, \u0026#34;Update docs\u0026#34;, \u0026#34;Fix that bug you\u0026#39;ve been ignoring\u0026#34;], \u0026#34;bob\u0026#34;: [\u0026#34;Stand-up meeting\u0026#34;, \u0026#34;Drink coffee\u0026#34;, \u0026#34;Question all life choices\u0026#34;], } tasks = fake_tasks.get(username, [\u0026#34;No tasks found. Either very productive or very lazy.\u0026#34;]) formatted = \u0026#34;\\n\u0026#34;.join(f\u0026#34;{i+1}. {t}\u0026#34; for i, t in enumerate(tasks)) return [types.TextContent(type=\u0026#34;text\u0026#34;, text=f\u0026#34;Tasks for {username}:\\n{formatted}\u0026#34;)] # ── get_weather ─────────────────────────────────────────── # Two HTTP calls: geocode the city name → get weather by lat/lon. # Uses urllib (stdlib) so no extra dependencies. if name == \u0026#34;get_weather\u0026#34;: city = arguments.get(\u0026#34;city\u0026#34;, \u0026#34;\u0026#34;) unit = arguments.get(\u0026#34;unit\u0026#34;, \u0026#34;celsius\u0026#34;) # default if AI omits it # Step 1 - turn city name into coordinates geo_url = ( \u0026#34;https://geocoding-api.open-meteo.com/v1/search\u0026#34; f\u0026#34;?name={urllib.parse.quote(city)}\u0026amp;count=1\u0026#34; ) with urllib.request.urlopen(geo_url) as r: geo = json.loads(r.read()) if not geo.get(\u0026#34;results\u0026#34;): # Tool ran fine, but the operation failed - use isError flag return [types.TextContent(type=\u0026#34;text\u0026#34;, text=f\u0026#39;City \u0026#34;{city}\u0026#34; not found.\u0026#39;)] result = geo[\u0026#34;results\u0026#34;][0] lat, lon, found_name = result[\u0026#34;latitude\u0026#34;], result[\u0026#34;longitude\u0026#34;], result[\u0026#34;name\u0026#34;] # Step 2 - fetch current weather for those coordinates wx_url = ( \u0026#34;https://api.open-meteo.com/v1/forecast\u0026#34; f\u0026#34;?latitude={lat}\u0026amp;longitude={lon}\u0026#34; f\u0026#34;¤t_weather=true\u0026amp;temperature_unit={unit}\u0026#34; ) with urllib.request.urlopen(wx_url) as r: wx = json.loads(r.read()) cw = wx[\u0026#34;current_weather\u0026#34;] symbol = \u0026#34;°C\u0026#34; if unit == \u0026#34;celsius\u0026#34; else \u0026#34;°F\u0026#34; return [types.TextContent( type=\u0026#34;text\u0026#34;, text=f\u0026#34;{found_name}: {cw[\u0026#39;temperature\u0026#39;]}{symbol}, wind {cw[\u0026#39;windspeed\u0026#39;]} km/h\u0026#34;, )] # Unknown tool - raise so MCP returns a proper protocol error raise ValueError(f\u0026#34;Unknown tool: {name}\u0026#34;) Part 4 — Wiring Up stdio Transport The final piece connects the server to the MCP host via stdin/stdout. stdio_server() returns two async streams; we hand them to app.run() and it takes care of the rest. The key thing to remember: never print() to stdout anywhere in your code — that channel belongs to MCP. Debug with sys.stderr only.\nserver.py — Part 4: entry point\nasync def main(): # stdio_server() yields (read_stream, write_stream) for the MCP host. # Everything inside this block is the server\u0026#39;s lifetime. async with stdio_server() as (read_stream, write_stream): print(\u0026#34;MCP server started\u0026#34;, file=sys.stderr) # stderr is safe for debug await app.run( read_stream, write_stream, app.create_initialization_options(), ) if __name__ == \u0026#34;__main__\u0026#34;: asyncio.run(main()) 💡Why async? MCP uses async I/O because servers often wait on multiple things at once — the host, network calls, file reads. Python\u0026rsquo;s asyncio lets a single thread handle all of that without blocking. You don\u0026rsquo;t need to understand the internals; just know that every handler must be async def, and use await before any I/O call.\nConnecting Your Server to Claude Desktop Once you’ve written your server.py, plugging it into Claude Desktop takes two minutes. Download Claude Desktop from claude.ai/download — it\u0026rsquo;s free and has an MCP client built in. Then locate the config file for your OS:\nRegister your server by adding it to that file:\nCLAUDE_DESKTOP_CONFIG.JSON\n{ \u0026#34;mcpServers\u0026#34;: { \u0026#34;my-first-mcp-server\u0026#34;: { \u0026#34;command\u0026#34;: \u0026#34;python\u0026#34;, \u0026#34;args\u0026#34;: [ \u0026#34;/absolute/path/to/my-first-mcp-server/server.py\u0026#34; ], \u0026#34;env\u0026#34;: { \u0026#34;MY_API_KEY\u0026#34;: \u0026#34;sk-...\u0026#34; // Pass secrets here; read with os.environ in Python } } } } Important: Use the absolute path to your server.py file. No tildes, no relative paths. Get the path with pwd on Mac/Linux or cd on Windows, and paste the full thing in.\nFully quit Claude Desktop (not just close the window) and reopen it. Look for the 🔧 tools icon in the chat input — that means MCP is connected. Then just ask Claude naturally: “Tell me a dad joke” or “What tasks does Alice have?” — it figures out on its own when to call your tools. No special syntax needed.\nBest Practices (a.k.a. Mistakes I’m Saving You From) Write excellent descriptions\nThe AI reads your tool descriptions to decide when to use them. Vague descriptions = confused AI = bad results. Be specific: what does the tool do, when should it be used, what do the arguments mean?\nNever print() to stdout for debug output\nIn stdio transport, stdout is the MCP communication channel. Any stray print() call will corrupt the protocol stream. Always write debug output to stderr: print(\u0026ldquo;debug\u0026rdquo;, file=sys.stderr).\nValidate your inputs\nThe AI sometimes passes unexpected values. Check that required arguments exist. Validate formats. Return helpful error messages rather than cryptic crashes.\nKeep tools focused\nOne tool, one job. Don’t build a do_everything() tool. Small, well-named tools are easier for the AI to reason about and call correctly. Think Unix philosophy: do one thing well.\nUse environment variables for secrets\nNever hardcode API keys. Pass them via the env key in your claude_desktop_config.json: \u0026ldquo;env\u0026rdquo;: { \u0026ldquo;API_KEY\u0026rdquo;: \u0026ldquo;your-key-here\u0026rdquo; }. Read with os.environ[\u0026ldquo;API_KEY\u0026rdquo;] in your server.\nThe MCP Ecosystem: You Don’t Have to Build Everything Here’s some great news: the open-source community has already built hundreds of MCP servers you can just plug in and use. Some highlights:\nFind these and more at github.com/modelcontextprotocol/servers — the official repo — and on mcp.so, a community directory. There’s a good chance what you need already exists.\nWrapping Up: Why This Matters MCP isn’t just a neat technical standard. It represents a fundamental shift in what AI assistants actually are.\nBefore MCP: AI as an oracle. You ask, it tells. Great for knowledge questions. Useless for action.\nAfter MCP: AI as an agent. You describe what you want, it figures out which tools to use, calls them, interprets the results, and gets the job done. The AI becomes an active participant rather than a fancy search engine.\nThe future isn’t AI that can describe how to fix your code. It’s AI that fixes your code, runs the tests, creates the PR, and messages your team. MCP is the foundation that makes all of that possible.\nYou’ve now seen the full picture: the history, the architecture, the wire format, and a working server from scratch. The next step is yours. Pick a tool you use daily — a database, an API, a local service — and wrap it in an MCP server. Once you’ve done it once, you’ll find excuses to do it for everything.\nHappy building. 🚀\n","permalink":"https://sukachvitally.github.io/posts/mcp-the-protocol-that-finally-taught-ai-to-stop-being-useless/","summary":"\u003cp\u003e\u003cem\u003eA beginner’s guide to Model Context Protocol — the thing that lets your AI assistant actually do stuff, not just talk about doing stuff.\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/mcp-the-protocol-that-finally-taught-ai-to-stop-being-useless/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003ePicture this: you’ve got the most brilliant assistant in the world. Knows everything. Speaks 50 languages. Can explain quantum physics and write poetry at the same time. But here’s the catch — \u003cstrong\u003ethey’re locked in a glass box\u003c/strong\u003e. They can see your computer screen. They can describe what they’d do if they could reach it. But touch anything? Nope. Read a file? Uh-uh. Check your calendar? Dream on.\u003c/p\u003e","title":"MCP: The Protocol That Finally Taught AI to Stop Being Useless"},{"content":"\nThe Agentic Loop (How Claude Actually Thinks) Before anything else, you need to understand what makes Claude Code different from your IDE’s autocomplete. Autocomplete guesses the next word. Claude Code runs a loop — it gathers context, takes action, verifies results, then repeats until the task is done. Think of it less like a text editor feature and more like a developer who works asynchronously while you make coffee.\nThe loop has three phases that blend together depending on the task:\nThe loop adapts to what you ask. A question might only hit phase 1. A bug fix cycles through all three phases multiple times. A full feature might chain dozens of tool calls before Claude surfaces for air. You’re part of the loop too — you can interrupt at any point with Ctrl+C and steer it in a different direction.\nTHE HONEST SUMMARY\nClaude Code is basically a language model with access to your entire computer strapped to it. It can read everything, run anything you could run from a terminal, and it just keeps going until it either finishes the task or you tell it to stop. This is either exhilarating or terrifying depending on how much you trust your test coverage.\nSessions \u0026amp; How Claude Remembers (Or Doesn’t) Here’s the thing about Claude: every session starts with a completely blank memory. No recollection of last week’s chat. No memory of that convention you mentioned. Tabula rasa. Every. Single. Time. If you don’t explicitly persist knowledge, it’s gone when the session closes.\nThis isn’t a bug — it’s how LLMs work. But Claude Code gives you two mechanisms to work around it:\nHow a session actually starts When you run claude in a directory, here\u0026rsquo;s what happens in the first few seconds before you\u0026rsquo;ve typed anything:\n**Settings load (all scopes)**Global ~/.claude/settings.json → project .claude/settings.json → local .claude/settings.local.json. Scalar settings use the most specific value; array settings (like permissions) merge across all scopes. MCP servers connectAll configured MCP servers from .mcp.json and ~/.claude.json start up. Tool schemas are deferred and load on demand. If a server fails to connect, Claude continues without it — it just loses those tools. CLAUDE.md files load into contextOrg-level → user-level (~/.claude/CLAUDE.md) → project-level (./CLAUDE.md) → local (./CLAUDE.local.md). Plus your ~/.claude/MEMORY.md auto memory. All of this costs context window tokens immediately. Plugins activateAll enabled plugins load their skills, hooks, MCP servers, and settings. Run /reload-plugins to activate changes mid-session. Session is readyYou see the prompt. Everything above has already happened. Claude now has your project context and is ready to work. CONTEXT WINDOW BUDGET\nEverything loaded at startup costs tokens before you type a single character. A bloated CLAUDE.md, a dozen MCP servers, and a large MEMORY.md can eat a significant chunk of your context window. Run /status to see current token usage. Use /compact when a session gets long.\nThe .claude Directory — Every Folder Explained The .claude/ folder is Claude Code\u0026rsquo;s configuration home in your project. There\u0026rsquo;s also ~/.claude/ in your home directory for global settings. Here\u0026rsquo;s the complete layout with every file and folder you\u0026rsquo;ll actually encounter:\nCOMMIT STRATEGY\nEverything marked committed should go in Git so your whole team gets the same Claude setup. Everything marked gitignored is personal — Claude automatically adds settings.local.json to your global Git ignore. Add CLAUDE.local.md to your project\u0026rsquo;s .gitignore manually.\nCLAUDE.md — The Full Story You’ve heard about CLAUDE.md. Everyone mentions it. But few people explain the entire system behind it, which has four separate scope levels that load in a specific order:\nThey load in that order — managed first, local last. More specific scopes appear later in the context window, giving them higher effective priority.\nWhat to actually put in CLAUDE.md The golden rule: add something to CLAUDE.md when Claude makes the same mistake a second time. Here’s a battle-tested example for a Spring Boot project:\nSIZE MATTERS\nTarget under 200 lines per CLAUDE.md. Longer files still load fully but adherence drops. If it’s getting long, split topic-specific rules into .claude/rules/ files and skills for complex procedures — they load conditionally and save your context window.\nAuto Memory (MEMORY.md) This is the magical part most people miss. Claude can write its own notes — automatically. When you correct Claude, it optionally remembers that correction for next session in ~/.claude/MEMORY.md. The first 200 lines or 25KB load at the start of every session.\nEnable it with /memory in session or toggle it in settings. Then: tell Claude something once, it remembers it forever (until you delete the entry). This is how you get a Claude that feels like it\u0026rsquo;s been on your team for months.\nsettings.json — The Enforcement Layer There’s a critical distinction most people miss: CLAUDE.md is guidance Claude reads.settings.json is configuration Claude Code enforces. One is a polite suggestion. The other is a locked door.\nSettings live in .claude/settings.json (project/team) and .claude/settings.local.json(personal overrides). Here\u0026rsquo;s what you can configure:\nALLOW VS DENY PRECEDENCE\nDeny rules always win over allow rules. If you allow Bash(kubectl *) and deny Bash(kubectl delete *), the delete command is blocked. Patterns support wildcards — Bash(npm test *)matches any command starting with npm test.\nrules/ — Scoped Instructions That Load On Demand Once your CLAUDE.md hits 200 lines, start splitting it into .claude/rules/. Rules are Markdown files that can load conditionally based on which files Claude is working with — only when it reads a file that matches the rule\u0026rsquo;s path pattern. This keeps your global context lean while ensuring Claude gets the right rules at the right time.\nTHE REAL POWER\nA rule file without paths: frontmatter loads every session like CLAUDE.md. A rule WITH paths:only loads when Claude reads a matching file. This means your Kafka rules don\u0026rsquo;t eat context window during a CLAUDE.md update, and your migration rules only appear when Claude actually opens a migration file.\nskills/ — Your Custom Slash Commands Skills are reusable prompts you or Claude can invoke by name using /skill-name. Think of them as saved workflows — things you do repeatedly that are more than a one-liner but less than a custom tool. You write the instructions once in a SKILL.md, and invoke it whenever you need it.\nNow you type /db-migrate and Claude walks through all seven steps. No more explaining the process from scratch every time. Skills also accept arguments via $ARGUMENTS:\nUsage: /review-pr 127 — Claude reviews MR #127 with your exact conventions baked in.\nSKILLS VS COMMANDS\nYou may see references to .claude/commands/ in older docs — this is the same mechanism as skills. For new workflows, use skills/ instead: same /name invocation, plus you can bundle supporting files alongside the SKILL.md.\nHooks — The Automation Engine Hooks are shell commands that fire automatically at specific points in Claude’s lifecycle. They’re deterministic — they always run when triggered, regardless of what Claude decides. This is the difference between “please format the file” (Claude might forget) and “PostToolUse hook runs Spotless” (it definitely happens).\nAvailable hook events PreToolUseFires before any tool call. Can block execution (exit 2). Use to protect files or validate commands.\nPostToolUseFires after a tool completes. Perfect for auto-formatting files after Claude edits them.\nNotificationFires when Claude needs your input. Desktop notification so you don’t stare at the terminal.\nSessionStartFires when a session begins. Inject additional context, set environment, run setup scripts.\nPreCompactFires before conversation compaction. Re-inject context that would otherwise be lost.\nStopFires when Claude finishes a response. Useful for post-response validation or logging.\nPractical examples for a Java/Spring Boot project EXIT CODES MATTER\nexit 0 = allow. exit 1 = warn Claude but let it continue. exit 2 = hard block — Claude gets told why and has to try something else. It\u0026rsquo;s the digital equivalent of a bouncer who also explains the dress code.\nMCP — Connecting Claude to the World Model Context Protocol (MCP) is an open standard for connecting AI to external tools. Without MCP, Claude can only see your filesystem and run terminal commands. With MCP, Claude can query your Jira board, check Sentry errors, read Slack messages, query your PostgreSQL database, and more — all without you copying and pasting anything.\nTHE MENTAL MODEL\nMCP is how Claude gets a login to your tools. Without it: “here’s a pasted error from Sentry.” With it: “Claude, check what errors happened in prod in the last hour and correlate them with my recent commits.”\nThree types of MCP servers Project-scoped MCP (.mcp.json) Put your team’s MCP servers in .mcp.json at the project root. This file gets committed to Git so everyone on the team gets the same tools:\nUser-scoped MCP (personal tools) For tools only you need (not your team), use the \u0026ndash;scope user flag. This writes to ~/.claude.json instead of .mcp.json:\nCHANNELS: MCP THAT PUSHES TO CLAUDE\nNormal MCP is pull — Claude queries a server when it needs something. Channels flip this: an MCP server pushes messages into your running session. This means Claude can react to CI results, monitoring alerts, or Slack messages while you’re doing something else. Enable with \u0026ndash;channelsflag on startup.\nPlugins — Package \u0026amp; Share Everything Everything you’ve built — skills, hooks, MCP servers, rules — can be packaged into a plugin and shared with your team, your org, or the whole world. Plugins are just directories with a .claude-plugin/plugin.json manifest. They\u0026rsquo;re the difference between \u0026ldquo;config that lives in one project\u0026rdquo; and \u0026ldquo;config that lives everywhere.\u0026rdquo;\nStandalone config vs plugins Plugin structure Installing a plugin THE REAL VALUE\nYour platform team builds a plugin once with all the Spring Boot conventions, Kafka rules, code review skills, and MCP servers your org uses. Every developer installs it once. Every project gets the same setup. No more “did you add the Flyway rules to your CLAUDE.md?” Slack messages.\nSubagents — Claude Spawning Claude This is the part where things get genuinely science-fiction. Subagents let Claude spin up other Claude instances to handle subtasks in parallel or in isolated contexts. The main Claude orchestrates; the subagents execute. Think distributed computing, but the workers are also AI and can also spawn more workers.\nSubagents can maintain their own MEMORY.md too. Imagine a \u0026ldquo;database specialist\u0026rdquo; subagent that remembers every schema change it\u0026rsquo;s ever helped with. That\u0026rsquo;s the direction this is heading.\nWHEN TO USE SUBAGENTS\nFor most day-to-day tasks, you don’t need subagents. They’re most useful when you have truly parallel workstreams (implement feature A and feature B simultaneously without conflicts), when you need a clean context for a specific task, or when you’re building automation pipelines that run without human interaction.\nComplete Quick Reference .claude/settings.json — permissions + hooks + model config. Enforced, not just read. CLAUDE.md — persistent instructions. Team version in ./CLAUDE.md, personal in ~/.claude/CLAUDE.md. MEMORY.md — auto-written by Claude. Persists learnings across sessions (200 lines / 25KB limit). .claude/rules/ — conditional instruction files. Use paths: frontmatter to load only for relevant files. .claude/skills/ — custom /commands. Each skill = a folder with SKILL.md. Accepts $ARGUMENTS. hooks — deterministic automation. PreToolUse (block/allow), PostToolUse (format), Notification (alert). .mcp.json — team MCP servers (committed). ~/.claude.json for personal servers. Plugins — packaged, shareable bundles of skills + hooks + MCP + settings. Install with /plugin install. Subagents — parallel Claude instances for complex multi-track workflows. Three isolation levels. /init — always run first in a new project. Generates CLAUDE.md automatically. /compact — compresses long sessions. /clear = fresh start (CLAUDE.md reloads). /mcp — shows server status + tool count. /hooks — shows configured hooks. /memory — edit CLAUDE.md. ","permalink":"https://sukachvitally.github.io/posts/claude-code-the-complete-guide-nobody-wrote/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/claude-code-the-complete-guide-nobody-wrote/image-01.png\"\u003e\u003c/p\u003e\n\u003ch3 id=\"the-agentic-loop-how-claude-actuallythinks\"\u003eThe Agentic Loop (How Claude Actually Thinks)\u003c/h3\u003e\n\u003cp\u003eBefore anything else, you need to understand what makes Claude Code different from your IDE’s autocomplete. Autocomplete guesses the next word. Claude Code runs a loop — it gathers context, takes action, verifies results, then repeats until the task is done. Think of it less like a text editor feature and more like a developer who works asynchronously while you make coffee.\u003c/p\u003e","title":"Claude Code: The Complete Guide Nobody Wrote"},{"content":"For developers who want an AI in their terminal and still have no idea what CLAUDE.md is. You’ve come to the right place.\n🤔 What Even IS Claude Code? Okay so you’ve heard of GitHub Copilot, right? Claude Code is like that, except instead of whispering suggestions into your IDE like a shy intern, it’s a full-blown agentic assistant that lives in your terminal and acts like a senior developer who has had three espressos and is ready to ship.\nIt can read your entire codebase, plan multi-step changes, edit multiple files, run your tests, and even commit code. All from the command line. No IDE plugin. No browser tab. Just you, your terminal, and your new AI best friend who never judges you for that 200-line method you wrote at 2am.\n🧠 Fun fact\nClaude Code holds up to 200,000 tokens of context in memory. That’s your entire project, your architecture docs, and probably your unfinished README — all at once. Copilot gets about 8,000. Do the math.\nThe key difference from other AI tools: Claude Code is agentic. It doesn’t just suggest — it does. Ask it to “add authentication to the user service” and it’ll actually read your code, figure out what you’re using, and make it happen. It’s the difference between a GPS giving you directions and a GPS that also drives the car.\n“I used to Google ‘how to reverse a string in Java’ every single time. Now Claude just does it for me. My Google history has never been cleaner.”\n💳 Prerequisites (Yes, You Need to Pay) Sorry, no free lunch here. But the good news is the cheapest plan already includes everything you need to get started:\n⚠️ Heads up\nIf you run a lot of Claude Code sessions simultaneously, you’ll burn through your rate limits faster than a frontend dev burning through console.log() statements. Claude Max ($100–200/month) exists for that use case.\n⚙️ Installation (The Easy Part, We Promise) One line. That’s it. No npm, no Homebrew tap, no reading a 47-step Medium article. Just paste this and go make a coffee while it installs:\nmacOS / Linux # This one line is doing more work than you did all morning curl -fsSL https://claude.ai/install.sh | bash # Make sure it worked claude --version Windows (PowerShell) # For our Windows friends. No judgment. (Okay, maybe a little.) irm https://claude.ai/install.ps1 | iex Homebrew (macOS, if you’re fancy) brew install --cask claude-code 🚫 Don’t do this\nThe old way — npm install -g @anthropic-ai/claude-code — is officially deprecated. If you used it before, run claude install to migrate. Your npm-installed past is a mistake you can fix.\nFirst launch: authenticate Run claude in any directory. A browser window opens, you log in with your Claude account, and boom — you\u0026rsquo;re in. If you\u0026rsquo;re behind a corporate firewall that blocks OAuth callbacks (classic), use an API key via the ANTHROPIC_API_KEY environment variable instead.\n💡 Pro tip\nIf it fails to authenticate, check your .claudeignore isn\u0026rsquo;t too aggressive. Actually wait — you don\u0026rsquo;t have one yet. Never mind, you\u0026rsquo;re fine. Keep going.\n🐣 Your First Session (Don’t Panic) Navigate to your project and type claude. You\u0026rsquo;ll see the Claude Code prompt. Here\u0026rsquo;s the crucial part: don\u0026rsquo;t just immediately say \u0026ldquo;build me a feature.\u0026rdquo; That\u0026rsquo;s like hiring a new developer and skipping the entire onboarding process. Claude needs context. Give it a tour first.\nRun /init— Let Claude scan your projectThis generates a starter CLAUDE.md file. Claude reads your directory structure, sniffs out your build system and test framework, and writes a first draft. It\u0026rsquo;s like Claude doing its own onboarding. Adorable and useful. Ask Claude to explain things before touching themTry: “Explain the architecture of this project.” or “How does authentication work here?” Claude builds a mental model of your code. You’ll get better results when it actually understands what it’s working with. Enable Plan Mode with Shift+Tab before any changeClaude will describe exactly what it intends to do before touching a single file. Think of it as asking a surgeon to show you the plan before they start cutting. Highly recommended. Review → Approve → Commit → RepeatRead the diff. Run your tests. Commit small. If Claude did something weird, a small commit is easy to revert. “I deleted the database” is much funnier as a story than an actual incident. ✨ The CLAUDE.md File (Where the Magic Lives) This is the most important thing in this whole guide. Read it twice.\nCLAUDE.md is a Markdown file you put in your project root. Claude reads it automatically at the start of every single session. It\u0026rsquo;s like a briefing document for your AI colleague — except unlike your actual team\u0026rsquo;s onboarding docs, people actually read this one.\nWithout it, Claude starts every session with zero context about your project, your conventions, your stack, your rules. With it, Claude starts every session knowing exactly how you work.\n🎯 The one-sentence summary: CLAUDE.md is your project’s brain dump, and Claude will actually remember and use everything you put in it — unlike that Wiki your team hasn’t touched since 2022.\nA real CLAUDE.md for a Spring Boot project Here’s a solid example for a Java/Spring Boot/PostgreSQL stack:\n# CLAUDE.MD ## Project REST API for customer order management. Java 17, Spring Boot 3, PostgreSQL 15, deployed on AWS EKS via Helm + ArgoCD. ## Commands - Build: `./mvnw clean package -DskipTests` - Test: `./mvnw test` - Run: `./mvnw spring-boot:run` - Lint: `./mvnw checkstyle:check` ## Architecture - Layered: Controller → Service → Repository (Spring Data JPA) - Async events via Spring ApplicationEventPublisher - DB migrations: Flyway (src/main/resources/db/migration) - All config via environment variables. No hardcoded secrets. Ever. ## Conventions - Constructor injection only. @Autowired on fields is a war crime. - All endpoints return ResponseEntity with standard error envelope - Unit tests: JUnit 5 + Mockito. Integration tests: @SpringBootTest + Testcontainers - Commits: imperative mood, max 72 chars (\u0026#34;Add order validation\u0026#34;, not \u0026#34;Added\u0026#34;) ## Hard Rules - Never modify existing Flyway migration files. Never. - Never commit .env or application-local.properties - No wildcard imports 💡 Pro tip\nCommit your CLAUDE.md to Git. Every teammate then gets the same Claude setup automatically. No more Claude giving one dev constructor injection advice and another dev @Autowired advice in the same codebase.\n📝 CLAUDE.md Best Practices (What Actually Works) Keep it under 100 lines Seriously. For every line you add, ask: “Would removing this cause Claude to make a mistake?” If no, delete it. A fat CLAUDE.md with 500 lines of noise is worse than a lean one with 60 lines of signal. Less is more. (Yes, the same rule applies to your commit messages.)\nUse the WHY / WHAT / HOW structure WHY — One paragraph on what the project exists to do. Domain context matters. Claude giving advice on an e-commerce API vs a healthcare API should feel different. WHAT — Tech stack, key dependencies, deployment target. Be specific about versions. HOW — Exact commands. Exact conventions. Your team’s particular quirks. The stuff Claude can’t infer from the code alone. ✅ Good things to put in CLAUDE.md Build, test, lint, run commands — saves Claude from guessing and running the wrong thing Architecture summary — layers, patterns, where important things live Non-obvious rules — things that would surprise a new dev (“never modify migration files”) Testing philosophy — unit vs integration, which frameworks, naming conventions Things that have burned you before — sprinkle your trauma in there 🚫 Things NOT to put in CLAUDE.md Code style rules your linter already enforces — don’t tell Claude twice what Checkstyle already screams about Secrets, passwords, API keys — you know this, but we’re saying it anyway Entire config files pasted in — link or summarise, don’t paste 200 lines of XML Your personal frustrations about legacy code — Claude doesn’t need your trauma dump Bonus: Sub-directory CLAUDE.md for big projects Monorepo? Complex project with distinct modules? Put a CLAUDE.md inside subdirectories too. Claude reads and merges them — the root file is the global rules, the subdirectory file adds module-specific context. Like a company handbook + a team-specific playbook.\n# SRC/NOTIFICATION-SERVICE/CLAUDE.MD ## Notification Service Handles email + push via AWS SES and SNS. Templates: src/main/resources/templates (Thymeleaf). ## Module Rules - All outbound calls MUST go through NotificationGateway - Do not call SES/SNS directly from service classes - Retry logic: Spring Retry handles it. Don\u0026#39;t add your own loops. 🙈 The .claudeignore File Like .gitignore but for Claude. It tells Claude which files to skip so it doesn\u0026rsquo;t waste context on your target/ folder and doesn\u0026rsquo;t accidentally read your .env file out loud during pair programming.\n# .CLAUDEIGNORE# Build output — Claude doesn\u0026#39;t need to read your compiled bytecode target/ build/ dist/ # Secrets - for the love of all that is holy .env .env.local application-local.properties *.pem *.key # Dependencies - it knows what Spring Boot is, trust it node_modules/ .gradle/ # Generated code (if you don\u0026#39;t want Claude accidentally \u0026#34;fixing\u0026#34; it) src/generated/ 🚫 Classic mistake\nAdding something too broad like *.java to .claudeignore. Now Claude can\u0026rsquo;t read any of your source files. It will be very polite about it but also completely useless. Be specific with your ignore patterns.\n🛡️ Plan Mode (Look Before You Leap) Plan Mode is what separates “Claude deleted half my project” stories from “Claude is amazing” stories. Before making any changes, Claude tells you exactly what it plans to do: which files it’ll touch, what changes it’ll make, and why. You say yes or no. Then it acts.\n⌨️ Just press Shift + Tab to toggle it on. That\u0026rsquo;s it. Best keyboard shortcut you\u0026rsquo;ll use today.\n😌 Wisdom\nAs a beginner, use Plan Mode for everything. As you build trust in Claude’s output quality (and your own CLAUDE.md), you can skip it for small, well-understood tasks. Think of it like asking your junior dev to write tests first vs. just letting your senior dev rip.\n🔄 The Holy Workflow: Explore → Plan → Implement → Commit This is Anthropic’s official recommended workflow. It sounds obvious but most beginners skip the first two steps, and that’s why they end up on Reddit asking why Claude rewrote their entire service layer.\n🔍 Explore — read before you writeAsk Claude to explain the relevant parts of your code before asking it to change anything. “How does order processing work in this project?” Claude builds a proper mental model. Future changes will actually fit your architecture. 📋 Plan — agree on the approachEnable Plan Mode. Describe what you want. Push back on the plan if something seems wrong. This is the cheapest place to catch misunderstandings — before any code is written. ⚙️ Implement — let Claude cookWith the plan approved, Claude implements. Watch the output. If something looks off, hit Ctrl+C. Don\u0026rsquo;t let it finish a 15-file change that you already know is wrong by file 3. ✅ Commit — small and oftenReview the diff. Run tests. Commit. Small commits are your insurance policy. “Revert one small commit” is a two-second job. “Unravel three hours of AI-generated changes” is a Thursday afternoon you’ll never get back. “I skipped the Explore step, asked Claude to ‘refactor auth’, and watched it replace my entire session management layer in 45 seconds. It was beautiful and terrifying in equal measure.”\n⌨️ Commands Cheat Sheet Type these inside an active Claude Code session (the ones starting with /):\nCI/CD / Scripting mode Claude Code has a --print flag for non-interactive use. Perfect for Jenkins pipelines, GitLab CI jobs, or any situation where you want Claude to run and exit without waiting for your input:\n# Run a single command and exit — great for pipelines claude --print \u0026#34;Summarise changes in the last 5 commits\u0026#34; # Works great in your Jenkinsfile or .gitlab-ci.yml claude --print \u0026#34;Check this changeset for obvious bugs\u0026#34; --no-color 💡 Bonus tip\nThe /clear command is your friend. When a conversation has gone on too long and Claude starts giving weird answers, clear it. CLAUDE.md re-loads automatically, so you lose the conversation but not the context. Fresh start, same configuration.\n🎒 The “I’m Late to a Meeting” Summary Install with one curl command — no Node.js, no npm, no fuss You need a paid plan — Claude Pro at $20/mo is enough to start Run /init in every new project. Do it now. Do it always. CLAUDE.md = Claude’s briefing doc. 50–100 lines, WHY/WHAT/HOW structure Commit CLAUDE.md to Git — the whole team benefits, zero extra effort Always use Plan Mode (Shift+Tab) before non-trivial changes Workflow: Explore → Plan → Implement → Commit. In that order. Always. Use .claudeignore to keep secrets and build junk out of Claude’s sight Keep commits small — your future self will thank you Update your CLAUDE.md as your project grows — stale instructions cause chaos ","permalink":"https://sukachvitally.github.io/posts/claude-code-a-beginners-guide-that-doesn-t-put-you-to-sleep/","summary":"\u003cp\u003eFor developers who want an AI in their terminal and still have no idea what CLAUDE.md is. You’ve come to the right place.\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/claude-code-a-beginners-guide-that-doesn-t-put-you-to-sleep/image-01.png\"\u003e\u003c/p\u003e\n\u003ch3 id=\"-what-even-is-claudecode\"\u003e🤔 What Even IS Claude Code?\u003c/h3\u003e\n\u003cp\u003eOkay so you’ve heard of GitHub Copilot, right? Claude Code is like that, except instead of whispering suggestions into your IDE like a shy intern, it’s a \u003cstrong\u003efull-blown agentic assistant that lives in your terminal\u003c/strong\u003e and acts like a senior developer who has had three espressos and is ready to ship.\u003c/p\u003e","title":"Claude Code: A Beginner’s Guide That Doesn’t Put You to Sleep"},{"content":"And why senior developers keep making them\nI’ve reviewed a lot of Spring Boot codebases. Startups burning through their runway trying to scale. Enterprises with ten-year-old monoliths being carved into microservices. Teams of three and teams of fifty. Different industries, different sizes, different levels of experience.\nThe mistakes are almost always the same.\nThat’s not an insult — it’s actually the interesting part. These aren’t beginner errors. Most of the teams I work with are smart, experienced engineers who made reasonable decisions under pressure. The problem is that Spring Boot is so productive, so frictionless, that bad patterns compound quietly. You ship fast. The tests pass. Production looks fine. And then six months later you’re staring at a 2,000-line controller and wondering how it got there.\nThis article is about the five patterns I see most often, why they happen, and — more importantly — how to fix them without rewriting everything from scratch.\nMistake 1: Putting business logic in controllers This is the most common one by a significant margin. It usually starts innocently: a small validation check here, a conditional there, a bit of data transformation before calling a service. Before long, your @RestController is making decisions it has no business making.\nHere’s what it looks like in the wild:\n@PostMapping(\u0026#34;/orders\u0026#34;) public ResponseEntity\u0026lt;OrderResponse\u0026gt; createOrder(@RequestBody OrderRequest request) { if (request.getItems().isEmpty()) { return ResponseEntity.badRequest().build(); } BigDecimal total = request.getItems().stream() .map(item -\u0026gt; item.getPrice().multiply(BigDecimal.valueOf(item.getQuantity()))) .reduce(BigDecimal.ZERO, BigDecimal::add); if (total.compareTo(new BigDecimal(\u0026#34;10000\u0026#34;)) \u0026gt; 0 \u0026amp;\u0026amp; !userService.isPremium(request.getUserId())) { return ResponseEntity.status(HttpStatus.FORBIDDEN).build(); } // ... and it keeps going } The controller now knows about pricing rules, user tiers, and order validation. None of that belongs here.\nWhy it happens: Controllers are the entry point, so they feel like the natural place to “handle” things. Also, injecting a service and calling one method feels like overkill for “just a quick check.”\nWhy it matters: Business logic in controllers can’t be unit tested without spinning up the web layer. It can’t be reused from a message consumer, a scheduled job, or a CLI command. It’s invisible to anyone who looks at your service layer expecting to understand the domain.\nThe fix: Your controller should do exactly three things — deserialise the request, call a service method, and serialise the response. Every conditional, every calculation, every rule goes into a service or domain object. If your controller has an ifstatement that isn\u0026rsquo;t about HTTP concerns, move it.\n@PostMapping(\u0026#34;/orders\u0026#34;) public ResponseEntity\u0026lt;OrderResponse\u0026gt; createOrder(@RequestBody OrderRequest request) { OrderResponse order = orderService.createOrder(request); return ResponseEntity.ok(order); } Boring. Perfect.\nMistake 2: Ignoring transaction boundaries Transaction management is one of those topics where a little knowledge is genuinely dangerous. Most developers know @Transactional exists. Fewer understand where to put it, what it actually does, and — critically — what it doesn\u0026rsquo;t do.\nThe most painful version of this mistake looks like this:\n@Service public class PaymentService { @Transactional public void processPayment(Long orderId) { Order order = orderRepository.findById(orderId).orElseThrow(); order.setStatus(OrderStatus.PAID); orderRepository.save(order); // This call is to an external HTTP service. // If it fails, the order status update is rolled back. // If it succeeds but the next line throws, you\u0026#39;ve charged // the customer but their order is still PENDING. paymentGateway.charge(order.getAmount()); inventoryService.reserve(order.getItems()); // can throw } } The transaction boundary wraps an external HTTP call and a separate service call. The atomicity guarantee you think you have doesn’t exist across those boundaries.\nWhy it happens: @Transactional feels like a safety net you throw over anything that looks important. It\u0026rsquo;s easy to apply and invisible when it\u0026rsquo;s wrong.\nWhy it matters: Inconsistent data is the hardest class of bug to diagnose and fix in production. By the time you notice, the damage is already done.\nThe fix: Be deliberate about what your transaction actually protects. Keep transactions short, keep them within a single data source, and handle external calls outside the transactional boundary. Consider the outbox pattern for operations that need to be reliably paired with an external call. And always ask: “if this line throws, what state am I leaving the database in?”\nAlso watch out for the self-invocation trap — calling a @Transactional method from within the same class bypasses the proxy and silently does nothing. This one has burned almost every team I\u0026rsquo;ve worked with.\nMistake 3: Overusing @Autowired everywhere Field injection is convenient. It’s also the option that makes your code harder to test, harder to reason about, and full of hidden dependencies you only discover at runtime.\n@Service public class ReportService { @Autowired private UserRepository userRepository; @Autowired private OrderRepository orderRepository; @Autowired private EmailService emailService; @Autowired private PdfGenerator pdfGenerator; @Autowired private AuditService auditService; ... } Five dependencies injected invisibly. You can’t instantiate this class in a test without a Spring context, or without reaching for @MockBean and a full application bootstrap. You also can\u0026rsquo;t see — at a glance — that this class might be doing too much.\nWhy it happens: Field injection is fewer lines of code and Spring supports it, so it feels fine. Early tutorials often show it this way and the habit sticks.\nWhy it matters: Constructor injection makes dependencies explicit and visible. It enables plain unit tests without a Spring context. It makes circular dependencies a compile-time error rather than a runtime surprise. And when a class has eight constructor parameters, that’s a signal — a service with eight dependencies probably has too many responsibilities.\nThe fix: Use constructor injection. Always. Since Spring 4.3 you don’t even need @Autowired on a single-constructor class. If you\u0026rsquo;re using Lombok, @RequiredArgsConstructor on final fields gives you this for free.\n@Service @RequiredArgsConstructor public class ReportService { private final UserRepository userRepository; private final OrderRepository orderRepository; private final EmailService emailService; ... } If the constructor starts to look unwieldy, take that as the feedback it is.\nMistake 4: No proper exception handling strategy Every codebase has exception handling. Very few have an exception handling strategy. The difference shows up as a patchwork of try-catch blocks scattered across service methods, inconsistent error responses from your API, and stack traces leaking into production responses.\nCommon signs you don’t have a strategy:\n// In one service: try { return userRepository.findById(id).orElseThrow(); } catch (Exception e) { log.error(\u0026#34;Error\u0026#34;, e); return null; // 🚩 silent failure } // In another: if (user == null) throw new RuntimeException(\u0026#34;User not found\u0026#34;); // 🚩 generic exception // In a third: } catch (UserNotFoundException e) { return ResponseEntity.status(500).body(\u0026#34;Something went wrong\u0026#34;); // 🚩 wrong status code Why it happens: Exception handling feels like plumbing — unglamorous work that gets deferred and then done inconsistently across a team over months or years.\nWhy it matters: Inconsistent error handling makes your API unpredictable for consumers, makes debugging painful, and creates security risk when internal details leak into responses.\nThe fix: Define a small hierarchy of domain exceptions (ResourceNotFoundException, ValidationException, ConflictException) and throw those deliberately rather than wrapping everything in RuntimeException. Then handle them all in one place using @RestControllerAdvice:\n@RestControllerAdvice public class GlobalExceptionHandler { @ExceptionHandler(ResourceNotFoundException.class) public ResponseEntity\u0026lt;ErrorResponse\u0026gt; handleNotFound(ResourceNotFoundException ex) { return ResponseEntity.status(HttpStatus.NOT_FOUND) .body(new ErrorResponse(\u0026#34;NOT_FOUND\u0026#34;, ex.getMessage())); } @ExceptionHandler(ValidationException.class) public ResponseEntity\u0026lt;ErrorResponse\u0026gt; handleValidation(ValidationException ex) { return ResponseEntity.status(HttpStatus.BAD_REQUEST) .body(new ErrorResponse(\u0026#34;VALIDATION_ERROR\u0026#34;, ex.getMessage())); } @ExceptionHandler(Exception.class) public ResponseEntity\u0026lt;ErrorResponse\u0026gt; handleUnexpected(Exception ex) { log.error(\u0026#34;Unexpected error\u0026#34;, ex); return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR) .body(new ErrorResponse(\u0026#34;INTERNAL_ERROR\u0026#34;, \u0026#34;An unexpected error occurred\u0026#34;)); } } One place. Consistent responses. No stack traces in production. Done.\nMistake 5: Treating application.properties as a dumping ground This one starts as a minor annoyance and grows into a genuine operational problem. A file that begins as twelve lines of database config becomes three hundred lines of undocumented magic numbers, environment-specific hacks, feature flags, and credentials that probably shouldn’t be there at all.\n# What does this do? Who added it? Is it still needed? some.service.timeout=5000 other.thing.enabled=true legacy.mode=false max.retry.count=3 feature.new-checkout=true internal.api.key=sk_live_abc123... Why it happens: application.properties is the path of least resistance for any config that doesn\u0026rsquo;t obviously belong somewhere else. Nobody wants to create a new @ConfigurationProperties class for one value.\nWhy it matters: At scale, unstructured configuration is an operational risk. Properties get duplicated across environments, credentials end up in version control, values change without anyone knowing what breaks, and onboarding a new developer becomes a treasure hunt.\nThe fix: Group related configuration using @ConfigurationProperties with typed, documented classes:\n@ConfigurationProperties(prefix = \u0026#34;payment\u0026#34;) @Validated public record PaymentProperties( @NotBlank String apiKey, @Min(1) @Max(30) int timeoutSeconds, @Min(0) int maxRetries ) {} This gives you type safety, validation on startup, IDE autocomplete, and a single place to document what each value does. Secrets should come from environment variables or a secrets manager — never committed to a repository. And if a property hasn’t been touched in six months, it probably shouldn’t exist.\nClosing: a quick checklist to self-audit your codebase You don’t need to fix everything at once. Run through this in an hour and you’ll know exactly where to focus.\nControllers\nDo any controllers contain if statements unrelated to HTTP status? Are there calculations or data transformations happening before a service call? Is any business rule enforced at the controller level? Transactions\nAre there @Transactional methods that call external HTTP services? Are there @Transactional methods calling other @Transactional methods on the same class? Do you know what state the database is in if any line in a transactional method throws? Dependency injection\nAre you using field injection (@Autowired on fields)? Can you instantiate your service classes in a unit test without a Spring context? Are there any services with more than five or six dependencies? Exception handling\nDo you have a @RestControllerAdvice that handles all exceptions in one place? Are you catching Exception in service methods and returning null or swallowing errors? Are stack traces or internal error messages reaching API responses in production? Configuration\nIs application.properties longer than 50 lines with no grouping or comments? Are any secrets or API keys committed to version control? Are there properties whose purpose isn’t immediately obvious? If you ticked more than three of those boxes, your codebase is normal — this is just what production software looks like after a few years of growth under pressure. The good news is that every one of these issues is fixable incrementally, without a rewrite, one pull request at a time.\n","permalink":"https://sukachvitally.github.io/posts/5-spring-boot-mistakes-i-see-in-every-codebase/","summary":"\u003cp\u003e\u003cem\u003eAnd why senior developers keep making them\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/5-spring-boot-mistakes-i-see-in-every-codebase/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eI’ve reviewed a lot of Spring Boot codebases. Startups burning through their runway trying to scale. Enterprises with ten-year-old monoliths being carved into microservices. Teams of three and teams of fifty. Different industries, different sizes, different levels of experience.\u003c/p\u003e\n\u003cp\u003eThe mistakes are almost always the same.\u003c/p\u003e\n\u003cp\u003eThat’s not an insult — it’s actually the interesting part. These aren’t beginner errors. Most of the teams I work with are smart, experienced engineers who made reasonable decisions under pressure. The problem is that Spring Boot is so productive, so frictionless, that bad patterns compound quietly. You ship fast. The tests pass. Production looks fine. And then six months later you’re staring at a 2,000-line controller and wondering how it got there.\u003c/p\u003e","title":"5 Spring Boot Mistakes I See in Every Codebase"},{"content":"\nHow distributed teams structure the Software Development Life Cycle — governance models, communication contracts, phase-by-phase responsibilities, and the hidden failure modes that kill outsourced projects before launch.\nWhat makes outsourced SDLC different Every software team runs some version of the SDLC — a structured sequence of phases that takes an idea from concept to running code. When delivery is handled by an external vendor, the same phases exist, but the dynamics between them change completely.\nIn a co-located team, context flows informally: a developer can tap the product manager on the shoulder, an architect can sketch on a whiteboard in the hallway. In an outsourced engagement, none of that happens. Every shared understanding has to be written down, formally agreed, and deliberately transmitted. What a co-located team resolves in a five-minute hallway conversation requires an async message thread, a clarification document, and a sign-off in outsourcing.\nThis is not a disadvantage — it is a discipline. Outsourced teams that develop rigorous documentation habits often produce more traceable, better-specified software than co-located teams where “everyone just knows” how things work. The SDLC is the framework that makes this discipline systematic.\nCORE PRINCIPLE\nThe client owns the what — product vision, business goals, acceptance criteria, and ultimate decision authority. The vendor owns the how — technical approach, tooling choices, implementation detail, and day-to-day engineering execution. The SDLC is the interface between them.\nFigure 1 — The seven SDLC phases in outsourced delivery. Click any phase to explore it. The dashed arc represents the feedback loop from maintenance back into discovery for the next cycle.\nThe dual-ownership model The most important structural concept in outsourced SDLC is dual ownership: two distinct parties hold responsibility for different layers of the project simultaneously. This is not a hierarchy — the client is not “above” the vendor — it is a division of accountability along a clear interface.\nConfusion about ownership is the single most common root cause of outsourced project failure. When both parties believe they are responsible for the same thing (or each believes the other is responsible), gaps appear and deliverables fall through.\nCLIENT SIDE Product ownership layer\nBusiness requirements and user needs Acceptance criteria and Definition of Done Backlog prioritisation and scope decisions UAT sign-off and production approval Regulatory and compliance requirements Budget authority and contract governance VENDOR SIDE Engineering execution layer\nTechnical architecture and tooling selection Sprint planning and capacity management Code quality, test coverage, and CI/CD Daily engineering decisions and trade-offs Infrastructure setup and monitoring Technical documentation and ADRs SHARED Interface \u0026amp; governance\nSprint reviews and retrospectives Risk register and escalation path Change request evaluation and impact Definition of Ready for sprint entry Release decision and go/no-go gates KEY ROLES Who sits at the interface\nClient: Product Owner + Technical Lead Vendor: Project Manager + Tech Lead These four people own the interface All escalations route through them first MOST COMMON FAILURE MODE\nProjects fail when the client Product Owner is not genuinely available. If the PO attends only one ceremony per sprint, ambiguities accumulate, the vendor makes assumptions, and by Month 3 the product being built is not the product the client imagined. PO availability is a project risk, not a courtesy.\nPhase 1 — discovery \u0026amp; planning Discovery is where the project is defined before a line of code is written. In outsourcing, this phase is often compressed or skipped entirely — and it is the single biggest predictor of whether the engagement will succeed. A well-run discovery produces a shared understanding of what is being built, why it matters, what it does not include, and how success will be measured.\nThe outputs of discovery are not optional. They form the contractual baseline for the engagement. Any change to scope relative to the discovery outputs is a change request — a formal negotiation, not a verbal agreement.\nDiscovery outputs — the mandatory artifact set Project Charter — business goals, success metrics, constraints, and named stakeholders with decision authority High-level roadmap — time-boxed phases with milestone deliverables and payment tranches tied to each Scope boundary document — an explicit list of what is in scope and what is explicitly out of scope for this engagement Technical feasibility assessment — vendor’s initial evaluation of risk, unknowns, and proposed technology approach Communication protocol — meeting cadence, tools, async response SLAs (Service Level Agreement), and escalation path Definition of Done — the conditions every deliverable must satisfy before the client signs off TIME INVESTMENT\nFor a six-month project, expect discovery to take two to four weeks and cost 5–8% of total budget. This is not overhead — it is insurance. Every hour spent in discovery typically saves three to five hours of rework later.\nFigure 2 — Ownership swimlane across SDLC phases. Click any phase row to explore the ownership detail. Every phase ends with a formal handoff artifact — nothing advances without a signed-off output.\nPhase 2 — requirements Requirements gathering in outsourcing is a negotiation, not a dictation. The client provides the business intent; the vendor translates it into implementable specifications and raises every ambiguity in writing. The output — typically a Software Requirements Specification or a groomed product backlog with acceptance criteria — is the contract for the build phase.\nThe two most common failure modes at this phase are: the client providing requirements that are too high-level (“the user should have a good experience”), and the vendor accepting ambiguous requirements without escalating. Both failures are preventable with process discipline.\nREQUIREMENTS QUALITY TEST\nEvery user story must answer three questions unambiguously: Who is the actor? What do they want to do? Why does it matter to the business? If any of the three is vague, the story is not ready for development. The vendor’s QA on requirements is as important as the client’s initial authorship.\nThe Requirements Clarification Log Every ambiguity the vendor identifies must be logged in a shared document — the Requirements Clarification Log — with a status of Open, Answered, or Accepted. This log is reviewed at the start of every sprint planning session. Any story with open clarifications cannot be pulled into the sprint.\nPhase 3 — system design System design is where technical architecture is decided and documented before implementation begins. In outsourcing, this phase serves an additional function: it is the moment when the vendor demonstrates technical competence and the client gains confidence (or identifies concerns) in the approach before committing to the build.\nThe key outputs are Architecture Decision Records — short documents that capture each significant technical decision, the alternatives considered, and the rationale for the chosen approach. ADRs are owned by the vendor but approved by the client’s technical lead. They survive team turnover and serve as institutional memory for the lifetime of the codebase.\nFigure 3 — System design phase: what goes in, what is decided, and what comes out. Every output requires client technical lead approval before development begins.\nPhases 4–5 — development and testing Development is typically the longest phase and the one where outsourcing relationships are most stress-tested. The vendor team is executing sprints; the client team is reviewing, testing, and making real-time decisions about scope. The quality of the interface between the two determines whether this phase runs smoothly or deteriorates into confusion and rework.\nTesting in an outsourced context has two layers: the vendor is responsible for technical correctness (unit tests, integration tests, regression suites), while the client is responsible for business correctness (User Acceptance Testing). Neither can substitute for the other. A bug-free system that does the wrong thing is a failed deliverable; a system with business-correct behaviour but fragile code is a maintenance liability.\nSprint planning SHARED CEREMONY\nStories are pulled from the refined backlog. Vendor estimates and commits. Client PO confirms priorities and clarifies ambiguities before sprint begins — not during it.\nDaily async standup VENDOR-LED, CLIENT-VISIBLE\nA structured async update (Slack, Linear, or similar) covering progress, plan, and blockers. Client can read but should not generate work requests through this channel.\nCode review protocol VENDOR PROCESS\nEvery pull request requires at least one peer review before merge. Client retains repository access and may review code, but day-to-day code review is a vendor responsibility.\nSprint review / demo SHARED CEREMONY\nThe vendor demos working software against the sprint goal. Client stakeholders provide feedback in the session. Verbal feedback is captured in writing before the session closes.\nVendor QA VENDOR-OWNED\nUnit, integration, and regression tests run in CI on every commit. Test coverage and pass rate are reported to the client at each sprint review as part of the standard dashboard.\nClient UAT CLIENT-OWNED\nClient team tests against acceptance criteria in the staging environment. UAT is time-boxed: a defined window, a defined scope, a written pass/fail result. Unbounded UAT stalls releases.\nTESTING ANTI-PATTERN TO AVOID\nNever treat UAT as a second round of development. If the client discovers a new requirement during UAT, it is a change request — not a bug. The distinction matters for both schedule and budget. Document the difference in the Definition of Done agreed in Phase 1.\nFigure 4 — The two-week sprint cycle. Ceremonies marked “shared” require both client and vendor attendance. Skipping shared ceremonies is the most reliable way to accumulate hidden misalignment.\nPhases 6–7 — deployment and maintenance Deployment is a production event, and in outsourcing it requires formal client approval before the vendor touches the production environment. This approval should be a written go/no-go decision, not a verbal message. The vendor provides a release runbook — a step-by-step guide to what is being deployed, in what order, with a tested rollback procedure for every step.\nMaintenance defines the ongoing relationship after launch. It should be governed by a Service Level Agreement that specifies response times for different severity levels, reporting cadence, and the process for prioritising maintenance requests against new feature work. Without a maintenance SLA, the post-launch period becomes a source of friction: the client expects immediate responses; the vendor has moved on to the next engagement.\nMAINTENANCE SLA TIERS — TYPICAL STRUCTURE\nP0 — Critical: system down or data loss risk. Response within 1 hour, resolution target 4 hours. P1 — High: major feature broken, no workaround. Response within 4 hours. P2 — Medium: feature degraded, workaround exists. Response within 1 business day. P3 — Low: cosmetic or minor issue. Response within 1 week.\nGovernance and sprint cadence Governance in outsourcing is not bureaucracy — it is the system that keeps the project honest. Without governance, small misalignments compound into large disputes. With governance, problems surface early when they are still cheap to fix.\nThe minimum viable governance framework for an outsourced engagement has three components: a sprint governance checklist run at every review, a monthly steering committee with executive-level attendance from both parties, and a risk register updated continuously and reviewed formally each month.\nSprint review governance checklist All committed stories are either accepted, carried forward with documented reason, or formally descoped — no stories silently dropped Sprint velocity is reported. If actual velocity is more than 20% below baseline, a root cause is documented and a recovery plan is presented Test coverage report is shared. No story is accepted without its associated tests in the CI suite Open bugs are counted and triaged. No P0 or P1 bug is carried forward without a written remediation commitment and target date Architecture Decision Records are up to date — any design decision made during the sprint is documented before the review closes Risk register is reviewed. New risks are assigned owners. Mitigations on existing risks are status-checked Next sprint’s stories are refined, estimated, and in a Definition-of-Ready state before sprint planning begins Any change requests received during the sprint are logged with estimated impact on scope, schedule, and budget — not silently absorbed Client retains unrestricted access to the repository, CI/CD pipeline, staging environment, and all project documentation THE MOST IMPORTANT GOVERNANCE RULE\nEvery verbal agreement made during a meeting must be captured in writing before the meeting ends — in the meeting notes, in the ticket, or in a follow-up message that is explicitly acknowledged. “We agreed in the call” is not an agreement in outsourcing. Writing is the agreement.\nFigure 5 — Three-tier governance cadence. Daily standups surface blockers in hours. Fortnightly reviews surface sprint-level misalignment in days. Monthly steering catches strategic drift before it becomes a crisis.\nClosing: what makes it work Outsourced SDLC succeeds when both parties treat the engagement as a collaboration rather than a transaction. The vendor is not a code factory executing a specification; they are a technical partner making hundreds of decisions per week that affect the client’s product. The client is not a passive recipient waiting for deliverables; they are an active participant whose availability, clarity, and decision speed directly determines the project’s velocity.\nThe SDLC framework described here — with its documentation discipline, formal handoffs, shared ceremonies, and governance cadence — is not bureaucracy for its own sake. It is the minimum structure needed to replace the informal coordination that happens naturally in a co-located team. Every element exists because its absence has been shown, repeatedly, to cause projects to fail.\n","permalink":"https://sukachvitally.github.io/posts/sdlc-in-an-outsourced-company-how-the-process-really-works/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/sdlc-in-an-outsourced-company-how-the-process-really-works/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eHow distributed teams structure the Software Development Life Cycle — governance models, communication contracts, phase-by-phase responsibilities, and the hidden failure modes that kill outsourced projects before launch.\u003c/p\u003e\n\u003ch3 id=\"what-makes-outsourced-sdlc-different\"\u003eWhat makes outsourced SDLC different\u003c/h3\u003e\n\u003cp\u003eEvery software team runs some version of the SDLC — a structured sequence of phases that takes an idea from concept to running code. When delivery is handled by an external vendor, the same phases exist, but the dynamics between them change completely.\u003c/p\u003e","title":"SDLC in an Outsourced Company: How the Process Really Works"},{"content":"\nEvery time two transactions run at the same time in PostgreSQL, something remarkable happens under the hood. Nobody locks the table. Nobody waits. Yet everyone sees a consistent view of the data. This is MVCC — Multi-Version Concurrency Control — and once you understand it, you’ll never look at your database the same way again.\nWhat is MVCC? Imagine a library where every book has a version number. When you check out a book to read it, the librarian doesn’t lock it away from everyone else. Instead, they hand you the version that existed at the moment you arrived. If someone else rewrites the book while you’re reading — you still see the original. You’re reading a snapshot in time.\nThat’s MVCC. Multi-Version Concurrency Control means PostgreSQL keeps multiple versions of each row simultaneously. Readers see a consistent snapshot of the database at the moment their transaction started. Writers create new versions without touching what readers see. Readers never block writers. Writers never block readers.\nIn PostgreSQL, every transaction lives in its own bubble of reality — a consistent snapshot of the world at the moment it began.\nThe problem it solves The naive approach to concurrent database access is locking — when someone reads a row, lock it so nobody can write it. When someone writes, lock the whole table. This works, but it turns your high-throughput database into a single-file queue. MVCC replaces most of those locks with versioned row history.\nThe core mechanism: transaction IDs Every transaction in PostgreSQL gets a monotonically increasing transaction ID (called xid). When a row is created or modified, PostgreSQL stamps it with the transaction IDs of who created it and who deleted it. These stamps — stored directly in the row header — are what make MVCC work.\n🔑 THE TWO HIDDEN COLUMNS EVERY ROW CARRIES\nxmin — the transaction ID that created (inserted) this row version.\nxmax — the transaction ID that deleted or superseded this row version. 0 means the row is still live.\nYou can actually query these: SELECT xmin, xmax, * FROM your_table;\nTwo transactions, one table, zero drama Let’s walk through exactly what happens when two transactions run at the same time. We’ll watch every step — from snapshot creation to commit — with the actual row versions PostgreSQL maintains internally.\nOur table: a simple accounts table. Alice has $1000. Transaction A is going to read her balance. Transaction B is going to update it. Watch how they never get in each other\u0026rsquo;s way.\nCREATE TABLE accounts ( id INT PRIMARY KEY, name TEXT, balance NUMERIC ); INSERT INTO accounts VALUES (1, \u0026#39;Alice\u0026#39;, 1000); -- Inserted by transaction xid=100, so xmin=100, xmax=0 -- What PostgreSQL actually stores internally: -- | xmin | xmax | id | name | balance | -- | 100 | 0 | 1 | Alice | 1000 | The timeline — step by step 💡 THE KEY INSIGHT\nWhen Transaction A reads Alice’s balance at T=180ms — after Transaction B has already modified it — it still sees $1000. Because when A started, B’s xid (102) was in the active transaction list. A’s snapshot says: “treat everything from xid=102 as if it doesn’t exist.” This is the snapshot isolation guarantee.\nHow PostgreSQL evaluates row visibility For every row a transaction encounters, PostgreSQL runs a visibility check. It’s essentially a set of rules against the row’s xmin and xmax:\nis_visible(row, my_snapshot): -- Rule 1: creator must have committed before my snapshot IF row.xmin NOT IN my_snapshot.committed: RETURN false -- creator not committed yet -- Rule 2: creator must not be in my active list IF row.xmin IN my_snapshot.active: RETURN false -- creator was in-flight when I started -- Rule 3: if deleted, deleter must NOT have committed before me IF row.xmax != 0: IF row.xmax IN my_snapshot.committed AND row.xmax NOT IN my_snapshot.active: RETURN false -- deleter committed → row is dead to me RETURN true -- this row version is visible to me Row versions on disk — what the heap actually looks like PostgreSQL stores rows in heap pages — 8KB chunks on disk. Every row version (called a tuple) has a header containing the MVCC metadata. When you UPDATE a row, the old tuple isn’t erased — it gets a new xmax stamp and a new tuple is inserted right next to it. Both live on the same page (or a new one if it\u0026rsquo;s full).\n📐 TUPLE HEADER FIELDS\nt_xmin — xid of the inserting transaction | t_xmax — xid of the deleting/updating transaction | t_ctid — physical location (page, offset) of the latest version of this row | t_infomask — bitfield of flags (committed? rolled back? frozen?)\nWatching the heap through an UPDATE Let’s trace the exact state of the heap after each operation on Alice’s row:\n↑ Transaction A (xid=101) reads Offset 1 only — Offset 2 has xmin=102 which is in A’s active set → invisible\n↑ Offset 1 is now a dead tuple — wasting space until VACUUM runs. New transactions see only Offset 2.\n⚠️ THE UPDATE IS ACTUALLY INSERT + LOGICAL DELETE\nPostgreSQL never updates a row in place. Every UPDATE creates a brand new tuple and marks the old one as dead by setting its xmax. This is why long-running UPDATE-heavy workloads cause table bloat — dead tuples accumulate until VACUUM reclaims the space.\nThe ctid chain Each old tuple’s t_ctid field points to the location of the next version. This creates a version chain that PostgreSQL follows to find the latest valid version of a row. You can inspect this with:\n-- See the hidden MVCC columns on any table SELECT ctid, -- physical location: (page, offset) xmin, -- creating transaction ID xmax, -- deleting transaction ID (0 = live) xmin::text::bigint, age(xmax::text::xid) AS xmax_age, * FROM accounts; -- Result during a concurrent update: -- ctid | xmin | xmax | id | name | balance -- (0,1) | 100 | 102 | 1 | Alice | 1000 ← dead, xmax set -- (0,2) | 102 | 0 | 1 | Alice | 1500 ← live, new version -- See how many dead tuples are in a table SELECT n_live_tup, n_dead_tup, last_autovacuum, last_vacuum FROM pg_stat_user_tables WHERE relname = \u0026#39;accounts\u0026#39;; How updates persist — from buffer to disk When Transaction B runs UPDATE accounts SET balance=1500, it doesn\u0026rsquo;t immediately go to the database file on disk. That would be catastrophically slow. Instead, PostgreSQL uses a layered architecture that keeps changes in memory until it\u0026rsquo;s safe and efficient to flush them.\nThe WAL — your durability guarantee The Write-Ahead Log is PostgreSQL’s crash safety mechanism. The rule is simple: before any dirty page can be written to the data file, its changes must first be recorded in the WAL and flushed to disk.\nWhen you COMMIT, PostgreSQL writes a commit record to WAL and calls fsync(). That\u0026rsquo;s what makes your data durable. If PostgreSQL crashes between commit and checkpoint, recovery replays the WAL to restore the committed state. Your $1500 is safe.\n📌 WHAT HAPPENS DURING THE TRANSACTION (BEFORE COMMIT)\nThe new tuple (xmin=102, balance=1500) sits in the shared buffer cache with a pending status. The WAL record has been written for crash recovery purposes, but the page isn\u0026rsquo;t flushed to the main data file yet. Other sessions can\u0026rsquo;t see the new version because xmin=102 is still in the active transaction list of their snapshots.\nCLOG — tracking commit status How does a transaction know whether xmin=102 is committed or rolled back? PostgreSQL maintains the Commit Log (CLOG) — a compact bit array where each transaction ID maps to one of four states:\nWhen a transaction commits, PostgreSQL updates the CLOG bit from IN_PROGRESS to COMMITTED. This is the moment other transactions can start seeing the new tuple. The CLOG update is fast — it\u0026rsquo;s just a bit flip in a shared-memory structure — and PostgreSQL caches the status of recently-seen transactions in tuple header hint bits to avoid CLOG lookups on every read.\n-- Session A: start a transaction and update BEGIN; SELECT pg_backend_pid(), txid_current(); -- pid=12345, txid=105 UPDATE accounts SET balance=1500 WHERE id=1; -- From Session B (while A is still open): SELECT xmin, xmax, balance FROM accounts WHERE id=1; -- xmin=100, xmax=105, balance=1000 ← sees OLD tuple -- The new tuple (xmin=105) exists but is invisible (xid=105 active) -- From Session A itself: SELECT xmin, xmax, balance FROM accounts WHERE id=1; -- xmin=105, xmax=0, balance=1500 ← sees its OWN pending change -- Now commit in Session A: COMMIT; -- CLOG[105] = COMMITTED -- Session B reads again: SELECT xmin, xmax, balance FROM accounts WHERE id=1; -- xmin=105, xmax=0, balance=1500 ← now sees the new version VACUUM — the janitor that nobody talks about Dead tuples don’t disappear on their own. Every UPDATE and DELETE leaves dead row versions in the heap. If nothing cleaned them up, your tables would grow forever. Enter VACUUM — PostgreSQL\u0026rsquo;s background cleanup process.\nVACUUM does several things: it marks dead tuples as free space so new rows can reuse the pages, it updates pg_stat_user_tables with current dead tuple counts, and critically, it advances the transaction ID horizon to prevent XID wraparound — which is the apocalyptic scenario where PostgreSQL has cycled through all 4 billion transaction IDs and can\u0026rsquo;t tell if old data is newer or older than current data.\n-- Standard VACUUM — marks dead space, doesn\u0026#39;t return to OS VACUUM accounts; -- VACUUM FULL — rewrites table, returns space to OS (locks table!) VACUUM FULL accounts; -- VACUUM ANALYZE — vacuum + update query planner statistics VACUUM ANALYZE accounts; -- Check autovacuum health — run this regularly in prod SELECT schemaname, relname AS table_name, n_live_tup, n_dead_tup, ROUND(n_dead_tup::numeric / NULLIF(n_live_tup, 0) * 100, 1) AS dead_pct, last_autovacuum, autovacuum_count FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 10; -- If dead_pct \u0026gt; 20%, autovacuum might not be keeping up -- Tune: autovacuum_vacuum_scale_factor (default 0.2 = 20%) The mental model MVCC in PostgreSQL is built on four simple ideas that compose into something powerful:\nEvery row version is stamped with the transaction that created it (xmin) and the one that deleted it (xmax). Every transaction gets a snapshot — a list of committed transactions at the moment the snapshot is taken. The snapshot determines what’s visible. Writers create new versions, they never overwrite old ones. Readers and writers coexist without locking. Dead versions accumulate until VACUUM reclaims them. Monitor your dead tuple ratio in production. THE ONE THING TO REMEMBER\n“Readers never block writers, writers never block readers.” Everything else in MVCC is the mechanism that makes that true. Understand the snapshot, understand xmin/xmax, and you understand 80% of what PostgreSQL is doing under every query you write.\n","permalink":"https://sukachvitally.github.io/posts/mvcc-invisible-superpower-inside-every-query/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/mvcc-invisible-superpower-inside-every-query/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eEvery time two transactions run at the same time in PostgreSQL, something remarkable happens under the hood. Nobody locks the table. Nobody waits. Yet everyone sees a consistent view of the data. This is MVCC — Multi-Version Concurrency Control — and once you understand it, you’ll never look at your database the same way again.\u003c/p\u003e\n\u003ch3 id=\"what-ismvcc\"\u003eWhat is MVCC?\u003c/h3\u003e\n\u003cp\u003eImagine a library where every book has a version number. When you check out a book to read it, the librarian doesn’t lock it away from everyone else. Instead, they hand you the version that existed at the moment you arrived. If someone else rewrites the book while you’re reading — you still see the original. You’re reading a snapshot in time.\u003c/p\u003e","title":"MVCC - invisible superpower inside every query"},{"content":"\nBinary serialization sounds scary until you realize it’s just a very efficient way of saying “I’ll describe the data once, then send only the bytes I need.” Let’s demystify Avro — from raw bytes to Schema Registry to versioning — with real Java code and zero hand-waving.\nWhat is Avro, and why should you care? Avro is Apache’s answer to “JSON is too fat and XML makes me cry.” It’s a binary serialization formatthat stores data alongside a schema — a JSON document that describes the structure of your data. Think of it as a very strict contract between producer and consumer.\n“Avro is what happens when you tell JSON to stop carrying around its key names every single time.”\nUnlike JSON, Avro doesn’t repeat field names with every record. The schema is sent once (or stored in a registry), and the actual data is pure binary — compact, fast, and type-safe. Your Java objects become a stream of bytes that know exactly what they are.\n🎭 THE ELEVATOR PITCH\nImagine sending a spreadsheet. JSON is like printing the column headers on every single row. Avro sends the headers once, then just the values. 10 million rows later, your network team buys you coffee.\nWhy not just use JSON? JSON is great for REST APIs where humans might read the data. But for high-throughput systems — millions of messages per second — JSON is a bandwidth and CPU hog. Avro gives you:\n~5–10x smaller payload than equivalent JSON Schema evolution — add fields without breaking consumers Type safety — no more \u0026quot;count\u0026quot;: \u0026quot;42\u0026quot; sneaking into your integers Code generation — compile-time safety with generated Java POJOs Where you’ll actually meet Avro in the wild You’re not going to randomly pick Avro for your REST API (please don’t). It shines in specific contexts, almost always involving large volumes of structured data moving through a pipeline.\nThe main places you’ll encounter Avro:\nKafka messaging — the most common use case. Avro is the default serialization format in Confluent’s ecosystem. Apache Spark — efficient columnar-style storage and processing of large datasets. Hadoop / HDFS — Avro files are a first-class citizen in big data pipelines. Microservice event buses — anywhere events need schema contracts enforced at runtime. Data pipelines (Flink, Debezium) — CDC and streaming ETL jobs love Avro’s compactness. A real Avro schema, from scratch An Avro schema is just a JSON file that describes your record. Add it to your project under src/main/avro/ and the Maven/Gradle plugin generates Java classes for you. Let\u0026rsquo;s model a simple user event.\n{ \u0026#34;type\u0026#34;: \u0026#34;record\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;UserEvent\u0026#34;, \u0026#34;namespace\u0026#34;: \u0026#34;com.example.events\u0026#34;, \u0026#34;doc\u0026#34;: \u0026#34;Fired every time a user does something interesting\u0026#34;, \u0026#34;fields\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;userId\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;long\u0026#34;, \u0026#34;doc\u0026#34;: \u0026#34;Internal user ID\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;username\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;action\u0026#34;, \u0026#34;type\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;enum\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;Action\u0026#34;, \u0026#34;symbols\u0026#34;: [\u0026#34;LOGIN\u0026#34;, \u0026#34;LOGOUT\u0026#34;, \u0026#34;PURCHASE\u0026#34;, \u0026#34;VIEW\u0026#34;] } }, { \u0026#34;name\u0026#34;: \u0026#34;timestamp\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;long\u0026#34;, \u0026#34;logicalType\u0026#34;: \u0026#34;timestamp-millis\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;metadata\u0026#34;, \u0026#34;type\u0026#34;: [\u0026#34;null\u0026#34;, {\u0026#34;type\u0026#34;: \u0026#34;map\u0026#34;, \u0026#34;values\u0026#34;: \u0026#34;string\u0026#34;}], \u0026#34;default\u0026#34;: null // nullable field — always put null first in union! } ] } ⚠️ THE NULL UNION TRAP\nWhen you have a nullable field, the union must be [\u0026quot;null\u0026quot;, \u0026quot;YourType\u0026quot;] with null first AND \u0026quot;default\u0026quot;: null. Get this wrong and schema evolution will bite you later. First type in the union = default type.\nSpring Boot setup with Maven \u0026lt;!-- Avro dependency --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.apache.avro\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;avro\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.11.3\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- For Kafka: Confluent serializer --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.confluent\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;kafka-avro-serializer\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;7.5.0\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- Code generation plugin --\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.apache.avro\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;avro-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.11.3\u0026lt;/version\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;phase\u0026gt;generate-sources\u0026lt;/phase\u0026gt; \u0026lt;goals\u0026gt;\u0026lt;goal\u0026gt;schema\u0026lt;/goal\u0026gt;\u0026lt;/goals\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;sourceDirectory\u0026gt;src/main/avro\u0026lt;/sourceDirectory\u0026gt; \u0026lt;outputDirectory\u0026gt;target/generated-sources\u0026lt;/outputDirectory\u0026gt; \u0026lt;stringType\u0026gt;String\u0026lt;/stringType\u0026gt; \u0026lt;!-- use java.lang.String not CharSequence --\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; Using the generated class in Spring Boot @Service public class KafkaProducerService { private final KafkaTemplate\u0026lt;String, UserEvent\u0026gt; kafkaTemplate; public void sendUserEvent(long userId, String username, Action action) { UserEvent event = UserEvent.newBuilder() // generated Builder pattern .setUserId(userId) .setUsername(username) .setAction(action) .setTimestamp(System.currentTimeMillis()) .setMetadata(null) .build(); kafkaTemplate.send(\u0026#34;user-events\u0026#34;, String.valueOf(userId), event); } } What serialized Avro actually looks like Let’s lift the curtain. When Avro serializes a record, it produces a compact binary stream. No field names. No wasted space. Just bytes, in the order the schema defines them.\nSerializing manually (no Kafka, no registry) import org.apache.avro.io.*; import org.apache.avro.specific.SpecificDatumWriter; UserEvent event = UserEvent.newBuilder() .setUserId(42L) .setUsername(\u0026#34;alice\u0026#34;) .setAction(Action.LOGIN) .setTimestamp(1700000000000L) .setMetadata(null) .build(); // Serialize to bytes ByteArrayOutputStream out = new ByteArrayOutputStream(); DatumWriter\u0026lt;UserEvent\u0026gt; writer = new SpecificDatumWriter\u0026lt;\u0026gt;(UserEvent.class); BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out, null); writer.write(event, encoder); encoder.flush(); byte[] bytes = out.toByteArray(); System.out.println(\u0026#34;Size: \u0026#34; + bytes.length + \u0026#34; bytes\u0026#34;); System.out.println(\u0026#34;Hex: \u0026#34; + HexFormat.of().formatHex(bytes)); // Output: 54 0a 616c 6963 65 00 80 b2 ... // That\u0026#39;s: userId=42, \u0026#34;alice\u0026#34;, action=0(LOGIN), timestamp... What those bytes actually mean Let’s decode the binary output of our UserEvent record visually. Avro uses zigzag encoding for integers and length-prefixed strings.\nZIGZAG ENCODING\nAvro encodes integers using zigzag + variable-length encoding. Small numbers take fewer bytes: the number 42 becomes 0x54 (one byte!). -1 becomes 0x01. Large numbers can take up to 10 bytes. There are no wasted sign bits. Clever.\nWith Confluent Schema Registry (magic byte!) When using the Confluent serializer with a Schema Registry, 5 extra bytes are prepended:\nDeserialization, step by step Deserialization is the reverse journey: bytes → Java object. The key insight is that Avro needs two schemas: the writer schema (what was used to write the data) and the reader schema (what your current code expects). When they match, it’s trivial. When they differ, Avro resolves the differences. That’s the magic of schema evolution.\nimport org.apache.avro.io.*; import org.apache.avro.specific.SpecificDatumReader; byte[] bytes = /* bytes from somewhere */; DatumReader\u0026lt;UserEvent\u0026gt; reader = new SpecificDatumReader\u0026lt;\u0026gt;(UserEvent.class); BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(bytes, null); UserEvent event = reader.read(null, decoder); System.out.println(event.getUserId()); // 42 System.out.println(event.getUsername()); // \u0026#34;alice\u0026#34; System.out.println(event.getAction()); // LOGIN // With writer/reader schema resolution (different versions): Schema writerSchema = /* schema that was used to write */; Schema readerSchema = UserEvent.getClassSchema(); // your current version DatumReader\u0026lt;UserEvent\u0026gt; resolver = new SpecificDatumReader\u0026lt;\u0026gt;(writerSchema, readerSchema); // Avro will fill defaults for new fields, skip unknown fields // application.yml // spring.kafka.consumer.value-deserializer: // io.confluent.kafka.serializers.KafkaAvroDeserializer // spring.kafka.consumer.properties: // schema.registry.url: http://localhost:8081 // specific.avro.reader: true ← generate specific class, not GenericRecord @KafkaListener(topics = \u0026#34;user-events\u0026#34;, groupId = \u0026#34;my-service\u0026#34;) public void handleUserEvent(UserEvent event) { // Already deserialized! Spring + Confluent did all the heavy lifting. log.info(\u0026#34;User {} performed {}\u0026#34;, event.getUsername(), event.getAction()); userEventProcessor.process(event); } How to debug raw Avro — it’s not as dark as it looks You got a byte array. Your consumer is blowing up. The logs say something unhelpful like EOFException: Unexpected end of stream. Welcome to Avro debugging. Let\u0026rsquo;s fix this.\n🔍 THE DEBUGGING MINDSET\nRaw Avro bytes without a schema are like IKEA furniture without instructions. You can tell something should go somewhere, but good luck figuring out which end is up. Always start with the schema.\nTool 1: avro-tools (official CLI) # Download: https://avro.apache.org/releases.html wget https://downloads.apache.org/avro/avro-1.11.3/java/avro-tools-1.11.3.jar # Inspect an .avro file (has schema embedded in header) java -jar avro-tools-1.11.3.jar tojson --pretty my-data.avro # Get the schema from an .avro file java -jar avro-tools-1.11.3.jar getschema my-data.avro # Convert JSON → Avro binary (useful for test data) java -jar avro-tools-1.11.3.jar fromjson \\ --schema-file user-event.avsc \\ input.json \u0026gt; output.avro # Check if a file is valid Avro java -jar avro-tools-1.11.3.jar cat my-data.avro | head -5 Tool 2: Decode in Java with GenericRecord // GenericRecord = schema-driven map-like object. No generated class needed. byte[] suspiciousBytes = /* from Kafka, file, network */; // If you have the schema: Schema schema = new Schema.Parser().parse(new File(\u0026#34;user-event.avsc\u0026#34;)); DatumReader\u0026lt;GenericRecord\u0026gt; reader = new GenericDatumReader\u0026lt;\u0026gt;(schema); BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(suspiciousBytes, null); GenericRecord record = reader.read(null, decoder); // Walk all fields dynamically schema.getFields().forEach(field -\u0026gt; System.out.printf(\u0026#34;%-20s = %s%n\u0026#34;, field.name(), record.get(field.name())) ); // Or convert to JSON for readability String json = record.toString(); // GenericRecord.toString() returns JSON System.out.println(json); // {\u0026#34;userId\u0026#34;: 42, \u0026#34;username\u0026#34;: \u0026#34;alice\u0026#34;, \u0026#34;action\u0026#34;: \u0026#34;LOGIN\u0026#34;, ...} Tool 3: Hex dump + manual decode public static void hexDump(byte[] bytes) { StringBuilder sb = new StringBuilder(); for (int i = 0; i \u0026lt; bytes.length; i++) { if (i % 16 == 0) sb.append(String.format(\u0026#34;%n%04X: \u0026#34;, i)); sb.append(String.format(\u0026#34;%02X \u0026#34;, bytes[i] \u0026amp; 0xFF)); } System.out.println(sb); } // Check for Confluent magic byte if (bytes[0] == 0x00) { int schemaId = ByteBuffer.wrap(bytes, 1, 4).getInt(); System.out.println(\u0026#34;Confluent format, schema ID: \u0026#34; + schemaId); // actual payload starts at byte[5] byte[] payload = Arrays.copyOfRange(bytes, 5, bytes.length); } Check the schema type without full deserialization Sometimes you receive Avro bytes on a shared topic and need to decide which deserializer to use — before you know the type. Don’t deserialize everything just to throw it away. Peek at the bytes first.\n// Confluent wire format: [0x00][schema-id (4 bytes)][avro payload] public Optional\u0026lt;String\u0026gt; peekSchemaName(byte[] bytes, SchemaRegistryClient registry) { if (bytes == null || bytes.length \u0026lt; 5 || bytes[0] != 0x00) { return Optional.empty(); // not Confluent Avro } int schemaId = ByteBuffer.wrap(bytes, 1, 4).getInt(); try { SchemaMetadata meta = registry.getSchemaMetadata(schemaId); return Optional.of(meta.getSchemaType()); // \u0026#34;AVRO\u0026#34; } catch (Exception e) { return Optional.empty(); } } // Get the full schema without deserializing the payload public Schema getSchemaFromBytes(byte[] bytes, SchemaRegistryClient registry) throws Exception { int schemaId = ByteBuffer.wrap(bytes, 1, 4).getInt(); return new Schema.Parser().parse( registry.getById(schemaId).toString() ); } // Route messages by schema name - dispatch pattern public void routeMessage(byte[] bytes) { int schemaId = ByteBuffer.wrap(bytes, 1, 4).getInt(); Schema schema = registry.getById(schemaId); switch (schema.getName()) { case \u0026#34;UserEvent\u0026#34; -\u0026gt; handleUser(deserialize(bytes, UserEvent.class)); case \u0026#34;OrderEvent\u0026#34; -\u0026gt; handleOrder(deserialize(bytes, OrderEvent.class)); case \u0026#34;PaymentEvent\u0026#34;-\u0026gt; handlePayment(deserialize(bytes, PaymentEvent.class)); default -\u0026gt; log.warn(\u0026#34;Unknown schema: {}\u0026#34;, schema.getName()); } } Schema Registry — and can you live without it? Schema Registry is a central server that stores and versions your Avro schemas. Instead of embedding the full schema in every message (that would defeat the purpose), you store an integer ID and look up the schema when needed.\nCan you work without Schema Registry? Yes, absolutely. Here are your options:\n✅ Option 1: Embed schema in file Avro container files (.avro) embed the schema in the file header. Works great for batch jobs, HDFS, and Spark. No registry needed.\n✅ Option 2: Schema in config / classpath Ship the.avsc file with your app. Every service owns a copy. Simple but schema evolution becomes manual coordination across teams.\n⚠️ Option 3: Embed schema in every message Technically works but kills the whole point of Avro. Your 15-byte payload is now 800 bytes. Don’t do this for streaming.\n✅ Option 4: Use Schema Registry The right choice for Kafka. Confluent’s is free and open-source. Provides compatibility checks, versioning, and central governance. Worth the ops overhead.\nspring: kafka: bootstrap-servers: localhost:9092 producer: value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer consumer: value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer properties: schema.registry.url: http://localhost:8081 # Compatibility mode: BACKWARD, FORWARD, FULL, NONE avro.compatibility.level: BACKWARD specific.avro.reader: true # auto.register.schemas: false ← disable in prod! # Let CI/CD register schemas, not the app at startup Performance — good and bad Avro model examples Avro is fast by default, but a poorly designed schema can negate all the advantages. The main culprits: excessive unions, deeply nested records, and string fields where you should use enums.\n😱 Bad Avro schema { \u0026#34;name\u0026#34;: \u0026#34;status\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34; // use enum! }, { \u0026#34;name\u0026#34;: \u0026#34;amount\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34; // use double! }, { \u0026#34;name\u0026#34;: \u0026#34;tags\u0026#34;, \u0026#34;type\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;array\u0026#34;, \u0026#34;items\u0026#34;: { // nested unions in \u0026#34;type\u0026#34;: [ // arrays = slow \u0026#34;null\u0026#34;, \u0026#34;string\u0026#34;, \u0026#34;int\u0026#34; ] } } } ✅ Good Avro schema { \u0026#34;name\u0026#34;: \u0026#34;status\u0026#34;, \u0026#34;type\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;enum\u0026#34;, // int index = 1 byte \u0026#34;name\u0026#34;: \u0026#34;Status\u0026#34;, \u0026#34;symbols\u0026#34;: [\u0026#34;ACTIVE\u0026#34;,\u0026#34;INACTIVE\u0026#34;] } }, { \u0026#34;name\u0026#34;: \u0026#34;amount\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;double\u0026#34; // fixed 8 bytes }, { \u0026#34;name\u0026#34;: \u0026#34;tags\u0026#34;, \u0026#34;type\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;array\u0026#34;, \u0026#34;items\u0026#34;: \u0026#34;string\u0026#34; // homogeneous! }, \u0026#34;default\u0026#34;: [] } Performance rules of thumb CACHE YOUR WRITERS AND READERS\nDatumWriter and DatumReader are expensive to create. Cache them as singletons or Spring beans. Creating one per message under high load will degrade performance significantly — it involves schema parsing and reflection setup.\nAvro vs JSON — the honest comparison 🏆 THE VERDICT\nUse JSON for REST APIs, config files, and anything humans read. Use Avro for event streaming, big data pipelines, and anywhere you have a schema contract between services producing and consuming millions of records. They’re not competitors — they’re different tools for different jobs.\nVersioning — how Avro handles change This is where Avro genuinely shines over JSON. Schema evolution is a first-class feature. The rules are simple: always add fields with defaults, never remove required fields, never change types.\nCompatibility modes Safe schema changes // v1 schema { \u0026#34;name\u0026#34;: \u0026#34;userId\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;long\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;username\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;action\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;...\u0026#34; } // v2 schema - SAFE: added optional field with default { \u0026#34;name\u0026#34;: \u0026#34;userId\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;long\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;username\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;action\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;...\u0026#34; }, { \u0026#34;name\u0026#34;: \u0026#34;userAgent\u0026#34;, \u0026#34;type\u0026#34;: [\u0026#34;null\u0026#34;, \u0026#34;string\u0026#34;], \u0026#34;default\u0026#34;: null }, // ✅ { \u0026#34;name\u0026#34;: \u0026#34;version\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;int\u0026#34;, \u0026#34;default\u0026#34;: 1 } // ✅ // v2 reading v1 data → userAgent=null, version=1 (from defaults) // v1 reading v2 data → userAgent and version ignored Dangerous schema changes // ❌ BREAKING: renaming a field { \u0026#34;name\u0026#34;: \u0026#34;user_id\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;long\u0026#34; } // was \u0026#34;userId\u0026#34; — old data unreadable // ❌ BREAKING: changing type { \u0026#34;name\u0026#34;: \u0026#34;userId\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;string\u0026#34; } // was \u0026#34;long\u0026#34; - parse error // ❌ BREAKING: removing a required field // (field without default - old data still has it, decoder confused) // ✅ \u0026#34;Renaming\u0026#34; safely via aliases { \u0026#34;name\u0026#34;: \u0026#34;user_id\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;long\u0026#34;, \u0026#34;aliases\u0026#34;: [\u0026#34;userId\u0026#34;] // reader can find old \u0026#34;userId\u0026#34; field by alias } // ✅ Adding to enum - put new values at the END \u0026#34;symbols\u0026#34;: [\u0026#34;LOGIN\u0026#34;, \u0026#34;LOGOUT\u0026#34;, \u0026#34;PURCHASE\u0026#34;, \u0026#34;VIEW\u0026#34;, \u0026#34;SHARE\u0026#34;] // SHARE is new // Old consumers reading new data get the default enum value The versioning workflow with Schema Registry GOLDEN RULE OF AVRO VERSIONING\nSet auto.register.schemas=false in production. Let your CI/CD pipeline register schemas after a compatibility check. If a developer\u0026rsquo;s schema breaks backwards compatibility, you want to catch it in the PR — not at 3am in production.\nVersion badges: what each change means ✅ Non-breaking Add optional field with default · Add enum value at end · Add alias · Add doc string\n⚠️ Compatible (FULL) Remove optional field · Widen numeric type (int→long) with resolution\n❌ Breaking Rename field · Change type · Remove required field · Reorder enum values · Add required field\nThe cheat sheet ","permalink":"https://sukachvitally.github.io/posts/apache-avro-without-the-confusion/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/apache-avro-without-the-confusion/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eBinary serialization sounds scary until you realize it’s just a very efficient way of saying “I’ll describe the data once, then send only the bytes I need.” Let’s demystify Avro — from raw bytes to Schema Registry to versioning — with real Java code and zero hand-waving.\u003c/p\u003e\n\u003ch3 id=\"what-is-avro-and-why-should-youcare\"\u003eWhat is Avro, and why should you care?\u003c/h3\u003e\n\u003cp\u003eAvro is Apache’s answer to “JSON is too fat and XML makes me cry.” It’s a binary serialization formatthat stores data alongside a schema — a JSON document that describes the structure of your data. Think of it as a very strict contract between producer and consumer.\u003c/p\u003e","title":"Apache Avro without the Confusion"},{"content":"\nYou’ve dealt with threads in production. You’ve seen the race conditions, the deadlocks, the “works on my machine” moments. This is a casual, example-first refresher — not a textbook.\nWhat even is a Thread? A thread is the smallest unit of execution. Your JVM always has at least one — the main thread. Every thread has its own call stack, but they share heap memory, which is exactly why concurrency is both powerful and terrifying.\n“Two threads walk into a bar. The bartender asks who’s first. Neither knows.”\nThere are two classic ways to create a thread. Extend Thread, or implement Runnable. In practice you\u0026rsquo;ll almost always use Runnable — or better yet, a lambda.\nCreating threads // Option A: extend Thread (old school, limits inheritance) class MyThread extends Thread { public void run() { System.out.println(\u0026#34;I\u0026#39;m running on: \u0026#34; + Thread.currentThread().getName()); } } // Option B: Runnable (preferred - separates task from thread) Runnable task = () -\u0026gt; System.out.println(\u0026#34;Hello from a lambda thread!\u0026#34;); Thread t = new Thread(task, \u0026#34;my-worker\u0026#34;); t.start(); // spawns a new OS thread - DON\u0026#39;T call run() directly! // calling run() just executes it on the current thread Thread lifecycle A thread goes through these states. It’s not always a straight line — it can bounce between BLOCKED, WAITING, and RUNNABLE while alive.\nStopping a thread \u0026amp; join() Java deprecated Thread.stop() long ago because it left objects in broken states. The right way is to use an interrupt flag and check it cooperatively inside your loop.\nStopping a thread gracefully + join() Thread worker = new Thread(() -\u0026gt; { while (!Thread.currentThread().isInterrupted()) { doSomeWork(); } System.out.println(\u0026#34;Thread stopping cleanly\u0026#34;); }); worker.start(); Thread.sleep(2000); // let it run 2 seconds worker.interrupt(); // sets the interrupt flag - the thread will see it worker.join(); // current thread blocks until worker is TERMINATED worker.join(5000); // or with a 5s timeout - safer System.out.println(\u0026#34;Worker is done, continuing...\u0026#34;); // If your thread sleeps, InterruptedException is thrown when interrupted Runnable sleeper = () -\u0026gt; { try { Thread.sleep(10_000); } catch (InterruptedException e) { Thread.currentThread().interrupt(); // re-set the flag! System.out.println(\u0026#34;Woken up early, cleaning up\u0026#34;); } }; CLASSIC MISTAKE\nWhen catching InterruptedException, always call Thread.currentThread().interrupt()afterward. Catching the exception clears the interrupt flag — if you don\u0026rsquo;t re-set it, the rest of the call stack will never know the thread was interrupted.\nSynchronization, monitors, wait \u0026amp; notify Every Java object has a built-in monitor (also called an intrinsic lock). When you mark a method or block as synchronized, only one thread can hold that lock at a time. Everyone else queues up.\nclass BankAccount { private double balance = 1000.0; // synchronized method - lock is the BankAccount instance public synchronized void withdraw(double amount) { if (amount \u0026gt; balance) throw new IllegalStateException(\u0026#34;Insufficient funds\u0026#34;); balance -= amount; } // synchronized block - more fine-grained, shorter lock duration public void deposit(double amount) { doSomeNonCriticalWork(); // runs without lock synchronized (this) { balance += amount; // only this part is protected } } } wait(), notify(), notifyAll() These methods let threads communicate through a shared object. wait() releases the lock and puts the thread to sleep. notify() wakes one waiting thread. notifyAll() wakes them all. Classic producer-consumer:\nProducer-Consumer with wait/notify class MessageBus { private String message; private boolean hasMessage = false; public synchronized void produce(String msg) throws InterruptedException { while (hasMessage) { wait(); // buffer full - release lock and sleep } message = msg; hasMessage = true; notifyAll(); // wake up consumer(s) } public synchronized String consume() throws InterruptedException { while (!hasMessage) { wait(); // nothing to consume - release lock and sleep } hasMessage = false; notifyAll(); // wake up producer return message; } } // Always use while() not if() - spurious wakeups are real! ALWAYS USE WHILE, NOT IF !!!\nWrap wait() in a while loop that re-checks the condition. Spurious wakeups (where a thread wakes up for no reason) are a real thing in the JVM spec. A while guards against them; an ifdoesn\u0026rsquo;t.\nTypes of Locks synchronized is simple but limited. java.util.concurrent.locks gives you more power: try to acquire without blocking, set timeouts, choose fairness policy, and separate read from write access.\nReentrantLock lock = new ReentrantLock(); // default: non-fair ReentrantLock fairLock = new ReentrantLock(true); // threads served in order // Always unlock in finally! lock.lock(); try { doWork(); } finally { lock.unlock(); } // tryLock - non-blocking. Perfect for avoiding deadlocks if (lock.tryLock(100, TimeUnit.MILLISECONDS)) { try { doWork(); } finally { lock.unlock(); } } else { System.out.println(\u0026#34;Couldn\u0026#39;t get the lock - moving on\u0026#34;); } // ReadWriteLock - readers don\u0026#39;t block each other ReadWriteLock rwLock = new ReentrantReadWriteLock(); void read() { rwLock.readLock().lock(); // many threads can read simultaneously try { /* read */ } finally { rwLock.readLock().unlock(); } } void write(String val) { rwLock.writeLock().lock(); // exclusive - blocks all readers too try { /* write */ } finally { rwLock.writeLock().unlock(); } } Lock Conditions — upgrade your wait/notify Conditions are wait()/notify() but tied to a specific ReentrantLock. The big win: you can have multiple conditions per lock — wake only the producers, or only the consumers, not everyone.\nLock Conditions — targeted wake-up ReentrantLock lock = new ReentrantLock(); Condition notFull = lock.newCondition(); // producers wait here Condition notEmpty = lock.newCondition(); // consumers wait here Queue\u0026lt;String\u0026gt; buffer = new LinkedList\u0026lt;\u0026gt;(); int capacity = 10; void produce(String item) throws InterruptedException { lock.lock(); try { while (buffer.size() == capacity) notFull.await(); buffer.add(item); notEmpty.signal(); // wake ONE consumer, not everyone } finally { lock.unlock(); } } String consume() throws InterruptedException { lock.lock(); try { while (buffer.isEmpty()) notEmpty.await(); String item = buffer.poll(); notFull.signal(); // wake ONE producer return item; } finally { lock.unlock(); } } Volatile — the visibility keyword Without volatile, threads can cache variables in CPU registers and never see updates made by other threads. volatile tells the JVM: \u0026ldquo;always read this from and write this to main memory.\u0026rdquo;\nclass Worker implements Runnable { private volatile boolean running = true; // always read from main memory public void run() { while (running) { doWork(); } System.out.println(\u0026#34;Stopped!\u0026#34;); } public void stop() { running = false; // visible to all threads immediately } } // volatile also prevents instruction reordering across the write // Useful for the \u0026#34;double-checked locking\u0026#34; singleton pattern: class Singleton { private static volatile Singleton instance; public static Singleton getInstance() { if (instance == null) { synchronized (Singleton.class) { if (instance == null) { instance = new Singleton(); // volatile needed here } } } return instance; } } VOLATILE ≠ ATOMIC\nvolatile int count; count++; is still NOT thread-safe. Increment is three operations (read, add, write). Use AtomicInteger instead. volatile only guarantees visibility, not compound atomicity.\nAtomic variables, CAS \u0026amp; CPU cache java.util.concurrent.atomic classes use Compare-And-Swap (CAS) — a single CPU instruction. It checks if a value is what you expect; if yes, swaps it atomically. If no (another thread changed it), it retries. No locks. No blocking. Very fast.\nAtomicInteger counter = new AtomicInteger(0); counter.incrementAndGet(); // returns new value (++count) counter.getAndIncrement(); // returns old value (count++) counter.addAndGet(5); // counter += 5 counter.compareAndSet(5, 10); // if 5, set to 10 - returns true/false // AtomicReference for any object AtomicReference\u0026lt;String\u0026gt; ref = new AtomicReference\u0026lt;\u0026gt;(\u0026#34;old\u0026#34;); ref.compareAndSet(\u0026#34;old\u0026#34;, \u0026#34;new\u0026#34;); // thread-safe swap // LongAdder - faster than AtomicLong under HIGH contention // Uses multiple cells internally to reduce CAS collisions LongAdder adder = new LongAdder(); adder.increment(); adder.add(42L); long total = adder.sum(); // slightly eventual - sums all cells // AtomicStampedReference - fixes the ABA problem // (value changes A→B→A; CAS wrongly thinks nothing changed) AtomicStampedReference\u0026lt;String\u0026gt; stamped = new AtomicStampedReference\u0026lt;\u0026gt;(\u0026#34;A\u0026#34;, 0); stamped.compareAndSet(\u0026#34;A\u0026#34;, \u0026#34;B\u0026#34;, 0, 1); // must match stamp too Daemon threads A daemon thread is a background thread that the JVM will kill as soon as all non-daemon (user) threads finish. It doesn’t prevent the JVM from exiting. Think: garbage collector, log flusher, heartbeat pinger.\nUser thread (default) JVM waits for it to finish. Use for your app’s main work — processing requests, writing to DB, etc.\nDaemon thread JVM kills it abruptly when all user threads exit. Use for background housekeeping that should NOT block shutdown.\nThread heartbeat = new Thread(() -\u0026gt; { while (true) { sendPing(); Thread.sleep(5000); } }); heartbeat.setDaemon(true); // MUST be set before start()! heartbeat.start(); // When main thread ends → JVM exits → heartbeat is killed // No waiting, no graceful shutdown, just poof // Check if a thread is daemon: boolean isDaemon = heartbeat.isDaemon(); // true DON’T DO I/O IN DAEMON THREADS\nDaemon threads get killed mid-operation — no finally blocks, no cleanup. Never do database writes, file flushes, or any state-critical work in a daemon thread. It will be cut off without warning.\nExecutorService \u0026amp; Future Creating raw threads is wasteful — each costs ~1MB of stack. ExecutorService manages a pool of reusable threads. You submit tasks; it runs them. Done.\nCallable is like Runnable but it can return a value and throw checked exceptions. Submitting a Callable gives you a Future — a promise for the result.\nExecutorService pool = Executors.newFixedThreadPool(4); // Submit a Callable - returns a Future Future\u0026lt;String\u0026gt; future = pool.submit(() -\u0026gt; { Thread.sleep(1000); return \u0026#34;Data from DB\u0026#34;; // can return values! }); System.out.println(\u0026#34;Doing other stuff while DB query runs...\u0026#34;); // get() blocks until done - or throws on error/timeout String result = future.get(5, TimeUnit.SECONDS); System.out.println(result); // invokeAll - submit a batch, wait for ALL List\u0026lt;Callable\u0026lt;Integer\u0026gt;\u0026gt; tasks = List.of( () -\u0026gt; computeA(), () -\u0026gt; computeB(), () -\u0026gt; computeC() ); List\u0026lt;Future\u0026lt;Integer\u0026gt;\u0026gt; results = pool.invokeAll(tasks, 10, TimeUnit.SECONDS); // invokeAny - return first result, cancel the rest (hedged requests!) Integer fastest = pool.invokeAny(tasks); // Always shut down! An ExecutorService keeps the JVM alive pool.shutdown(); if (!pool.awaitTermination(30, TimeUnit.SECONDS)) { pool.shutdownNow(); } Thread Pool types Java gives you factory methods for common pool configurations. In production, you’ll often configure ThreadPoolExecutor directly to know exactly what you\u0026rsquo;re running on.\nnewFixedThreadPool(n) N threads, unlimited queue. Good for CPU-bound work. Risk: queue grows unbounded under sustained load.\nnewCachedThreadPool() Unlimited threads, 60s idle timeout. Good for bursty short tasks. Danger: thread explosion under sustained load.\nnewSingleThreadExecutor() One thread, sequential. Like a lock-free serial queue. Great for ordered processing.\nnewScheduledThreadPool(n) Delay and periodic tasks. Replacement for Timer— one exception won\u0026rsquo;t kill future runs.\nThreadPoolExecutor executor = new ThreadPoolExecutor( 4, // corePoolSize — always-alive threads 16, // maxPoolSize — burst capacity 60L, TimeUnit.SECONDS, // idle thread keepalive new LinkedBlockingQueue\u0026lt;\u0026gt;(200), // bounded queue — prevents OOM r -\u0026gt; { Thread t = new Thread(r, \u0026#34;api-worker-\u0026#34; + counter.incrementAndGet()); t.setDaemon(true); return t; }, new ThreadPoolExecutor.CallerRunsPolicy() // backpressure: caller runs it ); // Rejection policies when queue is full: // AbortPolicy → throws RejectedExecutionException (default) // CallerRunsPolicy → caller thread executes the task (natural throttle) // DiscardPolicy → silently drops the task (risky!) // DiscardOldestPolicy → drops the oldest queued task and retries // Monitor your pool in production: executor.getActiveCount(); // threads currently working executor.getQueue().size(); // tasks waiting - this growing = bottleneck! executor.getCompletedTaskCount(); // total completed tasks ForkJoinPool \u0026amp; recursive tasks Regular thread pools are bad at recursive divide-and-conquer: a thread splits a task into subtasks and then blocks waiting for them — wasting a thread. ForkJoinPool uses work stealing: idle threads steal subtasks from busy threads\u0026rsquo; queues. Nobody sits idle.\nimport java.util.concurrent.*; class SumTask extends RecursiveTask\u0026lt;Long\u0026gt; { private final int[] data; private final int from, to; static final int THRESHOLD = 1000; // split until chunk is small enough SumTask(int[] data, int from, int to) { this.data = data; this.from = from; this.to = to; } protected Long compute() { if (to - from \u0026lt;= THRESHOLD) { long sum = 0; for (int i = from; i \u0026lt; to; i++) sum += data[i]; return sum; } int mid = (from + to) / 2; SumTask left = new SumTask(data, from, mid); SumTask right = new SumTask(data, mid, to); left.fork(); // push left to thread pool long rightResult = right.compute(); // run right inline return left.join() + rightResult; // join left result } } ForkJoinPool fjp = ForkJoinPool.commonPool(); // shared JVM-wide pool long total = fjp.invoke(new SumTask(bigArray, 0, bigArray.length)); // RecursiveAction for void tasks (no return value) CountDownLatch A CountDownLatch is a one-time gate. You initialize it with a count. Threads call countDown() to decrement it. Anyone waiting on await() is released when it hits zero. Think of it as a race starting pistol — or waiting for all microservices to warm up before accepting traffic.\nint workerCount = 5; CountDownLatch latch = new CountDownLatch(workerCount); ExecutorService pool = Executors.newFixedThreadPool(workerCount); for (int i = 0; i \u0026lt; workerCount; i++) { final int id = i; pool.submit(() -\u0026gt; { try { processChunk(id); } finally { latch.countDown(); // ALWAYS in finally - even on exception } }); } latch.await(); // main thread blocks here // latch.await(10, TimeUnit.SECONDS); // or with timeout System.out.println(\u0026#34;All chunks processed!\u0026#34;); // Starting gun pattern - release all threads simultaneously CountDownLatch startGun = new CountDownLatch(1); for (int i = 0; i \u0026lt; 10; i++) { new Thread(() -\u0026gt; { startGun.await(); // all 10 threads wait here runBenchmark(); }).start(); } startGun.countDown(); // fires - all 10 released at the same instant Semaphore A Semaphore controls access to a limited pool of resources. Think: connection pool with N connections, or a rate limiter that caps concurrent requests. It maintains a count of permits. Threads acquire permits; if none are available, they block. When done, they release.\n// Only 3 DB connections available Semaphore dbPool = new Semaphore(3); void queryDatabase() throws InterruptedException { dbPool.acquire(); // blocks if all 3 permits are taken try { runQuery(); } finally { dbPool.release(); // always release in finally! } } // tryAcquire - non-blocking version if (dbPool.tryAcquire(500, TimeUnit.MILLISECONDS)) { try { runQuery(); } finally { dbPool.release(); } } else { returnCachedResult(); // graceful fallback } // Semaphore(1) = mutex (similar to synchronized, but acquirable across methods) Semaphore mutex = new Semaphore(1); // Fair semaphore - threads served in arrival order (no starvation) Semaphore fair = new Semaphore(5, true); CopyOnWrite collections CopyOnWriteArrayList and CopyOnWriteArraySet trade write performance for read performance. Every write creates a fresh copy of the underlying array. Readers always see a consistent snapshot. Zero locking on reads.\nPerfect for… Event listener lists. Configuration snapshots. Anything read many times, updated rarely (like once per deployment).\nTerrible for… Large collections with frequent writes. Every add/remove copies the entire array — O(n) memory allocation per write.\nCopyOnWriteArrayList\u0026lt;EventListener\u0026gt; listeners = new CopyOnWriteArrayList\u0026lt;\u0026gt;(); // Write - creates a new internal array, old readers unaffected listeners.add(myListener); listeners.remove(oldListener); // Read - snapshot, safe without any locking // Iterating CopyOnWriteArrayList never throws ConcurrentModificationException for (EventListener l : listeners) { l.onEvent(event); // concurrent add/remove won\u0026#39;t break this loop } // Regular ArrayList under the same scenario would need: List\u0026lt;EventListener\u0026gt; synced = Collections.synchronizedList(new ArrayList\u0026lt;\u0026gt;()); synchronized (synced) { // must manually sync iteration! for (EventListener l : synced) l.onEvent(event); } ThreadLocal ThreadLocal gives each thread its own isolated copy of a variable. No sharing, no synchronization needed. Classic uses: per-request user context, database connection per thread, or formatters (which aren\u0026rsquo;t thread-safe).\n// Static ThreadLocal — each thread gets its own value static ThreadLocal\u0026lt;User\u0026gt; currentUser = new ThreadLocal\u0026lt;\u0026gt;(); // Set at the start of a request (e.g. in a filter/interceptor) currentUser.set(authenticatedUser); // Anywhere deep in the call stack - no need to pass the user around User user = currentUser.get(); // CRITICAL: always clean up! Thread pools reuse threads. // Old value from a previous request will leak to the next! try { currentUser.set(user); handleRequest(); } finally { currentUser.remove(); // prevents memory leaks in thread pools } // ThreadLocal with default value ThreadLocal\u0026lt;SimpleDateFormat\u0026gt; formatter = ThreadLocal.withInitial(() -\u0026gt; new SimpleDateFormat(\u0026#34;yyyy-MM-dd\u0026#34;)); // InheritableThreadLocal - child threads inherit parent\u0026#39;s value InheritableThreadLocal\u0026lt;String\u0026gt; traceId = new InheritableThreadLocal\u0026lt;\u0026gt;(); traceId.set(\u0026#34;req-abc-123\u0026#34;); new Thread(() -\u0026gt; System.out.println(traceId.get())).start(); // \u0026#34;req-abc-123\u0026#34; MEMORY LEAK TRAP\nIn a thread pool, threads are reused indefinitely. If you set a ThreadLocal and never call remove(), the value stays attached to that thread forever — causing memory leaks and data leaking between requests. Always clean up in a finally block.\nVirtual Threads (Java 21) Traditional platform threads map 1:1 to OS threads. 1 MB stack. ~10,000 max before things get painful. Virtual threads are JVM-managed — lightweight, cheap to create by the million. When a virtual thread blocks (I/O, sleep, lock), the JVM unmounts it from the carrier OS thread and parks it. The carrier is free to run another virtual thread. Magic.\n// Create a single virtual thread Thread vt = Thread.ofVirtual().name(\u0026#34;my-vt\u0026#34;).start(() -\u0026gt; handleRequest()); // Virtual thread per task - executor backed by virtual threads try (ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor()) { for (Request req : incoming) { executor.submit(() -\u0026gt; handle(req)); // each gets a new virtual thread } } // auto-shutdown on close // Blocking I/O is now FINE - JVM unmounts the virtual thread while waiting // No need for reactive/async code just to avoid blocking // Check if a thread is virtual: Thread.currentThread().isVirtual(); // true // What NOT to do with virtual threads: // 1. Don\u0026#39;t pin with synchronized + blocking I/O (use ReentrantLock instead) // 2. Don\u0026#39;t use thread pools of virtual threads (defeats the purpose) // 3. ThreadLocal still works but be careful with large values at scale WHEN TO USE VIRTUAL THREADS\nThey shine for I/O-bound, high-concurrency work — HTTP servers, database calls, file reads. They don’t help CPU-bound work (you’re still limited by CPU cores). If you’re using Spring Boot 3.2+, you can enable virtual threads with a single property: spring.threads.virtual.enabled=true.\nScopedValue (Java 25) ScopedValue is the modern answer to ThreadLocal. The problems with ThreadLocal: mutable (anyone can change it), inherited unboundedly (leaks to child threads), and requires manual cleanup. ScopedValue is immutable per scope, automatically scoped to a bounded call tree, and works naturally with virtual threads and structured concurrency.\nThreadLocal Mutable. Lives as long as the thread (or until remove()). Can leak. Doesn\u0026rsquo;t compose well with virtual threads at scale.\nScopedValue Immutable within scope. Auto-cleaned when scope exits. No leaks. Designed for virtual threads and structured concurrency.\n// Declare a ScopedValue (like a typed key) static final ScopedValue\u0026lt;User\u0026gt; CURRENT_USER = ScopedValue.newInstance(); // Bind a value for the duration of a lambda ScopedValue.where(CURRENT_USER, authenticatedUser) .run(() -\u0026gt; { processRequest(); // can read CURRENT_USER callServiceLayer(); // and so can anything it calls }); // scope exits → binding is gone. No cleanup needed. // Read it anywhere inside the scope void processRequest() { User user = CURRENT_USER.get(); // works! System.out.println(\u0026#34;Processing for: \u0026#34; + user.name()); } // Re-bind for child scope (doesn\u0026#39;t mutate parent) ScopedValue.where(CURRENT_USER, adminUser) .run(() -\u0026gt; { User u = CURRENT_USER.get(); // adminUser here }); // Back here: CURRENT_USER is still authenticatedUser // Works great with structured concurrency (Java 21+) try (var scope = new StructuredTaskScope.ShutdownOnFailure()) { scope.fork(() -\u0026gt; callServiceA()); // CURRENT_USER is inherited scope.fork(() -\u0026gt; callServiceB()); // automatically - no ThreadLocal tricks scope.join().throwIfFailed(); } STATUS\nScopedValue is finalized in Java 25 (JEP 487). It was in preview since Java 21. If you’re on Java 21–24, add --enable-preview to use it. It pairs naturally with StructuredTaskScope for structured concurrency.\nCheat sheet — pick your tool ","permalink":"https://sukachvitally.github.io/posts/java-concurrency-without-the-headache/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/java-concurrency-without-the-headache/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eYou’ve dealt with threads in production. You’ve seen the race conditions, the deadlocks, the “works on my machine” moments. This is a casual, example-first refresher — not a textbook.\u003c/p\u003e\n\u003ch3 id=\"what-even-is-athread\"\u003eWhat even is a Thread?\u003c/h3\u003e\n\u003cp\u003eA thread is the smallest unit of execution. Your JVM always has at least one — the main thread. Every thread has its own call stack, but they share heap memory, which is exactly why concurrency is both powerful and terrifying.\u003c/p\u003e","title":"Java Concurrency without the headache"},{"content":"\nDevelopers love the idea of a single, elegant SQL query that fetches everything in one go. One round‑trip to the database. One response. One perfect solution.\nExcept… in Hibernate, that “perfect” query can quietly turn into a CPU‑burning, memory‑eating disaster.\nThis hidden performance trap is known as the dehydration problem — when Hibernate tries to hydrate (map) a multiplied result set into your object graph. The SQL is fast, but the ORM mapping becomes painfully slow.\nLet’s break down why this happens, how to spot it, and how to avoid it.\nThe Real Enemy: Cartesian Explosion The dehydration problem starts with a simple fact:\nWhen you join multiple one‑to‑many or many‑to‑many relations, the database multiplies rows.\nExample domain:\n+---------------------+ | Order | |---------------------| | id | | date | | status | |---------------------| | items (IDs only) | +---------------------+ +---------------------+ | OrderItem | |---------------------| | id | | orderId | | productId | | quantity | +---------------------+ +---------------------+ | Product | |---------------------| | id | | name | | price | |---------------------| | images (IDs only) | | tags (IDs only) | +---------------------+ A typical join:\nSELECT * FROM orders o LEFT JOIN order_item oi ON oi.order_id = o.id LEFT JOIN product p ON p.id = oi.product_id WHERE o.id = :id; Looks harmless. Runs fast. DB returns rows instantly.\nBut if:\nOrder has 20 items Each product has 3 images Each product has 2 tags Then the join becomes:\n20 items × 3 images × 2 tags = 120 rows The database doesn’t care — it just streams rows. Hibernate, however, must reconstruct:\n1 Order 20 OrderItems 20 Products 60 Images 40 Tags …from 120 multiplied rows.\nThis is where the trouble begins.\nWhy Hibernate Suffers: Hydration Is Expensive Hibernate must:\ndeduplicate entities merge rows into collections maintain identity maps resolve proxies apply lazy/eager rules build the entire object graph This means:\nHigh CPU usage High memory consumption GC pressure Slow endpoints Thread pool starvation Sometimes even OutOfMemoryError The SQL query is fast. The ORM hydration is slow.\nThis is the dehydration problem in action.\nA Concrete Example: Order → OrderItems → Product Entities (simplified) class Order { @OneToMany(mappedBy = \u0026#34;order\u0026#34;, fetch = FetchType.EAGER) List\u0026lt;OrderItem\u0026gt; items; } class OrderItem { @ManyToOne(fetch = FetchType.EAGER) Product product; } class Product { @OneToMany(fetch = FetchType.LAZY) List\u0026lt;ProductImage\u0026gt; images; @ManyToMany(fetch = FetchType.LAZY) List\u0026lt;Tag\u0026gt; tags; } Even if you write a simple query, Hibernate loads the entire graph because of greedy relations.\nThe result? A single “smart” query becomes a Cartesian explosion that Hibernate must painfully hydrate.\nWhy Developers Fall Into This Trap Because the SQL looks good.\nThe query is fast. The DB plan is efficient. The response time is low. But Hibernate is not a database. It’s an object graph builder.\nORM hydration is the bottleneck — not the SQL.\nThe Hidden Culprit: Greedy Relations The dehydration problem almost always appears when entities contain:\n@OneToMany(fetch = EAGER) @ManyToOne(fetch = EAGER) @ManyToMany(fetch = EAGER) large collections deep object graphs bidirectional relations Even if you write a simple query, Hibernate loads everything.\nThis is why EAGER fetching is dangerous.\nWhy “One Query” Is Not Always Better Developers often assume:\n“One query is faster than multiple queries.”\nNot in ORM world.\nSometimes 3–5 small queries are dramatically faster than one huge join.\nExample:\nLoad Order Load OrderItems Load Products Load Images Load Tags Each query returns a small, clean dataset. Hibernate hydrates them efficiently.\nThis is often the correct solution.\nHow to Fix the Dehydration Problem 1. Replace joins with multiple smaller queries Load only what you need, when you need it.\nHibernate handles small result sets extremely well.\n2. Use DTO projections instead of entities @Query(\u0026#34;\u0026#34;\u0026#34; SELECT new OrderItemDTO(oi.id, p.name, p.price) FROM OrderItem oi JOIN oi.product p WHERE oi.order.id = :orderId \u0026#34;\u0026#34;\u0026#34;) DTOs avoid hydration entirely.\n3. Avoid EAGER fetching Use:\n@OneToMany(fetch = LAZY) And load explicitly.\n4. Apply DDD principles In Domain‑Driven Design:\nAggregates should be small Services should load only the data they own Bounded contexts should not hydrate entire graphs If your Order aggregate loads:\nitems products images tags categories suppliers …it’s too big.\nDDD teaches:\nLoad only what the use case needs.\nHow to Detect the Problem in Production Look for:\nhigh CPU usage high memory usage slow endpoints with “fast” SQL GC pauses thread pool exhaustion huge JSON responses thread dumps showing org.hibernate.loader.Loader.processResultSet This is a clear sign of dehydration.\nConclusion The dehydration problem is one of the most common — and most misunderstood — performance issues in Hibernate. A single “smart” query can silently trigger a massive Cartesian explosion, causing Hibernate to burn CPU and memory while trying to hydrate an oversized object graph.\nKey takeaways: SQL can be fast while Hibernate is slow Cartesian explosions kill performance EAGER relations are dangerous Multiple small queries are often faster DTOs and read models avoid hydration entirely DDD helps keep aggregates small and focused If you design your domain and queries with these principles in mind, you’ll avoid one of the most painful ORM performance traps — and your production services will thank you.\n","permalink":"https://sukachvitally.github.io/posts/when-one-fast-query-becomes-a-production-nightmare-the-dehydration-problem-in-hibernate/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/when-one-fast-query-becomes-a-production-nightmare-the-dehydration-problem-in-hibernate/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eDevelopers love the idea of a single, elegant SQL query that fetches everything in one go. One round‑trip to the database. One response. One perfect solution.\u003c/p\u003e\n\u003cp\u003eExcept… in Hibernate, that “perfect” query can quietly turn into a \u003cstrong\u003eCPU‑burning, memory‑eating disaster\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eThis hidden performance trap is known as the \u003cstrong\u003edehydration problem\u003c/strong\u003e — when Hibernate tries to hydrate (map) a multiplied result set into your object graph. The SQL is fast, but the ORM mapping becomes painfully slow.\u003c/p\u003e","title":"When One “Fast” Query Becomes a Production Nightmare: The Dehydration Problem in Hibernate"},{"content":"\nEvery developer has faced the same painful situation: a huge task from a Business Analyst that doesn’t align with the technical architecture, is full of gaps, or ignores corner cases entirely.\nSome developers implement only the “happy path.” Others follow the requirements literally, even when the logic becomes unnecessarily complicated. And after implementation, QA finds a long list of bugs — mostly in those forgotten corner cases.\nWhy does this happen? Because the business flow is unclear, incomplete, or not aligned with how the system actually works.\nSo the real question is: what can we do about it?\nThe Problem: Requirements Without Technical Reality When requirements are unclear, developers start asking dozens of questions. This often leads to:\nendless discussions huge text descriptions nobody wants to read BA writing tasks without proper refinement no long‑term vision of how the feature will evolve unpredictable complexity during implementation In many cases, BAs don’t have the full requirements in advance. So developers cannot predict how to make the code extensible or in which direction the feature will grow.\nA Practical Example: Promo Code Validation Imagine a simple feature:\n“User enters a promo code during registration.”\nBA writes a task:\nValidate promo code Apply bonus Show error if invalid But they don’t know:\nvalidation is done by another service promo codes can expire some promo codes require KYC some promo codes are region‑restricted some promo codes cannot be used during registration Developers implement the happy path. QA finds 12 bugs. Business says: “We didn’t think about these cases.”\nThis is exactly where technical documentation with business flow saves the project.\nStep 1: Start With a Clean Page Create a technical document for the new feature.\nInclude:\ngeneral information goals links to BA tasks assumptions non‑goals This becomes the foundation for the entire team.\nStep 2: Introduce Domain Models Before writing any code, define the domain model.\nExample Domain Model (PlantUML) This already clarifies:\nwhat data is needed what validations exist what dependencies are involved Step 3: Use Sequence Diagrams to Show the Real Flow This is where PlantUML becomes your best friend.\nExample Sequence Diagram This diagram shows:\ndependencies order of operations conditional logic where delays may occur where errors may appear BAs can read it and finally understand the technical reality. Developers get a clear, visual flow instead of long text. QA gets a map of what to test.\nStep 4: Add Technical Restrictions and Limitations This is the most important part.\nExample Restrictions Promo validation must be asynchronous. KYC service has 500ms latency. Bonus Service rejects expired or region‑restricted promos. User Service cannot override Bonus Service decisions. Promo cannot be applied after registration is completed. This prevents unrealistic expectations and protects the team from unnecessary refactoring later.\nStep 5: Review With the BA Team Schedule a short call.\nWalk through:\nthe diagrams the flow the limitations the corner cases BAs will usually adjust the business flow after seeing the technical reality. Together, you choose the best solution — not just the fastest or the most “business‑friendly.”\nAfter the review, get approval. Now you have an official document that covers the entire feature.\nStep 6: Use the Document as the Task Description Instead of writing long Jira descriptions, simply:\nadd a short introduction attach a link to the documentation include the relevant documentation block Every developer can open the document and see the full picture.\nThis also enables TDD or BDD because the flow is fully defined.\nStep 7: Maintain the Document After Release When the feature goes to production:\ncreate a “Bugs” section add every issue found note which ones are not bugs update the flow if needed Support teams can start from this document before asking questions. The documentation evolves naturally and stays alive.\nFinal Benefit BAs become aware of all technical limitations. Next time, there are no surprises like “we need a total refactoring.” And if refactoring is required, it becomes a shared responsibility — not just a development problem.\n","permalink":"https://sukachvitally.github.io/posts/business-flow-as-a-part-of-technical-documentation-why-tech-leads-should-own-it/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/business-flow-as-a-part-of-technical-documentation-why-tech-leads-should-own-it/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eEvery developer has faced the same painful situation: a huge task from a Business Analyst that doesn’t align with the technical architecture, is full of gaps, or ignores corner cases entirely.\u003c/p\u003e\n\u003cp\u003eSome developers implement only the “happy path.” Others follow the requirements literally, even when the logic becomes unnecessarily complicated. And after implementation, QA finds a long list of bugs — mostly in those forgotten corner cases.\u003c/p\u003e\n\u003cp\u003eWhy does this happen? Because the business flow is unclear, incomplete, or not aligned with how the system actually works.\u003c/p\u003e","title":"Business Flow as a Part of Technical Documentation: Why Tech Leads Should Own It"},{"content":"\nLegacy systems are rarely “just bad code”. Usually, they’re the result of years of rushed decisions, unclear requirements, and survival‑mode delivery.\nI’ve faced this situation more than once, and developed a practical methodology that lets you improve the system while still shipping features — without a big‑bang rewrite.\nThe three core problems In troubled projects, the same three issues appear again and again:\nBusiness logic is complex and undocumented You can’t be sure if a “better” solution is actually correct, because nobody fully understands the current behaviour. The team is under pressure and chooses the easiest path Deadlines dominate. Developers implement quick fixes instead of proper solutions, increasing entropy over time. Domain logic is scattered across the codebase There is no clear separation of concerns. The same domain is implemented in multiple places, in different ways. You can’t fix this with a single refactor. You need a strategy.\nStep 1: Enforce integration tests in CI The first move is simple but non‑negotiable:\nIntroduce integration tests and make them part of the CI pipeline. The pipeline must fail if tests fail. At this stage, don’t aim for perfect coverage. Your goal is to freeze current behaviour so you can safely change internals later.\nFocus on functional/integration tests, not unit tests. Unit tests are too tied to current class structure, which you plan to change.\nStep 2: Require coverage for all new code Don’t try to cover the entire legacy system at once — that’s a trap.\nInstead:\nIntroduce a rule: all new classes (or significantly new logic) must have at least 80% coverage. Keep a file like legacy-classes.txt with all existing uncovered classes. Configure your coverage tool to ignore those legacy classes for now. This way:\nLegacy code is tolerated but contained. New code is held to a higher standard. Over time, the “good” part of the system grows. Step 3: Touch legacy? Then test it. Add one more rule for the team —if you modify legacy code, you must:\nremove the affected classes from legacy-classes.txt add proper tests for them. This gradually converts legacy areas into tested, safer zones. You don’t stop feature work — you just attach improvement to change.\nAfter a few months, you’ll notice something important: your coverage and confidence have increased without a big refactor.\nStep 4: Identify and remove dead code Once you have better test coverage and more confidence, start cleaning up.\nA simple approach:\nMark suspected legacy code as@Deprecated. Add an aspect or interceptor that logs every call to deprecated methods in production. Wait a couple of months and analyse the logs. If a deprecated method hasn’t been called in production for a long time, it’s probably dead. Review it carefully — and then remove it.\nThis reduces noise and makes the remaining code easier to reason about.\nStep 5: Group logic by domain Now that the system is safer and cleaner, you can start addressing the real problem: domain logic scattered everywhere.\nSteps:\nIdentify the domains in your system (billing, users, orders, CRM, etc.). Move domain logic into dedicated modules or packages. Keep the behaviour the same — your integration tests should still pass. You don’t need heavy refactoring yet. Just physically group related logic so each domain has a clear “home”.\nDomain‑Driven Design (DDD) concepts help here, but you don’t need to implement full DDD to benefit from bounded contexts and clear boundaries.\nStep 6: Isolate data access per domain Once domains are clearer, start isolating data access:\nCreate domain services that own the logic for a specific domain. Each service uses its own repositories and exposes domain models, not raw entities. Keep SQL queries inside the domain boundary, not spread across controllers and helpers. At this stage, you may hit performance issues. That’s normal.\nTactics:\nKeep your database normalized as a baseline. Where performance is critical, introduce denormalized read models or NoSQL stores for specific use cases. Ensure denormalized structures are updated within the same transaction or via reliable async mechanisms. The key is to understand the trade‑offs, not blindly optimize.\nStep 7: Refine service boundaries and APIs Now your services are more coherent. Time to make them consistent:\nEach service should own one domain model and work with one type of domain DTO. Controllers, listeners, and tasks handle API‑level concerns (validation, mapping, HTTP, messaging). Services focus on business rules, not transport details. Your controllers may look a bit messy for a while. That’s acceptable. The important part is that business logic is now in one place.\nAt this point, you can start documenting the core flows and validating them with product owners. You’re no longer guessing — you have structure.\nStep 8: Evolve your public API As your internal model stabilizes, you’ll notice:\nsome APIs are redundant, some can be unified, some no longer reflect the real domain. Instead of breaking clients, introduce API versioning:\nDesign a cleaner v2 API aligned with your new domain model. Keep v1 for backward compatibility for a while. Gradually migrate consumers to v2. This lets you modernize without forcing a painful, immediate migration.\nStep 9: Extract microservices only when ready Only now — after:\ntests are in place, dead code is removed, domains are clear, services are cohesive …does it make sense to talk about microservices.\nYou can:\nidentify services that are truly independent, give them their own database, migrate them out carefully. There will still be risks: network latency, distributed transactions, operational overhead. But now you’re moving from a position of understanding, not chaos.\nFinal thought This methodology is not about hero refactors or perfect architecture. It’s about creating safety first, then improving structure step by step, while the business keeps moving.\n","permalink":"https://sukachvitally.github.io/posts/from-legacy-chaos-to-sustainable-architecture-a-step-by-step-refactoring-strategy/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/from-legacy-chaos-to-sustainable-architecture-a-step-by-step-refactoring-strategy/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eLegacy systems are rarely “just bad code”. Usually, they’re the result of years of rushed decisions, unclear requirements, and survival‑mode delivery.\u003c/p\u003e\n\u003cp\u003eI’ve faced this situation more than once, and developed a practical methodology that lets you \u003cstrong\u003eimprove the system while still shipping features\u003c/strong\u003e — without a big‑bang rewrite.\u003c/p\u003e\n\u003ch3 id=\"the-three-coreproblems\"\u003eThe three core problems\u003c/h3\u003e\n\u003cp\u003eIn troubled projects, the same three issues appear again and again:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eBusiness logic is complex and undocumented\u003c/strong\u003e You can’t be sure if a “better” solution is actually correct, because nobody fully understands the current behaviour.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eThe team is under pressure and chooses the easiest path\u003c/strong\u003e Deadlines dominate. Developers implement quick fixes instead of proper solutions, increasing entropy over time.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDomain logic is scattered across the codebase\u003c/strong\u003e There is no clear separation of concerns. The same domain is implemented in multiple places, in different ways.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eYou can’t fix this with a single refactor. You need a \u003cstrong\u003estrategy\u003c/strong\u003e.\u003c/p\u003e","title":"From Legacy Chaos to Sustainable Architecture: A Step‑by‑Step Refactoring Strategy"},{"content":"\nModern systems evolve fast. New features appear, requirements change, and data models grow in unexpected directions. If you’ve ever worked with a “god table” — a huge table with dozens or hundreds of columns — you know how painful schema evolution can be:\nevery new feature requires a migration many columns stay empty for most rows the table becomes harder to maintain the schema no longer reflects real business logic One way to avoid this is to use polymorphism in the database through a key–value (EAV) table. Instead of adding new columns, you store flexible attributes as rows.\nThis article explains how this pattern works, when it’s useful, and where it can go wrong — with a simple example you can study and experiment with.\n🔍 What Problem Does This Solve? Imagine you have a table products:\nid | name | price | color | size | weight | material | warranty | …\nAs the business grows, new product types appear:\nelectronics clothes furniture digital goods Each type needs different attributes. You end up adding more and more columns:\nscreen_size cpu ram fabric_type dimensions download_url license_key Most of these columns are NULL for most rows. Migrations become constant. The schema becomes a mess.\nThe Polymorphic Key–Value Solution Instead of adding new columns, we store attributes in a simple, flexible table:\nproduct_attributes SQL schema: CREATE TABLE product_attributes ( id SERIAL PRIMARY KEY, product_id INTEGER NOT NULL, key VARCHAR(100) NOT NULL, value TEXT NOT NULL, type VARCHAR(20) NOT NULL, -- string, number, boolean, json created_at TIMESTAMP DEFAULT NOW() ); Now you can store any attribute without touching the schema.\nSimple Example: Storing Dynamic Product Data Insert attributes for a T‑shirt: INSERT INTO product_attributes (product_id, key, value, type) VALUES (10, \u0026#39;color\u0026#39;, \u0026#39;red\u0026#39;, \u0026#39;string\u0026#39;), (10, \u0026#39;size\u0026#39;, \u0026#39;XL\u0026#39;, \u0026#39;string\u0026#39;), (10, \u0026#39;material\u0026#39;, \u0026#39;cotton\u0026#39;, \u0026#39;string\u0026#39;); Insert attributes for a laptop: INSERT INTO product_attributes (product_id, key, value, type) VALUES (11, \u0026#39;cpu\u0026#39;, \u0026#39;i7\u0026#39;, \u0026#39;string\u0026#39;), (11, \u0026#39;ram\u0026#39;, \u0026#39;16GB\u0026#39;, \u0026#39;string\u0026#39;), (11, \u0026#39;screen_size\u0026#39;, \u0026#39;15.6\u0026#39;, \u0026#39;number\u0026#39;); Query: Get all attributes for a product SELECT key, value FROM product_attributes WHERE product_id = 11; Query: Find all products with a specific attribute SELECT product_id FROM product_attributes WHERE key = \u0026#39;color\u0026#39; AND value = \u0026#39;red\u0026#39;; This is the essence of polymorphism: one table, many types of data, no schema changes.\nBenefits of This Approach 1. No more schema migrations New attributes = new rows, not new columns.\nThis is extremely useful when:\nrequirements change often product types vary you don’t control the schema (SaaS, plugins, integrations) 2. Flexible and future‑proof You can add attributes dynamically:\nfrom admin panels from configuration from user input from external systems No DBA involvement needed.\n3. Works well for sparse data If only 5% of products have a screen_size, why create a column for everyone?\n4. Great for prototyping You can start fast without designing a perfect schema.\n5. Easy to extend with JSON You can store complex values:\nINSERT INTO product_attributes (product_id, key, value, type) VALUES (12, \u0026#39;dimensions\u0026#39;, \u0026#39;{\u0026#34;w\u0026#34;: 100, \u0026#34;h\u0026#34;: 50}\u0026#39;, \u0026#39;json\u0026#39;); Pitfalls and How to Avoid Them This pattern is powerful — but dangerous if misused.\n1. Harder to query Filtering becomes more complex:\nSELECT p.id FROM products p JOIN product_attributes a1 ON a1.product_id = p.id AND a1.key = \u0026#39;color\u0026#39; AND a1.value = \u0026#39;red\u0026#39; JOIN product_attributes a2 ON a2.product_id = p.id AND a2.key = \u0026#39;size\u0026#39; AND a2.value = \u0026#39;XL\u0026#39;; Solution: Use materialized views or caching for common queries.\n2. No strong typing Everything is stored as text unless you enforce types manually.\nSolution: Use a type column and validate in the application layer.\n3. Harder to enforce constraints You can’t easily enforce:\nrequired attributes allowed values foreign keys Solution: Use application‑level validation or attribute metadata tables.\n4. Performance issues at scale A product with 50 attributes = 50 rows. A million products = 50 million rows.\nSolution:\nadd indexes on (product_id) and (key, value) shard by product type use caching (Redis) 5. Overuse leads to chaos If everything becomes key–value, you lose structure.\nSolution: Use this pattern only for dynamic or optional attributes. Keep core fields in the main table.\nWhen Should You Use This Pattern? Good use cases: product catalogs CMS metadata user preferences feature flags IoT sensor data audit logs dynamic forms Bad use cases: financial transactions core business entities data requiring strict validation heavy analytical workloads Final Thoughts Polymorphism in database design — especially through a key–value table — is a powerful tool for building flexible, evolving systems. It helps avoid schema bloat, reduces migration overhead, and supports dynamic data models.\nBut like any abstraction, it must be used intentionally.\nUse it for dynamic, optional, or unpredictable attributes. Avoid it for core, validated, business‑critical data.\nIf you balance flexibility with structure, this pattern becomes a reliable foundation for scalable, maintainable systems — especially in fast‑moving domains like e‑commerce, gaming, SaaS, and content platforms.\n","permalink":"https://sukachvitally.github.io/posts/polymorphism-in-database-design-a-simple-key-value-approach-for-evolving-schemas/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/polymorphism-in-database-design-a-simple-key-value-approach-for-evolving-schemas/image-01.png\"\u003e\u003c/p\u003e\n\u003cp\u003eModern systems evolve fast. New features appear, requirements change, and data models grow in unexpected directions. If you’ve ever worked with a “god table” — a huge table with dozens or hundreds of columns — you know how painful schema evolution can be:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eevery new feature requires a migration\u003c/li\u003e\n\u003cli\u003emany columns stay empty for most rows\u003c/li\u003e\n\u003cli\u003ethe table becomes harder to maintain\u003c/li\u003e\n\u003cli\u003ethe schema no longer reflects real business logic\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eOne way to avoid this is to use \u003cstrong\u003epolymorphism in the database\u003c/strong\u003e through a \u003cstrong\u003ekey–value (EAV) table\u003c/strong\u003e. Instead of adding new columns, you store flexible attributes as rows.\u003c/p\u003e","title":"Polymorphism in Database Design: A Simple Key–Value Approach for Evolving Schemas"},{"content":"\nProblem: SPA applications face challenges with SEO as they typically return index.html for all requests. While one solution involves proxying all requests through a backend application to render SPA layout or traditional HTML for robots, this introduces complexity and disadvantages.\nKubernetes ingress Fortunately, Kubernetes’ Ingress controller provides logic to address this issue without additional complexity. Annotations can be used to configure routing for the Ingress.\nAs an example, let’s redirect all robot requests to our seo-service.\nIngress Controller Example: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: main-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: $uri nginx.ingress.kubernetes.io/server-snippet: | set $prerender 0; if ($http_user_agent ~* \u0026#34;googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest|slackbot|vkShare|W3C_Validator\u0026#34;) { set $prerender 1; } if ($uri ~* \u0026#34;\\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)\u0026#34;) { set $prerender 0; } if ($prerender = 1) { rewrite ^(.*)$ /seo$1 break; } spec: rules: - host: yoursite.org http: paths: - path: /api pathType: Prefix backend: service: name: backend-service port: number: 30200 - path: /seo pathType: Prefix backend: service: name: seo-service port: number: 30201 - path: / pathType: Prefix backend: service: name: frontend-service port: number: 30202 Explanation: Services:\nbackend-service: Backend application providing API for SPA. seo-service: Backend application rendering traditional HTML for robots. frontend-service: Static storage serving SPA files (images, CSS, JS, etc.). Rules Block:\nResponsible for routing requests based on their paths. Annotation Block:\nAnnotations are handled before routing, allowing us to modify routing. nginx.ingress.kubernetes.io/rewrite-target: $uri: Adds the request path to each routed request. nginx.ingress.kubernetes.io/server-snippet: Contains logic to route robot requests to the seo-service by adding the /seo prefix. Conclusion: In conclusion, Kubernetes Ingress controllers offer an efficient way to make SPAs more SEO-friendly with minimal effort.\n","permalink":"https://sukachvitally.github.io/posts/spa-seo-routing-with-kubernetes-ingress-controller/","summary":"\u003cp\u003e\u003cimg loading=\"lazy\" src=\"https://sukachvitally.github.io/posts/spa-seo-routing-with-kubernetes-ingress-controller/image-01.png\"\u003e\u003c/p\u003e\n\u003ch3 id=\"problem\"\u003eProblem:\u003c/h3\u003e\n\u003cp\u003eSPA applications face challenges with SEO as they typically return \u003ccode\u003eindex.html\u003c/code\u003e for all requests. While one solution involves proxying all requests through a backend application to render SPA layout or traditional HTML for robots, this introduces complexity and disadvantages.\u003c/p\u003e\n\u003ch3 id=\"kubernetes-ingress\"\u003eKubernetes ingress\u003c/h3\u003e\n\u003cp\u003eFortunately, Kubernetes’ Ingress controller provides logic to address this issue without additional complexity. Annotations can be used to configure routing for the Ingress.\u003c/p\u003e\n\u003cp\u003eAs an example, let’s redirect all robot requests to our \u003ccode\u003eseo-service\u003c/code\u003e.\u003c/p\u003e","title":"SPA SEO Routing with Kubernetes Ingress Controller"},{"content":"Hi, I\u0026rsquo;m Vitalii 👋\nI\u0026rsquo;m a Java developer, and I\u0026rsquo;ve been writing backend code for more than 12 years: mostly Spring Boot, microservices, PostgreSQL, Kafka and everything that keeps them running in the cloud.\nMy favourite hobby is keeping up with what\u0026rsquo;s new: new Java releases, new tools, new trends. Lately that\u0026rsquo;s been a lot of AI tooling like Claude Code, MCP and RAG. Whenever I learn something, I try it out for real, and then write about it here. If something took me a weekend of digging to understand, I\u0026rsquo;d like it to take you twenty minutes of reading.\nWhat you\u0026rsquo;ll find here How Java actually works under the hood: the memory model, concurrency, builds PostgreSQL stuff I ran into in production Spring and Hibernate gotchas AI tools for developers, tried hands-on Notes on how teams and projects really work When I\u0026rsquo;m not writing backend code I make games! Game development is my other hobby. You can check out what I\u0026rsquo;ve built on Google Play.\nSay hi Medium: @sukach.v.v GitHub: SukachVitally LinkedIn: vitalii-sukach ","permalink":"https://sukachvitally.github.io/about/","summary":"\u003cp\u003eHi, I\u0026rsquo;m \u003cstrong\u003eVitalii\u003c/strong\u003e 👋\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;m a Java developer, and I\u0026rsquo;ve been writing backend code for more than 12 years: mostly Spring Boot, microservices, PostgreSQL, Kafka and everything that keeps them running in the cloud.\u003c/p\u003e\n\u003cp\u003eMy favourite hobby is keeping up with what\u0026rsquo;s new: new Java releases, new tools, new trends. Lately that\u0026rsquo;s been a lot of AI tooling like Claude Code, MCP and RAG. Whenever I learn something, I try it out for real, and then write about it here. If something took me a weekend of digging to understand, I\u0026rsquo;d like it to take you twenty minutes of reading.\u003c/p\u003e","title":"About me"}]