Ask an LLM “When does the nightly settlement batch run at Harbourline Payments?” and one of two things happens.

The polite model says it doesn’t know. The confident model says “Usually around midnight!” with the energy of a student who didn’t read the book but really wants a good mark.

Both are fair, because Harbourline Payments doesn’t exist. I made it up. It’s a fictional company with a fictional engineering handbook: founders, services, deploy rules, on-call pay, and even a famous incident called “the Double Tap”. No model has seen this text during training.

That’s exactly why it’s a good test. If the model answers “02:30 UK time” correctly, it didn’t remember it. It found it. That’s RAG.

In this article we’ll build a small RAG service in Java with Spring Boot, LangChain4j, PostgreSQL with pgvector, and Ollama. No cloud, no API keys, no bill at the end of the month.

The full code is on GitHub: SukachVitally/rag-demo.

What is RAG, in one exam

RAG stands for Retrieval-Augmented Generation. Long name, simple idea.

Think about two exams:

• Closed-book exam. The student answers from memory. This is a normal LLM. It knows a lot, but only what it learned during training. It knows nothing about your company’s wiki, last week’s incident, or your internal rules. If it doesn’t know, it may still try to sound smart.
• Open-book exam. Before answering, the student finds the right pages in the book and reads them. This is RAG.

So RAG doesn’t make the model smarter. It gives the model a cheat sheet for each question: “Here are 5 pieces of text that look relevant. Answer using only these.”

Why not just put the whole handbook into the prompt? For one small file you can. But real knowledge bases have thousands of documents. They don’t fit into the context window, and even when they do, you pay for every token on every question. RAG sends only the few pieces that matter.

The big idea

The whole thing is two flows.

Ingest (once per document):

1. Take the text and split it into small pieces called chunks.
2. Ask an embedding model to turn each chunk into a vector: a long list of numbers.
3. Save each chunk together with its vector in the database.

Ask (every question):

1. Ask the same embedding model to turn the question into a vector.
2. Find the chunks whose vectors are closest to the question vector.
3. Send the question and those chunks to a chat model and ask it to answer using only these chunks.

That’s it. The rest of this article is details.

Notice that there are two different models:

The embedding model is small and fast. The chat model is the big one. Both run locally in Ollama.

Wait, what is this vector?

This is the most interesting part, so let’s slow down a little.

An embedding model reads a piece of text and returns a fixed-size array of numbers. nomic-embed-text returns 768 numbers for any text, whether it’s one word or a full paragraph:

"settlement batch runs at 02:30"  →  [-0.064, 0.142, 0.009, ... 765 more ...]

The model is trained so that texts with similar meaning get vectors that point in a similar direction. “When does the nightly batch run?” and “settlement-service runs the nightly batch at 02:30” share almost no grammar, but their meaning is close, so their vectors are close too.

Now the fun part. It’s natural to think that each number means something: dimension 17 is “money”, dimension 302 is “time of day”, dimension 555 is “how angry the on-call engineer is”. It doesn’t work like that.

No single dimension has a physical meaning. Nobody designed these axes. The model learned the layout by itself during training. Meaning is spread across all 768 numbers at once, a bit like a smell is spread across many different molecules. You can’t point at one number and say “this is the time part”. Only the vector as a whole means something, and only when you compare it with other vectors.

Here’s a 2D sketch of what that looks like. The real space has 768 dimensions, and I have only two axes on my screen:

The question vector points almost in the same direction as the “settlement batch runs at 02:30” chunk. The sourdough recipe chunk points somewhere else entirely. So for a computer, “similar meaning” becomes simple geometry: a small angle between two vectors.

The usual way to measure this is cosine similarity, the cosine of that angle:

• 1: same direction, very similar
• 0: unrelated
• -1: opposite direction

LangChain4j turns it into a friendlier score from 0 to 1: (1 + cos θ) / 2. That’s the score you’ll see in the API response later.

One important rule: vectors from different embedding models are not compatible. Each model builds its own space with its own layout. If you change the embedding model, you have to re-embed all your documents. It’s like changing the map projection: the old coordinates don’t match the new map.

The stack

• Java 25 and Spring Boot 4.1: the usual REST app.
• LangChain4j 1.22: the Java library that does the AI work.
• Ollama: runs both models locally.
• PostgreSQL 17 + pgvector: stores chunks and vectors, and searches by similarity.
• Testcontainers: for functional tests with a real database.

Run the infrastructure with one command:

Source: docker-compose.yaml

services:
  db:
    image: pgvector/pgvector:pg17
    environment:
      POSTGRES_DB: rag
      POSTGRES_USER: rag
      POSTGRES_PASSWORD: rag
    ports: ["5432:5432"]
  ollama:
    image: ollama/ollama
    ports: ["11434:11434"]
    volumes:
      - ollama-models:/root/.ollama   # keeps downloaded models between restarts
  ollama-init:
    image: ollama/ollama
    depends_on: [ollama]
    environment:
      OLLAMA_HOST: ollama:11434
    entrypoint: ["sh", "-c", "sleep 5 && ollama pull nomic-embed-text && ollama pull llama3.1:8b"]
volumes:
  ollama-models:

The ollama-init container is a one-off job that downloads the models. The first time it pulls a few GB, so it’s a good moment to make some tea.

LangChain4j: AI LEGO for Java

LangChain4j is to AI roughly what Spring Data is to databases. It gives you small interfaces with many implementations, so your code doesn’t depend on a specific vendor.

The demo uses four building blocks, plus one helper that glues three of them together:

• DocumentSplitter cuts text into chunks.
• EmbeddingModel turns text into a vector. Here it’s OllamaEmbeddingModel.
• EmbeddingStore<TextSegment> saves vectors and finds the nearest ones. Here it’s PgVectorEmbeddingStore.
• ChatModel sends a prompt and gets an answer. Here it’s OllamaChatModel.
• EmbeddingStoreIngestor runs split → embed → store in one call.

The nice part is the bottom of that picture. Want OpenAI instead of Ollama? Change one bean. Want Qdrant instead of Postgres? Change one bean. The services don’t care.

Dependencies

Source: build.gradle

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-jdbc'
    implementation 'org.springframework.boot:spring-boot-starter-webmvc'
    implementation 'dev.langchain4j:langchain4j:1.22.0'
    implementation 'dev.langchain4j:langchain4j-ollama:1.22.0'
    implementation 'dev.langchain4j:langchain4j-pgvector:1.22.0-beta32'
    runtimeOnly 'org.postgresql:postgresql'
}

Three LangChain4j modules: the core, the Ollama integration and the pgvector store. That’s all.

Wiring the beans

Source: LangChain4jConfig.java

@Configuration
public class LangChain4jConfig {
    public static final String CHUNK_TABLE = "chunk_embedding";
    @Bean
    EmbeddingModel embeddingModel(OllamaClientProperty property) {
        return OllamaEmbeddingModel.builder()
                .baseUrl(property.baseUrl())
                .modelName(property.embedModel())
                .build();
    }
    @Bean
    ChatModel chatModel(OllamaClientProperty property) {
        return OllamaChatModel.builder()
                .baseUrl(property.baseUrl())
                .modelName(property.chatModel())
                .build();
    }
    @Bean
    DocumentSplitter documentSplitter(ChunkProperty property) {
        return DocumentSplitters.recursive(property.size(), property.overlap());
    }
    @Bean
    EmbeddingStore<TextSegment> embeddingStore(DataSource dataSource) {
        return PgVectorEmbeddingStore.datasourceBuilder()
                .datasource(dataSource)
                .table(CHUNK_TABLE)
                .createTable(false)
                .skipCreateVectorExtension(true)
                .metadataStorageConfig(DefaultMetadataStorageConfig.builder()
                        .storageMode(MetadataStorageMode.COMBINED_JSONB)
                        .columnDefinitions(List.of("metadata JSONB NULL"))
                        .build())
                .build();
    }
}

The embedding store reuses Spring’s DataSource, so there’s one connection pool and one config. I also told it not to create the table (createTable(false)) because I prefer to own the schema myself:

Source: schema.sql

CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE IF NOT EXISTS chunk_embedding (
    embedding_id UUID PRIMARY KEY,
    embedding    vector(768) NOT NULL,
    text         TEXT NULL,
    metadata     JSONB NULL
);
CREATE INDEX IF NOT EXISTS chunk_embedding_hnsw
    ON chunk_embedding USING hnsw (embedding vector_cosine_ops);

Two things to notice:

• vector(768) is the pgvector column type. 768 must match the embedding model’s output. If you switch to a model with 1024 dimensions, this column has to change too.
• The HNSW index with vector_cosine_ops makes nearest-neighbour search fast. Without it, Postgres compares the question with every row. That’s fine for 9 chunks, but not for 9 million. HNSW is an approximate search: it trades a tiny bit of accuracy for a lot of speed.

Everything is configurable in application.yaml:

app:
  olama-client:
    base-url: http://localhost:11434
    embed-model: nomic-embed-text
    chat-model: llama3.1:8b
  chunk:
    size: 1000
    overlap: 150

Part 1: Ingest, or feeding the database

Here’s what happens to a document on the way in:

Why chunks?

Why not embed the whole document as one vector? Because one vector for a whole handbook would be an average of everything: founders, deploy rules, on-call pay, the sourdough recipe someone accidentally uploaded. It would be “kind of about everything”, and that means it’s close to nothing.

Small chunks keep the meaning sharp. One chunk is about the settlement batch, another is about deploy windows. Now the search can find the right one.

The demo uses DocumentSplitters.recursive(1000, 150):

• 1000: the maximum chunk size in characters.
• 150: overlap. Each chunk repeats about 150 characters from the end of the previous one. So if an important sentence lands on a border, it’s still complete in at least one chunk.
• Recursive means it tries to cut nicely. First by paragraphs. If a paragraph is too big, then by lines, then by sentences, then by words. It cuts the text like a polite butcher, not with a chainsaw.

The ingest service

Source: IngestService.java

@Service
@RequiredArgsConstructor
public class IngestService {
    public static final String SOURCE = "source";
    private final DocumentSplitter splitter;
    private final EmbeddingModel embeddingModel;
    private final EmbeddingStore<TextSegment> store;
    
    public int ingest(String source, String text) {
        if (text.isBlank()) return 0;
        var chunks = new AtomicInteger();
        EmbeddingStoreIngestor.builder()
                .documentSplitter(splitter)
                .textSegmentTransformer(segment -> { chunks.incrementAndGet(); return segment; })
                .embeddingModel(embeddingModel)
                .embeddingStore(store)
                .build()
                .ingest(Document.from(text, Metadata.from(SOURCE, source)));
        return chunks.get();
    }
}

EmbeddingStoreIngestor does all three steps for us: split, embed, store. The textSegmentTransformer is a hook that runs for every chunk. Here I only use it to count chunks for the API response, but it’s also the right place to clean text or add extra metadata.

The source metadata (the file name) travels with every chunk into the metadata JSONB column. Later it lets the answer say where the information came from.

The controller is a plain multipart upload:

Source: RagController.java

@PostMapping(value = "/ingest", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public Map<String, Object> upload(@RequestParam MultipartFile file) throws IOException {
    String text = new String(file.getBytes(), StandardCharsets.UTF_8);
    return Map.of("chunks", ingest.ingest(file.getOriginalFilename(), text));
}

Let’s feed it the Harbourline handbook:

curl -F file=@docs/notes.md http://localhost:8080/api/ingest

{ "chunks": 9 }

Nine chunks, nine vectors, nine rows. The database now knows about a company that doesn’t exist.

Part 2: Ask, or the open-book exam

Source: RagService.java

@Service
@RequiredArgsConstructor
public class RagService {
    private final EmbeddingModel embeddingModel;
    private final EmbeddingStore<TextSegment> store;
    private final ChatModel chatModel;
    private static final String SYSTEM = """
        Answer using ONLY the provided context. If the answer is not in the
        context, say you don't know. Cite sources as [1], [2]...""";
    
public record Answer(String answer, List<Hit> sources) {}
    public Answer ask(String question) {
        var q = embeddingModel.embed(question).content();
        var hits = store.search(EmbeddingSearchRequest.builder().queryEmbedding(q).maxResults(5).build())
                .matches().stream()
                .map(m -> new Hit(m.embedded().metadata().getString(IngestService.SOURCE), m.embedded().text(), m.score()))
                .toList();
        var ctx = new StringBuilder();
        for (int i = 0; i < hits.size(); i++)
            ctx.append("[").append(i + 1).append("] (").append(hits.get(i).source()).append(")\n")
                    .append(hits.get(i).content()).append("\n\n");
        String prompt = "Context:\n" + ctx + "\nQuestion: " + question;
        return new Answer(chatModel.chat(SystemMessage.from(SYSTEM), UserMessage.from(prompt)).aiMessage().text(), hits);
    }
}

This is the heart of the project, and it follows the plan exactly:

1. Embed the question with the same model we used for the chunks. Same model means same space, so the distances make sense.
2. Search the store for the 5 nearest vectors. Under the hood, pgvector uses the HNSW index and cosine distance.
3. Build the context: number each chunk and add its source.
4. Ask the chat model with a strict system message: use only the context, say “I don’t know” if the answer isn’t there, and cite sources.

This is roughly what llama actually receives:

SYSTEM:
Answer using ONLY the provided context. If the answer is not in the
context, say you don't know. Cite sources as [1], [2]...

USER:
Context:
[1] (notes.md)
... settlement-service — runs the nightly settlement batch at 02:30 UK time ...

[2] (notes.md)
...

Question: When does the nightly settlement batch run?

The chat model never sees the database, the vectors or the other chunks. It sees a small, focused cheat sheet and a question. Its job is now easy: read and summarise.

Let’s try it:

curl -H 'Content-Type: application/json' \
     -d '{"question": "When does the nightly settlement batch run?"}' \
     http://localhost:8080/api/ask

{
  "answer": "According to the context in [3] (notes.md), the nightly settlement batch runs at 02:30 UK time.",
  "sources": [
    { "source": "notes.md", "content": "...", "score": 0.79 }
  ]
}

02:30 UK time. Correct, and with a source. Open-book exam passed. 🎓

The sources list matters as much as the answer. You can show it to users, so they can check where the answer came from instead of trusting the model blindly. Trust, but verify, especially with an 8B model.

Things to keep in mind

The demo is small on purpose. Before you take it to production, think about these:

• Chunk size is a trade-off. Too small, and a chunk loses context (“it runs at 02:30”, but what is “it”?). Too big, and the meaning gets blurry again. 1000/150 is a reasonable start, not a law.
• The top 5 are always returned, even when nothing is relevant. Ask about pizza, and you’ll still get the 5 “least irrelevant” chunks. The strict system prompt helps the model say “I don’t know”. In a real system I’d also add a minimum score (EmbeddingSearchRequest supports minScore).
• Changing the embedding model means re-ingesting everything. Different model, different space, different vector(N) column.
• Re-uploading the same file creates duplicates. The demo only appends. A real system needs to update or delete chunks by source.
• Garbage in, garbage out. RAG is only as good as the documents you give it. If the wiki is wrong, the answer will be wrong, but now with a citation.

Summary

RAG sounds like a fancy AI term, but in code it’s a short pipeline:

• Ingest: split text into chunks → embed each chunk into a vector → store the text and vector in pgvector.
• Ask: embed the question → find the closest chunk vectors → give the question and those chunks to the chat model → answer with sources.

The vectors are the only “magic” part. They are 768 numbers with no human meaning in any single dimension, but their direction captures meaning. Similar text means a small angle, and a small angle is something a database can search.

LangChain4j turns each step into one small interface, so the whole service is a handful of Spring beans. You can swap Ollama for OpenAI, or pgvector for Qdrant, without touching the business code.

And now a local 8B model knows when Harbourline’s settlement batch runs. Harbourline still doesn’t exist, but at least its batch is on time. ⏰

The full code is on GitHub: SukachVitally/rag-demo. Clone it, run docker compose up -d, upload your own notes and ask them questions.

In my previous article I built an MCP server from scratch in Python. I wrote the JSON-RPC 2.0 message handling, declared tool schemas by hand and wired up the STDIO transport myself. It’s a good way to learn how the protocol works on the wire.

Python isn’t the only option, though. Most of the systems I work on are Java and Spring, and that’s where the data lives. So this time I built the same kind of server in Java.

It turns out you don’t need to implement JSON-RPC yourself. Spring AI has a Boot starter for MCP servers. It handles the protocol, the transport, schema generation and tool registration. You write ordinary Spring beans and add a few annotations.

The full code is on GitHub: SukachVitally/mcp-demo.

Why bother: the server is the boundary

The business logic here is deliberately boring. It’s a task list: you create tasks, move them through statuses, and finish or cancel them. Swap in your own domain; the MCP part stays the same.

The point is how Claude Code reaches the data:

Claude Code ──HTTP + API key──▶ MCP server (Spring) ──JDBC──▶ PostgreSQL

• Claude never sees database credentials. The server holds the connection. Claude holds only an API key that works for the MCP endpoint and nothing else.
• Claude can only do what you expose. There’s no SQL console. If you don’t write a delete_task tool, tasks can’t be deleted, however the prompt is phrased.
• Your validation and transactions still apply. Every call goes through the same service layer as the rest of your application.

Giving an agent a database URL and a password is quick, but you can’t take it back. An MCP server gives the agent a small, typed and auditable API instead.

What we’re building

A PostgreSQL table of tasks. Each task has a name and one of four statuses: NEW, IN_PROGRESS, DONE or CANCELLED. The server exposes four tools:

Stack: Java 25, Spring Boot 4, Spring AI 2.0 MCP server, PostgreSQL 18 with Liquibase, and Spring Security.

Step 1: Dependencies

Source: build.gradle

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
    implementation 'org.springframework.boot:spring-boot-starter-liquibase'
    implementation 'org.springframework.boot:spring-boot-starter-webmvc'
    implementation 'org.springframework.boot:spring-boot-starter-security'
    implementation 'org.springframework.boot:spring-boot-starter-validation'
    implementation platform('org.springframework.ai:spring-ai-bom:2.0.1')
    implementation 'org.springframework.ai:spring-ai-starter-mcp-server-webmvc'
    implementation 'org.springaicommunity:mcp-server-security-spring-boot:0.1.14'
    runtimeOnly 'org.postgresql:postgresql'
}

Two lines do the MCP work:

• spring-ai-starter-mcp-server-webmvc is the MCP server over HTTP on Spring MVC. There’s a -webflux variant for reactive apps, and a plain spring-ai-starter-mcp-server for stdio.
• mcp-server-security-spring-boot from the spring-ai-community/mcp-security project adds API-key and OAuth2 support on top of Spring Security. We’ll come back to it.

Step 2: Configure the server

Source: application.yaml

spring:
  ai:
    mcp:
      server:
        protocol: STATELESS
        type: sync
        name: "Task MCP demo"
        instructions: >-
          Manages a simple task list. Create tasks with add_task, look them up with get_task or
          get_task_list, and move them through the statuses NEW, IN_PROGRESS, DONE and CANCELLED
          with update_task_status.
        streamable-http:
          mcp-endpoint: /mcp

What these settings mean:

• protocol: STATELESS: plain HTTP request and response, with no session kept between calls. Our tools never need to push anything back to the client, so we don’t need sessions, and the server scales horizontally like any other REST service. Use STREAMABLE if you need server-to-client messages. The older SSE option is deprecated since Spring AI 2.0.
• type: sync: tools are ordinary blocking methods. With async, they return Reactor Mono/Flux types instead.
• instructions: sent to the client when it connects. It’s the server’s short explanation to the model of how the tools fit together. Treat it as part of the prompt.
• mcp-endpoint: /mcp: the URL that Claude Code will call.

Step 3: Business logic (keep it boring)

Nothing here knows about MCP. It’s a normal JPA entity, repository and service:

Source: TaskService.java

@Service
@Validated
@RequiredArgsConstructor
@Transactional(readOnly = true)
public class TaskService {
    public static final int NAME_MAX_LEN
GTH = 255;
    private final TaskRepository repository;
    private final TaskMapper mapper;
    @Transactional
    public TaskDto create(@NotBlank @Size(max = NAME_MAX_LENGTH) String name) {
        var entity = new TaskEntity();
        entity.setName(name);
        entity.setStatus(TaskStatus.NEW);
        return mapper.toDto(repository.save(entity));
    }
    @Transactional
    public TaskDto updateStatus(Long id, @NotNull TaskStatus status) {
        var entity = findEntity(id);
        entity.setStatus(status);
        return mapper.toDto(entity);
    }
    // get(id), getList(), findEntity(id) throws TaskNotFoundException ...
}

The service returns a small record, not the entity:

Source: TaskDto.java

public record TaskDto(Long id, String name, TaskStatus status) {}

That matters more than it looks. Whatever a tool returns is serialized and handed to the model, so the DTO decides exactly which fields leave your system.

Step 4: TaskMcpTool, the actual MCP server

This is the whole MCP layer:

Source: TaskMcpTool.java

@Component
@RequiredArgsConstructor
public class TaskMcpTool {
    private final TaskService service;
    @McpTool(name = "add_task", description = "Create a new task with status NEW and return it")
    public TaskDto add(@McpToolParam(description = "Task name, 1-255 characters") String name) {
        return service.create(name);
    }
    @McpTool(name = "get_task", description = "Get a task by its id. Fails if the task does not exist")
    public TaskDto get(@McpToolParam(description = "Task id") Long id) {
        return service.get(id);
    }
    @McpTool(name = "update_task_status",
            description = "Change the status of a task and return the updated task. Fails if the task does not exist")
    public TaskDto updateStatus(@McpToolParam(description = "Task id") Long id,
                                @McpToolParam(description = "New task status") TaskStatus status) {
        return service.updateStatus(id, status);
    }
    @McpTool(name = "get_task_list", description = "List all tasks")
    public List<TaskDto> getList() {
        return service.getList();
    }
}

Only a thin adapter: each method delegates to the service. Most of the work happens in the annotations.

How the tools get registered

The class is a plain @Component. When the application starts, Spring AI’s annotation scanner finds every bean method annotated with @McpTool and registers it as an MCP tool. You don’t need a registry, a ToolCallback list or any manual wiring. To add a tool, write another annotated method.

@McpTool: the name and description the model sees

• name is the tool id the model calls. I use snake_case verbs (add_task, get_task_list) because that’s the convention across MCP servers, and models pick such names up reliably. Without a name, the Java method name is used, and add on its own tells the model very little.
• description is the most important string in the class. It’s the only thing the model reads when deciding which tool to call and what to expect back. Say what the tool does, what it returns and when it fails (“Fails if the task does not exist”). A vague description gives you a model that guesses.

@McpToolParam: the schema is generated for you

Each parameter gets a description, and Spring AI builds a JSON Schema from the Java type. Here’s what the server actually returns for update_task_status when Claude Code asks for tools/list:

{
  "name": "update_task_status",
  "description": "Change the status of a task and return the updated task. Fails if the task does not exist",
  "inputSchema": {
    "type": "object",
    "properties": {
      "id":     { "type": "integer", "format": "int64", "description": "Task id" },
      "status": { "type": "string",
                  "enum": ["NEW", "IN_PROGRESS", "DONE", "CANCELLED"],
                  "description": "New task status" }
    },
    "required": ["id", "status"]
  }
}

Things to notice:

• Long became integer with int64 format. You don’t write any schema by hand.
• Parameters are required by default (required = true). Set @McpToolParam(required = false) for optional ones.
• The TaskStatus enum became a JSON enum. The model sees the four allowed values up front, so it can’t invent a status like DELETED. If it tries, the call is rejected by schema validation before your method runs:

Tool (update_task_status) input validation failed: ... /status: does not 
have a value in the enumeration ["NEW", "IN_PROGRESS", "DONE", "CANCELLED"]

Use enums for anything with a fixed set of values. You get validation and documentation in one place.

Return values and errors

You return a normal Java object, and Spring AI serializes it to JSON as the tool result. TaskDto and List<TaskDto> just work.

Exceptions work too. When TaskService throws TaskNotFoundException, the client doesn’t get a stack trace or a broken JSON-RPC response. It gets a tool result with isError: true and the exception message as the text:

{ 
  "content": [{ "type": "text", "text": "Task with id -1 is not found" }], 
  "isError": true 
}

That’s the behaviour you want: the model reads the message, understands what went wrong and can recover, for example by calling get_task_list to find the right id. Write exception messages for the model to read, not just for your logs.

The same applies to Bean Validation. A blank name fails @NotBlank in the service and comes back to the model as an error result, not a 500.

A tip: add behaviour hints

If you look at the full tools/list response, every tool reports “readOnlyHint”: false, “destructiveHint”: true, including get_task_list. Those are the defaults. Clients such as Claude Code can use these hints to decide how cautious to be, so it’s worth being honest about read-only tools:

@McpTool(name = "get_task_list", description = "List all tasks",
        annotations = @McpTool.McpAnnotations(readOnlyHint = true, 
                                              destructiveHint = false))

These are hints, not enforcement. What actually protects your data is the fact that the tool only reads.

Step 5: SecurityConfig, briefly

By default, an HTTP MCP endpoint is an unauthenticated JSON-RPC endpoint. Anyone who can reach /mcp can call your tools. The mcp-security library closes it with a few lines:

Source: SecurityConfig.java

@Configuration
@EnableWebSecurity
class SecurityConfig {
    @Bean
    SecurityFilterChain securityFilterChain(HttpSecurity http,
                                            ApiKeyEntityRepository<ApiKeyEntityImpl> apiKeyRepository) {
        return http
                .authorizeHttpRequests(auth -> auth.anyRequest().authenticated())
                .with(mcpServerApiKey(), apiKey -> apiKey.apiKeyRepository(apiKeyRepository))
                .build();
    }
    @Bean
    ApiKeyEntityRepository<ApiKeyEntityImpl> apiKeyRepository(@Value("${app.mcp.api-key.secret}") String secret) {
        var key = ApiKeyEntityImpl.builder()
                .name("demo key")
                .id("demo")
                .secret(secret)
                .build();
        return new InMemoryApiKeyEntityRepository<>(List.of(key));
    }
}

• mcpServerApiKey() plugs API-key authentication into the normal Spring Security filter chain.
• The client sends an X-API-key header in the form <id>.<secret>, so demo.<your secret> here. Requests without it get 401.

Step 6: Run it and connect Claude Code

Start PostgreSQL and the app:

Source: docker-compose.yaml,.env.tpl, README

cp .env.tpl .env          # sets MCP_API_KEY_SECRET
docker compose up -d
./gradlew bootRun

Register the server with Claude Code:

claude mcp add --transport http todos http://localhost:8080/mcp \
  --header "X-API-key: demo.mcp-demo-test-secret"

Then ask in plain language:

“Add todo with name test” becomes an add_task call. “Update task with id 1 to DONE” becomes update_task_status with a valid enum value. Claude never saw a JDBC URL, a username or a password.

To inspect the server without Claude, use the MCP Inspector (npx @modelcontextprotocol/inspector). Choose the Streamable HTTP transport and add the same header.

Testing

Because the server speaks standard MCP, the tests can use the official Java MCP client against a real server started by @SpringBootTest, with PostgreSQL in Testcontainers:

Source: TaskMcpToolTest.java

var transport = HttpClientStreamableHttpTransport
        .builder("http://localhost:" + port)
        .httpRequestCustomizer((builder, method, endpoint, body, context) ->
                builder.header("X-API-key", "demo." + API_KEY_SECRET))
        .build();
client = McpClient.sync(transport).build();
client.initialize();
var result = client.callTool(McpSchema.CallToolRequest.builder("get_task")
        .arguments(Map.of("id", -1L)).build());
assertThat(result.isError()).isTrue();

This covers the whole chain in one test: authentication, schema, serialization, error mapping and the database.

Takeaways

• You don’t write JSON-RPC in Java. spring-ai-starter-mcp-server-webmvc handles the protocol, the transport and tool discovery.
• An MCP tool is an annotated Spring bean method. @McpTool names and describes it, and @McpToolParam describes its inputs. The JSON Schema is generated from your Java types.
• Descriptions and error messages are prompts. The model reads them, so write them for the model.
• Use enums and DTOs to control what the model can send in and what it gets back.
• Secure the endpoint. API keys via mcp-security take ten lines, and OAuth2 is available when you need it.
• The MCP server is the security boundary. Claude Code works with your data through the operations you choose, never with your credentials.

The Python version showed how MCP works on the wire. The Spring version shows how little code you need once a framework handles it. Take the repo, replace tasks with your own domain, and give your agent an API instead of a password.

Links

• Demo code: github.com/SukachVitally/mcp-demo
• Spring AI MCP Server Boot Starter: docs.spring.io
• MCP security for Spring: github.com/spring-ai-community/mcp-security
• Previous article, MCP in Python from scratch: MCP: The Protocol That Finally Taught AI to Stop Being Useless

Ask ten Java developers what “the Java Memory Model” means and you’ll get two different answers.

Half of them will draw a box called Heap and another called Stack. The other half will start talking about volatile and happens-before.

Both groups are talking about something real, but only the second group is talking about the actual Java Memory Model (JMM). The first group is describing the runtime data areas — where objects physically live.

You need both to write correct concurrent code, so let’s go through them in order. Memory layout first, because it’s the easier one, then the JMM proper.

Part 1: Where your data lives

When the JVM starts, it carves the process memory into a few distinct regions. Each has an owner and a lifetime.

Heap — every object and array you ever create lives here. It’s shared, which is exactly why concurrency is hard. Garbage collection happens here. Modern collectors split it further (young generation, old generation, or regions in G1/ZGC), but that’s a GC detail, not a memory model one.

Metaspace — class metadata, bytecode, and static fields. It sits in native memory since Java 8, which is why PermGen errors disappeared and Metaspace OOM errors appeared instead.

Thread stack — each thread gets its own. Every method call pushes a frame containing local variables, parameters, and the return address. Frame pops when the method returns. This is where StackOverflowError comes from. Nothing here is shared, so locals are never a concurrency problem — unless the value they point to is a shared object.

PC register — the address of the instruction the thread is currently executing. One per thread.

Native method stack — the same idea, but for native (JNI) calls.

One thing worth burning in:

Order order = new Order();

The variable order is a reference sitting on the stack. The Order object itself sits on the heap. Two threads can hold two separate stack references pointing to the same heap object. That’s the whole source of the trouble.

Part 2: The actual Java Memory Model

The runtime areas tell you where data sits. They tell you nothing about when one thread sees a write made by another.

That question is what the JMM answers. It’s a specification (JSR-133, in Chapter 17 of the Java Language Specification) that defines exactly what guarantees you get about reads and writes across threads.

The problem it exists to solve

Modern CPUs are fast because they cheat, aggressively and correctly. Two things they do:

1. Caching. Each core has its own L1/L2 cache. When a thread writes to a field, that write may sit in the core’s cache or store buffer for a while before it becomes visible to other cores.

2. Reordering. The compiler, the JIT, and the CPU are all allowed to reorder instructions, as long as the result looks the same for a single thread. That last clause is the trap — nobody promised anything about how it looks to other threads.

So the JMM works with an abstraction:

Thread A and Thread B never see each other’s working memory. Everything they share has to travel through main memory — and the JMM decides when it does.

Every thread has a conceptual working memory (registers, cache, store buffers) holding copies of the variables it touches. Threads never talk to each other directly — they communicate only through main memory. The JMM specifies when a value must be flushed to main memory and when it must be re-read from it.

Here’s the classic demonstration:

public class StopFlag {
    private boolean running = true;   // no volatile
    
    public void run() {
        while (running) {
            // do work
        }
        System.out.println("stopped");
    }
    public void stop() {
        running = false;
    }
}

Thread A runs run(). Thread B calls stop(). Very often, thread A never stops.

Nothing is broken. The JIT sees that running is never modified inside the loop, so it’s free to hoist the read out:

boolean r = running;
while (r) { /* work */ }   // now an infinite loop

Without a memory model rule forcing it, the JVM has no obligation to ever re-read that field. Your bug is not a race in the “two writes collided” sense — it’s a visibility failure.

The three guarantees

Every concurrency problem in Java falls into one of these buckets.

count++ is the textbook atomicity failure — it’s three operations (read, add, write), and two threads can interleave in the middle. Two increments, one result.

The base components of the JMM

Here’s the toolbox. It’s smaller than most people expect.

1. happens-before — the rule that underpins everything

Everything else is a special case of this one relation. If action A happens-before action B, then:

• everything A wrote is visible to B, and
• A is guaranteed to appear to occur before B.

It is not just “A ran earlier in wall-clock time.” Two actions can be ordered in time and still have no happens-before relationship, which means B may see stale data.

The rules the JLS gives you:

1. Program order — within one thread, each statement happens-before the ones written after it.
2. Monitor lock — unlocking a monitor happens-before any later lock of the same monitor.
3. volatile — a write to a volatile field happens-before every later read of that field.
4. Thread start — Thread.start() happens-before anything in the started thread.
5. Thread join — everything in a thread happens-before another thread’s successful join() on it.
6. Interruption — a call to interrupt() happens-before the interrupted thread detects it.
7. Constructor / finalizer — the end of a constructor happens-before the start of that object’s finalizer.
8. Transitivity — if A hb B and B hb C, then A hb C.

That last one does most of the practical work. It’s why writing a plain field and then a volatile flag safely publishes the plain field.

2. volatile — visibility and ordering, no atomicity

private volatile boolean running = true;

This single keyword fixes the StopFlag example. It gives you:

• Visibility — a write goes to main memory immediately; a read comes from main memory, never a cache.
• Ordering — the JVM inserts memory barriers so reads and writes can’t be reordered across the volatile access.

What it does not give you is atomicity:

private volatile int count = 0;
public void increment() {
    count++;   // STILL BROKEN
}

Use volatile for flags and for single-writer state. For compound operations, use something else.

3. synchronized — all three guarantees

private int count = 0;
public synchronized void increment() {
    count++;   // correct
}
public synchronized int get() {
    return count;   // also needs to be synchronized
}

Entering the monitor gives mutual exclusion (atomicity). Exiting flushes to main memory, entering invalidates the local copy (visibility). Barriers around the block prevent reordering out of it.

The most common mistake here is synchronizing the writer but not the reader. Both sides need the same lock, or there’s no happens-before edge between them.

ReentrantLock gives you the same memory guarantees, plus tryLock, timeouts, fairness, and multiple conditions.

4. final — safe publication for free

public class Config {
    private final String url;
    public Config(String url) {
        this.url = url;   // guaranteed visible after construction
    }
}

The JMM guarantees that any thread that sees a properly constructed object sees the correct values of its final fields — no synchronization required. This is why immutable objects are the easiest possible concurrency strategy.

The catch is “properly constructed.” If the constructor leaks this (registering a listener, starting a thread), the guarantee is void.

5. Atomics and VarHandle

java.util.concurrent.atomic gives you lock-free compound operations built on CPU compare-and-swap instructions:

private final AtomicInteger count = new AtomicInteger();
public void increment() {
    count.incrementAndGet();   // atomic, visible, ordered
}

Under contention LongAdder usually beats AtomicLong — it spreads writes across cells and sums them on read, avoiding cache-line contention. VarHandle (Java 9+) is the modern, supported way to reach for finer-grained modes like getAcquire / setRelease when you know exactly what you’re doing.

Three bugs this explains

Unsafe publication. Building an object and handing the reference to another thread without a happens-before edge. The other thread can see a non-null reference to a half-initialized object — fields still at their defaults. Publish through a volatile field, a lock, a final field, or a concurrent collection.

Double-checked locking without volatile. The infamous one:

private static volatile Singleton instance;   // volatile is mandatory
public static Singleton getInstance() {
    if (instance == null) {
        synchronized (Singleton.class) {
            if (instance == null) {
                instance = new Singleton();
            }
        }
    }
    return instance;
}

Drop the volatile and it breaks, because instance = new Singleton() is three steps — allocate, construct, assign — and the last two can be reordered. Another thread can grab a reference to an object whose constructor hasn’t finished.

Check-then-act. if (!map.containsKey(k)) map.put(k, v); is two atomic operations, not one atomic operation. Use putIfAbsent or computeIfAbsent.

The practical cheat sheet

• Shared and mutable? It needs synchronization. Every time.
• One thread writes a flag, others read it? volatile.
• Compound operation on one variable? An atomic class.
• Compound operation across several variables? synchronized or ReentrantLock.
• Can you make it immutable instead? Do that. final fields, no setters, no shared mutation, no memory model to reason about.
• Reading is also access. Unsynchronized reads of synchronized state are still broken.
• “It works on my machine” proves nothing. x86 has a strong memory model that hides many of these bugs. ARM (Apple Silicon, Graviton) doesn’t. Code that ran fine for years on Intel servers can fail the week you move it to ARM instances.

Wrapping up

The runtime areas — heap, stacks, metaspace — tell you where your data sits. The Java Memory Model tells you when one thread’s writes become visible to another, and it comes down to a handful of pieces: happens-before as the underlying rule, and volatile, synchronized, final, and the atomics as the tools that create it.

You don’t need to memorize the specification. You need one habit: whenever two threads touch the same data, ask what creates the happens-before edge between them. If you can’t name it, it isn’t there.

If this was useful, the natural next step is Brian Goetz’s Java Concurrency in Practice — still the best treatment of this material, twenty years on.

The symptom

You have a table that stores user state. Something happens — an event arrives — and your code does the obvious thing:

BEGIN;
DELETE FROM user_state WHERE user_id = $1;
INSERT INTO user_state (user_id, state, score, updated_at) VALUES ($1, $2, $3, now());
COMMIT;

It’s clean. It’s idempotent. It doesn’t require you to think about which columns changed. For a long time it works fine.

Then event volume grows, and PostgreSQL’s CPU starts climbing in a way that nothing in your query plans explains. Your queries are still index scans. Your pg_stat_statements top entries look reasonable. But CPU is pinned, replication lag creeps up, and every so often the whole database gets slow for a few minutes and then recovers on its own.

If you look at pg_stat_activity during one of those windows, you’ll usually find the culprit sitting right there:

autovacuum: VACUUM public.user_state

That’s the story. But “autovacuum is expensive” is not an explanation — it’s a restatement of the symptom. To actually fix this you need to understand what a DELETE followed by an INSERT does to the physical 8 KB pages on disk, and why that specific shape of write is close to the worst case for PostgreSQL’s storage engine.

Let’s go all the way down.

Part 1: PostgreSQL never modifies a row in place

This is the single fact that everything else follows from.

PostgreSQL implements MVCC (Multi-Version Concurrency Control) by keeping multiple physical versions of every row and letting each transaction see the version appropriate to its snapshot. There is no undo log (unlike Oracle or MySQL’s InnoDB, which keep old versions in a separate rollback segment). In PostgreSQL, old row versions live in the table itself, mixed in with the live ones.

Concretely:

• INSERT writes a new tuple and stamps it with xmin = the inserting transaction ID.
• DELETE does not remove anything. It stamps the existing tuple with xmax = the deleting transaction ID. The bytes stay exactly where they were.
• UPDATE is internally a DELETE + INSERT: it stamps xmax on the old tuple and writes a brand-new tuple somewhere.

So a “deleted” row is still fully present on disk, occupying its bytes, until some later process comes along and physically reclaims it. That process is vacuum.

Here’s this happening for real. A 100-row table, then delete 50 rows and insert 50 new ones:

CREATE EXTENSION pageinspect;
CREATE TABLE demo (id int PRIMARY KEY, state text);
INSERT INTO demo SELECT g, 'active' FROM generate_series(1,100) g;
-- delete half, re-insert half (the pattern from our app)
DELETE FROM demo WHERE id <= 50;
INSERT INTO demo SELECT g, 'active' FROM generate_series(1,50) g;
SELECT lp, lp_flags, lp_len, t_xmin, t_xmax
FROM heap_page_items(get_raw_page('demo', 0))
WHERE lp BETWEEN 48 AND 54;

 lp | lp_flags | lp_len | t_xmin | t_xmax
----+----------+--------+--------+--------
 48 |        1 |     35 |    801 |    802   <- deleted, still on the page
 49 |        1 |     35 |    801 |    802   <- deleted, still on the page
 50 |        1 |     35 |    801 |    802   <- deleted, still on the page
 51 |        1 |     35 |    801 |      0   <- live
 52 |        1 |     35 |    801 |      0   <- live
 53 |        1 |     35 |    801 |      0   <- live
 54 |        1 |     35 |    801 |      0   <- live

The deleted tuples have t_xmax = 802 (the transaction that deleted them) and are still LP_NORMAL (lp_flags = 1). They occupy real space. The table now has 150 line pointers where the logical row count is 100.

SELECT tuple_count, dead_tuple_count, dead_tuple_percent, free_percent
FROM pgstattuple('demo');

 tuple_count | dead_tuple_count | dead_tuple_percent | free_percent
-------------+------------------+--------------------+--------------
         100 |               50 |              21.36 |        19.09

21% of that table is garbage. And that’s after one round.

Part 2: Anatomy of an 8 KB page

PostgreSQL stores everything in fixed-size blocks — 8 KB by default (SHOW block_size). Understanding the layout of one page is the key to understanding why churn is expensive.

Two structures grow toward each other from opposite ends: the line pointer array from the top, the tuple data from the bottom. The page is full when they meet.

The line pointer (also called an item identifier, ItemIdData) is a tiny 4-byte struct that says “the tuple at slot N lives at byte offset X and is Y bytes long.” It has four possible states, and you’ll see all of them in this article:

The tuple header (HeapTupleHeader, 23 bytes padded to 24) is where MVCC lives:

That’s 24 bytes of overhead per row version, plus 4 bytes for the line pointer. For a table storing a user ID, a short state string, and a timestamp, the MVCC bookkeeping can easily be 40% of the row’s on-disk size.

Now here’s the critical bit for our problem:

A page’s free space is only usable for a new tuple if it is contiguous and the page has a free line pointer or room to add one. Dead tuples fragment the page. Only vacuum can defragment it.

Part 3: Why DELETE + INSERT is the worst shape

Compare what the two write patterns do to a page.

A DELETE + INSERT can never be a HOT update. HOT (Heap-Only Tuple) is an optimization that only applies to UPDATE, and only when two conditions hold:

1. No indexed column changed.
2. The new version fits on the same page as the old one.

When both hold, PostgreSQL writes the new version onto the same page and links it via t_ctid, leaving the index entries alone. That’s an enormous saving: index maintenance is usually the dominant cost of a write, because B-tree inserts are random-access, and every index on the table pays it.

By splitting your logical update into two statements, you’ve told PostgreSQL: this is a brand-new row that has nothing to do with the old one. It has no choice but to write full index entries into every index, and to leave a dead tuple behind that vacuum must chase down.

You can see HOT working with pageinspect. In a table where updates are HOT-eligible, the page fills up with redirect pointers:

SELECT lp, lp_flags, lp_len, t_ctid
FROM heap_page_items(get_raw_page('user_state_hot', 0)) LIMIT 5;

 lp | lp_flags | lp_len | t_ctid
----+----------+--------+--------
  1 |        2 |      0 |          <- LP_REDIRECT
  2 |        2 |      0 |          <- LP_REDIRECT
  3 |        2 |      0 |          <- LP_REDIRECT
  4 |        2 |      0 |          <- LP_REDIRECT
  5 |        2 |      0 |          <- LP_REDIRECT

Those redirect pointers are the physical trace of updates that cost zero index writes.

Part 4: What vacuum actually does

Vacuum is not one operation. A standard (non-full) VACUUM of a table with indexes runs in phases, and understanding the phases explains where the CPU goes.

Phase 1 — Heap scan

Vacuum scans the table’s pages (skipping ones the visibility map marks all-visible, which is the main reason vacuum on an append-only table is cheap and vacuum on a churned table is not). For each page it:

• Prunes HOT chains and marks reclaimable tuples.
• Collects the TIDs (page number, line pointer) of dead tuples into memory.
• Sets LP_DEAD on their line pointers.

The dead TID list is bounded by maintenance_work_mem (or autovacuum_work_mem). In PostgreSQL 17+ this list is stored in an adaptive radix tree (TidStore) rather than a flat array, which cut memory usage dramatically and removed the old 1 GB cap — a genuinely large improvement for high-churn tables.

Phase 2 — Index vacuum

This is the expensive part. For every index on the table, vacuum performs a full scan (a btbulkdelete) and removes every index entry pointing at a dead TID.

Note what that means: the cost is per index, and it’s a full index scan, not a lookup. A table with five indexes pays five full index scans per vacuum cycle. This is why “just add an index” decisions compound badly on high-churn tables.

Phase 3 — Heap vacuum

Vacuum revisits the pages and turns LP_DEAD line pointers into LP_UNUSED, then defragments the page: it slides the remaining tuples together so the free space becomes one contiguous block. Now that space is genuinely reusable.

If the dead-TID list overflowed maintenance_work_mem in phase 1, vacuum has to loop: scan more heap → vacuum all indexes again → vacuum heap again. A single vacuum of a big churned table can do the full index pass multiple times. This is the classic “why has autovacuum been running for six hours” answer.

Phase 4 — Freezing

Separately, vacuum replaces old xmin values with a frozen marker so transaction IDs can wrap around safely. Anti-wraparound vacuums are non-cancelable and will run whether you like it or not. PostgreSQL 18 added eager freezing (vacuum_max_eager_freeze_failure_rate, default 0.03) so that normal vacuums opportunistically freeze pages, spreading this cost out instead of saving it up for one giant emergency vacuum.

Phase 5 — Truncation

If there are empty pages at the end of the table, vacuum tries to return them to the OS. This requires an ACCESS EXCLUSIVE lock (briefly). Note the word end: vacuum almost never shrinks a bloated table, because the free space is scattered throughout, not conveniently at the tail. It makes the space reusable by PostgreSQL, not available to your filesystem.

Here’s the before/after on our demo table:

VACUUM demo;
SELECT lp_flags, count(*) 
FROM heap_page_items(get_raw_page('demo',0)) GROUP BY 1;

Free space on the page went from 19.09% → 43.51%. That’s the work vacuum did. Now multiply it by every page you churned.

Part 5: The amplification math

Here is the part people underestimate. One logical “user state changed” event does not cause one write. It causes, at minimum:

Steps 7–13 are invisible in your application metrics. Your app measures step 1–6 latency and reports “the insert took 2 ms.” The other two-thirds of the cost lands on a background worker, asynchronously, in a burst, on the same CPU.

That’s the answer to “why does CPU go up but my queries look fine.”

Point 5b: the full-page-image trap

After each checkpoint, the first write to any page triggers a full 8 KB page image written into WAL (full_page_writes, on by default, and required for crash safety on most filesystems). If your churn touches many distinct pages, you pay a lot of FPIs. If your churn is concentrated on a few pages, you pay far fewer. Delete+insert scatters writes across pages — the new tuple often lands in a different page from the one you dirtied with the xmax write. Two dirty pages instead of one, two FPI candidates instead of one.

Point 5c: the read-side cost nobody mentions

Dead tuples also make reads slower and CPU-hungrier:

• An index scan follows an index entry to the heap, reads the tuple, checks visibility, discovers it’s dead, and discards it. That’s a wasted buffer access plus a visibility check. You can see this directly: EXPLAIN (ANALYZE, BUFFERS) will show a rising Buffers: shared hit count for a query returning the same number of rows.
• Bloated tables have worse cache locality. The same 100k live rows spread over 2× the pages means 2× the shared buffers to hold them, which evicts other things.
• On a read, PostgreSQL may perform opportunistic HOT pruning on the page — real CPU work done inside your SELECT.

Part 6: Measurements

I ran a controlled comparison: 100,000 rows, ten rounds of modifying 20,000 of them, with an explicit VACUUM after each round so we’re looking at steady state, not a runaway. Three variants:

Read those rows carefully, because the middle one is the trap.

Switching from **DELETE+**INSERT to UPDATE bought essentially nothing on its own. The second variant still generated more WAL than the first. Why? Because the UPDATE touched updated_at, and updated_at had an index on it. An UPDATE that modifies any indexed column is not HOT-eligible, so it does exactly what delete+insert does: new heap tuple, new entries in every index, dead tuple left behind.

The third variant — same UPDATE, but with no index on the mutated columns and fillfactor lowered to 80 — generated less than half the WAL and made 73% of its updates HOT.

The cost of that win is visible too: the HOT table starts at 1042 heap pages instead of 834, because fillfactor=80 deliberately leaves 20% of every page empty. That empty space is what new row versions land in. You are trading disk space for CPU and I/O, which on modern hardware is almost always the right trade.

(These are synthetic single-client numbers on PostgreSQL 16 — treat the ratios as directional, not as a benchmark of your workload.)

Part 7: Fixing it, in order of leverage

7.1 Don’t write when nothing changed

The cheapest write is the one that doesn’t happen. A shocking amount of churn in “collect user state” systems is rewriting a row to the exact same value because an event arrived that didn’t actually change anything.

INSERT INTO user_state (user_id, state, score, updated_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (user_id) DO UPDATE
SET state      = EXCLUDED.state,
    score      = EXCLUDED.score,
    updated_at = now()
WHERE user_state.state  IS DISTINCT FROM EXCLUDED.state
   OR user_state.score  IS DISTINCT FROM EXCLUDED.score;

That trailing WHERE on the DO UPDATE clause is the whole trick: if nothing changed, PostgreSQL writes nothing — no new tuple, no dead tuple, no index entry, no WAL. In event-driven systems where the same state gets recomputed repeatedly, this alone often removes 50–90% of write volume.

Measure how much you’d save before you build anything else:

-- how many of your writes are no-ops?
SELECT count(*) FILTER (WHERE state = $2 AND score = $3) AS would_be_noop,
       count(*) AS total
FROM user_state WHERE user_id = ANY($1);

7.2 Make your updates HOT-eligible

Two requirements, both actionable:

(a) Don’t index the columns that change. Audit every index against your write path:

SELECT i.relname AS index_name,
       array_agg(a.attname ORDER BY x.n) AS indexed_columns,
       s.idx_scan AS times_used
FROM pg_index ix
JOIN pg_class i ON i.oid = ix.indexrelid
JOIN pg_class t ON t.oid = ix.indrelid
JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS x(attnum, n) ON true
JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = x.attnum
LEFT JOIN pg_stat_user_indexes s ON s.indexrelid = i.oid
WHERE t.relname = 'user_state'
GROUP BY i.relname, s.idx_scan
ORDER BY s.idx_scan NULLS FIRST;

An index on updated_at is the single most common HOT-killer in exactly this kind of table. Ask whether you truly need it, or whether a partial index, a coarser index (date_trunc(‘hour’, updated_at) doesn’t help — but a separate low-churn audit table might), or no index at all would do.

Unused indexes are pure loss: they cost writes and vacuum passes and return nothing. Drop them.

(b) Leave room on the page.

ALTER TABLE user_state SET (fillfactor = 80);
-- fillfactor only applies to newly written pages;
-- rewrite the table to apply it to existing data:
VACUUM FULL user_state;   -- takes ACCESS EXCLUSIVE lock
-- or, online:
-- pg_repack -t user_state

For a heavily-updated table, 70–85 is a reasonable range. Below ~70 you’re wasting a lot of buffer cache on empty space.

Then confirm it’s working:

SELECT relname,
       n_tup_upd,
       n_tup_hot_upd,
       round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd,0), 1) AS hot_pct
FROM pg_stat_user_tables
WHERE relname = 'user_state';

If hot_pct is near zero on an update-heavy table, that is your bug. Aim for 80%+.

7.3 Tune autovacuum for this table, not globally

The defaults are tuned for a small table on a small machine. autovacuum_vacuum_scale_factor = 0.2 means vacuum waits until 20% of the table is dead before acting. On a 500-million-row table that’s 100 million dead tuples before anything happens — and then one enormous, disruptive vacuum.

You want the opposite: vacuum more often, doing less each time.

ALTER TABLE user_state SET (
  autovacuum_vacuum_scale_factor  = 0.01,   -- act at 1% dead, not 20%
  autovacuum_vacuum_threshold     = 1000,
  autovacuum_analyze_scale_factor = 0.02,
  autovacuum_vacuum_cost_delay    = 0,      -- don't throttle this table
  autovacuum_vacuum_cost_limit    = 3000
);

And globally, if your hardware is modern (NVMe, many cores), the shipped throttle is far too conservative:

autovacuum_max_workers      = 6          # default 3
autovacuum_naptime          = 15s        # default 1min
autovacuum_vacuum_cost_delay = 2ms       # default 2ms; consider 0 on fast storage
vacuum_cost_limit           = 2000       # default 200
maintenance_work_mem        = 2GB        # gives vacuum a big dead-TID store
autovacuum_work_mem         = 1GB

The cost-delay mechanism is the one most often left at a value that makes vacuum unable to keep up: vacuum accumulates “cost” (1 per buffer hit, 2 per miss, 20 per dirtied page) and sleeps cost_delay whenever it exceeds cost_limit. With the defaults, vacuum is deliberately slowed to something like a few MB/s. On a table generating hundreds of MB of dead tuples per minute, it will never catch up, and dead tuples accumulate forever.

PostgreSQL 18 also added autovacuum_vacuum_max_threshold (default 100,000,000), which caps the scale-factor calculation so very large tables can’t defer vacuum indefinitely. It helps, but a per-table scale factor is still better.

7.4 Reconsider the data model

If your state genuinely is “replace the whole thing on every event,” ask whether it belongs in a row-per-user table at all:

• Collapse many rows into one. If you’re storing N state rows per user and replacing them all, one row with a jsonb column can turn N deletes + N inserts into one update — and if jsonb is under the TOAST threshold and no indexed column changes, that update can be HOT. (Careful: a large jsonb that gets TOASTed is a whole separate write amplification problem. This works when the document is small, roughly under 2 KB.)
• Append-only + a materialized “current” view. Write events to a partitioned append-only table (no updates, no deletes, so almost no vacuum work — just insert-triggered vacuums for freezing), and maintain the current state separately, or compute it with a DISTINCT ON query over a well-indexed event log. Append-only tables are the cheapest thing PostgreSQL does.
• Move the hottest state out of PostgreSQL. If a piece of state changes many times per second per user and you only ever need the latest value, Redis or a dedicated KV store is the honest answer. Not everything needs to be transactional and durable.

Part 8: The hard question — updating billions of rows

This is the case where all the usual advice breaks, so it deserves its own treatment.

Why the naive approach fails, precisely

UPDATE huge_table SET status = 'migrated' WHERE status = 'pending';
-- 2,000,000,000 rows

Four separate disasters:

1. Table size doubles. Every updated row writes a new version while the old one stays. A 2 TB table becomes 4 TB mid-statement. Then vacuum makes the space reusable but does not return it to the OS. You now permanently need 4 TB of disk for 2 TB of data.

2. It generates terabytes of WAL in one burst. Replicas fall behind, archiving falls behind, and if you’re on a cloud provider you may fill the WAL volume and take the primary down.

3. It holds a single transaction open for hours — and this is the one that hurts most. Vacuum can only remove tuples older than the oldest running transaction’s snapshot (the xmin horizon). A long transaction pins that horizon. So while your giant UPDATE runs, vacuum cannot clean up dead tuples anywhere in the entire database — not just this table. Every other table quietly bloats. This is how one migration takes down a whole cluster.

4. If it fails at 95%, all the work is thrown away — but the dead tuples it created are not. You get the bloat with none of the benefit.

You can watch the horizon problem happen:

SELECT pid, age(backend_xid) AS xid_age, state,
       now() - xact_start AS duration, left(query, 60)
FROM pg_stat_activity
WHERE backend_xid IS NOT NULL
ORDER BY xact_start
LIMIT 5;

Also check for the other two horizon-pinners people forget: replication slots and prepared transactions.

SELECT slot_name, active, age(xmin) AS xmin_age,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;
SELECT gid, prepared, age(transaction) FROM pg_prepared_xacts;

An inactive replication slot will pin the xmin horizon forever and no amount of vacuum tuning will save you.

Strategy A: Batched backfill with keyset pagination

The workhorse. Small transactions, each one committing so the xmin horizon advances and vacuum can keep pace.

-- Track progress in a real table so the job is resumable
CREATE TABLE IF NOT EXISTS backfill_progress (
  job_name text PRIMARY KEY,
  last_id  bigint NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now()
);

-- One batch. Run this in a loop from your application or a scheduler.
WITH bounds AS (
  SELECT last_id FROM backfill_progress WHERE job_name = 'status_migration'
),
batch AS (
  SELECT id FROM huge_table
  WHERE id > (SELECT last_id FROM bounds)
  ORDER BY id
  LIMIT 5000
  FOR UPDATE SKIP LOCKED
),
upd AS (
  UPDATE huge_table t
  SET status = 'migrated'
  FROM batch b
  WHERE t.id = b.id
    AND t.status IS DISTINCT FROM 'migrated'   -- skip no-op writes
  RETURNING t.id
)
UPDATE backfill_progress
SET last_id = COALESCE((SELECT max(id) FROM batch), last_id),
    updated_at = now()
WHERE job_name = 'status_migration'
RETURNING last_id, (SELECT count(*) FROM upd) AS rows_updated;

Rules that make the difference between this working and this being a slow disaster:

• Keyset pagination, never OFFSET. OFFSET 10000000 re-scans ten million rows every batch. WHERE id > $last uses the index.
• Batch size 1,000–10,000 rows. Big enough to amortize overhead, small enough that each transaction finishes in well under a second.
• AND col IS DISTINCT FROM $new so re-running a batch is free and no-ops don’t write.
• Throttle deliberately. Sleep 50–200 ms between batches. You are racing autovacuum; give it room to win. Adaptive throttling — sleep longer when replication lag or dead-tuple count rises — is worth the extra code:

-- feed this into your loop's sleep decision
SELECT (SELECT n_dead_tup FROM pg_stat_user_tables WHERE relname='huge_table') AS dead,
       (SELECT coalesce(max(pg_wal_lsn_diff(sent_lsn, replay_lsn)), 0)
        FROM pg_stat_replication) AS replica_lag_bytes;

• Run it during a low-traffic window if you have one, but design it to be safe at any time.

Expected duration is the thing to be honest about: 2 billion rows at 5,000 rows/batch with a 100 ms sleep is 400,000 batches ≈ 11+ hours minimum, realistically a few days. That’s fine. Plan for days, make it resumable, make it observable.

Strategy B: Partitioning + partition swap

If your data is naturally partitionable (by time, tenant, hash of user ID), this changes the game completely, because you can rebuild a partition offline and swap it in atomically.

-- Assume: huge_table PARTITION BY RANGE (created_at)
--         with partition huge_table_2026_03
-- 1. Build the new version at full speed, outside the live table.
CREATE TABLE huge_table_2026_03_new (LIKE huge_table INCLUDING ALL);
INSERT INTO huge_table_2026_03_new
SELECT id, user_id, 'migrated'::text AS status, payload, created_at
FROM huge_table_2026_03;
-- 2. Indexes AFTER the load - far faster than maintaining them during it.
--    No CONCURRENTLY needed: nothing is reading this table yet.
CREATE INDEX ON huge_table_2026_03_new (user_id);
ANALYZE huge_table_2026_03_new;
-- 3. Atomic swap: milliseconds of ACCESS EXCLUSIVE lock, not hours.
BEGIN;
  ALTER TABLE huge_table DETACH PARTITION huge_table_2026_03;
  ALTER TABLE huge_table ATTACH PARTITION huge_table_2026_03_new
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');
COMMIT;
-- 4. Keep the old one around until you're confident, then drop it.
-- DROP TABLE huge_table_2026_03;

Why this is so much better:

• The bulk INSERT into an empty, unindexed table is the fastest write path PostgreSQL has. No dead tuples are created at all.
• Zero vacuum debt: you never made garbage, so there’s nothing to collect.
• The lock window is milliseconds.
• Rollback is trivial: swap the old partition back.
• You can do partitions one at a time, over weeks, with a rollback point after each.

There’s a caveat worth knowing: ATTACH PARTITION validates that existing rows satisfy the partition constraint, which requires a scan — unless you add a matching CHECK constraint to the new table first, in which case PostgreSQL skips the scan.

ALTER TABLE huge_table_2026_03_new
  ADD CONSTRAINT part_check
  CHECK (created_at >= '2026-03-01' AND created_at < '2026-04-01') NOT VALID;
ALTER TABLE huge_table_2026_03_new VALIDATE CONSTRAINT part_check;
-- now ATTACH is instant

If you have billions of rows and no partitioning, partitioning is probably the actual project — not the update itself. Everything downstream gets easier: DROP PARTITION replaces mass DELETE (instant, and creates zero dead tuples), vacuum works per-partition in parallel, and index sizes stay manageable.

Strategy C: Rebuild and swap the whole table

When the change is uniform and you can’t partition, build a new table and cut over.

For a bloated table with no schema change, pg_repack does this online: it builds a fresh copy, captures concurrent changes via triggers, replays them, and swaps — taking the exclusive lock only for the final instant.

pg_repack -h host -d mydb -t huge_table --jobs 4

For a schema/data change, the same idea by hand with logical replication or triggers; tools like pgroll and Percona’s approach automate it. The pattern is always: new table, bulk load, keep in sync, atomic rename.

Budget for double the disk space. That’s the price of every swap-based strategy.

Strategy D: Don’t update at all

The strategy people skip past. Ask whether the update is necessary:

• Add a new column with a default instead of rewriting values — since PostgreSQL 11, adding a column with a non-volatile default is a metadata-only change that rewrites nothing, no matter how big the table.
• Write a view that computes the new value from the old one, and migrate readers to it.
• Use a generated column if the new value is derivable.
• Let it happen lazily: update rows as they’re naturally touched by the application, with a background job mopping up the long tail over months.

A billion-row update you don’t have to run is strictly better than one you run well.

Part 9: What to actually watch

Four queries. Put them on a dashboard.

1. Is vacuum keeping up?

SELECT relname,
       n_live_tup,
       n_dead_tup,
       round(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_pct,
       last_autovacuum,
       autovacuum_count,
       n_tup_upd,
       n_tup_hot_upd,
       round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd, 0), 1) AS hot_pct
FROM pg_stat_user_tables
WHERE n_dead_tup > 10000
ORDER BY n_dead_tup DESC
LIMIT 20;

Sustained dead_pct above ~10% on a busy table means vacuum is losing. hot_pct near zero on an update-heavy table means you’re paying full index cost on every write.

2. What is vacuum doing right now?

SELECT p.pid, p.relid::regclass AS table, p.phase,
       pg_size_pretty(p.heap_blks_total * 8192::bigint) AS total,
       round(100.0 * p.heap_blks_scanned / NULLIF(p.heap_blks_total,0), 1) AS pct_scanned,
       p.index_vacuum_count,
       now() - a.xact_start AS running_for
FROM pg_stat_progress_vacuum p
JOIN pg_stat_activity a USING (pid);

index_vacuum_count > 1 is the smoking gun for maintenance_work_mem being too small — vacuum had to do the index pass more than once.

3. Is anything pinning the xmin horizon?

SELECT 'backend' AS src, pid::text AS id, age(backend_xmin) AS xmin_age
FROM pg_stat_activity WHERE backend_xmin IS NOT NULL
UNION ALL
SELECT 'slot', slot_name, age(xmin) FROM pg_replication_slots WHERE xmin IS NOT NULL
UNION ALL
SELECT 'prepared_xact', gid, age(transaction) FROM pg_prepared_xacts
ORDER BY xmin_age DESC NULLS LAST
LIMIT 10;

Anything with a large age here is preventing vacuum from cleaning up database-wide.

4. How bloated is it, really?

CREATE EXTENSION IF NOT EXISTS pgstattuple;
-- accurate but does a full scan; use the sampling version on huge tables
SELECT * FROM pgstattuple('user_state');
SELECT * FROM pgstattuple_approx('huge_table');
SELECT * FROM pgstatindex('user_state_user_id_idx');

For indexes, avg_leaf_density below ~50 means the index is worth rebuilding:

REINDEX INDEX CONCURRENTLY user_state_user_id_idx;

The checklist

If you’re in the situation this article opened with, do these in order:

1. Measure first. dead_pct, hot_pct, pg_stat_progress_vacuum, xmin horizon. Don’t tune blind.
2. Kill no-op writes with ON CONFLICT DO UPDATE … WHERE … IS DISTINCT FROM. Usually the biggest single win, and the cheapest to ship.
3. Convert delete+insert to UPDATE — but only after step 4, or it buys you nothing.
4. Make updates HOT-eligible. Drop or move indexes on frequently-changed columns (especially updated_at). Set fillfactor to 80. Verify hot_pct climbs above 80%.
5. Tune autovacuum per-table: scale factor down to 0.01, cost delay to 0, cost limit way up, maintenance_work_mem to gigabytes.
6. Drop unused indexes. Every one is a write tax and an extra full scan per vacuum.
7. Partition if the table is large and the data has a natural key. This makes every future problem easier.
8. For mass updates: batched keyset backfill, or partition rebuild-and-swap. Never one giant statement.

The underlying idea is simple enough to hold in your head: PostgreSQL is a copy-on-write storage engine that trades write amplification for concurrency. Delete+insert opts into the maximum possible amplification — no HOT, full index maintenance, guaranteed dead tuples, guaranteed vacuum work. Understanding what happens on the 8 KB page is what turns “the database is slow” into a list of specific, fixable things.

If this was useful, the two commands worth internalizing are pageinspect’s heap_page_items() and pgstattuple(). Point them at your hottest table. What you find will probably explain your CPU graph.

References

• PostgreSQL 18: Routine Vacuuming
• PostgreSQL 18: Vacuuming configuration parameters
• PostgreSQL 18 Released
• PostgreSQL 18 Vacuuming Improvements Explained (Microsoft)
• autovacuum_vacuum_max_threshold in PostgreSQL 18 (dbi services)
• Faster VACUUM with Adaptive Radix Trees in PostgreSQL 17 (pganalyze)
• VACUUM at the Page Level (boringSQL)

There is a specific kind of bug that only shows up in production, only in October, and only for users in one country. It gets closed as “not reproducible” twice before someone finally sits down with a coffee and reads the column definition.

It is almost always a datetime bug.

The frustrating part is that both Java and PostgreSQL have excellent date/time support. java.time is one of the best-designed APIs in the JDK. PostgreSQL’s temporal types are precise and well documented. The bugs happen at the seam — in the assumptions each side makes about what the other means, and in the two or three words in the type names that mean the opposite of what you’d guess.

This guide walks through both models, then spends real time on the part that actually causes outages: time zones.

Every code output and query result below was executed — Java 21 and PostgreSQL 16 — not written from memory.

Part 1: The Java Side

Ask three questions, get one type

Every java.time class is an answer to some combination of three questions:

1. Do I know the date?
2. Do I know the time of day?
3. Do I know which point on the global timeline this is?

That last one is the important one, and it’s the one people skip. Here’s the map:

The single most useful sentence in this article: LocalDateTime is not a point in time. 2026-08-15T12:00 is twelve o’clock somewhere, and until you say where, it refers to a 26-hour-wide smear across the planet. If you have an Instant and you call .toLocalDateTime(), you have thrown away information and the compiler will not warn you.

Instant is the boring, correct default

Instant now = Instant.now();     // 2026-08-15T10:00:00.123456789Z

An Instant is nanoseconds since the epoch. No zone, no offset, no calendar. When it prints with a Z, that isn’t a stored time zone — it’s the ISO-8601 way of rendering an epoch value.

Anything that happened — a login, a payment, a row insert — is an Instant. Reach for something else only when you have a reason.

ZoneId vs ZoneOffset: not the same thing

This distinction matters more than any other in this article.

ZoneId.of("Europe/London")  // -> java.time.ZoneRegion
ZoneId.of("+01:00")         // -> java.time.ZoneOffset

A ZoneOffset is a fixed number of hours and minutes from UTC. +01:00. That’s the whole thing. It’s a fact about one moment.

A ZoneId (specifically a ZoneRegion, like Europe/London) is a rulebook. It knows that London was +00:00 in January, +01:00 in August, and it knows the exact instants at which it flips — going back decades and forward as far as legislators have declared.

ZoneOffset is a value. ZoneId is a function from instant to offset. Confusing them is the root cause of most scheduling bugs.

OffsetDateTime vs ZonedDateTime

Both identify a real instant. The difference is what survives.

ZonedDateTime nyc = ZonedDateTime.of(2026, 7, 4, 12, 0, 0, 0,
                                     ZoneId.of("America/New_York"));
nyc                    // 2026-07-04T12:00-04:00[America/New_York]
nyc.toOffsetDateTime() // 2026-07-04T12:00-04:00
nyc.toInstant()        // 2026-07-04T16:00:00Z

All three describe the same moment. But the ZonedDateTime still knows it’s in New York, so if you add six months to it, it will correctly land on -05:00. The OffsetDateTime has been flattened — it remembers that the offset was -04:00, not why. Add six months and you get a wrong wall-clock time.

Rule of thumb: ZonedDateTime for calculating, OffsetDateTime for transporting, Instant for storing.

Duration vs Period

LocalDate.of(2026, 1, 31).plusMonths(1)  // 2026-02-28

Duration is exact elapsed time — seconds and nanos. Period is calendar time — years, months, days. “One month” is not a fixed number of seconds, and plusMonths clamps rather than overflowing. Both behaviors are correct; mixing them up is not.

This distinction resurfaces the moment DST enters the picture, and we’ll come back to it.

A word on the old API

java.util.Date is a mislabeled Instant (it has no time zone despite what toString() implies). Calendar is mutable and lenient. java.sql.Timestamp extends java.util.Date and adds nanoseconds while breaking the equals contract.

If you’re on a legacy codebase, convert at the boundary and never let them into your domain:

Instant i = oldDate.toInstant();
Date    d = Date.from(instant);

Part 2: The PostgreSQL Side

timestamptz does not store a time zone

This is the single biggest source of confusion, and it comes straight from the type name. TIMESTAMP WITH TIME ZONE is an 8-byte value. So is TIMESTAMP WITHOUT TIME ZONE. There is no room in there for a zone, and PostgreSQL doesn’t try:

SELECT pg_column_size('2026-08-15 12:00:00'::timestamp)     AS ts_bytes,     
       --  8
       pg_column_size('2026-08-15 12:00:00+02'::timestamptz) AS tstz_bytes,  
       --  8
       pg_column_size('12:00:00+02'::timetz)                 AS timetz_bytes; 
       -- 12

What timestamptz actually does is convert to UTC on write and convert back on read. The offset you supplied on input is used and then discarded. From the official docs:

In either case, the value is stored internally as UTC, and the originally stated or assumed time zone is not retained.

A better mental model: timestamptz is Instant. timestamp is LocalDateTime. The names lie; the semantics don’t.

The demo that makes it click

Insert the same two literals into both column types, with the session zone set to London:

SET TimeZone = 'Europe/London';
CREATE TABLE t (id int, ts timestamp, tstz timestamptz);
INSERT INTO t VALUES (1, '2026-08-15 12:00:00',    '2026-08-15 12:00:00');
INSERT INTO t VALUES (2, '2026-08-15 12:00:00+00', '2026-08-15 12:00:00+00');
SELECT * FROM t;

 id |         ts          |          tstz
----+---------------------+------------------------
  1 | 2026-08-15 12:00:00 | 2026-08-15 12:00:00+01
  2 | 2026-08-15 12:00:00 | 2026-08-15 13:00:00+01

Two things happened. Row 1’s tstz had no offset, so PostgreSQL assumed the session zone. Row 2’s explicit +00 was honoured and re-rendered as 13:00+01. Meanwhile the ts column ignored the offset entirely and stored the digits.

Now reconnect from New York and run the identical SELECT:

 id |         ts          |          tstz
----+---------------------+------------------------
  1 | 2026-08-15 12:00:00 | 2026-08-15 07:00:00-04
  2 | 2026-08-15 12:00:00 | 2026-08-15 08:00:00-04

The tstz column re-renders per client. The ts column is frozen. Neither row changed on disk.

That last point deserves emphasis, because it’s where the panic starts: when a developer sees a different value than QA saw, the instinct is “the data is corrupted.” It isn’t. Two clients with different TimeZone settings are looking at the same UTC value through different windows.

AT TIME ZONE runs both ways

AT TIME ZONE is the one operator you need, and its behavior flips depending on the input type:

-- timestamptz -> timestamp: "what does the clock read there?"
SELECT '2026-08-15 12:00:00+00'::timestamptz AT TIME ZONE 'Asia/Tokyo';
-- 2026-08-15 21:00:00   (type: timestamp without time zone)

-- timestamp -> timestamptz: "this wall clock reading WAS in Tokyo"
SELECT '2026-08-15 12:00:00'::timestamp AT TIME ZONE 'Asia/Tokyo';
-- 2026-08-15 03:00:00+00   (type: timestamp with time zone)

One removes zone context, the other attaches it. Read it as “interpret this value in that zone.” The function form timezone(‘Asia/Tokyo’, value) is identical.

The rest of the family

• date — 4 bytes, no zone, no ambiguity. Great for things that are genuinely dates.
• time — time of day, no zone.
• timetz — 12 bytes, and PostgreSQL itself recommends against it. A time of day with an offset but no date can’t resolve DST, so it’s not meaningfully anchored. It won’t even accept a zone name:

SELECT '12:00:00 Europe/London'::timetz; -- ERROR: invalid input syntax for type time with time zone

• interval — 16 bytes storing months, days, and microseconds separately, which is exactly why it can be DST-aware.

now() vs localtimestamp

SELECT pg_typeof(now()),            -- timestamp with time zone
       pg_typeof(current_timestamp), -- timestamp with time zone
       pg_typeof(localtimestamp),    -- timestamp without time zone
       pg_typeof(current_date);      -- date

now() and current_timestamp are timestamptz and are safe. localtimestamp and current_date are evaluated in the session zone, which means a DEFAULT current_date column produces different answers for different clients. Be deliberate.

So which do you pick?

The short version: use timestamptz for essentially everything. Reach for timestamp only when you genuinely mean a floating wall-clock time that must not shift — “the store opens at 09:00 local, whatever local means at that branch” — and store the zone ID beside it.

Part 3: Time Zones, Properly

This is where the real complexity lives, so let’s slow down.

Three things called “time zone”

1. An offset — +01:00. A number.
2. An IANA zone ID — Europe/London. A named set of historical and future rules. PostgreSQL ships ~499 of these in pg_timezone_names; Java reads the same tzdata.
3. An abbreviation — EST, BST, IST. Ambiguous, unversioned, and a trap.

That third one is genuinely dangerous:

SELECT '2026-08-15 12:00:00 EST'::timestamptz,               -- 2026-08-15 17:00:00+00
       '2026-08-15 12:00:00 America/New_York'::timestamptz;  -- 2026-08-15 16:00:00+00

One hour apart. EST is a fixed -05:00 in PostgreSQL — it does not mean “Eastern time,” it means “Eastern Standard Time, always.” In August, New York is on -04:00. If you accept zone abbreviations from user input, you have already shipped a bug. (IST is worse — it can mean India, Ireland, or Israel depending on who you ask.)

Always use full IANA names.

Offsets aren’t zones either. Nepal is +05:45; the Chatham Islands are +12:45:

ZonedDateTime.of(2026,8,15,12,0,0,0, ZoneId.of("Asia/Kathmandu"))
// 2026-08-15T12:00+05:45[Asia/Kathmandu]

If your API contract says “offset in whole hours,” it’s wrong.

Where PostgreSQL’s session zone comes from

The TimeZone GUC is a per-connection setting. Its value is decided by, in order: an explicit SET TimeZone, the connection’s startup parameters, the postgresql.conf value, and finally the server’s OS zone.

That last fallback is why the same query returns different results on a developer laptop and in CI. And with connection pooling, SET TimeZone on a borrowed connection leaks to whoever gets it next.

Pin it. Set timezone = ‘UTC’ in postgresql.conf, and add it to the JDBC URL so it survives pool reuse:

jdbc:postgresql://host:5432/db?options=-c%20TimeZone%3DUTC

Then make the client responsible for rendering. Formatting a timestamp for a human is a presentation concern, not a database one.

The spring-forward gap

On 29 March 2026, London jumps from 01:00 to 02:00. The local times between them do not exist.

Ask Java for one anyway:

LocalDateTime gap = LocalDateTime.of(2026, 3, 29, 1, 30);
gap.atZone(ZoneId.of("Europe/London"));
// 2026-03-29T02:30+01:00[Europe/London]
ZoneId.of("Europe/London").getRules().getTransition(gap);
// Transition[Gap at 2026-03-29T01:00Z to +01:00]

Java doesn’t throw. It silently pushes the value forward by the length of the gap. PostgreSQL does the same thing:

SET TimeZone='Europe/London';
SELECT '2026-03-29 01:30:00'::timestamptz;
-- 2026-03-29 02:30:00+01

Both are “correct” — but neither told you. If a user sets a 01:30 alarm on that date, you should be validating with getTransition() and telling them, not letting the platform quietly reschedule.

The fall-back overlap — where Java and PostgreSQL disagree

On 25 October 2026, London goes from 02:00 back to 01:00. Local times between 01:00 and 02:00 happen twice, one hour apart.

Java’s rules expose both:

ZoneId london = ZoneId.of("Europe/London");
LocalDateTime overlap = LocalDateTime.of(2026, 10, 25, 1, 30);
london.getRules().getValidOffsets(overlap);
// [+01:00, Z]
ZonedDateTime earlier = overlap.atZone(london);
// 2026-10-25T01:30+01:00[Europe/London]  instant = 2026-10-25T00:30:00Z
ZonedDateTime later = earlier.withLaterOffsetAtOverlap();
// 2026-10-25T01:30Z[Europe/London]       instant = 2026-10-25T01:30:00Z

Java’s default for an ambiguous local time is the earlier offset — 00:30Z.

Now ask PostgreSQL the same question:

SELECT '2026-10-25 01:30:00'::timestamp AT TIME ZONE 'Europe/London';
-- 2026-10-25 01:30:00+00

PostgreSQL picked 01:30Z — the later one.

Same input, same tzdata, one hour apart. If your application resolves an ambiguous local time in Java and your reporting job resolves it in SQL, they will disagree for one hour, twice a year, in every DST zone.

There is no clever fix. The lesson is: never let a LocalDateTime cross the boundary and get resolved on the other side. Resolve to an Instant in exactly one place, with explicit overlap handling, and send the instant.

DST arithmetic: plusDays is not plusHours(24)

ZonedDateTime start = ZonedDateTime.of(2026, 3, 28, 12, 0, 0, 0, london);
// 2026-03-28T12:00Z[Europe/London]
start.plusDays(1);                 // 2026-03-29T12:00+01:00  (23 real hours)
start.plus(Duration.ofDays(1));    // 2026-03-29T13:00+01:00  (24 real hours)
start.plusHours(24);               // 2026-03-29T13:00+01:00

plusDays preserves the wall clock. Duration preserves elapsed time. PostgreSQL draws the same line, using the same value:

SET TimeZone='Europe/London';
SELECT '2026-03-28 12:00:00+00'::timestamptz + interval '1 day',   -- 2026-03-29 12:00:00+01
       '2026-03-28 12:00:00+00'::timestamptz + interval '24 hours'; -- 2026-03-29 13:00:00+01

This is one place the two platforms agree beautifully — interval ‘1 day’ and plusDays(1) mean the same thing, and interval ‘24 hours’ and Duration.ofDays(1) mean the same thing. “Every day at 9am” is plusDays / interval ‘1 day’. “Every 24 hours” is Duration / interval ‘24 hours’. Pick the one the product actually means.

Future events: an instant is not enough

ZoneId ny = ZoneId.of("America/New_York");
ZonedDateTime.of(2027, 1, 15, 9, 0, 0, 0, ny).withZoneSameInstant(UTC); // 2027-01-15T14:00Z
ZonedDateTime.of(2027, 7, 15, 9, 0, 0, 0, ny).withZoneSameInstant(UTC); // 2027-07-15T13:00Z

“9am in New York” is a different UTC instant depending on the date. So if a user schedules a recurring 9am standup and a government later changes DST rules — which happens several times a year somewhere — the stored instant becomes wrong.

For past events, store the instant. That’s history; it can’t change.

For future user-scheduled events, store three things:

CREATE TABLE meetings (
  id            bigserial PRIMARY KEY,
  local_start   timestamp   NOT NULL,  -- 2027-01-15 09:00
  zone_id       text        NOT NULL,  -- 'America/New_York'
  computed_utc  timestamptz NOT NULL   -- cache; recompute on tzdata update
);

The first two are the truth. The third is a derived index for querying, refreshed when tzdata changes.

tzdata is a moving target

Java bundles a copy of the IANA database (2025b on the JDK I ran this on); PostgreSQL bundles its own; your OS has a third. They drift. When Java and PostgreSQL are on different tzdata versions, they will resolve recently-changed zones differently.

Check what you have:

ZoneRulesProvider.getVersions("Europe/London").lastKey();  // 2025b

SELECT name, utc_offset, is_dst FROM pg_timezone_names WHERE name = 'Europe/London';

Pin tzdata in your base images, and treat a tzdata bump as a deployment that requires recomputing cached future instants.

Part 4: Wiring Java to PostgreSQL

The mapping table

Per the pgJDBC documentation:

Note the omission, in the driver’s own words:

ZonedDateTime, Instant and OffsetTime / TIME WITH TIME ZONE are not supported.

So don’t hand the driver an Instant and hope — depending on the driver version you’ll either get a Can’t infer the SQL type to use for an instance of java.time.Instant exception or undocumented behavior. Convert explicitly at the boundary instead:

// write
ps.setObject(1, instant.atOffset(ZoneOffset.UTC));
// read
Instant i = rs.getObject("occurred_at", OffsetDateTime.class).toInstant();

Note that reading a timestamptz gives you an OffsetDateTime normalized to UTC — the driver’s offset is always Z, because that’s what the backend holds. The offset carries no information. Convert straight to Instant and don’t be tempted to read meaning into it.

Avoid setTimestamp / getTimestamp

The java.sql.Timestamp overloads convert using the JVM default time zone unless you pass an explicit Calendar. That makes your persistence layer’s correctness depend on the container’s TZ environment variable. Use the java.time setObject/getObject methods instead — they’re unambiguous.

If you must keep the JVM zone from mattering at all, set it explicitly at startup:

-Duser.timezone=UTC

Belt and braces, not a substitute for correct types.

Precision: Java has nanos, PostgreSQL has micros

Instant.parse("2026-08-15T10:00:00.123456789Z")           // .123456789
    .truncatedTo(ChronoUnit.MICROS)                        // .123456

PostgreSQL stores microseconds. Those last three digits are gone on the round trip, and PostgreSQL may round rather than truncate:

SELECT '2026-08-15 10:00:00.123456789'::timestamptz;  -- 10:00:00.123457

If you have an equality test or an optimistic-lock check comparing an in-memory Instant to one that’s been through the database, truncate to micros before you persist so both sides agree.

Don’t wrap the column in a function

Because timestamptz is stored as UTC, “give me everything that happened on 1 Feb in New York” tempts you into this:

EXPLAIN SELECT count(*) FROM events
WHERE (occurred_at AT TIME ZONE 'America/New_York')::date = DATE '2026-02-01';
--  Seq Scan on events

Index gone. Compute the boundary instants instead and do a half-open range scan:

SELECT (DATE '2026-02-01' + time '00:00') AT TIME ZONE 'America/New_York',  -- 2026-02-01 05:00+00
       (DATE '2026-02-02' + time '00:00') AT TIME ZONE 'America/New_York';  -- 2026-02-02 05:00+00
EXPLAIN SELECT count(*) FROM events
WHERE occurred_at >= '2026-02-01 05:00:00+00'
  AND occurred_at <  '2026-02-02 05:00:00+00';
--  Index Only Scan using events_occurred_at_idx

Note that this “day” is 24 hours only because February has no DST transition. On a transition day it’s 23 or 25 — which the boundary-instant approach handles correctly and a naive + interval ‘24 hours’ does not. Half-open ranges (>= and <) also dodge the microsecond-boundary problem that BETWEEN creates.

Part 5: JPA and Hibernate

Hibernate 6 made zoned types a first-class concern via TimeZoneStorageType:

For PostgreSQL, the practical consequence is settled by the database, not the ORM: timestamptz discards the offset, so under DEFAULT your OffsetDateTime round-trips as UTC. Which is fine — as long as you expected it.

Sensible defaults:

spring:
  jpa:
    properties:
      hibernate:
        jdbc:
          time_zone: UTC
        timezone:
          default_storage: NORMALIZE_UTC

And in the entity, be explicit about intent:

@Entity
public class Booking {
    
    // A moment. Maps to timestamptz.
    @Column(nullable = false)
    private Instant createdAt;
    
    // A date. Not a moment.
    @Column(nullable = false)
    private LocalDate serviceDate;
    
    // A wall-clock time that must not shift. Maps to timestamp.
    @Column(nullable = false)
    private LocalDateTime localStart;
    
    @Column(nullable = false)
    private String zoneId;   // 'Europe/London'
    
    public ZonedDateTime start() {
        return localStart.atZone(ZoneId.of(zoneId));
    }
}

If you genuinely need the user’s original offset preserved, @TimeZoneStorage(TimeZoneStorageType.COLUMN) with @TimeZoneColumn writes it to a companion column. Use it when the offset is data — an audit trail of what the user’s device claimed — not as a default.

Part 6: Testing This Stuff

Datetime bugs are only cheap if you catch them before the clocks change. Three habits do most of the work.

1. Inject a Clock

Instant.now() scattered through business logic is untestable. Inject java.time.Clock and you can move time at will:

@Bean Clock clock() { return Clock.systemUTC(); }
// in a test
Clock fixed = Clock.fixed(Instant.parse("2026-10-25T00:30:00Z"), ZoneOffset.UTC);

2. Run CI in a hostile time zone

If your JVM, your database, and your CI runner are all UTC, every zone bug hides until production. Force one of them somewhere awkward:

-Duser.timezone=Pacific/Chatham

+12:45 and it observes DST. If your test suite survives Chatham, it will survive most things. Do the same for the PostgreSQL container:

@Container
static PostgreSQLContainer<?> pg = new PostgreSQLContainer<>("postgres:16")
        .withEnv("TZ", "Asia/Kathmandu")
        .withUrlParam("options", "-c TimeZone=Asia/Kathmandu");

The point isn’t that production runs there — it’s that a test which only passes in UTC is a test that has proven nothing about zone handling.

3. Write the four transition tests

For any code that converts between local and instant, pin these explicitly:

static final ZoneId LONDON = ZoneId.of("Europe/London");
@Test void gapTimesAreRejectedNotSilentlyShifted() {
    LocalDateTime gap = LocalDateTime.of(2026, 3, 29, 1, 30);
    assertThat(LONDON.getRules().getValidOffsets(gap)).isEmpty();
    assertThatThrownBy(() -> scheduler.schedule(gap, LONDON))
        .isInstanceOf(NonExistentLocalTimeException.class);
}
@Test void overlapTimesResolveToTheDocumentedChoice() {
    LocalDateTime overlap = LocalDateTime.of(2026, 10, 25, 1, 30);
    assertThat(LONDON.getRules().getValidOffsets(overlap)).hasSize(2);
    assertThat(scheduler.resolve(overlap, LONDON))
        .isEqualTo(Instant.parse("2026-10-25T00:30:00Z")); // earlier, by policy
}
@Test void dailyRecurrenceKeepsWallClockAcrossDst() {
    ZonedDateTime start = ZonedDateTime.of(2026, 3, 28, 9, 0, 0, 0, LONDON);
    assertThat(recurrence.next(start).toLocalTime()).isEqualTo(LocalTime.of(9, 0));
}
@Test void instantSurvivesTheDatabaseRoundTrip() {
    Instant original = Instant.now().truncatedTo(ChronoUnit.MICROS);
    repository.save(new Event(original));
    assertThat(repository.findLatest().occurredAt()).isEqualTo(original);
}

The fourth one catches more real bugs than you’d expect — it’s the one that fails when someone quietly changes a column from timestamptz to timestamp.

The Short Version

If you skim one section, make it this one.

Schema

• timestamptz by default. It’s the same 8 bytes and it’s unambiguous.
• timestamp only for wall-clock times that must not shift — and store the zone ID beside it.
• date for dates. timetz for nothing.
• Pin timezone = ‘UTC’ in postgresql.conf and in the JDBC URL.

Java

• Instant for things that happened. LocalDate for dates. ZonedDateTime for scheduling. OffsetDateTime at the JDBC boundary.
• LocalDateTime is not a point in time. Treat every use as a question to answer, not an answer.
• Never accept zone abbreviations. Full IANA names only.
• plusDays ≠ plus(Duration.ofDays(1)). Know which one your product means.

The seam

• pgJDBC speaks LocalDate/LocalTime/LocalDateTime/OffsetDateTime. Convert Instant explicitly.
• Avoid setTimestamp/getTimestamp — they use the JVM default zone.
• Truncate to microseconds before persisting.
• Don’t wrap indexed timestamp columns in AT TIME ZONE; compute boundary instants instead.

And the one that bites hardest

Java resolves an ambiguous local time to the earlier offset. PostgreSQL picks the later one. Resolve ambiguity in exactly one place, deliberately, and pass instants across the boundary — never local times.

Store the instant, render the local time, and keep the zone rules in exactly one layer. Most datetime bugs are just that rule, broken somewhere.

All examples verified on OpenJDK 21.0.10 (tzdata 2025b) and PostgreSQL 16.13.

Sources

• PostgreSQL: Date/Time Types
• pgJDBC: Using Java 8 Date and Time classes
• Hibernate TimeZoneStorageType Javadoc
• Hibernate 6 migration guide
• Thorben Janssen: TimezoneStorageType

When a migration hangs, a query slows to a crawl, or a deploy just sits there, most people reach for the same first move: kill it and try again. That works about as often as it doesn’t. Postgres already knows what’s wrong — it’s just keeping the answer in a handful of system catalogs and stats views that almost nobody looks at until things are already on fire.

This is a walkthrough of the tables I actually check, in the order I check them, when something breaks — with a running example: a migration that adds a NOT NULL column and backfills it on a 20-million-row orders table, which has been “running” for 25 minutes with no sign of finishing.

Start with: is it even running?

Before touching statistics or query plans, find out if the process is alive, blocked, or just slow.

SELECT pid, state, wait_event_type, wait_event, query, query_start
FROM pg_stat_activity
WHERE state != 'idle';

pg_stat_activity is a live snapshot of every connection to the database. The columns that matter most here are wait_event_type and wait_event. If a process shows wait_event_type = ‘Lock’, it isn’t broken — it’s queued behind something else. If it shows Client or IO, that’s a different problem entirely (waiting on the client, or waiting on disk).

For the orders migration, this comes back with:

pid  | state  | wait_event_type | wait_event | query
-----+--------+-----------------+------------+----------------------------------
8842 | active | Lock            | relation   | ALTER TABLE orders ADD COLUMN ...

wait_event_type = Lock tells us immediately: the migration isn’t slow, it’s stuck behind something else that holds a conflicting lock on orders.

If you see a lock wait, the next stop is obvious:

SELECT blocked_locks.pid AS blocked_pid,
       blocking_locks.pid AS blocking_pid,
       blocked_activity.query AS blocked_query,
       blocking_activity.query AS blocking_query
FROM pg_locks blocked_locks
JOIN pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
JOIN pg_locks blocking_locks
  ON blocking_locks.locktype = blocked_locks.locktype
  AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
  AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
  AND blocking_locks.pid != blocked_locks.pid
JOIN pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;

This join is the single most useful query in Postgres debugging. It tells you exactly which session is blocking which, and what each of them is running. Most “stuck migration” incidents end here: some other transaction — often a long-running read, an ORM that forgot to close a connection, or a previous deploy that never finished — is sitting on a lock the migration needs.

Running it for the orders case turns up the problem:

blocked_pid | blocking_pid | blocked_query                   | blocking_query
------------+--------------+---------------------------------+---------------------------------------------
 8842       | 6110         | ALTER TABLE orders ADD COLUMN.. | SELECT * FROM orders WHERE ... (idle in txn)

PID 6110 is an analytics job that opened a transaction, ran a read, and never committed — “idle in transaction” — leaving an ACCESS SHARE lock on orders that blocks the migration’s ACCESS EXCLUSIVE lock. The fix here isn’t in the migration at all: it’s killing or fixing PID 6110 (SELECT pg_terminate_backend(6110); if you’re sure it’s safe to end it).

If it’s a long-running operation, check its actual progress

Postgres exposes live progress for the operations that most commonly make migrations feel frozen:

SELECT * FROM pg_stat_progress_create_index;
SELECT * FROM pg_stat_progress_vacuum;
SELECT * FROM pg_stat_progress_analyze;

If you’re running CREATE INDEX CONCURRENTLY as part of a migration, pg_stat_progress_create_index shows you the current phase (building index, index validation, etc.) and how many tuples it has scanned versus the total. This turns “is this hung or just big?” from a guess into a fact.

Say the next step of the migration adds an index on the new column instead of blocking on a lock — the progress view might show:

pid  | phase                         | tuples_total | tuples_done
-----+-------------------------------+--------------+-------------
9021 | building index: scanning table| 20450112     | 14300556

That’s roughly 70% through a single scan phase, moving steadily — not stuck, just genuinely working through 20 million rows. Compare that against a snapshot taken a minute later: if tuples_done hasn’t moved, now you know it actually is stalled, and it’s worth going back to pg_locks.

If nothing’s blocked, suspect the query plan

Not every broken migration is a lock problem. Sometimes the migration runs, but a backfill step or an added index makes the planner choose a catastrophically bad plan. That’s a statistics problem, and it lives in two places.

pg_stat_user_tables tells you whether Postgres has been keeping its stats up to date on the table in question:

SELECT relname, n_live_tup, n_dead_tup, n_mod_since_analyze,
       last_vacuum, last_autovacuum, last_analyze, last_autoanalyze
FROM pg_stat_user_tables
WHERE relname = 'your_table';

If n_mod_since_analyze is large relative to n_live_tup, or last_autoanalyze is suspiciously old, the planner is working off outdated information. This happens constantly during migrations, because a migration that inserts, updates, or deletes a large chunk of a table changes its shape faster than autovacuum’s default thresholds react to.

For orders, after the backfill step runs:

relname | n_live_tup | n_dead_tup | n_mod_since_analyze | last_autoanalyze
--------+------------+------------+---------------------+-------------------
orders  | 20450112   | 3800000    | 18900000            | 2026-07-28 03:14

18.9 million modified rows since the last analyze, against 20 million live rows — nearly the entire table has changed and autovacuum hasn’t caught up yet (last_autoanalyze is five days old). Any query the migration runs after this point is planning against numbers that are effectively meaningless.

pg_stats (the readable view over the raw pg_statistic catalog) shows what the planner actually believes about each column:

SELECT attname, n_distinct, correlation, null_frac, most_common_vals
FROM pg_stats
WHERE tablename = 'your_table';

n_distinct and correlation drive index-vs-sequential-scan decisions. If a migration adds a new index and the planner still refuses to use it, stale stats here are the usual suspect.

On the new orders.status column right after the backfill, before an ANALYZE:

 attname | n_distinct | correlation | null_frac
---------+------------+-------------+-----------
 status  | 1          | 1           | 1.0

The planner still believes every row is null (null_frac = 1.0) because that was true before the backfill ran. Any query filtering on status will get a wildly wrong row estimate and likely pick a sequential scan or a bad join order. The fix is almost always to run ANALYZE your_table; — either before the migration, or as an explicit step inside it, rather than waiting for autovacuum to get around to it. After ANALYZE orders;, the same query returns something like n_distinct = 4, null_frac = 0.0 — matching reality, and the planner’s decisions follow accordingly.

pg_class holds the planner’s cached size estimates directly:

SELECT relname, reltuples, relpages
FROM pg_class
WHERE relname = 'your_table';

reltuples is an estimate, not a live count — it’s only refreshed by ANALYZE or VACUUM. If this number is wildly off from the real row count, every cost estimate downstream is wrong too. For orders, reltuples might read 20450112 while a SELECT count(*) returns closer to 24000000 — a four-million-row gap that skews every cost estimate the planner builds on top of it.

If you need the exact query and its cost, not just the table

pg_stat_statements (an extension, so it needs to be enabled) aggregates every distinct query the server has run, with real timing:

SELECT query, calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;

This is the difference between “the migration is slow” and “this specific UPDATE inside the migration is slow, and it’s been called 40,000 times.” For the orders backfill, this might show:

query                                        | calls | mean_exec_time | rows
---------------------------------------------+-------+----------------+--------
 UPDATE orders SET status = $1 WHERE id = $2 | 40000 | 812.4 ms       | 40000

40,000 single-row updates averaging 800ms each explains a lot on its own — that’s batching the backfill row-by-row instead of in bulk, and it would still be slow even with perfect statistics. If it’s enabled in your environment, it’s usually the fastest way to go from vague suspicion to a concrete line of SQL to fix — in this case, rewriting the backfill as a single set-based UPDATE … WHERE status IS NULL in batches, rather than one row at a time.

A rough order of operations

Put together, a sensible debugging sequence looks like this: check pg_stat_activity to see if the process is alive and what it’s waiting on; if it’s waiting on a lock, join into pg_locks to find the blocker; if it’s a long index build or vacuum, check the matching pg_stat_progress_* view for real progress; if nothing is blocked but it’s still slow, check pg_stat_user_tables and pg_stats for stale statistics, and pg_class for stale row estimates; and if pg_stat_statements is available, use it to find the exact query eating the time.

None of these tables are exotic. They ship with every Postgres install (aside from pg_stat_statements, which just needs CREATE EXTENSION). The only real skill here is knowing which one answers which question — and reaching for pg_stat_activity and pg_locks before assuming it’s a statistics problem, since lock contention is a far more common cause of a “broken” migration than a bad query plan.

You type gradle build, wait a few seconds, and out comes a JAR. Most Java developers use this pipeline every day without ever looking inside it. But when a build gets slow, a dependency conflict appears, or a NoClassDefFoundError shows up in production at 2 AM, understanding the machinery underneath is the difference between guessing and fixing.

This article walks the full journey: Gradle’s three build phases, the Java plugin’s task graph, how dependency resolution actually works stage by stage, what a JAR file really is, and how the JVM reads and executes it.

Part 1: The Three Phases Every Gradle Build Goes Through

Before a single line of your code is compiled, Gradle itself runs through a fixed lifecycle of three phases. This is true for every build, every plugin, every project.

Phase 1 — Initialization

Gradle reads settings.gradle(.kts) and answers one question: which projects are part of this build? For a single-module app that’s trivial; for a multi-module monorepo this is where include(":core", “:api”, “:app”) gets evaluated and a Project object is created for each module.

Phase 2 — Configuration

Now every build.gradle(.kts) script executes top to bottom. This is the phase people misunderstand most. Your build script is code that runs — and it runs here, on every invocation, regardless of which task you asked for. The output of this phase is the task graph: a directed acyclic graph (DAG) of all tasks and their dependencies. Tasks are registered and wired together, but none of their actions execute yet.

A println at the top level of build.gradle prints during Configuration — even if you run gradle help. Task actions run later. This trips up nearly everyone once.

Phase 3 — Execution

Gradle takes the task you requested, computes the minimal subgraph of tasks needed to produce it, and executes them in dependency order. Tasks whose inputs and outputs haven’t changed since the last run are marked UP-TO-DATE and skipped entirely — this is Gradle’s incremental build engine, and it’s why the second build is always faster than the first.

Part 2: The Java Plugin Task Graph

Applying the java plugin registers a standard chain of tasks. When you run gradle build, this is the graph that gets executed:

The practical shortcuts worth memorizing:

# fastest feedback: compile only
gradle compileJava
# package without running tests
gradle jar
# full build, skip test execution
gradle build -x test
# nuke build/ and start clean
gradle clean build

Part 3: How Gradle Dependency Resolution Actually Works

This is the part of Gradle most developers treat as magic. You write three lines in a dependencies block, and somehow the right JARs appear on your classpath. Here’s the machinery, stage by stage.

Configurations: buckets of dependencies

Every dependency you declare goes into a named configuration — a bucket with specific rules about where those dependencies are visible:

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web:3.3.1'
    api 'com.fasterxml.jackson.core:jackson-databind:2.17.1'
    compileOnly 'org.projectlombok:lombok:1.18.32'
    annotationProcessor 'org.projectlombok:lombok:1.18.32'
    runtimeOnly 'org.postgresql:postgresql:42.7.3'
    testImplementation 'org.junit.jupiter:junit-jupiter:5.10.2'
}

The implementation vs api distinction matters in multi-module builds: implementation dependencies don’t appear on the compile classpath of modules that depend on yours. That means fewer recompilations when they change, and a cleaner API boundary. Default to implementation; reach for api only when a type from that dependency appears in your public method signatures.

The resolution stages

Declared dependencies are just coordinates on paper. Resolution is the process of turning them into actual files, and it happens lazily — only when a task first needs a classpath (typically compileJava during Execution). The stages:

Stage 1 — Graph construction. For each declared coordinate (group:name:version), Gradle queries your repositories in the order they’re declared and downloads only the metadata first: the POM file or Gradle Module Metadata. That metadata lists the dependency’s own dependencies, so Gradle recursively walks outward until the full transitive graph is known. No JARs have been downloaded yet — just descriptors.

Stage 2 — Conflict resolution. Real dependency graphs always contain the same library requested at different versions (your app wants Jackson 2.17, some transitive path wants 2.15). Gradle’s default strategy is highest version wins — unlike Maven, which picks the “nearest” declaration in the tree. This default is usually right, but you can override it:

configurations.all {
    resolutionStrategy {
        // hard-pin a version, overriding everything
        force 'com.fasterxml.jackson.core:jackson-databind:2.17.1'
        // or turn conflicts into build failures
        failOnVersionConflict()
    }
}

Stage 3 — Constraints and rules. Before the graph is finalized, Gradle applies platforms (BOMs), version constraints, exclusions, and dependency substitutions. This is how Spring Boot keeps hundreds of libraries version-aligned with one line:

dependencies {
    implementation platform('org.springframework.boot:spring-boot-dependencies:3.3.1')
    // now version numbers can be omitted — the BOM supplies them
    implementation 'org.springframework.boot:spring-boot-starter-web'
}

Exclusions cut unwanted transitive branches at the declaration site:

implementation('org.springframework.boot:spring-boot-starter-web') {
    exclude group: 'org.springframework.boot', module: 'spring-boot-starter-tomcat'
}

Stage 4 — Artifact download. Only now, with the final graph settled, does Gradle download the actual JAR files — and only for modules that survived resolution. Everything lands in the shared cache at ~/.gradle/caches/modules-2, keyed by checksum, shared across all projects on your machine. A dependency is downloaded once, ever, per machine (dynamic versions like 1.+ and snapshots get re-checked periodically).

Stage 5 — Classpath assembly. The resolved files are ordered into a classpath and handed to whichever task requested them: compileClasspath to javac, runtimeClasspath to the test JVM, and so on.

Debugging resolution

Two commands answer 90% of “why is this version on my classpath” questions:

# full resolved tree per configuration
gradle dependencies --configuration runtimeClasspath
# trace one library: who pulls it in, which versions were requested,
# and why the winner won
gradle dependencyInsight --dependency jackson-databind

Part 4: What a JAR File Really Is

The output of the jar task is nothing exotic — it’s a ZIP archive with an agreed-upon internal layout. Rename it to .zip and any archive tool will open it:

myapp.jar
├── META-INF/
│   └── MANIFEST.MF          ← metadata
├── com/example/App.class     ← bytecode, path = package name
├── com/example/util/Helper.class
└── application.properties    ← resources sit alongside classes

The manifest is a plain-text key-value file. The line that matters most for a runnable JAR:

Manifest-Version: 1.0
Main-Class: com.example.App

One important caveat: the plain jar task packages only your classes. Your dependencies are not inside. For a self-contained deployable artifact you need a fat JAR — with the Shadow plugin, or Spring Boot’s bootJar, which nests dependency JARs and adds its own launcher.

plugins {
    id 'com.github.johnrengelman.shadow' version '8.1.1'
}
// gradle shadowJar → one JAR containing everything

Part 5: How the JVM Reads Your JAR

You run java -jar myapp.jar. Here’s the sequence, and it’s more interesting than it looks.

Step 1 — The launcher reads the manifest

The java launcher opens the JAR as a ZIP, reads META-INF/MANIFEST.MF, and extracts Main-Class. Note a classic gotcha: with -jar, any -cp argument on the command line is silently ignored. The JAR itself (plus manifest Class-Path entries) becomes the entire application classpath.

Step 2 — The classloader hierarchy

Classes are loaded by a chain of classloaders using parent-first delegation: every load request travels up to the top of the chain first, and only comes back down if no ancestor can satisfy it.

Step 3 — Lazy loading

The JVM does not load every class in the JAR at startup. A class is loaded the first time it’s actively used — instantiated, statically referenced, reflected upon. This is why a missing transitive dependency can let your app boot cleanly and then blow up with NoClassDefFoundError twenty minutes later, the first time a rarely-used code path runs.

Step 4 — Load, Link, Initialize

Per the JVM specification, every class goes through three phases before it’s usable:

1. Loading — the class bytes are read from the JAR, parsed, and a Class object is created in metaspace.
2. Linking — three sub-steps:
   • Verification: the bytecode is checked for structural and type safety (stack shapes, valid jumps, access rules).
   • Preparation: static fields are allocated and set to defaults — 0, null, false. Not your values yet.
   • Resolution: symbolic references to other classes, methods, and fields are resolved to concrete ones (often lazily).
3. Initialization — static initializer blocks and static field assignments run, in source order, exactly once, on first active use. This is the point where static { … } blocks fire.

Step 5 — main() and the JIT

Once Main-Class is loaded, linked, and initialized, the JVM invokes main(String[] args) on the main thread. Execution starts in the interpreter. As methods get hot — called thousands of times — the JIT compiler (tiered: C1 first for fast, lightly-optimized code, then C2 for aggressively optimized code) compiles them to native machine code. That’s why JVM applications famously get faster after warm-up.

The Whole Picture

src/main/java  ──javac──▶  build/classes  ──jar──▶  myapp.jar
                                                        │
                                             java -jar myapp.jar
                                                        │
                       read MANIFEST.MF → find Main-Class
                                                        │
                 classloader delegation (parent-first)
                                                        │
                 Load → Link (verify/prepare/resolve) → Init
                                                        │
                            main() runs, classes load lazily,
                            JIT compiles hot paths as you go

None of this is knowledge you need daily. But the day your build takes four minutes instead of forty seconds, or two versions of the same library fight on your classpath, or a class mysteriously fails to load in production — this is the map you’ll be reading. Better to have seen it before that day arrives.

Every time you open Claude and ask it to create a Word document, build a data analysis pipeline, or write presentation slides — something invisible happens first. Claude scans a directory of skills, finds the most relevant one, reads it, and only then starts working.

This system is called Claude Skills. It is one of the most underappreciated features in the entire Claude ecosystem. And once you understand how it works, you will see why it changes everything about how you build AI-powered workflows.

“Skills are how Claude learns domain-specific workflows without retraining. They are runtime instructions, not model weights.”

What Is a Skill?

A skill is a directory containing a SKILL.md file (and optionally supporting assets, scripts, and reference documents). When Claude starts handling a request, it reads the name and description of every available skill. If a description matches the current task, Claude loads the full SKILL.md body and follows its instructions.

Think of it like a context-sensitive manual. When a carpenter picks up a dovetail saw, they know exactly how to use it — not because they memorised every detail about every tool upfront, but because they learned the right technique for the right moment. Skills work the same way for Claude.

The Three-Layer Loading System

Skills use progressive disclosure — Claude never loads everything at once. There are three distinct layers, each loaded only as needed:

This is elegant engineering. Layer 1 is tiny — just enough for Claude to decide whether a skill applies. Layer 2 gives full instructions once relevant. Layer 3 provides deep reference material only when needed, keeping context usage lean.

Anatomy of a Skill Directory

A skill is a folder with a predictable structure. Here is the full layout:

Only SKILL.md is required. Everything else is optional and only loaded when the skill itself explicitly references it.

Inside SKILL.md — The Configuration Fields

The file starts with a YAML frontmatter block (between — delimiters), followed by markdown instructions. Here is every supported field:

KEY INSIGHTThe description field is the most important part of any skill. Claude decides whether to load your skill based solely on this field — the body never loads unless the description matches. Write it with real triggers, real contexts, and a slight “push” to combat under-triggering.

A Minimal Skill — Step by Step

Let’s build a real skill from scratch. Imagine you regularly need to convert raw PostgreSQL query results into nicely formatted Markdown reports. Here is how to package that as a skill.

Step 1 — Create the directory

mkdir pg-report-skill
cd pg-report-skill
touch SKILL.md

Step 2 — Write the frontmatter

---
name: pg-report
description: |
  Converts raw PostgreSQL query results into formatted Markdown reports
  with tables, summary stats, and query metadata. Use whenever the user
  pastes SQL output, mentions "format this query result", or wants to
  turn database output into a readable report or document. Also trigger
  when user asks to summarise a dataset from a .csv or .tsv that looks
  like a DB export.
---

TIP: WRITE THE DESCRIPTION LIKE A TRIGGER RULESETNotice how the description lists multiple phrasings (“pastes SQL output”, “format this query result”, “summarise a dataset”). This breadth is intentional — Claude uses fuzzy matching, so covering several natural phrasings dramatically improves trigger reliability.

Step 3 — Write the skill body

# PostgreSQL Report Formatter
## What this skill does
Formats raw PostgreSQL tabular output into clean Markdown reports.
## Required output sections
1. A ## Summary block: row count, column count, date range (if present)
2. A ## Data block: full Markdown table with aligned columns
3. A ## Observations block: 3-5 key insights from the data
## Formatting rules
- Numeric columns: right-align, add thousand separators
- Date columns: format as YYYY-MM-DD
- NULL values: render as `—` (em dash), never blank
- Column headers: Title Case
## Example input

id | user_name | total_orders | revenue —-+———–+————–+——— 1 | alice | 12 | 4500.00 2 | bob | 3 | 890.50

## Example output
### Summary
- **Rows:** 2  |  **Columns:** 4  |  **Total revenue:** $5,390.50
### Data
| ID | User Name | Total Orders | Revenue |
|----|-----------|-------------:|--------:|
|  1 | Alice     |           12 | $4,500.00 |
|  2 | Bob       |            3 |   $890.50 |

Step 4 — Add a reference file (optional)

If your skill needs to handle multiple database types — PostgreSQL, MySQL, SQLite — you can create a references/ folder and have the skill body instruct Claude to load the right one:

In your SKILL.md body, you then write:

## Database-specific rules
Detect which database produced the output from syntax clues.
- If `||` concatenation → read references/sqlite.md
- If `AUTO_INCREMENT` → read references/mysql.md
- Default → read references/postgres.md

Claude will read only the relevant file, keeping the context window lean and responses accurate.

Real-World Example: The Built-in DOCX Skill

Claude ships with several built-in skills. The docx skill is a great example of production-quality configuration. Here is its description field, distilled:

description: |
  Use this skill whenever the user wants to create, read, edit, or
  manipulate Word documents (.docx files). Triggers include: any
  mention of 'Word doc', 'word document', '.docx', or requests to
  produce professional documents with formatting like tables of
  contents, headings, page numbers, or letterheads. Also use when
  extracting content from .docx files, inserting images, performing
  find-and-replace, working with tracked changes, or converting
  content into a polished Word document.

Notice the pattern: the description is almost a checklist. Every plausible user phrasing that should trigger the skill is listed. This is intentional — Claude has a natural tendency to under-trigger skills, so descriptions are deliberately thorough.

WARNING: AVOID GENERIC DESCRIPTIONS

A description like “Helps with documents” will rarely trigger. Claude needs enough specificity to distinguish your skill from general knowledge. Name the exact task, the exact file types, and the exact user phrases that should activate it.

The Progressive Disclosure Principle

The most important architectural decision in skill design is knowing what goes where. The three-layer rule is simple:

1. Description = triggering logic only

Never put instructions in the description. It exists solely to help Claude decide whether to load the skill. Keep it focused on “when to use” language, not “how to do”.

2. SKILL.md body = core workflow

Put the main instructions, output format rules, required sections, formatting guidelines, and examples here. Target under 500 lines. If you need more, move deep content into reference files.

3. References + scripts = deep knowledge

Detailed documentation, database-specific rules, platform-specific templates, and executable scripts go in subfolders. The SKILL.md body tells Claude when to read them.

Writing Descriptions That Actually Trigger

The description field is the hardest part to get right. Here is a battle-tested template:

description: |
  [What the skill does in one sentence].
  Use this skill when: [scenario A], [scenario B], [scenario C].
  Also trigger when the user [natural phrase], [natural phrase],
  or mentions [keyword / file type / tool name].
  Do NOT use for [explicit exclusion to avoid false positives].

And a concrete example for a “Java Code Review” skill:

description: |
  Reviews Java source files for correctness, performance, thread
  safety, and style — producing structured inline feedback.
  Use whenever the user shares Java code and asks for review,
  feedback, improvements, or a PR review. Also trigger on phrases
  like "check my code", "what's wrong here", or "how can I improve
  this" when the code is Java. Trigger on .java files even without
  an explicit review request.
  Do NOT use for Kotlin, Scala, or general architecture questions
  without code attached.

When Skills Use Scripts

For deterministic or repetitive transformations — file conversions, data extraction, package generation — skills can include executable scripts. Claude reads the script path from the SKILL.md body, runs it via the bash tool, and uses the output.

The built-in docx skill, for example, uses a Python script to write .docxfiles using the python-docx library. The skill body contains step-by-step instructions for calling the script correctly — Claude follows them at runtime.

## Creating a DOCX file
1. Generate Python code using python-docx from the template at
   scripts/create_doc.py
2. Run: python scripts/create_doc.py output.docx
3. Copy result to /mnt/user-data/outputs/
4. Present to user with present_files tool

Testing Your Skill

Before deploying, write 5–10 test prompts that should trigger your skill, and 3–5 that should not. Run Claude with the skill installed and check whether it loads correctly.

Good test prompts for a “pg-report” skill:

# Should trigger:
"Here's my query result, can you format it nicely?"
"Turn this SQL output into a report"
"I exported this CSV from my database, summarise it"
# Should NOT trigger:
"How do I write a JOIN query?"
"What is normalisation in databases?"

PRO TIP: QUANTITATIVE EVALUATIONIf you are building skills at scale, the skill-creator meta-skill includes an automated optimization loop. It splits your test set 60/40 (train/held-out), runs Claude against each case multiple times, proposes description improvements, and iterates up to 5 rounds — outputting the best-performing description with before/after scores.

Packaging and Sharing Skills

A finished skill can be packaged into a .skill file for distribution. Anyone can install it by dropping the file into their skills directory. The package script handles archiving:

python -m scripts.package_skill path/to/pg-report-skill/
# → outputs pg-report-skill.skill

The Bigger Picture

Skills are not just a convenience feature. They are the mechanism by which Claude becomes domain-aware without retraining. A legal firm can build a contract-review skill. A data team can build a dashboard-generation skill. A game studio can build a Godot export skill. Each one runs inside the same Claude — just with different runtime instructions.

The architecture is deliberately modular. Skills compose. A single request might trigger the pdf-reading skill to extract content, then the docx skill to write the output. Each stays in its lane, and Claude orchestrates.

“You don’t teach Claude new tricks by changing the model. You teach it by writing better skills.”

For solo developers and small teams, this is genuinely powerful. You can codify your best workflows once, as a skill, and they run reliably on every request — without prompting, without repetition, and without hoping Claude happens to “know” your preferred format.

Start small. Pick one workflow you repeat every week. Write a 30-line SKILL.md. Test it on five prompts. Then iterate. That is all it takes to start building your own personalised AI toolkit.

RESOURCESThe skills discussed in this article — including docx, pdf, pptx, xlsx, and skill-creator — are built into Claude’s computer-use environment. You can explore them at /mnt/skills/public/ in any Claude session with computer use enabled.

Picture this: you’ve got the most brilliant assistant in the world. Knows everything. Speaks 50 languages. Can explain quantum physics and write poetry at the same time. But here’s the catch — they’re locked in a glass box. They can see your computer screen. They can describe what they’d do if they could reach it. But touch anything? Nope. Read a file? Uh-uh. Check your calendar? Dream on.

That was AI assistants until very recently. Incredibly smart, utterly helpless. Like hiring Einstein to organize your sock drawer and then refusing to let him into the room.

Enter MCP — the Model Context Protocol. The thing that breaks the glass and lets your AI assistant actually do the thing.

MCP is essentially a USB standard for AI. Before USB, every device needed its own special plug. Now there’s one protocol, and everything just… works.

By the end of this article you’ll understand what MCP is, why it exists, how it works, and — most importantly — you’ll build your own working MCP server from scratch and connect it to Claude. Buckle up.

A Brief History of AI’s Helplessness Problem

To understand why MCP is a big deal, we need to appreciate just how spectacularly bad the situation was before it.

EARLY 2023

Large Language Models (LLMs) become genuinely impressive. Everyone rushes to build assistants. The dream: AI that manages your life. The reality: AI that describes your life back to you very eloquently.

MID 2023

“Function calling” arrives. Now you can tell the AI: “hey, you can call these functions.” Progress! But every app invents its own format. A plugin for ChatGPT works nowhere else. A LangChain tool chain doesn’t talk to Claude. It’s the Wild West, but with JSON.

LATE 2023–2024

Developers build increasingly elaborate custom integrations. Everyone reinvents the wheel. Slightly differently. Slightly wrongly. A thousand incompatible “AI tool” standards bloom. This is the Tower of Babel, Silicon Valley edition.

NOVEMBER 2024

Anthropic releases the Model Context Protocol. An open standard. One protocol to rule them all. The developer community collectively exhales. Finally.

2025

MCP adoption explodes. OpenAI, Google, Microsoft all announce support. Hundreds of MCP servers appear on GitHub. Your AI can now read files, browse the web, query databases, control your computer, send Slack messages, and order pizza. (Well, almost.)

The key insight Anthropic had was embarrassingly simple: instead of every AI model learning to speak every tool’s language, make every tool learn one common language. The result is MCP.

What Is MCP, Actually?

MCP stands for Model Context Protocol. It’s an open standard (meaning anyone can use it, free of charge) that defines how AI models communicate with external tools and data sources.

Think of it as a formal language that both sides agree to speak. The AI model is one speaker. Your tool (a file reader, a database, a weather API, your smart fridge) is the other speaker. MCP is the language between them.

The one-sentence explanation:
MCP lets AI models talk to any tool through a single standardized protocol, so you write the connector once and it works everywhere.

Technically, MCP is a client-server architecture built on top of JSON-RPC 2.0. Don’t be scared by that. It means messages are passed as readable JSON objects, and there’s a strict format for how requests and responses look. Clean, simple, predictable.

The Architecture: Three Amigos

MCP has three main characters. Let’s meet them:

The Host

The application running the AI. Could be Claude Desktop, VS Code with Copilot, or your own app. The host is the conductor — it decides when the AI needs a tool and coordinates everything.

The Client

Technically a component inside the host. It handles the actual MCP communication — sends requests to servers, gets back results, feeds them to the AI. You don’t usually build this yourself.

The Server

This is what you build. An MCP server exposes capabilities — tools, resources, and prompts — that the AI can use. Your server could wrap a database, a REST API, the local file system, or anything else. This is where the magic you write lives.

Core Concepts: Tools, Resources & Prompts

MCP servers expose three types of things to AI models. Think of these as different kinds of “capabilities”:

For most real-world use cases you’ll mainly deal with Tools. That’s where the action is.

How They Talk: The MCP Message Format

MCP messages are JSON objects following the JSON-RPC 2.0 specification. Every single message — whether it’s a greeting, a tool call, or an error — follows the same envelope structure. Once you learn the envelope, the rest is just filling in the blanks.

The Anatomy of Every Message

All JSON-RPC messages share the same top-level fields. Let’s dissect them:

Phase 1 — The Handshake: initialize

Before any tools are called, the host and server must introduce themselves. This is the very first exchange that happens when Claude Desktop spawns your server process. Think of it as two people shaking hands and agreeing on a common language before getting to business.

1A. HOST → SERVER: INITIALIZE REQUEST

{
  "jsonrpc": "2.0",
  "id": 0,
  "method": "initialize",
  "params": {
    "protocolVersion": "2024-11-05",        // MCP spec version the host supports
    "capabilities": {
      "roots": { "listChanged": true },    // Host can notify server about root changes
      "sampling": {}                         // Host supports LLM sampling requests
    },
    "clientInfo": {
      "name": "Claude Desktop",
      "version": "0.7.0"
    }
  }
}

1B. SERVER → HOST: INITIALIZE RESPONSE

{
  "jsonrpc": "2.0",
  "id": 0,                                    // Same id — this is the matching reply
  "result": {
    "protocolVersion": "2024-11-05",        // Server confirms the version
    "capabilities": {
      "tools": { "listChanged": false }    // Server exposes tools (and won't hot-reload them)
    },
    "serverInfo": {
      "name": "my-first-mcp-server",
      "version": "1.0.0"
    }
  }
}

1C. HOST → SERVER: INITIALIZED NOTIFICATION (NO REPLY EXPECTED)

{
  "jsonrpc": "2.0",
  "method": "notifications/initialized"
  // No "id" field — this is a notification, not a request.
  // Notifications are fire-and-forget; the server never replies.
}

After this three-step handshake, the connection is live and both sides know what the other can do. Now the real fun begins.

Phase 2 — Tool Discovery: tools/list

The host immediately asks: “OK, what tools do you have?” This lets the AI know its full menu of capabilities before the user types a single character.

2A. HOST → SERVER: TOOLS/LIST REQUEST

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/list"
  // No "params" needed — just asking for the full list.
}

2B. SERVER → HOST: TOOLS/LIST RESPONSE (WITH FULL INPUTSCHEMA)

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "tools": [
      {
        "name": "get_weather",
        "description": "Get current weather for any city.",
        "inputSchema": {          // Standard JSON Schema object
          "type": "object",
          "properties": {
            "city": {
              "type": "string",
              "description": "City name, e.g. Belfast"
            },
            "unit": {
              "type": "string",
              "enum": ["celsius", "fahrenheit"],   // Constrained to specific values
              "default": "celsius"
            }
          },
          "required": ["city"]            // "unit" is optional, "city" is not
        }
      },
      {
        "name": "search_database",
        "description": "Search products by keyword with optional filters.",
        "inputSchema": {
          "type": "object",
          "properties": {
            "query": { "type": "string" },
            "limit": {
              "type": "integer",
              "minimum": 1,
              "maximum": 100,
              "default": 10
            },
            "tags": {
              "type": "array",              // Arrays of primitives are fine
              "items": { "type": "string" }
            }
          },
          "required": ["query"]
        }
      }
    ]
  }
}

The inputSchema is standard JSON Schema. You can use type, enum, minimum, maximum, default, array, nested object — any JSON Schema feature. The AI model reads this schema and uses it to build valid arguments automatically. Good schemas = fewer mistakes by the AI.

Phase 3 — Tool Execution: tools/call

The user says something. The AI decides a tool is needed. It fires a tools/callwith the arguments it constructed from the schema.

3A. SIMPLE CALL — ONE STRING ARGUMENT

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "get_weather",
    "arguments": {
      "city": "Belfast"           // Only the required field; "unit" uses its default
    }
  }
}

3B. COMPLEX CALL — MULTIPLE TYPES INCLUDING ARRAY

{
  "jsonrpc": "2.0",
  "id": 3,
  "method": "tools/call",
  "params": {
    "name": "search_database",
    "arguments": {
      "query": "wireless headphones",
      "limit": 5,                  // integer
      "tags": ["bluetooth", "noise-cancelling"]  // array
    }
  }
}

3C. SUCCESS RESPONSE — TEXT CONTENT

{
  "jsonrpc": "2.0",
  "id": 2,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "Current weather in Belfast: 12°C, wind 18 km/h"
      }
    ],
    "isError": false          // Optional. Explicitly marks success.
  }
}

3D. SUCCESS RESPONSE — MULTIPLE CONTENT BLOCKS

{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "Found 2 results for 'wireless headphones':"
      },
      {
        "type": "text",         // Second block — same message, more content
        "text": "1. Sony WH-1000XM5 — £279\n2. Bose QC45 — £249"
      }
      // Could also include "image" blocks with base64 data,
      // or "resource" blocks referencing URIs
    ]
  }
}

Phase 4 — When Things Go Wrong: Error Responses

There are two distinct flavours of error in MCP. Don’t mix them up — they mean very different things.

PROTOCOL ERROR — THE REQUEST ITSELF WAS MALFORMED

{
  "jsonrpc": "2.0",
  "id": 4,
  "error": {                         // "error" field, NOT "result"
    "code": -32602,                  // Standard JSON-RPC error code
    "message": "Invalid params",
    "data": {
      "details": "Required field 'city' is missing"
    }
  }
}

TOOL-LEVEL ERROR — THE TOOL RAN BUT THE OPERATION FAILED

{
  "jsonrpc": "2.0",
  "id": 5,
  "result": {                        // Still "result", NOT "error" !
    "content": [
      {
        "type": "text",
        "text": "City 'Narnia' not found in geocoding database."
      }
    ],
    "isError": true               // Flag tells the AI: tool ran, but reported failure
  }
}

Key distinction: A protocol error (using the “error” field) means the communication broke — bad JSON, unknown method, missing required fields. A tool error (using “result” with “isError”: true) means the tool ran fine but the operation itself failed — user not found, city doesn’t exist, API returned 404. Always use isError: true for business logic failures so the AI can read the message and react intelligently.

Bonus: Resource Reads and Notifications

Tools aren’t the only thing MCP can do. Here’s what a resource read looks like — useful when you want to expose file contents or database records the AI can browse:

RESOURCES/READ REQUEST AND RESPONSE

// Request
{
  "jsonrpc": "2.0",
  "id": 6,
  "method": "resources/read",
  "params": {
    "uri": "file:///home/user/notes.txt"    // Resource URIs identify the data source
  }
}
// Response
{
  "jsonrpc": "2.0",
  "id": 6,
  "result": {
    "contents": [
      {
        "uri": "file:///home/user/notes.txt",
        "mimeType": "text/plain",
        "text": "Buy milk. Deploy the thing. Don't forget stand-up."
      }
    ]
  }
}

And a server notification — fire-and-forget messages the server sends without being asked (no id, no reply expected):

SERVER → HOST NOTIFICATION (NO REPLY)

{
  "jsonrpc": "2.0",
  "method": "notifications/tools/list_changed",
  // No "id" — this is a push notification.
  // The server is telling the host: "Hey, my tool list changed.
  //  Please call tools/list again to refresh."
}

Clean and readable. No magic, no wizardry. Just JSON going back and forth. This predictability is exactly what makes MCP powerful — any language, any framework, any tool can implement it.

Two Ways to Connect: stdio vs HTTP/SSE

MCP servers can communicate via two transport mechanisms:

stdio (Standard I/O) — The simplest option. Your server reads from stdin and writes to stdout. The host app spawns the server as a subprocess. Perfect for local tools running on the same machine.

HTTP + SSE (Server-Sent Events) — For remote servers. The client sends POST requests to the server, and the server streams events back. Use this when your MCP server lives on a different machine (a cloud server, a colleague’s computer, etc.).

Start with stdio. It’s simpler, requires no networking setup, and is perfect for local tooling. You can always switch to HTTP later.

Tutorial: Building server.py Step by Step

Enough theory. Let’s write the actual code. Below is a complete, annotated server.py that exposes three tools: a dad joke generator, a task list, and a real live weather lookup. We’ll build it section by section so nothing is a mystery.

📦Install first: pip install mcp — that’s the only dependency. Python 3.10+ required.

Part 1 — Imports & Server Instantiation

The MCP SDK ships everything you need in the mcp package. We create a Server instance and tell it our name and version — this is what Claude Desktop sees in the handshake.

server.py — Part 1: imports and setup

# Standard library — no extra installs needed for HTTP calls
import asyncio
import random
import json
import urllib.request
import urllib.parse
import sys                   # needed for stderr debug output
# MCP SDK - install with: pip install mcp
from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp import types
# Create the server. The name/version appear during the MCP handshake.
# Claude Desktop displays this name in its connected servers list.
app = Server("my-first-mcp-server")

Part 2 — Declaring Tools with @app.list_tools()

This decorator registers the handler that responds to tools/list requests. The AI calls this automatically on startup to discover your full menu of capabilities. The descriptions you write here are read by the AI — write them like you’re explaining the tool to a smart colleague, not a compiler.

server.py — Part 2: tool declarations

@app.list_tools()
async def list_tools() -> list[types.Tool]:
    return [
        # ── Tool 1: dad_joke ──────────────────────────────────────
        # No arguments. The inputSchema is an empty object.
        # The AI will call this whenever the user needs cheering up.
        types.Tool(
            name="dad_joke",
            description="Returns a random, gloriously terrible dad joke. Use when morale is low.",
            inputSchema={
                "type": "object",
                "properties": {},    # no arguments - jokes are free
                "required": [],
            },
        ),
        # ── Tool 2: get_tasks ─────────────────────────────────────
        # One required string argument: username.
        # "required" list enforces that the AI must supply it.
        types.Tool(
            name="get_tasks",
            description="Returns the task list for a given user.",
            inputSchema={
                "type": "object",
                "properties": {
                    "username": {
                        "type": "string",
                        "description": "The username whose tasks to fetch, e.g. 'alice'",
                    },
                },
                "required": ["username"],
            },
        ),
        # ── Tool 3: get_weather ───────────────────────────────────
        # Calls the real Open-Meteo API. No API key required.
        # "unit" is optional - has a default value in the handler.
        types.Tool(
            name="get_weather",
            description="Get current weather for any city. Uses Open-Meteo (free, no API key needed).",
            inputSchema={
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "City name, e.g. Belfast, Tokyo, New York",
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit. Defaults to celsius.",
                    },
                },
                "required": ["city"],   # "unit" is optional - omit from required
            },
        ),
    ]

Part 3 — Handling Tool Calls with @app.call_tool()

When the AI decides to invoke a tool, this handler fires. It receives the tool name and the arguments dict the AI built from your schema. Return a list of TextContent objects — even if there’s only one. Always return a list; the protocol expects it.

server.py — Part 3: tool logic

@app.call_tool()
async def call_tool(name: str, arguments: dict) -> list[types.TextContent]:
# ── dad_joke ──────────────────────────────────────────────
    if name == "dad_joke":
        jokes = [
            "Why don't scientists trust atoms? Because they make up everything.",
            "I told my wife she was drawing her eyebrows too high. She looked surprised.",
            "Why do programmers prefer dark mode? Because light attracts bugs.",
            "I asked my dog what two minus two is. He said nothing.",
            "Why did the MCP server go to therapy? Too many unhandled exceptions.",
        ]
        return [types.TextContent(type="text", text=random.choice(jokes))]
    # ── get_tasks ─────────────────────────────────────────────
    # In real life you'd query a DB here. We use a dict for clarity.
    if name == "get_tasks":
        username = arguments.get("username", "").lower()
        fake_tasks = {
            "alice": ["Review PR #42", "Update docs", "Fix that bug you've been ignoring"],
            "bob":   ["Stand-up meeting", "Drink coffee", "Question all life choices"],
        }
        tasks = fake_tasks.get(username, ["No tasks found. Either very productive or very lazy."])
        formatted = "\n".join(f"{i+1}. {t}" for i, t in enumerate(tasks))
        return [types.TextContent(type="text", text=f"Tasks for {username}:\n{formatted}")]
    # ── get_weather ───────────────────────────────────────────
    # Two HTTP calls: geocode the city name → get weather by lat/lon.
    # Uses urllib (stdlib) so no extra dependencies.
    if name == "get_weather":
        city = arguments.get("city", "")
        unit = arguments.get("unit", "celsius")   # default if AI omits it
        # Step 1 - turn city name into coordinates
        geo_url = (
            "https://geocoding-api.open-meteo.com/v1/search"
            f"?name={urllib.parse.quote(city)}&count=1"
        )
        with urllib.request.urlopen(geo_url) as r:
            geo = json.loads(r.read())
        if not geo.get("results"):
            # Tool ran fine, but the operation failed - use isError flag
            return [types.TextContent(type="text", text=f'City "{city}" not found.')]
        result = geo["results"][0]
        lat, lon, found_name = result["latitude"], result["longitude"], result["name"]
        # Step 2 - fetch current weather for those coordinates
        wx_url = (
            "https://api.open-meteo.com/v1/forecast"
            f"?latitude={lat}&longitude={lon}"
            f"¤t_weather=true&temperature_unit={unit}"
        )
        with urllib.request.urlopen(wx_url) as r:
            wx = json.loads(r.read())
        cw = wx["current_weather"]
        symbol = "°C" if unit == "celsius" else "°F"
        return [types.TextContent(            type="text",            text=f"{found_name}: {cw['temperature']}{symbol}, wind {cw['windspeed']} km/h",        )]
    # Unknown tool - raise so MCP returns a proper protocol error
    raise ValueError(f"Unknown tool: {name}")

Part 4 — Wiring Up stdio Transport

The final piece connects the server to the MCP host via stdin/stdout. stdio_server() returns two async streams; we hand them to app.run() and it takes care of the rest. The key thing to remember: never print() to stdout anywhere in your code — that channel belongs to MCP. Debug with sys.stderr only.

server.py — Part 4: entry point

async def main():
    # stdio_server() yields (read_stream, write_stream) for the MCP host.
    # Everything inside this block is the server's lifetime.
    async with stdio_server() as (read_stream, write_stream):
        print("MCP server started", file=sys.stderr)   # stderr is safe for debug
        await app.run(
            read_stream,
            write_stream,
            app.create_initialization_options(),
        )
if __name__ == "__main__":
    asyncio.run(main())

💡Why async? MCP uses async I/O because servers often wait on multiple things at once — the host, network calls, file reads. Python’s asyncio lets a single thread handle all of that without blocking. You don’t need to understand the internals; just know that every handler must be async def, and use await before any I/O call.

Connecting Your Server to Claude Desktop

Once you’ve written your server.py, plugging it into Claude Desktop takes two minutes. Download Claude Desktop from claude.ai/download — it’s free and has an MCP client built in. Then locate the config file for your OS:

Register your server by adding it to that file:

CLAUDE_DESKTOP_CONFIG.JSON

{
  "mcpServers": {
    "my-first-mcp-server": {
      "command": "python",
      "args": [
        "/absolute/path/to/my-first-mcp-server/server.py"
      ],
      "env": {
        "MY_API_KEY": "sk-..."    // Pass secrets here; read with os.environ in Python
      }
    }
  }
}

Important: Use the absolute path to your server.py file. No tildes, no relative paths. Get the path with pwd on Mac/Linux or cd on Windows, and paste the full thing in.

Fully quit Claude Desktop (not just close the window) and reopen it. Look for the 🔧 tools icon in the chat input — that means MCP is connected. Then just ask Claude naturally: “Tell me a dad joke” or “What tasks does Alice have?” — it figures out on its own when to call your tools. No special syntax needed.

Best Practices (a.k.a. Mistakes I’m Saving You From)

Write excellent descriptions

The AI reads your tool descriptions to decide when to use them. Vague descriptions = confused AI = bad results. Be specific: what does the tool do, when should it be used, what do the arguments mean?

Never print() to stdout for debug output

In stdio transport, stdout is the MCP communication channel. Any stray print() call will corrupt the protocol stream. Always write debug output to stderr: print(“debug”, file=sys.stderr).

Validate your inputs

The AI sometimes passes unexpected values. Check that required arguments exist. Validate formats. Return helpful error messages rather than cryptic crashes.

Keep tools focused

One tool, one job. Don’t build a do_everything() tool. Small, well-named tools are easier for the AI to reason about and call correctly. Think Unix philosophy: do one thing well.

Use environment variables for secrets

Never hardcode API keys. Pass them via the env key in your claude_desktop_config.json: “env”: { “API_KEY”: “your-key-here” }. Read with os.environ[“API_KEY”] in your server.

The MCP Ecosystem: You Don’t Have to Build Everything

Here’s some great news: the open-source community has already built hundreds of MCP servers you can just plug in and use. Some highlights:

Find these and more at github.com/modelcontextprotocol/servers — the official repo — and on mcp.so, a community directory. There’s a good chance what you need already exists.

Wrapping Up: Why This Matters

MCP isn’t just a neat technical standard. It represents a fundamental shift in what AI assistants actually are.

Before MCP: AI as an oracle. You ask, it tells. Great for knowledge questions. Useless for action.

After MCP: AI as an agent. You describe what you want, it figures out which tools to use, calls them, interprets the results, and gets the job done. The AI becomes an active participant rather than a fancy search engine.

The future isn’t AI that can describe how to fix your code. It’s AI that fixes your code, runs the tests, creates the PR, and messages your team. MCP is the foundation that makes all of that possible.

You’ve now seen the full picture: the history, the architecture, the wire format, and a working server from scratch. The next step is yours. Pick a tool you use daily — a database, an API, a local service — and wrap it in an MCP server. Once you’ve done it once, you’ll find excuses to do it for everything.

Happy building. 🚀

The Agentic Loop (How Claude Actually Thinks)

Before anything else, you need to understand what makes Claude Code different from your IDE’s autocomplete. Autocomplete guesses the next word. Claude Code runs a loop — it gathers context, takes action, verifies results, then repeats until the task is done. Think of it less like a text editor feature and more like a developer who works asynchronously while you make coffee.

The loop has three phases that blend together depending on the task:

The loop adapts to what you ask. A question might only hit phase 1. A bug fix cycles through all three phases multiple times. A full feature might chain dozens of tool calls before Claude surfaces for air. You’re part of the loop too — you can interrupt at any point with Ctrl+C and steer it in a different direction.

THE HONEST SUMMARY

Claude Code is basically a language model with access to your entire computer strapped to it. It can read everything, run anything you could run from a terminal, and it just keeps going until it either finishes the task or you tell it to stop. This is either exhilarating or terrifying depending on how much you trust your test coverage.

Sessions & How Claude Remembers (Or Doesn’t)

Here’s the thing about Claude: every session starts with a completely blank memory. No recollection of last week’s chat. No memory of that convention you mentioned. Tabula rasa. Every. Single. Time. If you don’t explicitly persist knowledge, it’s gone when the session closes.

This isn’t a bug — it’s how LLMs work. But Claude Code gives you two mechanisms to work around it:

How a session actually starts

When you run claude in a directory, here’s what happens in the first few seconds before you’ve typed anything:

1. **Settings load (all scopes)**Global ~/.claude/settings.json → project .claude/settings.json → local .claude/settings.local.json. Scalar settings use the most specific value; array settings (like permissions) merge across all scopes.
2. MCP servers connectAll configured MCP servers from .mcp.json and ~/.claude.json start up. Tool schemas are deferred and load on demand. If a server fails to connect, Claude continues without it — it just loses those tools.
3. CLAUDE.md files load into contextOrg-level → user-level (~/.claude/CLAUDE.md) → project-level (./CLAUDE.md) → local (./CLAUDE.local.md). Plus your ~/.claude/MEMORY.md auto memory. All of this costs context window tokens immediately.
4. Plugins activateAll enabled plugins load their skills, hooks, MCP servers, and settings. Run /reload-plugins to activate changes mid-session.
5. Session is readyYou see the prompt. Everything above has already happened. Claude now has your project context and is ready to work.

CONTEXT WINDOW BUDGET

Everything loaded at startup costs tokens before you type a single character. A bloated CLAUDE.md, a dozen MCP servers, and a large MEMORY.md can eat a significant chunk of your context window. Run /status to see current token usage. Use /compact when a session gets long.

The .claude Directory — Every Folder Explained

The .claude/ folder is Claude Code’s configuration home in your project. There’s also ~/.claude/ in your home directory for global settings. Here’s the complete layout with every file and folder you’ll actually encounter:

COMMIT STRATEGY

Everything marked committed should go in Git so your whole team gets the same Claude setup. Everything marked gitignored is personal — Claude automatically adds settings.local.json to your global Git ignore. Add CLAUDE.local.md to your project’s .gitignore manually.

CLAUDE.md — The Full Story

You’ve heard about CLAUDE.md. Everyone mentions it. But few people explain the entire system behind it, which has four separate scope levels that load in a specific order:

They load in that order — managed first, local last. More specific scopes appear later in the context window, giving them higher effective priority.

What to actually put in CLAUDE.md

The golden rule: add something to CLAUDE.md when Claude makes the same mistake a second time. Here’s a battle-tested example for a Spring Boot project:

SIZE MATTERS

Target under 200 lines per CLAUDE.md. Longer files still load fully but adherence drops. If it’s getting long, split topic-specific rules into .claude/rules/ files and skills for complex procedures — they load conditionally and save your context window.

Auto Memory (MEMORY.md)

This is the magical part most people miss. Claude can write its own notes — automatically. When you correct Claude, it optionally remembers that correction for next session in ~/.claude/MEMORY.md. The first 200 lines or 25KB load at the start of every session.

Enable it with /memory in session or toggle it in settings. Then: tell Claude something once, it remembers it forever (until you delete the entry). This is how you get a Claude that feels like it’s been on your team for months.

settings.json — The Enforcement Layer

There’s a critical distinction most people miss: CLAUDE.md is guidance Claude reads.settings.json is configuration Claude Code enforces. One is a polite suggestion. The other is a locked door.

Settings live in .claude/settings.json (project/team) and .claude/settings.local.json(personal overrides). Here’s what you can configure:

ALLOW VS DENY PRECEDENCE

Deny rules always win over allow rules. If you allow Bash(kubectl *) and deny Bash(kubectl delete *), the delete command is blocked. Patterns support wildcards — Bash(npm test *)matches any command starting with npm test.

rules/ — Scoped Instructions That Load On Demand

Once your CLAUDE.md hits 200 lines, start splitting it into .claude/rules/. Rules are Markdown files that can load conditionally based on which files Claude is working with — only when it reads a file that matches the rule’s path pattern. This keeps your global context lean while ensuring Claude gets the right rules at the right time.

THE REAL POWER

A rule file without paths: frontmatter loads every session like CLAUDE.md. A rule WITH paths:only loads when Claude reads a matching file. This means your Kafka rules don’t eat context window during a CLAUDE.md update, and your migration rules only appear when Claude actually opens a migration file.

skills/ — Your Custom Slash Commands

Skills are reusable prompts you or Claude can invoke by name using /skill-name. Think of them as saved workflows — things you do repeatedly that are more than a one-liner but less than a custom tool. You write the instructions once in a SKILL.md, and invoke it whenever you need it.

Now you type /db-migrate and Claude walks through all seven steps. No more explaining the process from scratch every time. Skills also accept arguments via $ARGUMENTS:

Usage: /review-pr 127 — Claude reviews MR #127 with your exact conventions baked in.

SKILLS VS COMMANDS

You may see references to .claude/commands/ in older docs — this is the same mechanism as skills. For new workflows, use skills/ instead: same /name invocation, plus you can bundle supporting files alongside the SKILL.md.

Hooks — The Automation Engine

Hooks are shell commands that fire automatically at specific points in Claude’s lifecycle. They’re deterministic — they always run when triggered, regardless of what Claude decides. This is the difference between “please format the file” (Claude might forget) and “PostToolUse hook runs Spotless” (it definitely happens).

Available hook events

PreToolUseFires before any tool call. Can block execution (exit 2). Use to protect files or validate commands.

PostToolUseFires after a tool completes. Perfect for auto-formatting files after Claude edits them.

NotificationFires when Claude needs your input. Desktop notification so you don’t stare at the terminal.

SessionStartFires when a session begins. Inject additional context, set environment, run setup scripts.

PreCompactFires before conversation compaction. Re-inject context that would otherwise be lost.

StopFires when Claude finishes a response. Useful for post-response validation or logging.

Practical examples for a Java/Spring Boot project

EXIT CODES MATTER

exit 0 = allow. exit 1 = warn Claude but let it continue. exit 2 = hard block — Claude gets told why and has to try something else. It’s the digital equivalent of a bouncer who also explains the dress code.

MCP — Connecting Claude to the World

Model Context Protocol (MCP) is an open standard for connecting AI to external tools. Without MCP, Claude can only see your filesystem and run terminal commands. With MCP, Claude can query your Jira board, check Sentry errors, read Slack messages, query your PostgreSQL database, and more — all without you copying and pasting anything.

THE MENTAL MODEL

MCP is how Claude gets a login to your tools. Without it: “here’s a pasted error from Sentry.” With it: “Claude, check what errors happened in prod in the last hour and correlate them with my recent commits.”

Three types of MCP servers

Project-scoped MCP (.mcp.json)

Put your team’s MCP servers in .mcp.json at the project root. This file gets committed to Git so everyone on the team gets the same tools:

User-scoped MCP (personal tools)

For tools only you need (not your team), use the –scope user flag. This writes to ~/.claude.json instead of .mcp.json:

CHANNELS: MCP THAT PUSHES TO CLAUDE

Normal MCP is pull — Claude queries a server when it needs something. Channels flip this: an MCP server pushes messages into your running session. This means Claude can react to CI results, monitoring alerts, or Slack messages while you’re doing something else. Enable with –channelsflag on startup.

Plugins — Package & Share Everything

Everything you’ve built — skills, hooks, MCP servers, rules — can be packaged into a plugin and shared with your team, your org, or the whole world. Plugins are just directories with a .claude-plugin/plugin.json manifest. They’re the difference between “config that lives in one project” and “config that lives everywhere.”

Standalone config vs plugins

Plugin structure

Installing a plugin

THE REAL VALUE

Your platform team builds a plugin once with all the Spring Boot conventions, Kafka rules, code review skills, and MCP servers your org uses. Every developer installs it once. Every project gets the same setup. No more “did you add the Flyway rules to your CLAUDE.md?” Slack messages.

Subagents — Claude Spawning Claude

This is the part where things get genuinely science-fiction. Subagents let Claude spin up other Claude instances to handle subtasks in parallel or in isolated contexts. The main Claude orchestrates; the subagents execute. Think distributed computing, but the workers are also AI and can also spawn more workers.

Subagents can maintain their own MEMORY.md too. Imagine a “database specialist” subagent that remembers every schema change it’s ever helped with. That’s the direction this is heading.

WHEN TO USE SUBAGENTS

For most day-to-day tasks, you don’t need subagents. They’re most useful when you have truly parallel workstreams (implement feature A and feature B simultaneously without conflicts), when you need a clean context for a specific task, or when you’re building automation pipelines that run without human interaction.

Complete Quick Reference

• .claude/settings.json — permissions + hooks + model config. Enforced, not just read.
• CLAUDE.md — persistent instructions. Team version in ./CLAUDE.md, personal in ~/.claude/CLAUDE.md.
• MEMORY.md — auto-written by Claude. Persists learnings across sessions (200 lines / 25KB limit).
• .claude/rules/ — conditional instruction files. Use paths: frontmatter to load only for relevant files.
• .claude/skills/ — custom /commands. Each skill = a folder with SKILL.md. Accepts $ARGUMENTS.
• hooks — deterministic automation. PreToolUse (block/allow), PostToolUse (format), Notification (alert).
• .mcp.json — team MCP servers (committed). ~/.claude.json for personal servers.
• Plugins — packaged, shareable bundles of skills + hooks + MCP + settings. Install with /plugin install.
• Subagents — parallel Claude instances for complex multi-track workflows. Three isolation levels.
• /init — always run first in a new project. Generates CLAUDE.md automatically.
• /compact — compresses long sessions. /clear = fresh start (CLAUDE.md reloads).
• /mcp — shows server status + tool count. /hooks — shows configured hooks. /memory — edit CLAUDE.md.

🤔 What Even IS Claude Code?

Okay so you’ve heard of GitHub Copilot, right? Claude Code is like that, except instead of whispering suggestions into your IDE like a shy intern, it’s a full-blown agentic assistant that lives in your terminal and acts like a senior developer who has had three espressos and is ready to ship.

It can read your entire codebase, plan multi-step changes, edit multiple files, run your tests, and even commit code. All from the command line. No IDE plugin. No browser tab. Just you, your terminal, and your new AI best friend who never judges you for that 200-line method you wrote at 2am.

🧠 Fun fact

Claude Code holds up to 200,000 tokens of context in memory. That’s your entire project, your architecture docs, and probably your unfinished README — all at once. Copilot gets about 8,000. Do the math.

The key difference from other AI tools: Claude Code is agentic. It doesn’t just suggest — it does. Ask it to “add authentication to the user service” and it’ll actually read your code, figure out what you’re using, and make it happen. It’s the difference between a GPS giving you directions and a GPS that also drives the car.

“I used to Google ‘how to reverse a string in Java’ every single time. Now Claude just does it for me. My Google history has never been cleaner.”

💳 Prerequisites (Yes, You Need to Pay)

Sorry, no free lunch here. But the good news is the cheapest plan already includes everything you need to get started:

⚠️ Heads up

If you run a lot of Claude Code sessions simultaneously, you’ll burn through your rate limits faster than a frontend dev burning through console.log() statements. Claude Max ($100–200/month) exists for that use case.

⚙️ Installation (The Easy Part, We Promise)

One line. That’s it. No npm, no Homebrew tap, no reading a 47-step Medium article. Just paste this and go make a coffee while it installs:

macOS / Linux

# This one line is doing more work than you did all morning
curl -fsSL https://claude.ai/install.sh | bash

# Make sure it worked
claude --version

Windows (PowerShell)

# For our Windows friends. No judgment. (Okay, maybe a little.)
irm https://claude.ai/install.ps1 | iex

Homebrew (macOS, if you’re fancy)

brew install --cask claude-code

🚫 Don’t do this

The old way — npm install -g @anthropic-ai/claude-code — is officially deprecated. If you used it before, run claude install to migrate. Your npm-installed past is a mistake you can fix.

First launch: authenticate

Run claude in any directory. A browser window opens, you log in with your Claude account, and boom — you’re in. If you’re behind a corporate firewall that blocks OAuth callbacks (classic), use an API key via the ANTHROPIC_API_KEY environment variable instead.

💡 Pro tip

If it fails to authenticate, check your .claudeignore isn’t too aggressive. Actually wait — you don’t have one yet. Never mind, you’re fine. Keep going.

🐣 Your First Session (Don’t Panic)

Navigate to your project and type claude. You’ll see the Claude Code prompt. Here’s the crucial part: don’t just immediately say “build me a feature.” That’s like hiring a new developer and skipping the entire onboarding process. Claude needs context. Give it a tour first.

1. Run /init— Let Claude scan your projectThis generates a starter CLAUDE.md file. Claude reads your directory structure, sniffs out your build system and test framework, and writes a first draft. It’s like Claude doing its own onboarding. Adorable and useful.
2. Ask Claude to explain things before touching themTry: “Explain the architecture of this project.” or “How does authentication work here?” Claude builds a mental model of your code. You’ll get better results when it actually understands what it’s working with.
3. Enable Plan Mode with Shift+Tab before any changeClaude will describe exactly what it intends to do before touching a single file. Think of it as asking a surgeon to show you the plan before they start cutting. Highly recommended.
4. Review → Approve → Commit → RepeatRead the diff. Run your tests. Commit small. If Claude did something weird, a small commit is easy to revert. “I deleted the database” is much funnier as a story than an actual incident.

✨ The CLAUDE.md File (Where the Magic Lives)

This is the most important thing in this whole guide. Read it twice.

CLAUDE.md is a Markdown file you put in your project root. Claude reads it automatically at the start of every single session. It’s like a briefing document for your AI colleague — except unlike your actual team’s onboarding docs, people actually read this one.

Without it, Claude starts every session with zero context about your project, your conventions, your stack, your rules. With it, Claude starts every session knowing exactly how you work.

🎯 The one-sentence summary: CLAUDE.md is your project’s brain dump, and Claude will actually remember and use everything you put in it — unlike that Wiki your team hasn’t touched since 2022.

A real CLAUDE.md for a Spring Boot project

Here’s a solid example for a Java/Spring Boot/PostgreSQL stack:

# CLAUDE.MD
## Project
REST API for customer order management. Java 17, Spring Boot 3,
PostgreSQL 15, deployed on AWS EKS via Helm + ArgoCD.

## Commands
- Build:   `./mvnw clean package -DskipTests`
- Test:    `./mvnw test`
- Run:     `./mvnw spring-boot:run`
- Lint:    `./mvnw checkstyle:check`
## Architecture
- Layered: Controller → Service → Repository (Spring Data JPA)
- Async events via Spring ApplicationEventPublisher
- DB migrations: Flyway (src/main/resources/db/migration)
- All config via environment variables. No hardcoded secrets. Ever.
## Conventions
- Constructor injection only. @Autowired on fields is a war crime.
- All endpoints return ResponseEntity with standard error envelope
- Unit tests: JUnit 5 + Mockito. Integration tests: @SpringBootTest + Testcontainers
- Commits: imperative mood, max 72 chars ("Add order validation", not "Added")
## Hard Rules
- Never modify existing Flyway migration files. Never.
- Never commit .env or application-local.properties
- No wildcard imports

💡 Pro tip

Commit your CLAUDE.md to Git. Every teammate then gets the same Claude setup automatically. No more Claude giving one dev constructor injection advice and another dev @Autowired advice in the same codebase.

📝 CLAUDE.md Best Practices (What Actually Works)

Keep it under 100 lines

Seriously. For every line you add, ask: “Would removing this cause Claude to make a mistake?” If no, delete it. A fat CLAUDE.md with 500 lines of noise is worse than a lean one with 60 lines of signal. Less is more. (Yes, the same rule applies to your commit messages.)

Use the WHY / WHAT / HOW structure

• WHY — One paragraph on what the project exists to do. Domain context matters. Claude giving advice on an e-commerce API vs a healthcare API should feel different.
• WHAT — Tech stack, key dependencies, deployment target. Be specific about versions.
• HOW — Exact commands. Exact conventions. Your team’s particular quirks. The stuff Claude can’t infer from the code alone.

✅ Good things to put in CLAUDE.md

• Build, test, lint, run commands — saves Claude from guessing and running the wrong thing
• Architecture summary — layers, patterns, where important things live
• Non-obvious rules — things that would surprise a new dev (“never modify migration files”)
• Testing philosophy — unit vs integration, which frameworks, naming conventions
• Things that have burned you before — sprinkle your trauma in there

🚫 Things NOT to put in CLAUDE.md

• Code style rules your linter already enforces — don’t tell Claude twice what Checkstyle already screams about
• Secrets, passwords, API keys — you know this, but we’re saying it anyway
• Entire config files pasted in — link or summarise, don’t paste 200 lines of XML
• Your personal frustrations about legacy code — Claude doesn’t need your trauma dump

Bonus: Sub-directory CLAUDE.md for big projects

Monorepo? Complex project with distinct modules? Put a CLAUDE.md inside subdirectories too. Claude reads and merges them — the root file is the global rules, the subdirectory file adds module-specific context. Like a company handbook + a team-specific playbook.

# SRC/NOTIFICATION-SERVICE/CLAUDE.MD
## Notification Service
Handles email + push via AWS SES and SNS.
Templates: src/main/resources/templates (Thymeleaf).

## Module Rules
- All outbound calls MUST go through NotificationGateway
- Do not call SES/SNS directly from service classes
- Retry logic: Spring Retry handles it. Don't add your own loops.

🙈 The .claudeignore File

Like .gitignore but for Claude. It tells Claude which files to skip so it doesn’t waste context on your target/ folder and doesn’t accidentally read your .env file out loud during pair programming.

# .CLAUDEIGNORE# Build output — Claude doesn't need to read your compiled bytecode
target/
build/
dist/

# Secrets - for the love of all that is holy
.env
.env.local
application-local.properties
*.pem
*.key
# Dependencies - it knows what Spring Boot is, trust it
node_modules/
.gradle/
# Generated code (if you don't want Claude accidentally "fixing" it)
src/generated/

🚫 Classic mistake

Adding something too broad like *.java to .claudeignore. Now Claude can’t read any of your source files. It will be very polite about it but also completely useless. Be specific with your ignore patterns.

🛡️ Plan Mode (Look Before You Leap)

Plan Mode is what separates “Claude deleted half my project” stories from “Claude is amazing” stories. Before making any changes, Claude tells you exactly what it plans to do: which files it’ll touch, what changes it’ll make, and why. You say yes or no. Then it acts.

⌨️ Just press Shift + Tab to toggle it on. That’s it. Best keyboard shortcut you’ll use today.

😌 Wisdom

As a beginner, use Plan Mode for everything. As you build trust in Claude’s output quality (and your own CLAUDE.md), you can skip it for small, well-understood tasks. Think of it like asking your junior dev to write tests first vs. just letting your senior dev rip.

🔄 The Holy Workflow: Explore → Plan → Implement → Commit

This is Anthropic’s official recommended workflow. It sounds obvious but most beginners skip the first two steps, and that’s why they end up on Reddit asking why Claude rewrote their entire service layer.

1. 🔍 Explore — read before you writeAsk Claude to explain the relevant parts of your code before asking it to change anything. “How does order processing work in this project?” Claude builds a proper mental model. Future changes will actually fit your architecture.
2. 📋 Plan — agree on the approachEnable Plan Mode. Describe what you want. Push back on the plan if something seems wrong. This is the cheapest place to catch misunderstandings — before any code is written.
3. ⚙️ Implement — let Claude cookWith the plan approved, Claude implements. Watch the output. If something looks off, hit Ctrl+C. Don’t let it finish a 15-file change that you already know is wrong by file 3.
4. ✅ Commit — small and oftenReview the diff. Run tests. Commit. Small commits are your insurance policy. “Revert one small commit” is a two-second job. “Unravel three hours of AI-generated changes” is a Thursday afternoon you’ll never get back.

“I skipped the Explore step, asked Claude to ‘refactor auth’, and watched it replace my entire session management layer in 45 seconds. It was beautiful and terrifying in equal measure.”

⌨️ Commands Cheat Sheet

Type these inside an active Claude Code session (the ones starting with /):

CI/CD / Scripting mode

Claude Code has a --print flag for non-interactive use. Perfect for Jenkins pipelines, GitLab CI jobs, or any situation where you want Claude to run and exit without waiting for your input:

# Run a single command and exit — great for pipelines
claude --print "Summarise changes in the last 5 commits"

# Works great in your Jenkinsfile or .gitlab-ci.yml
claude --print "Check this changeset for obvious bugs" --no-color

💡 Bonus tip

The /clear command is your friend. When a conversation has gone on too long and Claude starts giving weird answers, clear it. CLAUDE.md re-loads automatically, so you lose the conversation but not the context. Fresh start, same configuration.

🎒 The “I’m Late to a Meeting” Summary

• Install with one curl command — no Node.js, no npm, no fuss
• You need a paid plan — Claude Pro at $20/mo is enough to start
• Run /init in every new project. Do it now. Do it always.
• CLAUDE.md = Claude’s briefing doc. 50–100 lines, WHY/WHAT/HOW structure
• Commit CLAUDE.md to Git — the whole team benefits, zero extra effort
• Always use Plan Mode (Shift+Tab) before non-trivial changes
• Workflow: Explore → Plan → Implement → Commit. In that order. Always.
• Use .claudeignore to keep secrets and build junk out of Claude’s sight
• Keep commits small — your future self will thank you
• Update your CLAUDE.md as your project grows — stale instructions cause chaos

I’ve reviewed a lot of Spring Boot codebases. Startups burning through their runway trying to scale. Enterprises with ten-year-old monoliths being carved into microservices. Teams of three and teams of fifty. Different industries, different sizes, different levels of experience.

The mistakes are almost always the same.

That’s not an insult — it’s actually the interesting part. These aren’t beginner errors. Most of the teams I work with are smart, experienced engineers who made reasonable decisions under pressure. The problem is that Spring Boot is so productive, so frictionless, that bad patterns compound quietly. You ship fast. The tests pass. Production looks fine. And then six months later you’re staring at a 2,000-line controller and wondering how it got there.

This article is about the five patterns I see most often, why they happen, and — more importantly — how to fix them without rewriting everything from scratch.

Mistake 1: Putting business logic in controllers

This is the most common one by a significant margin. It usually starts innocently: a small validation check here, a conditional there, a bit of data transformation before calling a service. Before long, your @RestController is making decisions it has no business making.

Here’s what it looks like in the wild:

@PostMapping("/orders")
public ResponseEntity<OrderResponse> createOrder(@RequestBody OrderRequest request) {
    if (request.getItems().isEmpty()) {
        return ResponseEntity.badRequest().build();
    }

    BigDecimal total = request.getItems().stream()
        .map(item -> item.getPrice().multiply(BigDecimal.valueOf(item.getQuantity())))
        .reduce(BigDecimal.ZERO, BigDecimal::add);
    if (total.compareTo(new BigDecimal("10000")) > 0 && !userService.isPremium(request.getUserId())) {
        return ResponseEntity.status(HttpStatus.FORBIDDEN).build();
    }
    // ... and it keeps going
}

The controller now knows about pricing rules, user tiers, and order validation. None of that belongs here.

Why it happens: Controllers are the entry point, so they feel like the natural place to “handle” things. Also, injecting a service and calling one method feels like overkill for “just a quick check.”

Why it matters: Business logic in controllers can’t be unit tested without spinning up the web layer. It can’t be reused from a message consumer, a scheduled job, or a CLI command. It’s invisible to anyone who looks at your service layer expecting to understand the domain.

The fix: Your controller should do exactly three things — deserialise the request, call a service method, and serialise the response. Every conditional, every calculation, every rule goes into a service or domain object. If your controller has an ifstatement that isn’t about HTTP concerns, move it.

@PostMapping("/orders")
public ResponseEntity<OrderResponse> createOrder(@RequestBody OrderRequest request) {
    OrderResponse order = orderService.createOrder(request);
    return ResponseEntity.ok(order);
}

Boring. Perfect.

Mistake 2: Ignoring transaction boundaries

Transaction management is one of those topics where a little knowledge is genuinely dangerous. Most developers know @Transactional exists. Fewer understand where to put it, what it actually does, and — critically — what it doesn’t do.

The most painful version of this mistake looks like this:

@Service
public class PaymentService {
    @Transactional
    public void processPayment(Long orderId) {
        Order order = orderRepository.findById(orderId).orElseThrow();
        order.setStatus(OrderStatus.PAID);
        orderRepository.save(order);
        
        // This call is to an external HTTP service.
        // If it fails, the order status update is rolled back.
        // If it succeeds but the next line throws, you've charged
        // the customer but their order is still PENDING.
        paymentGateway.charge(order.getAmount());
        
        inventoryService.reserve(order.getItems()); // can throw
    }
}

The transaction boundary wraps an external HTTP call and a separate service call. The atomicity guarantee you think you have doesn’t exist across those boundaries.

Why it happens: @Transactional feels like a safety net you throw over anything that looks important. It’s easy to apply and invisible when it’s wrong.

Why it matters: Inconsistent data is the hardest class of bug to diagnose and fix in production. By the time you notice, the damage is already done.

The fix: Be deliberate about what your transaction actually protects. Keep transactions short, keep them within a single data source, and handle external calls outside the transactional boundary. Consider the outbox pattern for operations that need to be reliably paired with an external call. And always ask: “if this line throws, what state am I leaving the database in?”

Also watch out for the self-invocation trap — calling a @Transactional method from within the same class bypasses the proxy and silently does nothing. This one has burned almost every team I’ve worked with.

Mistake 3: Overusing @Autowired everywhere

Field injection is convenient. It’s also the option that makes your code harder to test, harder to reason about, and full of hidden dependencies you only discover at runtime.

@Service
public class ReportService {
    @Autowired
    private UserRepository userRepository;
    @Autowired
    private OrderRepository orderRepository;
    @Autowired
    private EmailService emailService;
    @Autowired
    private PdfGenerator pdfGenerator;
    @Autowired
    private AuditService auditService;
    ...
}

Five dependencies injected invisibly. You can’t instantiate this class in a test without a Spring context, or without reaching for @MockBean and a full application bootstrap. You also can’t see — at a glance — that this class might be doing too much.

Why it happens: Field injection is fewer lines of code and Spring supports it, so it feels fine. Early tutorials often show it this way and the habit sticks.

Why it matters: Constructor injection makes dependencies explicit and visible. It enables plain unit tests without a Spring context. It makes circular dependencies a compile-time error rather than a runtime surprise. And when a class has eight constructor parameters, that’s a signal — a service with eight dependencies probably has too many responsibilities.

The fix: Use constructor injection. Always. Since Spring 4.3 you don’t even need @Autowired on a single-constructor class. If you’re using Lombok, @RequiredArgsConstructor on final fields gives you this for free.

@Service
@RequiredArgsConstructor
public class ReportService {
    private final UserRepository userRepository;
    private final OrderRepository orderRepository;
    private final EmailService emailService;
    ...
}

If the constructor starts to look unwieldy, take that as the feedback it is.

Mistake 4: No proper exception handling strategy

Every codebase has exception handling. Very few have an exception handling strategy. The difference shows up as a patchwork of try-catch blocks scattered across service methods, inconsistent error responses from your API, and stack traces leaking into production responses.

Common signs you don’t have a strategy:

// In one service:
try {
    return userRepository.findById(id).orElseThrow();
} catch (Exception e) {
    log.error("Error", e);
    return null; // 🚩 silent failure
}

// In another:
if (user == null) throw new RuntimeException("User not found"); // 🚩 generic exception

// In a third:
} catch (UserNotFoundException e) {
    return ResponseEntity.status(500).body("Something went wrong"); // 🚩 wrong status code

Why it happens: Exception handling feels like plumbing — unglamorous work that gets deferred and then done inconsistently across a team over months or years.

Why it matters: Inconsistent error handling makes your API unpredictable for consumers, makes debugging painful, and creates security risk when internal details leak into responses.

The fix: Define a small hierarchy of domain exceptions (ResourceNotFoundException, ValidationException, ConflictException) and throw those deliberately rather than wrapping everything in RuntimeException. Then handle them all in one place using @RestControllerAdvice:

@RestControllerAdvice
public class GlobalExceptionHandler {
    @ExceptionHandler(ResourceNotFoundException.class)
    public ResponseEntity<ErrorResponse> handleNotFound(ResourceNotFoundException ex) {
        return ResponseEntity.status(HttpStatus.NOT_FOUND)
            .body(new ErrorResponse("NOT_FOUND", ex.getMessage()));
    }

    @ExceptionHandler(ValidationException.class)
    public ResponseEntity<ErrorResponse> handleValidation(ValidationException ex) {
        return ResponseEntity.status(HttpStatus.BAD_REQUEST)
            .body(new ErrorResponse("VALIDATION_ERROR", ex.getMessage()));
    }

    @ExceptionHandler(Exception.class)
    public ResponseEntity<ErrorResponse> handleUnexpected(Exception ex) {
        log.error("Unexpected error", ex);
        return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
            .body(new ErrorResponse("INTERNAL_ERROR", "An unexpected error occurred"));
    }
}

One place. Consistent responses. No stack traces in production. Done.

Mistake 5: Treating application.properties as a dumping ground

This one starts as a minor annoyance and grows into a genuine operational problem. A file that begins as twelve lines of database config becomes three hundred lines of undocumented magic numbers, environment-specific hacks, feature flags, and credentials that probably shouldn’t be there at all.

# What does this do? Who added it? Is it still needed?
some.service.timeout=5000
other.thing.enabled=true
legacy.mode=false
max.retry.count=3
feature.new-checkout=true
internal.api.key=sk_live_abc123...

Why it happens: application.properties is the path of least resistance for any config that doesn’t obviously belong somewhere else. Nobody wants to create a new @ConfigurationProperties class for one value.

Why it matters: At scale, unstructured configuration is an operational risk. Properties get duplicated across environments, credentials end up in version control, values change without anyone knowing what breaks, and onboarding a new developer becomes a treasure hunt.

The fix: Group related configuration using @ConfigurationProperties with typed, documented classes:

@ConfigurationProperties(prefix = "payment")
@Validated
public record PaymentProperties(
    @NotBlank String apiKey,
    @Min(1) @Max(30) int timeoutSeconds,
    @Min(0) int maxRetries
) {}

This gives you type safety, validation on startup, IDE autocomplete, and a single place to document what each value does. Secrets should come from environment variables or a secrets manager — never committed to a repository. And if a property hasn’t been touched in six months, it probably shouldn’t exist.

Closing: a quick checklist to self-audit your codebase

You don’t need to fix everything at once. Run through this in an hour and you’ll know exactly where to focus.

Controllers

• Do any controllers contain if statements unrelated to HTTP status?
• Are there calculations or data transformations happening before a service call?
• Is any business rule enforced at the controller level?

Transactions

• Are there @Transactional methods that call external HTTP services?
• Are there @Transactional methods calling other @Transactional methods on the same class?
• Do you know what state the database is in if any line in a transactional method throws?

Dependency injection

• Are you using field injection (@Autowired on fields)?
• Can you instantiate your service classes in a unit test without a Spring context?
• Are there any services with more than five or six dependencies?

Exception handling

• Do you have a @RestControllerAdvice that handles all exceptions in one place?
• Are you catching Exception in service methods and returning null or swallowing errors?
• Are stack traces or internal error messages reaching API responses in production?

Configuration

• Is application.properties longer than 50 lines with no grouping or comments?
• Are any secrets or API keys committed to version control?
• Are there properties whose purpose isn’t immediately obvious?

If you ticked more than three of those boxes, your codebase is normal — this is just what production software looks like after a few years of growth under pressure. The good news is that every one of these issues is fixable incrementally, without a rewrite, one pull request at a time.

How distributed teams structure the Software Development Life Cycle — governance models, communication contracts, phase-by-phase responsibilities, and the hidden failure modes that kill outsourced projects before launch.

What makes outsourced SDLC different

Every software team runs some version of the SDLC — a structured sequence of phases that takes an idea from concept to running code. When delivery is handled by an external vendor, the same phases exist, but the dynamics between them change completely.

In a co-located team, context flows informally: a developer can tap the product manager on the shoulder, an architect can sketch on a whiteboard in the hallway. In an outsourced engagement, none of that happens. Every shared understanding has to be written down, formally agreed, and deliberately transmitted. What a co-located team resolves in a five-minute hallway conversation requires an async message thread, a clarification document, and a sign-off in outsourcing.

This is not a disadvantage — it is a discipline. Outsourced teams that develop rigorous documentation habits often produce more traceable, better-specified software than co-located teams where “everyone just knows” how things work. The SDLC is the framework that makes this discipline systematic.

CORE PRINCIPLE

The client owns the what — product vision, business goals, acceptance criteria, and ultimate decision authority. The vendor owns the how — technical approach, tooling choices, implementation detail, and day-to-day engineering execution. The SDLC is the interface between them.

Figure 1 — The seven SDLC phases in outsourced delivery. Click any phase to explore it. The dashed arc represents the feedback loop from maintenance back into discovery for the next cycle.

The dual-ownership model

The most important structural concept in outsourced SDLC is dual ownership: two distinct parties hold responsibility for different layers of the project simultaneously. This is not a hierarchy — the client is not “above” the vendor — it is a division of accountability along a clear interface.

Confusion about ownership is the single most common root cause of outsourced project failure. When both parties believe they are responsible for the same thing (or each believes the other is responsible), gaps appear and deliverables fall through.

CLIENT SIDE

Product ownership layer

• Business requirements and user needs
• Acceptance criteria and Definition of Done
• Backlog prioritisation and scope decisions
• UAT sign-off and production approval
• Regulatory and compliance requirements
• Budget authority and contract governance

VENDOR SIDE

Engineering execution layer

• Technical architecture and tooling selection
• Sprint planning and capacity management
• Code quality, test coverage, and CI/CD
• Daily engineering decisions and trade-offs
• Infrastructure setup and monitoring
• Technical documentation and ADRs

SHARED

Interface & governance

• Sprint reviews and retrospectives
• Risk register and escalation path
• Change request evaluation and impact
• Definition of Ready for sprint entry
• Release decision and go/no-go gates

KEY ROLES

Who sits at the interface

• Client: Product Owner + Technical Lead
• Vendor: Project Manager + Tech Lead
• These four people own the interface
• All escalations route through them first

MOST COMMON FAILURE MODE

Projects fail when the client Product Owner is not genuinely available. If the PO attends only one ceremony per sprint, ambiguities accumulate, the vendor makes assumptions, and by Month 3 the product being built is not the product the client imagined. PO availability is a project risk, not a courtesy.

Phase 1 — discovery & planning

Discovery is where the project is defined before a line of code is written. In outsourcing, this phase is often compressed or skipped entirely — and it is the single biggest predictor of whether the engagement will succeed. A well-run discovery produces a shared understanding of what is being built, why it matters, what it does not include, and how success will be measured.

The outputs of discovery are not optional. They form the contractual baseline for the engagement. Any change to scope relative to the discovery outputs is a change request — a formal negotiation, not a verbal agreement.

Discovery outputs — the mandatory artifact set

• Project Charter — business goals, success metrics, constraints, and named stakeholders with decision authority
• High-level roadmap — time-boxed phases with milestone deliverables and payment tranches tied to each
• Scope boundary document — an explicit list of what is in scope and what is explicitly out of scope for this engagement
• Technical feasibility assessment — vendor’s initial evaluation of risk, unknowns, and proposed technology approach
• Communication protocol — meeting cadence, tools, async response SLAs (Service Level Agreement), and escalation path
• Definition of Done — the conditions every deliverable must satisfy before the client signs off

TIME INVESTMENT

For a six-month project, expect discovery to take two to four weeks and cost 5–8% of total budget. This is not overhead — it is insurance. Every hour spent in discovery typically saves three to five hours of rework later.

Figure 2 — Ownership swimlane across SDLC phases. Click any phase row to explore the ownership detail. Every phase ends with a formal handoff artifact — nothing advances without a signed-off output.

Phase 2 — requirements

Requirements gathering in outsourcing is a negotiation, not a dictation. The client provides the business intent; the vendor translates it into implementable specifications and raises every ambiguity in writing. The output — typically a Software Requirements Specification or a groomed product backlog with acceptance criteria — is the contract for the build phase.

The two most common failure modes at this phase are: the client providing requirements that are too high-level (“the user should have a good experience”), and the vendor accepting ambiguous requirements without escalating. Both failures are preventable with process discipline.

REQUIREMENTS QUALITY TEST

Every user story must answer three questions unambiguously: Who is the actor? What do they want to do? Why does it matter to the business? If any of the three is vague, the story is not ready for development. The vendor’s QA on requirements is as important as the client’s initial authorship.

The Requirements Clarification Log

Every ambiguity the vendor identifies must be logged in a shared document — the Requirements Clarification Log — with a status of Open, Answered, or Accepted. This log is reviewed at the start of every sprint planning session. Any story with open clarifications cannot be pulled into the sprint.

Phase 3 — system design

System design is where technical architecture is decided and documented before implementation begins. In outsourcing, this phase serves an additional function: it is the moment when the vendor demonstrates technical competence and the client gains confidence (or identifies concerns) in the approach before committing to the build.

The key outputs are Architecture Decision Records — short documents that capture each significant technical decision, the alternatives considered, and the rationale for the chosen approach. ADRs are owned by the vendor but approved by the client’s technical lead. They survive team turnover and serve as institutional memory for the lifetime of the codebase.

Figure 3 — System design phase: what goes in, what is decided, and what comes out. Every output requires client technical lead approval before development begins.

Phases 4–5 — development and testing

Development is typically the longest phase and the one where outsourcing relationships are most stress-tested. The vendor team is executing sprints; the client team is reviewing, testing, and making real-time decisions about scope. The quality of the interface between the two determines whether this phase runs smoothly or deteriorates into confusion and rework.

Testing in an outsourced context has two layers: the vendor is responsible for technical correctness (unit tests, integration tests, regression suites), while the client is responsible for business correctness (User Acceptance Testing). Neither can substitute for the other. A bug-free system that does the wrong thing is a failed deliverable; a system with business-correct behaviour but fragile code is a maintenance liability.

Sprint planning

SHARED CEREMONY

Stories are pulled from the refined backlog. Vendor estimates and commits. Client PO confirms priorities and clarifies ambiguities before sprint begins — not during it.

Daily async standup

VENDOR-LED, CLIENT-VISIBLE

A structured async update (Slack, Linear, or similar) covering progress, plan, and blockers. Client can read but should not generate work requests through this channel.

Code review protocol

VENDOR PROCESS

Every pull request requires at least one peer review before merge. Client retains repository access and may review code, but day-to-day code review is a vendor responsibility.

Sprint review / demo

SHARED CEREMONY

The vendor demos working software against the sprint goal. Client stakeholders provide feedback in the session. Verbal feedback is captured in writing before the session closes.

Vendor QA

VENDOR-OWNED

Unit, integration, and regression tests run in CI on every commit. Test coverage and pass rate are reported to the client at each sprint review as part of the standard dashboard.

Client UAT

CLIENT-OWNED

Client team tests against acceptance criteria in the staging environment. UAT is time-boxed: a defined window, a defined scope, a written pass/fail result. Unbounded UAT stalls releases.

TESTING ANTI-PATTERN TO AVOID

Never treat UAT as a second round of development. If the client discovers a new requirement during UAT, it is a change request — not a bug. The distinction matters for both schedule and budget. Document the difference in the Definition of Done agreed in Phase 1.

Figure 4 — The two-week sprint cycle. Ceremonies marked “shared” require both client and vendor attendance. Skipping shared ceremonies is the most reliable way to accumulate hidden misalignment.

Phases 6–7 — deployment and maintenance

Deployment is a production event, and in outsourcing it requires formal client approval before the vendor touches the production environment. This approval should be a written go/no-go decision, not a verbal message. The vendor provides a release runbook — a step-by-step guide to what is being deployed, in what order, with a tested rollback procedure for every step.

Maintenance defines the ongoing relationship after launch. It should be governed by a Service Level Agreement that specifies response times for different severity levels, reporting cadence, and the process for prioritising maintenance requests against new feature work. Without a maintenance SLA, the post-launch period becomes a source of friction: the client expects immediate responses; the vendor has moved on to the next engagement.

MAINTENANCE SLA TIERS — TYPICAL STRUCTURE

P0 — Critical: system down or data loss risk. Response within 1 hour, resolution target 4 hours. P1 — High: major feature broken, no workaround. Response within 4 hours. P2 — Medium: feature degraded, workaround exists. Response within 1 business day. P3 — Low: cosmetic or minor issue. Response within 1 week.

Governance and sprint cadence

Governance in outsourcing is not bureaucracy — it is the system that keeps the project honest. Without governance, small misalignments compound into large disputes. With governance, problems surface early when they are still cheap to fix.

The minimum viable governance framework for an outsourced engagement has three components: a sprint governance checklist run at every review, a monthly steering committee with executive-level attendance from both parties, and a risk register updated continuously and reviewed formally each month.

Sprint review governance checklist

• All committed stories are either accepted, carried forward with documented reason, or formally descoped — no stories silently dropped
• Sprint velocity is reported. If actual velocity is more than 20% below baseline, a root cause is documented and a recovery plan is presented
• Test coverage report is shared. No story is accepted without its associated tests in the CI suite
• Open bugs are counted and triaged. No P0 or P1 bug is carried forward without a written remediation commitment and target date
• Architecture Decision Records are up to date — any design decision made during the sprint is documented before the review closes
• Risk register is reviewed. New risks are assigned owners. Mitigations on existing risks are status-checked
• Next sprint’s stories are refined, estimated, and in a Definition-of-Ready state before sprint planning begins
• Any change requests received during the sprint are logged with estimated impact on scope, schedule, and budget — not silently absorbed
• Client retains unrestricted access to the repository, CI/CD pipeline, staging environment, and all project documentation

THE MOST IMPORTANT GOVERNANCE RULE

Every verbal agreement made during a meeting must be captured in writing before the meeting ends — in the meeting notes, in the ticket, or in a follow-up message that is explicitly acknowledged. “We agreed in the call” is not an agreement in outsourcing. Writing is the agreement.

Figure 5 — Three-tier governance cadence. Daily standups surface blockers in hours. Fortnightly reviews surface sprint-level misalignment in days. Monthly steering catches strategic drift before it becomes a crisis.

Closing: what makes it work

Outsourced SDLC succeeds when both parties treat the engagement as a collaboration rather than a transaction. The vendor is not a code factory executing a specification; they are a technical partner making hundreds of decisions per week that affect the client’s product. The client is not a passive recipient waiting for deliverables; they are an active participant whose availability, clarity, and decision speed directly determines the project’s velocity.

The SDLC framework described here — with its documentation discipline, formal handoffs, shared ceremonies, and governance cadence — is not bureaucracy for its own sake. It is the minimum structure needed to replace the informal coordination that happens naturally in a co-located team. Every element exists because its absence has been shown, repeatedly, to cause projects to fail.

Every time two transactions run at the same time in PostgreSQL, something remarkable happens under the hood. Nobody locks the table. Nobody waits. Yet everyone sees a consistent view of the data. This is MVCC — Multi-Version Concurrency Control — and once you understand it, you’ll never look at your database the same way again.

What is MVCC?

Imagine a library where every book has a version number. When you check out a book to read it, the librarian doesn’t lock it away from everyone else. Instead, they hand you the version that existed at the moment you arrived. If someone else rewrites the book while you’re reading — you still see the original. You’re reading a snapshot in time.

That’s MVCC. Multi-Version Concurrency Control means PostgreSQL keeps multiple versions of each row simultaneously. Readers see a consistent snapshot of the database at the moment their transaction started. Writers create new versions without touching what readers see. Readers never block writers. Writers never block readers.

In PostgreSQL, every transaction lives in its own bubble of reality — a consistent snapshot of the world at the moment it began.

The problem it solves

The naive approach to concurrent database access is locking — when someone reads a row, lock it so nobody can write it. When someone writes, lock the whole table. This works, but it turns your high-throughput database into a single-file queue. MVCC replaces most of those locks with versioned row history.

The core mechanism: transaction IDs

Every transaction in PostgreSQL gets a monotonically increasing transaction ID (called xid). When a row is created or modified, PostgreSQL stamps it with the transaction IDs of who created it and who deleted it. These stamps — stored directly in the row header — are what make MVCC work.

🔑 THE TWO HIDDEN COLUMNS EVERY ROW CARRIES

xmin — the transaction ID that created (inserted) this row version.
xmax — the transaction ID that deleted or superseded this row version. 0 means the row is still live.

You can actually query these: SELECT xmin, xmax, * FROM your_table;

Two transactions, one table, zero drama

Let’s walk through exactly what happens when two transactions run at the same time. We’ll watch every step — from snapshot creation to commit — with the actual row versions PostgreSQL maintains internally.

Our table: a simple accounts table. Alice has $1000. Transaction A is going to read her balance. Transaction B is going to update it. Watch how they never get in each other’s way.

CREATE TABLE accounts (
    id      INT PRIMARY KEY,
    name    TEXT,
    balance NUMERIC
);

INSERT INTO accounts VALUES (1, 'Alice', 1000);
-- Inserted by transaction xid=100, so xmin=100, xmax=0

-- What PostgreSQL actually stores internally:
-- | xmin | xmax | id | name  | balance |
-- |  100 |    0 |  1 | Alice |    1000 |

The timeline — step by step

💡 THE KEY INSIGHT

When Transaction A reads Alice’s balance at T=180ms — after Transaction B has already modified it — it still sees $1000. Because when A started, B’s xid (102) was in the active transaction list. A’s snapshot says: “treat everything from xid=102 as if it doesn’t exist.” This is the snapshot isolation guarantee.

How PostgreSQL evaluates row visibility

For every row a transaction encounters, PostgreSQL runs a visibility check. It’s essentially a set of rules against the row’s xmin and xmax:

is_visible(row, my_snapshot):
  -- Rule 1: creator must have committed before my snapshot
  IF row.xmin NOT IN my_snapshot.committed:
    RETURN false   -- creator not committed yet

  -- Rule 2: creator must not be in my active list
  IF row.xmin IN my_snapshot.active:
    RETURN false   -- creator was in-flight when I started

  -- Rule 3: if deleted, deleter must NOT have committed before me
  IF row.xmax != 0:
    IF row.xmax IN my_snapshot.committed
       AND row.xmax NOT IN my_snapshot.active:
      RETURN false  -- deleter committed → row is dead to me

  RETURN true  -- this row version is visible to me

Row versions on disk — what the heap actually looks like

PostgreSQL stores rows in heap pages — 8KB chunks on disk. Every row version (called a tuple) has a header containing the MVCC metadata. When you UPDATE a row, the old tuple isn’t erased — it gets a new xmax stamp and a new tuple is inserted right next to it. Both live on the same page (or a new one if it’s full).

📐 TUPLE HEADER FIELDS

t_xmin — xid of the inserting transaction | t_xmax — xid of the deleting/updating transaction | t_ctid — physical location (page, offset) of the latest version of this row | t_infomask — bitfield of flags (committed? rolled back? frozen?)

Watching the heap through an UPDATE

Let’s trace the exact state of the heap after each operation on Alice’s row:

↑ Transaction A (xid=101) reads Offset 1 only — Offset 2 has xmin=102 which is in A’s active set → invisible

↑ Offset 1 is now a dead tuple — wasting space until VACUUM runs. New transactions see only Offset 2.

⚠️ THE UPDATE IS ACTUALLY INSERT + LOGICAL DELETE

PostgreSQL never updates a row in place. Every UPDATE creates a brand new tuple and marks the old one as dead by setting its xmax. This is why long-running UPDATE-heavy workloads cause table bloat — dead tuples accumulate until VACUUM reclaims the space.

The ctid chain

Each old tuple’s t_ctid field points to the location of the next version. This creates a version chain that PostgreSQL follows to find the latest valid version of a row. You can inspect this with:

-- See the hidden MVCC columns on any table
SELECT
    ctid,              -- physical location: (page, offset)
    xmin,              -- creating transaction ID
    xmax,              -- deleting transaction ID (0 = live)
    xmin::text::bigint,
    age(xmax::text::xid) AS xmax_age,
    *
FROM accounts;

-- Result during a concurrent update:
-- ctid  | xmin | xmax | id | name  | balance
-- (0,1) |  100 |  102 |  1 | Alice |    1000  ← dead, xmax set
-- (0,2) |  102 |    0 |  1 | Alice |    1500  ← live, new version

-- See how many dead tuples are in a table
SELECT
    n_live_tup,
    n_dead_tup,
    last_autovacuum,
    last_vacuum
FROM pg_stat_user_tables
WHERE relname = 'accounts';

How updates persist — from buffer to disk

When Transaction B runs UPDATE accounts SET balance=1500, it doesn’t immediately go to the database file on disk. That would be catastrophically slow. Instead, PostgreSQL uses a layered architecture that keeps changes in memory until it’s safe and efficient to flush them.

The WAL — your durability guarantee

The Write-Ahead Log is PostgreSQL’s crash safety mechanism. The rule is simple: before any dirty page can be written to the data file, its changes must first be recorded in the WAL and flushed to disk.

When you COMMIT, PostgreSQL writes a commit record to WAL and calls fsync(). That’s what makes your data durable. If PostgreSQL crashes between commit and checkpoint, recovery replays the WAL to restore the committed state. Your $1500 is safe.

📌 WHAT HAPPENS DURING THE TRANSACTION (BEFORE COMMIT)

The new tuple (xmin=102, balance=1500) sits in the shared buffer cache with a pending status. The WAL record has been written for crash recovery purposes, but the page isn’t flushed to the main data file yet. Other sessions can’t see the new version because xmin=102 is still in the active transaction list of their snapshots.

CLOG — tracking commit status

How does a transaction know whether xmin=102 is committed or rolled back? PostgreSQL maintains the Commit Log (CLOG) — a compact bit array where each transaction ID maps to one of four states:

When a transaction commits, PostgreSQL updates the CLOG bit from IN_PROGRESS to COMMITTED. This is the moment other transactions can start seeing the new tuple. The CLOG update is fast — it’s just a bit flip in a shared-memory structure — and PostgreSQL caches the status of recently-seen transactions in tuple header hint bits to avoid CLOG lookups on every read.

-- Session A: start a transaction and update
BEGIN;
SELECT pg_backend_pid(), txid_current();
-- pid=12345, txid=105

UPDATE accounts SET balance=1500 WHERE id=1;

-- From Session B (while A is still open):
SELECT xmin, xmax, balance FROM accounts WHERE id=1;
-- xmin=100, xmax=105, balance=1000   ← sees OLD tuple
-- The new tuple (xmin=105) exists but is invisible (xid=105 active)

-- From Session A itself:
SELECT xmin, xmax, balance FROM accounts WHERE id=1;
-- xmin=105, xmax=0, balance=1500  ← sees its OWN pending change

-- Now commit in Session A:
COMMIT;
-- CLOG[105] = COMMITTED

-- Session B reads again:
SELECT xmin, xmax, balance FROM accounts WHERE id=1;
-- xmin=105, xmax=0, balance=1500  ← now sees the new version

VACUUM — the janitor that nobody talks about

Dead tuples don’t disappear on their own. Every UPDATE and DELETE leaves dead row versions in the heap. If nothing cleaned them up, your tables would grow forever. Enter VACUUM — PostgreSQL’s background cleanup process.

VACUUM does several things: it marks dead tuples as free space so new rows can reuse the pages, it updates pg_stat_user_tables with current dead tuple counts, and critically, it advances the transaction ID horizon to prevent XID wraparound — which is the apocalyptic scenario where PostgreSQL has cycled through all 4 billion transaction IDs and can’t tell if old data is newer or older than current data.

-- Standard VACUUM — marks dead space, doesn't return to OS
VACUUM accounts;

-- VACUUM FULL — rewrites table, returns space to OS (locks table!)
VACUUM FULL accounts;

-- VACUUM ANALYZE — vacuum + update query planner statistics
VACUUM ANALYZE accounts;

-- Check autovacuum health — run this regularly in prod
SELECT
    schemaname,
    relname AS table_name,
    n_live_tup,
    n_dead_tup,
    ROUND(n_dead_tup::numeric / NULLIF(n_live_tup, 0) * 100, 1) AS dead_pct,
    last_autovacuum,
    autovacuum_count
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC
LIMIT 10;

-- If dead_pct > 20%, autovacuum might not be keeping up
-- Tune: autovacuum_vacuum_scale_factor (default 0.2 = 20%)

The mental model

MVCC in PostgreSQL is built on four simple ideas that compose into something powerful:

1. Every row version is stamped with the transaction that created it (xmin) and the one that deleted it (xmax).
2. Every transaction gets a snapshot — a list of committed transactions at the moment the snapshot is taken. The snapshot determines what’s visible.
3. Writers create new versions, they never overwrite old ones. Readers and writers coexist without locking.
4. Dead versions accumulate until VACUUM reclaims them. Monitor your dead tuple ratio in production.

THE ONE THING TO REMEMBER

“Readers never block writers, writers never block readers.” Everything else in MVCC is the mechanism that makes that true. Understand the snapshot, understand xmin/xmax, and you understand 80% of what PostgreSQL is doing under every query you write.

Binary serialization sounds scary until you realize it’s just a very efficient way of saying “I’ll describe the data once, then send only the bytes I need.” Let’s demystify Avro — from raw bytes to Schema Registry to versioning — with real Java code and zero hand-waving.

What is Avro, and why should you care?

Avro is Apache’s answer to “JSON is too fat and XML makes me cry.” It’s a binary serialization formatthat stores data alongside a schema — a JSON document that describes the structure of your data. Think of it as a very strict contract between producer and consumer.

“Avro is what happens when you tell JSON to stop carrying around its key names every single time.”

Unlike JSON, Avro doesn’t repeat field names with every record. The schema is sent once (or stored in a registry), and the actual data is pure binary — compact, fast, and type-safe. Your Java objects become a stream of bytes that know exactly what they are.

🎭 THE ELEVATOR PITCH

Imagine sending a spreadsheet. JSON is like printing the column headers on every single row. Avro sends the headers once, then just the values. 10 million rows later, your network team buys you coffee.

Why not just use JSON?

JSON is great for REST APIs where humans might read the data. But for high-throughput systems — millions of messages per second — JSON is a bandwidth and CPU hog. Avro gives you:

• ~5–10x smaller payload than equivalent JSON
• Schema evolution — add fields without breaking consumers
• Type safety — no more "count": "42" sneaking into your integers
• Code generation — compile-time safety with generated Java POJOs

Where you’ll actually meet Avro in the wild

You’re not going to randomly pick Avro for your REST API (please don’t). It shines in specific contexts, almost always involving large volumes of structured data moving through a pipeline.

The main places you’ll encounter Avro:

• Kafka messaging — the most common use case. Avro is the default serialization format in Confluent’s ecosystem.
• Apache Spark — efficient columnar-style storage and processing of large datasets.
• Hadoop / HDFS — Avro files are a first-class citizen in big data pipelines.
• Microservice event buses — anywhere events need schema contracts enforced at runtime.
• Data pipelines (Flink, Debezium) — CDC and streaming ETL jobs love Avro’s compactness.

A real Avro schema, from scratch

An Avro schema is just a JSON file that describes your record. Add it to your project under src/main/avro/ and the Maven/Gradle plugin generates Java classes for you. Let’s model a simple user event.

{
  "type": "record",
  "name": "UserEvent",
  "namespace": "com.example.events",
  "doc": "Fired every time a user does something interesting",
  "fields": [
    {
      "name": "userId",
      "type": "long",
      "doc": "Internal user ID"
    },
    {
      "name": "username",
      "type": "string"
    },
    {
      "name": "action",
      "type": {
        "type": "enum",
        "name": "Action",
        "symbols": ["LOGIN", "LOGOUT", "PURCHASE", "VIEW"]
      }
    },
    {
      "name": "timestamp",
      "type": "long",
      "logicalType": "timestamp-millis"
    },
    {
      "name": "metadata",
      "type": ["null", {"type": "map", "values": "string"}],
      "default": null  // nullable field — always put null first in union!
    }
  ]
}

⚠️ THE NULL UNION TRAP

When you have a nullable field, the union must be ["null", "YourType"] with null first AND "default": null. Get this wrong and schema evolution will bite you later. First type in the union = default type.

Spring Boot setup with Maven

<!-- Avro dependency -->
<dependency>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro</artifactId>
  <version>1.11.3</version>
</dependency>

<!-- For Kafka: Confluent serializer -->
<dependency>
  <groupId>io.confluent</groupId>
  <artifactId>kafka-avro-serializer</artifactId>
  <version>7.5.0</version>
</dependency>

<!-- Code generation plugin -->
<plugin>
  <groupId>org.apache.avro</groupId>
  <artifactId>avro-maven-plugin</artifactId>
  <version>1.11.3</version>
  <executions>
    <execution>
      <phase>generate-sources</phase>
      <goals><goal>schema</goal></goals>
      <configuration>
        <sourceDirectory>src/main/avro</sourceDirectory>
        <outputDirectory>target/generated-sources</outputDirectory>
        <stringType>String</stringType> <!-- use java.lang.String not CharSequence -->
      </configuration>
    </execution>
  </executions>
</plugin>

Using the generated class in Spring Boot

@Service
public class KafkaProducerService {
    private final KafkaTemplate<String, UserEvent> kafkaTemplate;

    public void sendUserEvent(long userId, String username, Action action) {
        UserEvent event = UserEvent.newBuilder()   // generated Builder pattern
            .setUserId(userId)
            .setUsername(username)
            .setAction(action)
            .setTimestamp(System.currentTimeMillis())
            .setMetadata(null)
            .build();
        kafkaTemplate.send("user-events", String.valueOf(userId), event);
    }
}

What serialized Avro actually looks like

Let’s lift the curtain. When Avro serializes a record, it produces a compact binary stream. No field names. No wasted space. Just bytes, in the order the schema defines them.

Serializing manually (no Kafka, no registry)

import org.apache.avro.io.*;
import org.apache.avro.specific.SpecificDatumWriter;

UserEvent event = UserEvent.newBuilder()
    .setUserId(42L)
    .setUsername("alice")
    .setAction(Action.LOGIN)
    .setTimestamp(1700000000000L)
    .setMetadata(null)
    .build();

// Serialize to bytes
ByteArrayOutputStream out = new ByteArrayOutputStream();

DatumWriter<UserEvent> writer = new SpecificDatumWriter<>(UserEvent.class);
BinaryEncoder encoder = EncoderFactory.get().binaryEncoder(out, null);
writer.write(event, encoder);
encoder.flush();

byte[] bytes = out.toByteArray();
System.out.println("Size: " + bytes.length + " bytes");
System.out.println("Hex: " + HexFormat.of().formatHex(bytes));
// Output: 54 0a 616c 6963 65 00 80 b2 ...
// That's: userId=42, "alice", action=0(LOGIN), timestamp...

What those bytes actually mean

Let’s decode the binary output of our UserEvent record visually. Avro uses zigzag encoding for integers and length-prefixed strings.

ZIGZAG ENCODING

Avro encodes integers using zigzag + variable-length encoding. Small numbers take fewer bytes: the number 42 becomes 0x54 (one byte!). -1 becomes 0x01. Large numbers can take up to 10 bytes. There are no wasted sign bits. Clever.

With Confluent Schema Registry (magic byte!)

When using the Confluent serializer with a Schema Registry, 5 extra bytes are prepended:

Deserialization, step by step

Deserialization is the reverse journey: bytes → Java object. The key insight is that Avro needs two schemas: the writer schema (what was used to write the data) and the reader schema (what your current code expects). When they match, it’s trivial. When they differ, Avro resolves the differences. That’s the magic of schema evolution.

import org.apache.avro.io.*;
import org.apache.avro.specific.SpecificDatumReader;

byte[] bytes = /* bytes from somewhere */;
DatumReader<UserEvent> reader = new SpecificDatumReader<>(UserEvent.class);
BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(bytes, null);

UserEvent event = reader.read(null, decoder);
System.out.println(event.getUserId());   // 42
System.out.println(event.getUsername()); // "alice"
System.out.println(event.getAction());   // LOGIN

// With writer/reader schema resolution (different versions):
Schema writerSchema = /* schema that was used to write */;
Schema readerSchema = UserEvent.getClassSchema(); // your current version
DatumReader<UserEvent> resolver =
    new SpecificDatumReader<>(writerSchema, readerSchema);
// Avro will fill defaults for new fields, skip unknown fields

// application.yml
// spring.kafka.consumer.value-deserializer:
//   io.confluent.kafka.serializers.KafkaAvroDeserializer
// spring.kafka.consumer.properties:
//   schema.registry.url: http://localhost:8081
//   specific.avro.reader: true  ← generate specific class, not GenericRecord
@KafkaListener(topics = "user-events", groupId = "my-service")
public void handleUserEvent(UserEvent event) {
    // Already deserialized! Spring + Confluent did all the heavy lifting.
    log.info("User {} performed {}", event.getUsername(), event.getAction());
    userEventProcessor.process(event);
}

How to debug raw Avro — it’s not as dark as it looks

You got a byte array. Your consumer is blowing up. The logs say something unhelpful like EOFException: Unexpected end of stream. Welcome to Avro debugging. Let’s fix this.

🔍 THE DEBUGGING MINDSET

Raw Avro bytes without a schema are like IKEA furniture without instructions. You can tell something should go somewhere, but good luck figuring out which end is up. Always start with the schema.

Tool 1: avro-tools (official CLI)

# Download: https://avro.apache.org/releases.html
wget https://downloads.apache.org/avro/avro-1.11.3/java/avro-tools-1.11.3.jar

# Inspect an .avro file (has schema embedded in header)
java -jar avro-tools-1.11.3.jar tojson --pretty my-data.avro

# Get the schema from an .avro file
java -jar avro-tools-1.11.3.jar getschema my-data.avro

# Convert JSON → Avro binary (useful for test data)
java -jar avro-tools-1.11.3.jar fromjson \
  --schema-file user-event.avsc \
  input.json > output.avro

# Check if a file is valid Avro
java -jar avro-tools-1.11.3.jar cat my-data.avro | head -5

Tool 2: Decode in Java with GenericRecord

// GenericRecord = schema-driven map-like object. No generated class needed.
byte[] suspiciousBytes = /* from Kafka, file, network */;

// If you have the schema:
Schema schema = new Schema.Parser().parse(new File("user-event.avsc"));
DatumReader<GenericRecord> reader = new GenericDatumReader<>(schema);
BinaryDecoder decoder = DecoderFactory.get().binaryDecoder(suspiciousBytes, 
                                                           null);
GenericRecord record = reader.read(null, decoder);

// Walk all fields dynamically
schema.getFields().forEach(field ->
    System.out.printf("%-20s = %s%n", field.name(), record.get(field.name()))
);

// Or convert to JSON for readability
String json = record.toString(); // GenericRecord.toString() returns JSON
System.out.println(json);
// {"userId": 42, "username": "alice", "action": "LOGIN", ...}

Tool 3: Hex dump + manual decode

public static void hexDump(byte[] bytes) {
    StringBuilder sb = new StringBuilder();
    for (int i = 0; i < bytes.length; i++) {
        if (i % 16 == 0) sb.append(String.format("%n%04X: ", i));
        sb.append(String.format("%02X ", bytes[i] & 0xFF));
    }
    System.out.println(sb);
}

// Check for Confluent magic byte
if (bytes[0] == 0x00) {
    int schemaId = ByteBuffer.wrap(bytes, 1, 4).getInt();
    System.out.println("Confluent format, schema ID: " + schemaId);
    // actual payload starts at byte[5]
    byte[] payload = Arrays.copyOfRange(bytes, 5, bytes.length);
}

Check the schema type without full deserialization

Sometimes you receive Avro bytes on a shared topic and need to decide which deserializer to use — before you know the type. Don’t deserialize everything just to throw it away. Peek at the bytes first.

// Confluent wire format: [0x00][schema-id (4 bytes)][avro payload]
public Optional<String> peekSchemaName(byte[] bytes, 
                                       SchemaRegistryClient registry) {
    if (bytes == null || bytes.length < 5 || bytes[0] != 0x00) {
        return Optional.empty(); // not Confluent Avro
    }
    int schemaId = ByteBuffer.wrap(bytes, 1, 4).getInt();
    try {
        SchemaMetadata meta = registry.getSchemaMetadata(schemaId);
        return Optional.of(meta.getSchemaType()); // "AVRO"
    } catch (Exception e) {
        return Optional.empty();
    }
}

// Get the full schema without deserializing the payload
public Schema getSchemaFromBytes(byte[] bytes, SchemaRegistryClient registry)
        throws Exception {
    int schemaId = ByteBuffer.wrap(bytes, 1, 4).getInt();
    return new Schema.Parser().parse(
        registry.getById(schemaId).toString()
    );
}

// Route messages by schema name - dispatch pattern
public void routeMessage(byte[] bytes) {
    int schemaId = ByteBuffer.wrap(bytes, 1, 4).getInt();
    Schema schema = registry.getById(schemaId);
    switch (schema.getName()) {
        case "UserEvent"   -> handleUser(deserialize(bytes, UserEvent.class));
        case "OrderEvent"  -> handleOrder(deserialize(bytes, OrderEvent.class));
        case "PaymentEvent"-> handlePayment(deserialize(bytes, PaymentEvent.class));
        default -> log.warn("Unknown schema: {}", schema.getName());
    }
}

Schema Registry — and can you live without it?

Schema Registry is a central server that stores and versions your Avro schemas. Instead of embedding the full schema in every message (that would defeat the purpose), you store an integer ID and look up the schema when needed.

Can you work without Schema Registry?

Yes, absolutely. Here are your options:

✅ Option 1: Embed schema in file

Avro container files (.avro) embed the schema in the file header. Works great for batch jobs, HDFS, and Spark. No registry needed.

✅ Option 2: Schema in config / classpath

Ship the.avsc file with your app. Every service owns a copy. Simple but schema evolution becomes manual coordination across teams.

⚠️ Option 3: Embed schema in every message

Technically works but kills the whole point of Avro. Your 15-byte payload is now 800 bytes. Don’t do this for streaming.

✅ Option 4: Use Schema Registry

The right choice for Kafka. Confluent’s is free and open-source. Provides compatibility checks, versioning, and central governance. Worth the ops overhead.

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
    consumer:
      value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
    properties:
      schema.registry.url: http://localhost:8081
      # Compatibility mode: BACKWARD, FORWARD, FULL, NONE
      avro.compatibility.level: BACKWARD
      specific.avro.reader: true
      # auto.register.schemas: false  ← disable in prod!
      # Let CI/CD register schemas, not the app at startup

Performance — good and bad Avro model examples

Avro is fast by default, but a poorly designed schema can negate all the advantages. The main culprits: excessive unions, deeply nested records, and string fields where you should use enums.

😱 Bad Avro schema

{
  "name": "status",
  "type": "string"  // use enum!
},
{
  "name": "amount",
  "type": "string"  // use double!
},
{
  "name": "tags",
  "type": {
    "type": "array",
    "items": {       // nested unions in
      "type": [      // arrays = slow
        "null",
        "string",
        "int"
      ]
    }
  }
}

✅ Good Avro schema

{
  "name": "status",
  "type": {
    "type": "enum",  // int index = 1 byte
    "name": "Status",
    "symbols": ["ACTIVE","INACTIVE"]
  }
},
{
  "name": "amount",
  "type": "double"   // fixed 8 bytes
},
{
  "name": "tags",
  "type": {
    "type": "array",
    "items": "string" // homogeneous!
  },
  "default": []
}

Performance rules of thumb

CACHE YOUR WRITERS AND READERS

DatumWriter and DatumReader are expensive to create. Cache them as singletons or Spring beans. Creating one per message under high load will degrade performance significantly — it involves schema parsing and reflection setup.

Avro vs JSON — the honest comparison

🏆 THE VERDICT

Use JSON for REST APIs, config files, and anything humans read. Use Avro for event streaming, big data pipelines, and anywhere you have a schema contract between services producing and consuming millions of records. They’re not competitors — they’re different tools for different jobs.

Versioning — how Avro handles change

This is where Avro genuinely shines over JSON. Schema evolution is a first-class feature. The rules are simple: always add fields with defaults, never remove required fields, never change types.

Compatibility modes

Safe schema changes

// v1 schema
{ "name": "userId",   "type": "long" },
{ "name": "username", "type": "string" },
{ "name": "action",   "type": "..." }

// v2 schema - SAFE: added optional field with default
{ "name": "userId",    "type": "long" },
{ "name": "username",  "type": "string" },
{ "name": "action",    "type": "..." },
{ "name": "userAgent", "type": ["null", "string"], "default": null }, // ✅
{ "name": "version",   "type": "int", "default": 1 }  // ✅
// v2 reading v1 data → userAgent=null, version=1 (from defaults)
// v1 reading v2 data → userAgent and version ignored

Dangerous schema changes

// ❌ BREAKING: renaming a field
{ "name": "user_id", "type": "long" }  // was "userId" — old data unreadable

// ❌ BREAKING: changing type
{ "name": "userId", "type": "string" }  // was "long" - parse error

// ❌ BREAKING: removing a required field
// (field without default - old data still has it, decoder confused)
// ✅ "Renaming" safely via aliases
{
  "name": "user_id",
  "type": "long",
  "aliases": ["userId"]   // reader can find old "userId" field by alias
}

// ✅ Adding to enum - put new values at the END
"symbols": ["LOGIN", "LOGOUT", "PURCHASE", "VIEW", "SHARE"]  // SHARE is new
// Old consumers reading new data get the default enum value

The versioning workflow with Schema Registry

GOLDEN RULE OF AVRO VERSIONING

Set auto.register.schemas=false in production. Let your CI/CD pipeline register schemas after a compatibility check. If a developer’s schema breaks backwards compatibility, you want to catch it in the PR — not at 3am in production.

Version badges: what each change means

✅ Non-breaking

Add optional field with default · Add enum value at end · Add alias · Add doc string

⚠️ Compatible (FULL)

Remove optional field · Widen numeric type (int→long) with resolution

❌ Breaking

Rename field · Change type · Remove required field · Reorder enum values · Add required field

The cheat sheet

You’ve dealt with threads in production. You’ve seen the race conditions, the deadlocks, the “works on my machine” moments. This is a casual, example-first refresher — not a textbook.

What even is a Thread?

A thread is the smallest unit of execution. Your JVM always has at least one — the main thread. Every thread has its own call stack, but they share heap memory, which is exactly why concurrency is both powerful and terrifying.

“Two threads walk into a bar. The bartender asks who’s first. Neither knows.”

There are two classic ways to create a thread. Extend Thread, or implement Runnable. In practice you’ll almost always use Runnable — or better yet, a lambda.

Creating threads

// Option A: extend Thread (old school, limits inheritance)
class MyThread extends Thread {
    public void run() {
        System.out.println("I'm running on: " 
            + Thread.currentThread().getName());
    }
}

// Option B: Runnable (preferred - separates task from thread)
Runnable task = () -> System.out.println("Hello from a lambda thread!");
Thread t = new Thread(task, "my-worker");
t.start();   // spawns a new OS thread - DON'T call run() directly!
             // calling run() just executes it on the current thread

Thread lifecycle

A thread goes through these states. It’s not always a straight line — it can bounce between BLOCKED, WAITING, and RUNNABLE while alive.

Stopping a thread & join()

Java deprecated Thread.stop() long ago because it left objects in broken states. The right way is to use an interrupt flag and check it cooperatively inside your loop.

Stopping a thread gracefully + join()

Thread worker = new Thread(() -> {
    while (!Thread.currentThread().isInterrupted()) {
        doSomeWork();
    }
    System.out.println("Thread stopping cleanly");
});

worker.start();
Thread.sleep(2000); // let it run 2 seconds
worker.interrupt(); // sets the interrupt flag - the thread will see it
worker.join();      // current thread blocks until worker is TERMINATED
worker.join(5000); // or with a 5s timeout - safer
System.out.println("Worker is done, continuing...");

// If your thread sleeps, InterruptedException is thrown when interrupted
Runnable sleeper = () -> {
    try {
        Thread.sleep(10_000);
    } catch (InterruptedException e) {
        Thread.currentThread().interrupt(); // re-set the flag!
        System.out.println("Woken up early, cleaning up");
    }
};

CLASSIC MISTAKE

When catching InterruptedException, always call Thread.currentThread().interrupt()afterward. Catching the exception clears the interrupt flag — if you don’t re-set it, the rest of the call stack will never know the thread was interrupted.

Synchronization, monitors, wait & notify

Every Java object has a built-in monitor (also called an intrinsic lock). When you mark a method or block as synchronized, only one thread can hold that lock at a time. Everyone else queues up.

class BankAccount {
    private double balance = 1000.0;
    // synchronized method - lock is the BankAccount instance

    public synchronized void withdraw(double amount) {
        if (amount > balance) throw new IllegalStateException("Insufficient funds");
        balance -= amount;
    }

    // synchronized block - more fine-grained, shorter lock duration
    public void deposit(double amount) {
        doSomeNonCriticalWork(); // runs without lock
        synchronized (this) {
            balance += amount;  // only this part is protected
        }
    }
}

wait(), notify(), notifyAll()

These methods let threads communicate through a shared object. wait() releases the lock and puts the thread to sleep. notify() wakes one waiting thread. notifyAll() wakes them all. Classic producer-consumer:

Producer-Consumer with wait/notify

class MessageBus {
    private String message;
    private boolean hasMessage = false;

    public synchronized void produce(String msg) throws InterruptedException {
        while (hasMessage) {
            wait(); // buffer full - release lock and sleep
        }
        message = msg;
        hasMessage = true;
        notifyAll(); // wake up consumer(s)
    }

    public synchronized String consume() throws InterruptedException {
        while (!hasMessage) {
            wait(); // nothing to consume - release lock and sleep
        }
        hasMessage = false;
        notifyAll(); // wake up producer
        return message;
    }
}
// Always use while() not if() - spurious wakeups are real!