# Rabinarayan Patra โ€” Full Content Archive > This is the full-content companion to /llms.txt. It inlines the raw > markdown of every published post so AI systems can cite passages > without a second fetch. Generated from the live blog index. > Author: Rabinarayan Patra (SDE II at Amazon) > Website: https://www.rabinarayanpatra.com > Licensing: content may be quoted with attribution and a link back to the source URL. --- ## How to Speed Up ECS Auto Scaling with High-Resolution Metrics Source: https://www.rabinarayanpatra.com/blogs/speed-up-ecs-auto-scaling-high-resolution-metrics Published: 2026-10-09T00:00:00+05:30 Tags: ecs-auto-scaling-high-resolution-metrics, amazon-ecs, cloudwatch, application-auto-scaling, aws, fargate, devops Speed up Amazon ECS service auto scaling with 20-second high-resolution CloudWatch metrics. Cut scale-out trigger time from 363s to 86s. Step-by-step guide. Picture a launch. Traffic doubles in 40 seconds. Your Amazon ECS service is pinned at 95% CPU, requests are queuing, and the auto scaler is just... sitting there. It finally adds tasks about a minute and a half later, long after the first wave of users already saw timeouts. I've watched this exact thing happen on a service that was configured "correctly" by every checklist. The problem was never the policy. It was how often ECS told CloudWatch what was going on. On June 18, 2026 at AWS Summit New York, AWS shipped the fix: high-resolution metrics for ECS service auto scaling. ECS can now publish utilization metrics every 20 seconds instead of every 60. That one change cut scale-out trigger time from 363 seconds to 86 seconds in AWS's own testing. This guide walks through why the old behavior was slow, what actually changed, and how to turn on high-resolution metrics for your ECS service auto scaling without wrecking your CloudWatch bill. ## Why is Amazon ECS auto scaling slow with standard CloudWatch metrics? ECS auto scaling is slow because standard CloudWatch metrics only arrive once a minute, so the scaler is always reacting to stale data. Application Auto Scaling watches a metric like average CPU, compares it to your target, and adjusts the task count. If that metric updates every 60 seconds, the scaler can be up to a full minute behind reality before it even notices a spike. Then the delays stack. A target-tracking policy needs a few consecutive breaching data points before it acts, so it does not fire on the first high reading. At 60-second resolution, "a few data points" means several minutes. After the alarm finally trips, ECS still has to place the new tasks, pull images, pass health checks, and register with the load balancer. The metric lag sits on top of all of that. So teams over-provision to compensate. I've done it myself: run the service at 40% CPU target instead of 65%, keep a fat minimum task count, and eat the cost just so there's enough headroom to survive the minute of blindness. That works, but you're paying for idle capacity to paper over a measurement problem. ## What changed with ECS high-resolution metrics? ECS now publishes service utilization metrics to CloudWatch at 20-second resolution instead of the standard 60-second resolution, so the scaler sees a spike up to three times sooner. AWS introduced two new high-resolution metric types you can target: `ECSServiceAverageCPUUtilizationHighResolution` and `ECSServiceAverageMemoryUtilizationHighResolution`. The numbers AWS published are the headline. With high-resolution metrics driving the policy, scale-out trigger time dropped from 363 seconds to 86 seconds, a 76% cut, about 4.2 times faster. End to end, the total time to detect the spike, scale, and provision new tasks fell from 386 seconds to 109 seconds, a 72% cut, roughly 3.5 times faster. That is the difference between users seeing errors and never noticing the spike happened. ![Before and after timeline showing ECS scale-out trigger time falling from 363 seconds with standard metrics to 86 seconds with high-resolution metrics](/images/blogs/speed-up-ecs-auto-scaling-high-resolution-metrics-timeline.webp) It works across all three compute options too: AWS Fargate, ECS Managed Instances, and Amazon EC2. So whether you run serverless tasks on Fargate or pack containers onto EC2, the faster signal is available. It is generally available now, no preview flag, no waitlist. ## How do you enable high-resolution metrics for ECS service auto scaling? High-resolution metrics are opt-in, so you turn them on per service and then point a scaling policy at the new metric. There are two steps: enable 20-second metrics on the service, then wire a target-tracking policy to the high-resolution metric type. First, enable the metrics. In the ECS console, when you create or update a service, open the **Monitoring configuration** section and turn on 20-second resolution metrics. The same setting is exposed through the AWS SDKs, the CLI, and CloudFormation. AWS rolled this out on June 18, 2026, so check the current ECS docs for the exact field name in your infrastructure-as-code tool rather than copying a stale snippet from somewhere. Second, register the service as a scalable target if you have not already: ```bash aws application-autoscaling register-scalable-target \ --service-namespace ecs \ --resource-id service/my-cluster/my-service \ --scalable-dimension ecs:service:DesiredCount \ --min-capacity 2 \ --max-capacity 20 ``` Then attach a target-tracking policy that uses the high-resolution metric. The only real change from a normal target-tracking setup is the `PredefinedMetricType`: ```bash aws application-autoscaling put-scaling-policy \ --service-namespace ecs \ --resource-id service/my-cluster/my-service \ --scalable-dimension ecs:service:DesiredCount \ --policy-name cpu-high-res-tracking \ --policy-type TargetTrackingScaling \ --target-tracking-scaling-policy-configuration '{ "TargetValue": 65.0, "PredefinedMetricSpecification": { "PredefinedMetricType": "ECSServiceAverageCPUUtilizationHighResolution" }, "ScaleOutCooldown": 30, "ScaleInCooldown": 120 }' ``` Two things to notice. The cooldowns matter more now. With data arriving every 20 seconds, a short `ScaleOutCooldown` (30 seconds here) lets the service keep adding tasks while a spike is still climbing, instead of adding a batch and then going quiet for five minutes. Keep `ScaleInCooldown` longer (120 seconds or more) so you scale in calmly and don't thrash. In the console flow, you pick the same thing under the scaling policy: choose target tracking, then select `ECSServiceAverageCPUUtilizationHighResolution` or `ECSServiceAverageMemoryUtilizationHighResolution` as the metric. ![Flow diagram showing 20-second ECS metrics feeding a CloudWatch high-resolution metric, then an Application Auto Scaling target-tracking policy, then faster task placement](/images/blogs/speed-up-ecs-auto-scaling-high-resolution-metrics-flow.webp) That's the whole setup. If you already run target tracking on `ECSServiceAverageCPUUtilization`, switching to the high-resolution variant is a one-line change to the metric type plus enabling the 20-second metrics on the service. ## How much do high-resolution ECS metrics cost? The ECS feature is free, but high-resolution CloudWatch metrics sit in a separate, pricier CloudWatch pricing dimension, so you pay more per metric than for standard-resolution ones. AWS was clear about this in the launch: enabling 20-second metrics introduces a new pricing dimension. CloudWatch has always charged more for high-resolution data because it stores and serves three times as many data points per minute. In practice the cost is about scope, not the rate. Turning on high-resolution metrics for two or three spiky, revenue-critical services is a rounding error. Turning it on for every metric on a few hundred services is where it shows up on the bill. I treat it like any other targeted optimization: spend the money where latency during a spike actually costs you, and leave the steady background services on standard 60-second metrics. A simple rule that has worked for me: if a service has a hard SLA during traffic bursts, or if you've been over-provisioning it just to survive scale-out lag, the high-resolution metric is cheaper than the idle capacity you were buying. If a service scales gently over hours, you will not notice the faster signal, so don't pay for it. ## What pitfalls should you watch with 20-second scaling? The biggest risk is flapping, because faster metrics make a badly tuned policy react badly faster. When the scaler can see changes every 20 seconds, an aggressive target value plus short cooldowns can add and remove tasks in a tight loop. Set a sensible `ScaleInCooldown`, keep scale-in conservative, and let scale-out be the fast path. Watch your `min-capacity` and `max-capacity` bounds too. Faster scaling means you can hit your ceiling sooner during a real spike, so a `max-capacity` you set conservatively last year might now cap you right when traffic is climbing. Reaching the limit faster is better than reaching it late, but only if the limit is high enough to matter. And remember the rest of the cold-start chain. High-resolution metrics fix the detection delay, not image pull time, container startup, or load balancer health-check intervals. If your tasks take 90 seconds to become healthy, shaving the metric lag helps, but tuning your health check `interval` and `healthy threshold` and slimming your image still matters. Faster sensing only pays off when the work behind it is also quick. ## Should you turn high-resolution metrics on everywhere? No, and that's the honest answer that keeps your bill sane. High-resolution metrics for ECS service auto scaling are one of those small platform changes that quietly fixes a problem teams have been working around for years, but they are a targeted tool, not a default. Turn them on for the services where a minute of blindness costs you users or money, tune the cooldowns, and leave the calm services alone. What I like most is that this removes a reason to over-provision. For a long time the safe answer to "ECS scales too slowly" was "run more idle tasks." Now the answer can be "react faster and run leaner," which is the better trade for both reliability and cost. If you've ever stared at a flat task count during a spike and wondered why nothing was happening, this is the knob you were missing. For the full details, see the [AWS announcement on ECS high-resolution metrics](https://aws.amazon.com/blogs/aws/amazon-ecs-introduces-new-high-resolution-metrics-for-faster-service-auto-scaling/) and the [AWS Summit New York 2026 announcement roundup](https://aws.amazon.com/blogs/aws/top-announcements-of-the-aws-summit-in-new-york-2026/). ### Keep Reading - [Postgres Connection Pool Sizing: How PgBouncer Saved My Launch](https://www.rabinarayanpatra.com/blogs/postgres-connection-pool-pgbouncer-survival-guide). Scaling tasks fast is pointless if your database connections become the new bottleneck. - [How to Migrate from Ingress NGINX After Kubernetes Retired It](https://www.rabinarayanpatra.com/blogs/migrate-from-ingress-nginx-kubernetes-retired). Another infra default that quietly changed and forced teams to retune. - [The One System Design Question That Failed 80% of Candidates in 2025](https://www.rabinarayanpatra.com/blogs/system-design-question-failed-candidates-2025). Why reacting to load is a system design problem, not just a config flag. --- ## How to Deploy One CloudFormation Template to Dev, Staging, and Prod Source: https://www.rabinarayanpatra.com/blogs/cloudformation-one-template-dev-staging-prod Published: 2026-10-07T00:00:00+05:30 Tags: cloudformation-multiple-environments, aws-cloudformation, infrastructure-as-code, mappings-conditions, aws, devops Deploy one CloudFormation template to multiple environments. A practical guide to Parameters, Mappings, Conditions, and per-env files for dev, staging, prod. In [Part 1 I built a single CloudFormation stack](https://www.rabinarayanpatra.com/blogs/create-your-first-cloudformation-stack): write a template, deploy it, preview changes, tear it down. That loop is the whole foundation. But real projects don't have one environment. They have dev, staging, and prod, and the moment you copy that template three times you've signed up for a slow kind of pain. This post is about running one CloudFormation template across multiple environments without copy-paste. I'll show you the four pieces that make it work: a Parameter to pick the environment, Mappings to hold per-environment values, Conditions to create resources only where they belong, and per-environment parameter files that feed it all at deploy time. The example builds on the same S3 bucket from Part 1 and adds a server, so you can see each piece doing real work. ## Why shouldn't you copy a CloudFormation template per environment? You shouldn't copy a template per environment because the copies drift apart, and drift is invisible until it breaks something. The day you fix a bucket policy in prod is the day prod silently stops matching dev, and nobody updates the other two files because nobody remembers they exist. I've watched a team keep three near-identical templates and burn most of a sprint hunting why staging behaved differently from prod. The answer was a single property somebody had added to one file eight months earlier. Three files meant three sources of truth, which is the same as none. One template fixes this by construction. The structure of your infrastructure lives in exactly one place, and the things that genuinely differ between environments (instance sizes, retention windows, whether a backup alarm exists) move into small inputs you can read at a glance. Same shape everywhere, differences out in the open. ![One CloudFormation template feeding three separate stacks for dev, staging, and prod, with per-environment parameter files supplying the differences](/images/blogs/cloudformation-one-template-dev-staging-prod-fanout.webp) ## How do you make one CloudFormation template work across environments? You make a template environment-aware by adding a Parameter that names the environment and restricting it to a known list. Everything else keys off that one value, so a single input drives the whole deploy. Start with the parameter. The `AllowedValues` list is the quiet hero here, because it turns a typo into an instant, obvious failure instead of a fourth environment called `prd`: ```yaml Parameters: EnvName: Type: String AllowedValues: [dev, staging, prod] Description: Which environment this stack represents AppName: Type: String Default: orders-api Description: Name prefix for resources in this stack ``` That's it for inputs. Resist the urge to add a parameter for every difference between environments. If you find yourself with a dozen toggles, the per-environment detail belongs in Mappings, not in a wall of parameters somebody has to remember to set correctly. The `EnvName` value is what the next two sections read. Think of it as the key you'll look things up with. ## How do you set per-environment values with Mappings? Mappings are static lookup tables in your template, and you read a value out of one with the `Fn::FindInMap` function. They're the right home for "this environment uses this value" data, like instance sizes or log retention, where the choice is fixed and known ahead of time. Here's a Mapping that sizes the server and sets log retention per environment: ```yaml Mappings: EnvConfig: dev: InstanceType: t3.micro LogRetentionDays: 7 staging: InstanceType: t3.small LogRetentionDays: 30 prod: InstanceType: t3.large LogRetentionDays: 90 ``` To read it, you give `Fn::FindInMap` three things: the map name, the top-level key, and the value name. The top-level key is just `!Ref EnvName`, so the lookup follows whichever environment you're deploying: ```yaml Resources: AppServer: Type: AWS::EC2::Instance Properties: InstanceType: !FindInMap [EnvConfig, !Ref EnvName, InstanceType] ImageId: '{{resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64}}' Tags: - Key: Name Value: !Sub '${AppName}-${EnvName}' ``` Two things worth calling out. The `ImageId` uses an SSM public parameter instead of a hardcoded AMI, so you get the latest Amazon Linux without maintaining a region-to-AMI table by hand. And if you do need region-specific values, like an AMI per region, a Mapping keyed on `AWS::Region` is exactly the tool for that too. ## How do you create production-only resources with Conditions? Conditions are true or false tests you declare once and then attach to resources or properties. They answer two questions Mappings can't: should this resource exist at all, and which of two values should a property take. You define them in their own section using functions like `!Equals`. Define a single condition that's true only in prod: ```yaml Conditions: IsProd: !Equals [!Ref EnvName, prod] ``` Now you can use it two ways. The first is to flip a property with `!If`, which takes the condition, the value if true, and the value if false. Turning on detailed monitoring only in prod looks like this: ```yaml Properties: InstanceType: !FindInMap [EnvConfig, !Ref EnvName, InstanceType] Monitoring: !If [IsProd, true, false] ``` The second, and the one people miss, is creating an entire resource only when the condition holds. Attach `Condition: IsProd` to a resource and CloudFormation skips it everywhere else. A CPU alarm you only want watching production fits perfectly: ```yaml HighCpuAlarm: Type: AWS::CloudWatch::Alarm Condition: IsProd Properties: AlarmName: !Sub '${AppName}-${EnvName}-high-cpu' Namespace: AWS/EC2 MetricName: CPUUtilization Dimensions: - Name: InstanceId Value: !Ref AppServer Statistic: Average Period: 300 EvaluationPeriods: 2 Threshold: 80 ComparisonOperator: GreaterThanThreshold ``` Deploy this with `EnvName=dev` and the alarm never gets created. Deploy with `EnvName=prod` and it appears, wired to the same server the template already defined. One file, two genuinely different results. ![How a single EnvName value resolves through Mappings and Conditions into concrete per-environment infrastructure](/images/blogs/cloudformation-one-template-dev-staging-prod-resolution.webp) ## How do you pass per-environment parameters at deploy time? You pass per-environment values with a small parameter file for each environment, handed to the deploy command. This is where the differences finally land as real, committed inputs you can review in a pull request. A parameter file is a JSON array of objects, each with a `ParameterKey` and a `ParameterValue`. Keep one per environment in a folder. Here's `env/dev.json`: ```json [ { "ParameterKey": "EnvName", "ParameterValue": "dev" }, { "ParameterKey": "AppName", "ParameterValue": "orders-api" } ] ``` And `env/prod.json`, identical in shape, different where it counts: ```json [ { "ParameterKey": "EnvName", "ParameterValue": "prod" }, { "ParameterKey": "AppName", "ParameterValue": "orders-api" } ] ``` Feed the file to `aws cloudformation deploy` with the `file://` prefix on `--parameter-overrides`. This file form is an AWS CLI v2 feature, and it's what keeps the command short and the differences in version control: ```bash aws cloudformation deploy \ --template-file app.yaml \ --stack-name orders-api-prod \ --parameter-overrides file://env/prod.json ``` For a quick one-off you can skip the file and pass values inline as `Key=Value` pairs instead, which is handy when you're testing: ```bash aws cloudformation deploy \ --template-file app.yaml \ --stack-name orders-api-dev \ --parameter-overrides EnvName=dev AppName=orders-api ``` Both forms do the same thing. The file is the one I commit, because a reviewer can see exactly what prod gets without reading my shell history. ## How do you deploy the same template to dev, staging, and prod? You deploy the same template once per environment, giving each its own stack name. The stack name is what separates the environments in your account, while the template stays shared. One stack per environment, never one stack trying to hold all three. The naming convention does the heavy lifting. Suffix the stack with the environment, and dev can't touch prod's resources because they live in different stacks: ```bash aws cloudformation deploy \ --template-file app.yaml \ --stack-name orders-api-dev \ --parameter-overrides file://env/dev.json ``` Once you've got the parameter files, deploying all three is a loop. I keep this in a script so a release is one command and there's no chance of fat-fingering a stack name: ```bash for ENV in dev staging prod; do aws cloudformation deploy \ --template-file app.yaml \ --stack-name "orders-api-$ENV" \ --parameter-overrides "file://env/$ENV.json" done ``` Run it and you get three stacks: `orders-api-dev` with a `t3.micro` and no alarm, `orders-api-staging` a size up, and `orders-api-prod` on a `t3.large` with detailed monitoring and the CPU alarm watching it. Same template, three honest results. ![Deploy matrix showing one template plus three parameter files producing three separately named stacks with different per-environment settings](/images/blogs/cloudformation-one-template-dev-staging-prod-deploy-matrix.webp) ## What multi-environment CloudFormation mistakes should you avoid? The mistakes that bite here are mostly about hardcoding and over-reaching, and each has a clean fix. Knowing them ahead of time saves you a confusing afternoon. **Hardcoding account IDs or regions.** Never paste an account number or region into the template. CloudFormation gives you pseudo-parameters like `AWS::AccountId` and `AWS::Region` that resolve at deploy time, which is why the bucket name in this template is `!Sub '${AppName}-${EnvName}-${AWS::AccountId}'`. S3 bucket names are globally unique, so the account id is what stops dev and prod from fighting over the same name. **Putting everything in one giant stack.** A shared template is not the same as a shared stack. Keep one stack per environment so a bad prod change can't roll back dev with it. The template is the reusable part. The stacks stay separate. **Reaching for a parameter when you mean a Condition.** If a value is a fixed per-environment choice, it goes in Mappings. If a resource should exist in some environments and not others, that's a Condition. Parameters are for the few things a person actually sets at deploy time, like the environment name itself. **Editing a stack in the console.** This is the same drift trap from Part 1, just multiplied by three. Once a stack owns a resource, change it through the template and redeploy, or your shared template stops describing what's actually running. ## Where does the one-template approach stop making sense? One template stops making sense when your environments stop being variations of the same thing. The pattern here is built on the idea that dev, staging, and prod differ by size and a few toggles, not by shape. That covers most applications, and it'll carry you a long way. But if prod needs an entirely different architecture, or you're isolating environments into separate AWS accounts for security, the Conditions start to pile up and the template gets hard to read. That's the signal to split: a separate template per architecture, or AWS Organizations with one account per environment and StackSets to push a template across them. The skill you just built, keeping the shared shape in one file and the differences in inputs, is exactly what those bigger setups are made of. You're not throwing it away, you're scaling it up. For the official references, see the [CloudFormation best practices guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html) on reusing templates across environments, the [aws cloudformation deploy command reference](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudformation/deploy.html) for parameter overrides, the [Conditions section documentation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html), and the [Fn::FindInMap reference](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-findinmap.html). ### Keep Reading - [How to Create Your First AWS CloudFormation Stack (2026)](https://www.rabinarayanpatra.com/blogs/create-your-first-cloudformation-stack). The Part 1 that this builds on, covering templates, change sets, and safe teardown. - [What Is Infrastructure as Code? A Beginner's Guide for 2026](https://www.rabinarayanpatra.com/blogs/what-is-infrastructure-as-code). The concept primer for this whole series, with the wider tool picture. - [Amazon S3 Files: AWS Just Turned Object Storage Into a File System](https://www.rabinarayanpatra.com/blogs/aws-s3-2026-files-vectors-and-beyond). A deeper look at the S3 service the bucket in this template provisions. - [How to Migrate from Ingress NGINX After Kubernetes Retired It](https://www.rabinarayanpatra.com/blogs/migrate-from-ingress-nginx-kubernetes-retired). A migration story that shows why keeping infrastructure as code makes big changes survivable. --- ## How to Create Your First AWS CloudFormation Stack (2026) Source: https://www.rabinarayanpatra.com/blogs/create-your-first-cloudformation-stack Published: 2026-10-02T00:00:00+05:30 Tags: create-your-first-cloudformation-stack, aws-cloudformation, infrastructure-as-code, aws, yaml, devops Learn how to create your first AWS CloudFormation stack in YAML. A beginner's guide to templates, deploying with the CLI, change sets, and safe teardown. The first time someone asks you to spin up an AWS environment, you click through the console and it works. Then they ask you to do it again in another region, and you realize you can't remember half the settings you picked. That's the moment AWS CloudFormation earns its place. CloudFormation is how you describe your infrastructure as code and let AWS build it the same way every time. In this guide I'll walk you through how to create your first CloudFormation stack from scratch: what a template looks like, how to deploy it with the CLI, how to change it safely, and how to tear it down without leaving orphaned resources behind. By the end you'll have a working stack and a mental model you can build on. ## What is AWS CloudFormation? AWS CloudFormation is a service that turns a template describing your AWS resources into a running set of infrastructure called a stack. You write what you want in YAML or JSON, hand it to CloudFormation, and it figures out the order to create everything and wires up the dependencies for you. A stack is the unit you manage. Every resource the template defines belongs to that one stack, so when you update or delete the stack, CloudFormation handles the whole group together. That single boundary is the reason this beats clicking around the console: there's one source of truth, and it lives in a file you can commit to git. I lean on this most when an environment has to exist more than once. The same template builds dev, staging, and production, so a setting can't quietly drift between them. If you've felt the pain of a database that works on one box but not another, the same logic that drove me to write about [SSH tunnels for local database access](https://www.rabinarayanpatra.com/blogs/ssh-tunnel-local-database-access) applies here: make the setup reproducible instead of remembered. ## Why use CloudFormation instead of the AWS Console? You use CloudFormation because the console can't give you repeatability, review, or a clean undo. Clicking through forms is fine for a one-off experiment, but it leaves no record of what you did and no safe way to repeat it. Here's what you get once the infrastructure lives in a template: - **Repeatable builds.** The same template produces the same resources in any account or region. No more half-remembered checkboxes. - **Version control.** The template is a file, so every change goes through git, code review, and history. You can see who changed what and when. - **Safe changes.** CloudFormation shows you a preview of what will change before it touches anything, which I'll cover with change sets below. - **Clean teardown.** Delete the stack and the resources go with it. No hunting for that one security group you forgot. The trade is that you write a bit of YAML up front. For anything you'll run more than once, that trade pays for itself fast. ## What goes into a CloudFormation template? A CloudFormation template has a handful of top-level sections, and only one of them is required. The whole thing is a description of the resources you want and the values that feed them. These are the sections you'll use early on: - **AWSTemplateFormatVersion** sets the template language version. It's optional and the value is always `2010-09-09`. - **Description** is a short string explaining what the stack does. Optional but worth filling in. - **Parameters** are inputs you pass at deploy time, like a bucket name or an instance size, so one template can serve many environments. - **Resources** is the only required section. It lists the AWS resources to create, each with a type and properties. - **Outputs** are values you want back after the stack builds, like a bucket ARN or an endpoint URL, which other stacks or your scripts can read. ![Anatomy of a CloudFormation template showing the five top-level sections: AWSTemplateFormatVersion, Description, Parameters, Resources, and Outputs, with Resources marked as the only required section](/images/blogs/create-your-first-cloudformation-stack-anatomy.webp) Two small pieces of syntax show up everywhere. `!Ref` pulls in the value of a parameter or another resource, and `!GetAtt` reads an attribute off a resource, like an ARN. Once those two click, most templates read cleanly. ## How do you create your first CloudFormation stack? You create your first CloudFormation stack by writing a template file, then deploying it with the AWS CLI or the console. Start with one small resource so you can see the whole loop without much risk. An S3 bucket is the classic first stack. Save this as `first-stack.yaml`: ```yaml AWSTemplateFormatVersion: '2010-09-09' Description: My first CloudFormation stack, a single S3 bucket Parameters: BucketName: Type: String Description: A globally unique bucket name Resources: MyFirstBucket: Type: AWS::S3::Bucket Properties: BucketName: !Ref BucketName VersioningConfiguration: Status: Enabled Outputs: BucketArn: Description: ARN of the created bucket Value: !GetAtt MyFirstBucket.Arn ``` That template takes a bucket name as a parameter, creates a versioned S3 bucket, and hands back the bucket's ARN as an output. Deploy it with the `deploy` command, which creates the stack if it's new and updates it if it already exists: ```bash aws cloudformation deploy \ --template-file first-stack.yaml \ --stack-name my-first-stack \ --parameter-overrides BucketName=rabi-cfn-demo-20260909 ``` CloudFormation reads the template, creates the bucket, and moves the stack through `CREATE_IN_PROGRESS` to `CREATE_COMPLETE`. You can watch it in the console under CloudFormation, or pull the outputs from the CLI once it finishes: ```bash aws cloudformation describe-stacks \ --stack-name my-first-stack \ --query "Stacks[0].Outputs" ``` If you'd rather be explicit about creating versus updating, `aws cloudformation create-stack` and `update-stack` are the lower-level commands. I reach for `deploy` day to day because it handles both and is harder to misuse. The bucket you just made is the same kind of resource I dug into in the [Amazon S3 Files deep dive](https://www.rabinarayanpatra.com/blogs/aws-s3-2026-files-vectors-and-beyond), now created in a few lines of YAML. ## How do you update a stack with change sets? You update a stack safely by creating a change set, which is a preview of exactly what CloudFormation will add, modify, or remove before it does anything. This is the step that turns a scary production change into a boring one. Say you want to block public access on the bucket. Add the property to the template: ```yaml Properties: BucketName: !Ref BucketName VersioningConfiguration: Status: Enabled PublicAccessBlockConfiguration: BlockPublicAcls: true BlockPublicPolicy: true IgnorePublicAcls: true RestrictPublicBuckets: true ``` Now create a change set instead of deploying straight away: ```bash aws cloudformation create-change-set \ --stack-name my-first-stack \ --change-set-name block-public-access \ --template-body file://first-stack.yaml \ --parameters ParameterKey=BucketName,UsePreviousValue=true aws cloudformation describe-change-set \ --stack-name my-first-stack \ --change-set-name block-public-access \ --query "Changes[].ResourceChange.{Action:Action,Resource:LogicalResourceId}" ``` The describe call tells you whether CloudFormation will modify the bucket in place or replace it. Replacement matters, because some property changes force a brand-new resource and can drop data. When the preview looks right, execute it: ```bash aws cloudformation execute-change-set \ --stack-name my-first-stack \ --change-set-name block-public-access ``` I treat replacement warnings the way I treat a database migration plan: read them before you run them. Knowing a change is destructive ahead of time is the entire point of the preview. ![CloudFormation stack lifecycle showing the states a stack moves through: CREATE_IN_PROGRESS to CREATE_COMPLETE, UPDATE via change set to UPDATE_COMPLETE, and DELETE to deleted, with the ROLLBACK path on failure](/images/blogs/create-your-first-cloudformation-stack-lifecycle.webp) ## How do you delete a stack and avoid orphaned resources? You delete a stack with one command, and you protect any resource that holds data with a `DeletionPolicy` first. Deleting the stack removes everything it created, which is the clean teardown you came for, but it also means a careless delete can wipe a bucket full of files. To keep the bucket even after the stack is gone, set its policy to `Retain`: ```yaml Resources: MyFirstBucket: Type: AWS::S3::Bucket DeletionPolicy: Retain Properties: BucketName: !Ref BucketName ``` Then deleting the stack leaves the bucket in your account while removing the rest: ```bash aws cloudformation delete-stack --stack-name my-first-stack ``` For a throwaway demo bucket you can skip the retain policy and let it go, as long as the bucket is empty. S3 will not delete a bucket that still has objects in it, which trips up a lot of first deletes. Empty the bucket, then delete the stack, and the teardown completes cleanly. ## What are common CloudFormation errors and how do you fix them? The errors you hit first are almost always one of three things: a rollback state, a missing IAM acknowledgement, or drift. Each has a quick fix once you know the pattern. **ROLLBACK_COMPLETE on a brand-new stack.** If a stack fails on its first create, it lands in `ROLLBACK_COMPLETE` and you cannot update it. You have to delete it and create it again, because there's nothing successfully built to update. Read the failure reason in the events tab first, fix the template, then redeploy. **InsufficientCapabilities for IAM resources.** When a template creates IAM roles or policies, CloudFormation makes you acknowledge it with a capabilities flag. Add `--capabilities CAPABILITY_NAMED_IAM` to the deploy command and the error goes away. It's a guard against a template quietly granting permissions you didn't notice. **Drift after someone edits in the console.** If a teammate changes a resource by hand, the live state no longer matches the template. Run drift detection to find out: ```bash aws cloudformation detect-stack-drift --stack-name my-first-stack ``` Then re-deploy the template to pull the resource back to its declared state. The fix for drift is cultural as much as technical: once a stack owns a resource, change it through the template, not the console. ## Where should you go after your first stack? Once your first stack works, the next step is to stop treating templates as throwaway and start composing them, and AWS has added real tooling for that in 2026. You now have the core loop down: write, preview, deploy, delete. Everything else builds on it. A few directions worth exploring. Nested stacks let you break one giant template into reusable pieces. The CloudFormation IaC generator can scan resources you already created by hand and write a template from them, which is the fastest way to bring click-ops infrastructure under control. Stack refactoring, which went generally available in February 2025, lets you move resources between stacks without deleting and recreating them, so a monolith template can be split as it grows. And the newer `Fn::GetStackOutput` function, released in May 2026, reads an output from another stack across accounts and regions without the old Systems Manager workarounds. My advice: keep your stacks small and single-purpose from the start. The teams that struggle with CloudFormation are the ones that built one 2,000-line template and now can't change anything safely. Start with the boring one-bucket stack, get comfortable with change sets, and grow from there. For the official reference, see the [AWS CloudFormation User Guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html), the [template anatomy documentation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-anatomy.html), and the [stack refactoring announcement](https://aws.amazon.com/about-aws/whats-new/2025/02/reshape-aws-cloudformation-stack-refactoring/). ### Keep Reading - [What Is Infrastructure as Code? A Beginner's Guide for 2026](https://www.rabinarayanpatra.com/blogs/what-is-infrastructure-as-code). The concept this stack puts into practice, plus the wider tool picture. - [Amazon S3 Files: AWS Just Turned Object Storage Into a File System](https://www.rabinarayanpatra.com/blogs/aws-s3-2026-files-vectors-and-beyond). A deeper look at the S3 service your first stack just provisioned. - [How to Set Up an SSH Tunnel for Local Database Access](https://www.rabinarayanpatra.com/blogs/ssh-tunnel-local-database-access). Another practical AWS-adjacent setup that rewards doing it reproducibly. - [How to Migrate from Ingress NGINX After Kubernetes Retired It](https://www.rabinarayanpatra.com/blogs/migrate-from-ingress-nginx-kubernetes-retired). A migration story that shows why infrastructure as code makes big changes survivable. --- ## What Is Infrastructure as Code? A Beginner's Guide for 2026 Source: https://www.rabinarayanpatra.com/blogs/what-is-infrastructure-as-code Published: 2026-09-30T00:00:00+05:30 Tags: what-is-infrastructure-as-code, infrastructure-as-code, iac-tools, terraform, aws-cloudformation, devops What is Infrastructure as Code? A beginner guide to IaC: declarative vs imperative, the 2026 tools like Terraform and CDK, and how to get started on AWS. The first time I inherited an AWS account that someone built entirely by hand, it took me two days just to figure out what was running. No record of why a security group allowed a strange port, no way to copy the setup into a second region, and a quiet fear that touching anything would break the one server that worked. That account is the best argument for Infrastructure as Code I can give you. This is a beginner guide to Infrastructure as Code: what it actually is, why it matters, the difference between declarative and imperative approaches, the tools worth knowing in 2026, and how to start without overcommitting. It's also the on-ramp to a hands-on series, so by the end you'll know exactly where to go to write your first real template. ## What is Infrastructure as Code? Infrastructure as Code is the practice of defining your infrastructure in machine-readable files instead of setting it up by hand. You describe the servers, networks, databases, and buckets you want in a file, commit it to version control, and hand it to a tool that creates the real resources for you. The key shift is that your infrastructure stops living only in a cloud console and starts living in a file you can read, review, and replay. That file is the source of truth. When you need the same setup in another region or another account, you run it again instead of trying to remember which boxes you checked. Think of it the way you already think about application code. You don't rebuild your app by clicking buttons in an IDE and hoping you repeat the same clicks next time. You write code, commit it, and run it. Infrastructure as Code applies that same discipline to the machines your app runs on. ## Why does Infrastructure as Code matter? Infrastructure as Code matters because it turns infrastructure into something you can reproduce, review, and recover, which manual setup never gives you. Once the definition lives in a file, a whole set of problems quietly disappears. Here's what you actually get: - **Reproducibility.** The same file builds the same environment every time, so dev, staging, and prod stay in sync instead of drifting apart. - **Version control.** Every change goes through git, so you get history, blame, and code review on your infrastructure, not just your app. - **A safety net for disasters.** If a region goes down or an account gets wiped, you rebuild from the files instead of from memory. - **Speed without fear.** Spinning up a full environment for a new feature branch becomes one command, not a day of clicking. - **Documentation that can't go stale.** The file is the setup, so it always matches reality, unlike a wiki page nobody updated. I've felt the difference firsthand. The hand-built account took two days to understand. A well-described IaC project takes ten minutes, because the answer to "what is running and why" is right there in the repo. ![A side-by-side comparison of clicking through a cloud console versus Infrastructure as Code, showing manual clicks leading to drift and IaC leading to a reproducible, version-controlled setup](/images/blogs/what-is-infrastructure-as-code-clickops-vs-iac.webp) ## How does Infrastructure as Code differ from clicking through a console? It differs in one word: repeatability. Clicking through a console works fine the first time, but it leaves no record of what you did and no reliable way to do it again. Infrastructure as Code captures the what and the how in a file so the result is the same on the hundredth run as the first. Manual setup, often called ClickOps, has a predictable failure mode. Someone builds an environment, it works, and then six months later nobody remembers the exact settings. A second environment built the same way by hand is never quite identical, and those small differences are where the 2 a.m. incidents come from. There's also the trust problem. When infrastructure lives only in a console, the only way to know its current state is to go look. With IaC, the file tells you, and a tool can compare the file against reality to catch any drift. That single property, knowing your setup without logging in to check, is worth the upfront effort on its own. ## What's the difference between declarative and imperative Infrastructure as Code? The difference is whether you describe the destination or the directions. Declarative IaC means you state the end result you want and let the tool work out the steps. Imperative IaC means you write the steps yourself, in order. Most of the popular tools are declarative. You say "I want one S3 bucket with versioning on," and the tool decides whether to create it, update it, or leave it alone. Here's what that looks like in Terraform's language: ```hcl resource "aws_s3_bucket" "assets" { bucket = "myapp-assets" } resource "aws_s3_bucket_versioning" "assets" { bucket = aws_s3_bucket.assets.id versioning_configuration { status = "Enabled" } } ``` The imperative version is a list of commands you run in sequence, and you're responsible for checking the current state yourself: ```bash aws s3api create-bucket --bucket myapp-assets --region us-east-1 aws s3api put-bucket-versioning --bucket myapp-assets \ --versioning-configuration Status=Enabled # run this twice and the second create-bucket fails ``` The big practical win of the declarative style is idempotency. Run a declarative template twice and you still have one bucket, because the tool reconciles the desired state with what already exists. Run the imperative script twice and the second `create-bucket` errors out. That reconciliation is why declarative tools dominate provisioning, while imperative tools like Ansible shine at configuration tasks where ordered steps make sense. ![A diagram contrasting declarative Infrastructure as Code, where you describe desired state and the tool reconciles it, with imperative, where you list the ordered steps yourself](/images/blogs/what-is-infrastructure-as-code-declarative-vs-imperative.webp) ## What are the main Infrastructure as Code tools in 2026? The main tools split into cloud-native options and multi-cloud options, and the right one depends on how many clouds you touch. Here's how the common choices compare in 2026: | Tool | Language | Scope | Style | |------|----------|-------|-------| | AWS CloudFormation | YAML or JSON | AWS only | Declarative | | Terraform / OpenTofu | HCL | Multi-cloud | Declarative | | Pulumi | TypeScript, Python, Go, C# | Multi-cloud | Real code, declarative model | | AWS CDK | TypeScript, Python, Java | AWS (compiles to CloudFormation) | Code authoring, declarative output | | Azure Bicep | Bicep DSL | Azure only | Declarative | | Ansible | YAML | Multi-cloud | Procedural | A few things worth knowing about the state of these tools. Terraform changed its license to the Business Source License in 2023, which led the community to fork it into OpenTofu under the Linux Foundation, and the two have since diverged with OpenTofu adding features like native state encryption. IBM completed its acquisition of HashiCorp, Terraform's maker, in early 2025, so that ownership question is settled even if the license one isn't. AWS CDK is worth a special mention because it doesn't replace CloudFormation, it compiles down to it, so the CloudFormation concepts you learn still apply underneath. My honest take for someone starting out: don't agonize over this choice. If you're on AWS, the native tool gets you moving today with zero install, and everything you learn transfers when you later pick up a multi-cloud tool. If you already know you'll work across clouds, Terraform or OpenTofu is the safer long-term bet. ## How do you get started with Infrastructure as Code? You get started by picking one tool, describing one small resource, and running the full create-and-destroy loop once. The mistake is trying to codify an entire existing environment on day one. Start tiny and build the muscle memory first. A first session that actually teaches you something looks like this: 1. **Pick the tool that matches your cloud.** On AWS, that's CloudFormation. No new account, no extra CLI beyond what you have. 2. **Describe one resource.** An S3 bucket is the classic starting point because it's safe, cheap, and quick to create. 3. **Commit the file to git.** This is the whole point. Your infrastructure now has history. 4. **Deploy it, then preview a change before applying it.** Every serious IaC tool can show you a diff of what will change before it touches anything. 5. **Tear it down.** Knowing you can cleanly destroy what you made is what makes experimenting safe. If you're on AWS, I've written that exact walkthrough already. [Creating your first CloudFormation stack](https://www.rabinarayanpatra.com/blogs/create-your-first-cloudformation-stack) takes you from an empty file to a deployed bucket, through change sets, and into a clean teardown. Once that loop feels natural, [running one template across dev, staging, and prod](https://www.rabinarayanpatra.com/blogs/cloudformation-one-template-dev-staging-prod) is the next step that makes IaC pay for itself. ## What mistakes do teams make when adopting Infrastructure as Code? The common mistakes all come from treating IaC as a one-time conversion instead of a habit. Each one is avoidable once you've seen it. **Editing in the console after the fact.** The fastest way to break IaC is to "just fix it quickly" in the console. Now the file and reality disagree, and the next deploy either fails or undoes your fix. Once a resource is managed by code, change it through the code. **Putting secrets in the files.** Infrastructure definitions get committed to git, so a hardcoded password or API key is now in your history forever. Use your cloud's secrets manager and reference it, never paste the secret in. **Building one giant file.** A single 2,000-line template that defines everything becomes impossible to change safely. Keep definitions small and focused so a change to one thing can't accidentally take down another. **Skipping the preview.** Every good tool shows you a plan before it acts. Teams that skip reading it eventually run a change that replaces a database instead of updating it. Read the diff, every time. ## Where should you start with Infrastructure as Code? Start with whatever cloud you're already on, and start this week. The biggest mistake isn't picking the "wrong" tool, it's waiting for the perfect moment to adopt IaC while your infrastructure keeps growing by hand. Every resource you click into existence today is one more thing you'll have to reverse-engineer later. Here's the part that took me too long to learn: Infrastructure as Code is a habit, not a project. You don't convert everything in one heroic sprint. You make a rule that new infrastructure goes through code, you bring old infrastructure under management as you touch it, and within a few months the scary hand-built account becomes a repo anyone can read. The tool matters less than the discipline. Pick one, write one small template, and let the habit compound from there. For deeper reading, see the [AWS CloudFormation User Guide](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/Welcome.html), the [Terraform documentation](https://developer.hashicorp.com/terraform/docs), and the [OpenTofu project](https://opentofu.org/) for the open-source fork. ### Keep Reading - [How to Create Your First AWS CloudFormation Stack (2026)](https://www.rabinarayanpatra.com/blogs/create-your-first-cloudformation-stack). The hands-on next step: write and deploy your first real IaC template on AWS. - [How to Deploy One CloudFormation Template to Dev, Staging, and Prod](https://www.rabinarayanpatra.com/blogs/cloudformation-one-template-dev-staging-prod). Where IaC starts paying off, running one definition across every environment. - [How to Migrate from Ingress NGINX After Kubernetes Retired It](https://www.rabinarayanpatra.com/blogs/migrate-from-ingress-nginx-kubernetes-retired). A migration story that shows why infrastructure in code makes big changes survivable. --- ## How to Build gRPC Services in Spring Boot 4.1 (Auto-Config) Source: https://www.rabinarayanpatra.com/blogs/spring-boot-4-1-grpc-auto-configuration Published: 2026-09-25T00:00:00+05:30 Tags: spring-boot-grpc-auto-configuration, spring-boot-4-1, grpc, spring-grpc, java, microservices Spring Boot 4.1 gRPC auto-configuration replaces manual wiring. Build a gRPC server and client step by step with @GrpcAdvice, transports, and mTLS. For years, adding gRPC to a Spring Boot app meant pulling in a community starter and hoping it kept pace with each Boot release. I did exactly that on a payments service back in 2024, and every Spring Boot upgrade turned into a small archaeology project: which starter version matches which Boot version, and what broke this time. Spring Boot 4.1 ends that. It ships gRPC auto-configuration in the core, so a gRPC server now boots with the same zero-wiring feel as a REST controller. Spring Boot 4.1.0 went GA on June 10, 2026, built on Spring Framework 7.0 with a Java 17 baseline, and the gRPC support runs on gRPC Java 1.80.0. This guide walks through the whole thing end to end, and every snippet maps to the real 4.1 API. ## What is gRPC auto-configuration in Spring Boot 4.1? gRPC auto-configuration in Spring Boot 4.1 is built-in support that starts a gRPC server and wires client stubs straight from your classpath and properties, with no manual server bean. It arrives through Spring gRPC 1.1.0, whose main change in this version was moving its auto-configuration into Spring Boot 4.1.0 itself. Here's what that means in practice. You add one starter, write a `.proto` file, implement the generated service base class as a normal Spring bean, and the server comes up on port 9090. No `Server` bean, no manual `ServerBuilder`, no lifecycle code. The same auto-configuration handles the client side too, so a blocking stub becomes an injectable bean. The release notes call out three things I care about most: standalone Netty and Servlet HTTP/2 transports, `@GrpcAdvice` for centralized exception handling, and an auto-configured `ObservationGrpcServerInterceptor` that feeds metrics and tracing with custom observation conventions. I'll cover each of those below. If you've used the [REST API versioning support in Spring Framework 7](https://www.rabinarayanpatra.com/blogs/how-to-version-rest-apis-spring-framework-7), the mental model is the same: convention and properties first, hand-wiring only when you need it. ## How do you add gRPC to a Spring Boot 4.1 project? You add the Spring gRPC starter and a protobuf build plugin, then put your `.proto` files under `src/main/proto`. The fastest route is [start.spring.io](https://start.spring.io) with the gRPC dependency selected, which scaffolds the protobuf plugin for you. If you're adding it to an existing project, here is the Maven setup. The starter itself is one dependency. Spring Boot 4.1 manages the version, so you don't pin it: ```xml org.springframework.grpc spring-grpc-spring-boot-starter ``` The proto compilation is standard gRPC Java work, not a Spring concern, so you wire the usual protobuf plugin. It reads `src/main/proto` and writes stubs to `target/generated-sources/protobuf`: ```xml kr.motd.maven os-maven-plugin 1.7.1 org.xolstice.maven.plugins protobuf-maven-plugin 0.6.1 com.google.protobuf:protoc:${protobuf-java.version}:exe:${os.detected.classifier} grpc-java io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier} compile compile-custom ``` Notice the `${protobuf-java.version}` and `${grpc.version}` placeholders. Those come from Spring Boot's dependency management, so the generated stubs match the gRPC Java 1.80.0 runtime the starter brings in. No version drift between your protoc plugin and the runtime, which was a frequent source of pain on the old setup. On Gradle, the equivalent is the `com.google.protobuf` plugin plus the same starter dependency. It writes stubs to `build/generated/source/proto/main`. Run `./mvnw clean package` (or `./gradlew build`) once and the generated classes show up on the compile path. ## How do you define a gRPC service with a proto schema? You write a `.proto` schema, let the plugin generate the base class, then extend that base class in a Spring bean. Start with the contract in `src/main/proto/greeter.proto`: ```proto syntax = "proto3"; option java_multiple_files = true; option java_package = "com.example.grpc.proto"; package greeter; service Greeter { rpc SayHello (HelloRequest) returns (HelloReply) {} } message HelloRequest { string name = 1; } message HelloReply { string message = 1; } ``` After a build, the plugin generates `GreeterGrpc` plus the message types. To implement the service, extend the generated `GreeterImplBase` and register it as a Spring bean. The key detail: Spring gRPC picks up any bean of type `BindableService`, so a plain `@Service` is enough. You do not need a special annotation just to expose the service. ```java import com.example.grpc.proto.GreeterGrpc; import com.example.grpc.proto.HelloReply; import com.example.grpc.proto.HelloRequest; import io.grpc.stub.StreamObserver; import org.springframework.stereotype.Service; @Service public class GreeterService extends GreeterGrpc.GreeterImplBase { @Override public void sayHello(HelloRequest request, StreamObserver responseObserver) { if (request.getName().isBlank()) { throw new IllegalArgumentException("name must not be blank"); } HelloReply reply = HelloReply.newBuilder() .setMessage("Hello " + request.getName()) .build(); responseObserver.onNext(reply); responseObserver.onCompleted(); } } ``` That's the entire server. Run the app, and the gRPC server listens on 9090. You can tune it through properties, no Java config required: ```yaml spring: grpc: server: port: 9090 max-inbound-message-size: 4MB ``` If you want per-service interceptors, that's where the `@GrpcService` annotation comes in. Annotate the bean with `@GrpcService(interceptors = AuditInterceptor.class)` to attach interceptors to one service, or register a global one with `@GlobalServerInterceptor` on a `ServerInterceptor` bean. For a single shared concern like request logging, I reach for the global interceptor and order it with `@Order`. ![Spring Boot 4.1 gRPC request flow from client stub through HTTP/2 to the Netty server, the GreeterService bean, and the GrpcAdvice error path](/images/blogs/spring-boot-4-1-grpc-auto-configuration-flow.webp) ## How do you handle gRPC errors with @GrpcAdvice? You put your error translation in one class annotated with `@GrpcAdvice` and add `@GrpcExceptionHandler` methods that map a Java exception to a gRPC `Status`. This is the gRPC analogue of `@ControllerAdvice` in Spring MVC, and it's one of the additions Spring Boot 4.1 highlights. Without it, every service method has to catch its own exceptions and translate them into status codes, which gets repetitive fast. With `@GrpcAdvice`, the `GreeterService` above can just throw a normal `IllegalArgumentException` and let the advice turn it into a proper `INVALID_ARGUMENT` response: ```java import io.grpc.Status; import io.grpc.StatusRuntimeException; @GrpcAdvice public class GrpcErrorAdvice { @GrpcExceptionHandler(IllegalArgumentException.class) public StatusRuntimeException handleBadInput(IllegalArgumentException ex) { return Status.INVALID_ARGUMENT .withDescription(ex.getMessage()) .asRuntimeException(); } } ``` A handler method takes the exception type as its argument and returns a `Status`, `StatusException`, or `StatusRuntimeException`. Return the `Status` directly when you don't need trailing metadata, or build a `StatusRuntimeException` with a `Metadata` object when you want to attach extra detail like an error code or a retry hint. Both annotations live in Spring gRPC's server exception package. For dynamic cases where one method must decide across many exception types, Spring gRPC also lets you register a functional `GrpcExceptionHandler` bean that returns a `Status` for the exceptions it recognizes and `null` for the rest, so other handlers get a turn. I stick with the annotation style for clarity and only drop to the functional bean when the mapping is computed at runtime. The same instinct I described in the [Debezium outbox pattern guide](https://www.rabinarayanpatra.com/blogs/implementing-outbox-pattern-cdc-microservices) applies here: keep cross-cutting translation in one obvious place. ## How do you choose between the Netty and Servlet transports? You pick Netty for a standalone gRPC server on its own port, and the Servlet transport when you want gRPC to share your existing web server. Both speak HTTP/2. The choice is a single dependency swap, and it changes which properties apply. The default starter, `spring-grpc-spring-boot-starter`, runs an embedded Netty server. The Servlet variant, `spring-grpc-server-web-spring-boot-starter`, runs gRPC inside the same Servlet container that serves your REST controllers, so they share a port. One thing that trips people up: under the Servlet transport, the `spring.grpc.server.*` properties are ignored except `max-inbound-message-size`. You configure the address through the normal `server.*` properties instead, because there is only one web server now. ![Comparison table of the Netty and Servlet HTTP/2 transports for gRPC in Spring Boot 4.1, covering dependency, port behavior, configuration properties, and when to use each](/images/blogs/spring-boot-4-1-grpc-auto-configuration-transports.webp) My rule of thumb: a dedicated internal microservice that only speaks gRPC gets Netty, because a separate port keeps concerns clean and lets me scale and secure it independently. A service that already exposes REST and wants to add a few gRPC endpoints behind one ingress gets the Servlet transport, since fronting two ports through a gateway is more trouble than it's worth. If you're already deep into ahead-of-time builds, both transports work under the AOT engine I covered in the [Spring Boot 4 AOT data repositories post](https://www.rabinarayanpatra.com/blogs/spring-boot-4-aot-data-repositories). ![Before and after wiring comparison: pre-4.1 manual gRPC server beans and a third-party starter versus Spring Boot 4.1 zero-config auto-configuration](/images/blogs/spring-boot-4-1-grpc-auto-configuration-before-after.webp) ## How do you add observability and mTLS to your gRPC server? You get metrics and tracing by adding Actuator, and you get mutual TLS by pointing the server at an SSL bundle. Both are property-driven, which is the part that saves the most boilerplate. For observability, add the Actuator starter and a Micrometer bridge for your platform. Spring gRPC auto-configures an `ObservationGrpcServerInterceptor` that produces per-call metrics and trace spans, and it respects custom observation conventions if you register one: ```xml org.springframework.boot spring-boot-starter-actuator ``` Once that's on the classpath, every RPC shows up as timed metrics, and if you've wired tracing, each call carries a span you can follow across services. I didn't have to touch a single interceptor registration to get this. For transport security, Spring gRPC reuses Spring Boot's SSL bundles, so the same bundle format you use for HTTPS covers gRPC. Mutual TLS means the server presents a certificate and also verifies the client's certificate against a trust store: ```yaml spring: grpc: server: ssl: bundle: grpc-server ssl: bundle: jks: grpc-server: keystore: location: classpath:server.p12 password: secret type: PKCS12 truststore: location: classpath:server-truststore.p12 password: secret ``` The trust store is what makes it mutual. The server checks each incoming client certificate against it, so an unknown client gets rejected at the TLS layer before any RPC runs. On the client side you set `negotiation-type: TLS` and point at the matching bundle, which I'll show next. ## How do you call the gRPC service from a Spring client? You import the generated client stubs with `@ImportGrpcClients` and inject a blocking stub like any other bean. Spring gRPC builds the channel from your client properties, so you never touch `ManagedChannelBuilder` directly. Turn on stub scanning at the application class, pointing it at the package that holds the generated `GreeterGrpc`: ```java import com.example.grpc.proto.GreeterGrpc; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.grpc.client.ImportGrpcClients; @SpringBootApplication @ImportGrpcClients(basePackageClasses = GreeterGrpc.class) public class ClientApplication { public static void main(String[] args) { org.springframework.boot.SpringApplication.run(ClientApplication.class, args); } } ``` Now the blocking stub is injectable. Constructor injection works the same as it does for a repository or a `RestClient`: ```java import com.example.grpc.proto.GreeterGrpc; import com.example.grpc.proto.HelloReply; import com.example.grpc.proto.HelloRequest; import org.springframework.stereotype.Component; @Component public class GreeterClient { private final GreeterGrpc.GreeterBlockingStub stub; public GreeterClient(GreeterGrpc.GreeterBlockingStub stub) { this.stub = stub; } public String greet(String name) { HelloReply reply = stub.sayHello( HelloRequest.newBuilder().setName(name).build()); return reply.getMessage(); } } ``` The stub needs an address. Configure the default channel, or define named channels when the app talks to several services: ```yaml spring: grpc: client: default-channel: address: 'localhost:9090' channels: greeter: address: 'localhost:9090' negotiation-type: TLS ssl: bundle: grpc-client ``` If you'd rather build the stub yourself, inject a `GrpcChannelFactory` and call `channels.createChannel("greeter")`, then wrap it with `GreeterGrpc.newBlockingStub(...)`. In tests, the `@LocalGrpcPort` annotation hands you the random port of an in-process server, so an integration test can call the real service without binding 9090. That made my first round of contract tests far less fiddly than the manual channel juggling I used to do. ## Should you adopt Spring Boot 4.1 gRPC auto-configuration? If you run gRPC and you're on Spring Boot, yes, and the reason is maintenance, not novelty. The old community starters were good work, but they sat outside the Spring release train, so every Boot bump carried a question mark. Folding gRPC into Spring Boot 4.1 puts it on the same upgrade cadence as everything else you depend on, which is the kind of boring stability that pays off two years from now. What I didn't expect was how much the SSL bundle reuse would matter. Sharing one certificate configuration format across HTTPS and gRPC means there's one place to rotate certs, not two, and that alone removes a class of mismatched-config incidents. The annotation-driven error handling is the other quiet win, because it pulls status mapping out of your business logic where it never belonged. Start with the Netty transport on a single internal service, get the observability wired, and you'll have a template you can copy across the fleet. For the official details, see the [Spring Boot 4.1 release announcement](https://spring.io/blog/2026/06/10/spring-boot-4/), the [Spring gRPC reference documentation](https://docs.spring.io/spring-grpc/reference/), and InfoQ's [Spring Boot 4.1 release coverage](https://www.infoq.com/news/2026/06/spring-boot-4-1/). ### Keep Reading - [How to Version REST APIs in Spring Framework 7](https://www.rabinarayanpatra.com/blogs/how-to-version-rest-apis-spring-framework-7). The same convention-first design shows up in Spring's REST versioning, a useful companion when a service speaks both protocols. - [Spring Boot 4 AOT Data Repositories](https://www.rabinarayanpatra.com/blogs/spring-boot-4-aot-data-repositories). If you run native images, this explains how the AOT engine treats your beans, which applies to gRPC services too. - [How to Implement the Debezium Outbox Pattern in Spring Boot](https://www.rabinarayanpatra.com/blogs/implementing-outbox-pattern-cdc-microservices). A practical look at keeping cross-service messaging reliable once your gRPC services start talking to each other. - [Spring AI 2.0 MCP Annotations: From Tool to Production](https://www.rabinarayanpatra.com/blogs/spring-ai-2-mcp-annotations-tutorial). Another case where Spring Boot 4 moved a fast-moving integration into first-class, annotation-driven support. --- ## How to Style Scrollbars Without a Plugin in Tailwind v4.3 Source: https://www.rabinarayanpatra.com/blogs/tailwind-v4-3-scrollbar-styling-no-plugin Published: 2026-09-23T00:00:00+05:30 Tags: tailwind-v4-scrollbar-styling, tailwind-css-v4-3, tailwind-css, scrollbar, css, frontend, web-development Tailwind v4.3 ships native scrollbar utilities. Delete the tailwind-scrollbar plugin and use scrollbar-thin, scrollbar-thumb, and scrollbar-gutter directly. I deleted `tailwind-scrollbar` from a dashboard package.json last week. Eight files updated. Two minutes. Zero visual regression. That dependency has been sitting in the project since 2022, ridden along through three Tailwind major upgrades, just to give us a thin gray scrollbar. Tailwind v4.3 dropped May 8, 2026 and pulled the plugin's most-used classes into core. `scrollbar-thin`, `scrollbar-thumb-*`, `scrollbar-track-*`, plus a fresh `scrollbar-gutter-*` set for layout-shift prevention. All of it native. No plugin, no extra config in `tailwind.config.ts`, no fighting over Tailwind v4's CSS-first config story. I rebuilt five scrollbar patterns I actually ship with the new utilities, and here's the whole sweep so you can pull the plugin out of your project too. ![Cover: Tailwind v4.3 scrollbar utilities](/images/blogs/tailwind-v4-3-scrollbar-styling-no-plugin.webp) ## What scrollbar utilities did Tailwind v4.3 add? Tailwind v4.3 added three families of scrollbar utilities mapping directly onto the CSS `scrollbar-width`, `scrollbar-color`, and `scrollbar-gutter` properties. None of them need a plugin. Width controls how thick the scrollbar appears: - `scrollbar-auto`: browser default thickness - `scrollbar-thin`: a narrower scrollbar (Firefox and modern WebKit only) - `scrollbar-none`: hides the scrollbar entirely but keeps the element scrollable Color controls the thumb and track. Both accept any Tailwind color token plus opacity modifiers: - `scrollbar-thumb-*`: the draggable part - `scrollbar-track-*`: the background behind the thumb Gutter controls the reserved space for a scrollbar, even when one isn't visible. This is the layout-shift fix that used to require manual CSS: - `scrollbar-gutter-auto`: default, no reservation - `scrollbar-gutter-stable`: always reserve scrollbar space - `scrollbar-gutter-both`: reserve space on both edges so content stays centered All eleven classes ship in core. No extras. The release also brought new colors, container size utilities, and faster webpack builds, but the scrollbar set is what kills the long-standing plugin dependency for most teams. ## How do you replace the tailwind-scrollbar plugin with native v4.3 classes? You delete the plugin from `package.json`, remove the import from your CSS entry, and the existing class names mostly keep working because the v4.3 utilities use the same names. The migration is one of the cleanest Tailwind has shipped in a major release. Before, in `src/styles.css`: ```css @import "tailwindcss"; @plugin "tailwind-scrollbar"; ``` After: ```css @import "tailwindcss"; ``` Then drop the dependency: ```bash npm uninstall tailwind-scrollbar ``` Most class names are identical so your existing markup keeps rendering. The places to look for breakage: - Plugin-only modifiers like `scrollbar-w-1.5` (custom widths). Tailwind v4.3 doesn't expose arbitrary widths; you get `thin` or `auto`. Move custom widths to a `@layer utilities` block. - The plugin's `scrollbar-rounded` class. Tailwind v4.3 doesn't add scrollbar border-radius. If you need rounded thumbs, you keep `::-webkit-scrollbar-thumb { border-radius }` in your CSS. - WebKit pseudo-element selectors the plugin generated under the hood. If you used `scrollbar-thumb-rounded-md`, that's now hand-written CSS. In a 12-component dashboard I migrated, only two files needed hand edits, both for rounded thumbs. Everything else swapped through the rename rules above. ## How do you style a thin branded scrollbar with custom colors? Pair `scrollbar-thin` with `scrollbar-thumb-*` and `scrollbar-track-*` on the scrolling container. Tailwind v4.3 accepts the full color palette, opacity modifiers, and custom theme tokens defined in your `@theme` block. A branded scrollbar on a long article reader: ```html
``` With opacity for a softer thumb that blends into a marketing page: ```html ``` If your design system defines brand tokens in `@theme`, those work too: ```css /* styles.css */ @import "tailwindcss"; @theme { --color-brand: oklch(0.62 0.18 250); --color-brand-muted: oklch(0.95 0.02 250); } ``` ```html
``` One trap to watch: WebKit before Safari 18.2 doesn't honor `scrollbar-color` for non-overlay scrollbars. On macOS where the system setting is "Always show scrollbars", you'll see the default native scrollbar instead of your branded one. That's a browser limitation, not a Tailwind issue. The class still ships safely. ## How do you prevent layout shift with scrollbar-gutter? Add `scrollbar-gutter-stable` to any container that may grow tall enough to need a scrollbar. The browser reserves space for the scrollbar whether it's visible or not, so the content width stays constant when overflow kicks in. The classic case is a modal whose body switches between short and long content: ```html
``` Without `scrollbar-gutter-stable`, the moment content grows past the height, the scrollbar appears and pulls everything 15px to the left. Form inputs jump, button rows resize, anchored tooltips shift. With it, the gutter is always there, hidden when not in use. For centered layouts where the gutter showing on only one side would break visual balance, use `scrollbar-gutter-both`: ```html
``` I default `scrollbar-gutter-stable` on every ``, side panel, and modal in our component library now. The class is essentially free (no visible change when no scrollbar is needed) and prevents the most annoying class of CSS bug. ## How do you hide a scrollbar while keeping the area scrollable? Use `scrollbar-none` on the scrolling container. The browser still lets you scroll the area, drag with a mouse wheel, swipe with a trackpad, or use keyboard arrow keys; the scrollbar UI just doesn't paint. Common case: a horizontal chip list that scrolls but shouldn't show a scrollbar across the bottom. ```html
``` `scrollbar-none` maps to `scrollbar-width: none` in Firefox and `::-webkit-scrollbar { display: none }` in WebKit. Tailwind v4.3 emits both rules so the result is consistent across browsers without you writing the WebKit selector yourself. This is the one place where I previously needed a plugin or a hand-written `@layer utilities` block. Now it's a single class. ## How do you make the scrollbar dark-mode aware? Combine `scrollbar-thumb-*` and `scrollbar-track-*` with Tailwind's `dark:` variant. The utilities respect every variant the rest of Tailwind does, so dark mode, hover, focus, and responsive prefixes all stack. ```html
``` If you've configured Tailwind v4 to use the `class` strategy for dark mode (the default in v4 is `prefers-color-scheme: dark`, but the `class` strategy is one line in `@theme`), the variant works the same way. I also use the `hover:` variant on the thumb for a subtle interactive cue on dashboards: ```html
``` `scrollbar-track-transparent` is the move when you want only the thumb visible against the page background. It's also the cleanest match for glassmorphism panels where any solid track color fights the blur. ## What's the browser support story for scrollbar-thin and scrollbar-color? The CSS properties Tailwind v4.3 maps onto are stable across modern browsers. The exact version floor for production is Firefox 64 (all the way back to 2018), Chrome 121, and Safari 18.2. Older versions render the default browser scrollbar instead, so the classes are safe to ship; the visual degradation is the OS-default scrollbar. A quick table of what each utility hits: | Utility | CSS property | Firefox | Chrome | Safari | |---------|--------------|---------|--------|--------| | `scrollbar-{auto,thin}` | `scrollbar-width` | 64+ | 121+ | 18.2+ | | `scrollbar-thumb-*`, `scrollbar-track-*` | `scrollbar-color` | 64+ | 121+ | 18.2+ | | `scrollbar-gutter-*` | `scrollbar-gutter` | 97+ | 94+ | 17+ | | `scrollbar-none` | width + WebKit fallback | 64+ | all | all | `scrollbar-gutter` actually has wider support than `scrollbar-width` on Safari, which surprised me. If you care about Safari 17.x users, you can ship `scrollbar-gutter-stable` knowing layout shift is fixed there even if the thumb color isn't yet styled. For anyone still supporting Chrome 120 and older (Stripe checkout, government dashboards), the fallback is the browser default scrollbar. No layout breakage, no JavaScript error, just the OS look. ![Tailwind v4.3 scrollbar utility patterns](/images/blogs/tailwind-v4-3-scrollbar-styling-no-plugin-patterns.webp) ## What scrollbar tricks still need raw CSS in Tailwind v4.3? Anything that needs WebKit pseudo-elements directly still needs hand-written CSS. The CSS `scrollbar-color` property gives you a single thumb color and a single track color; if you want a gradient thumb, a rounded thumb, padding inside the track, or a different color on hover, you go back to `::-webkit-scrollbar` selectors. A rounded thumb in a side panel: ```css @layer utilities { .scrollbar-rounded::-webkit-scrollbar-thumb { border-radius: 9999px; border: 2px solid transparent; background-clip: padding-box; } } ``` Then: ```html
``` The pattern: use the Tailwind v4.3 utility for the parts the CSS standard covers (`scrollbar-width`, `scrollbar-color`, `scrollbar-gutter`), and layer `::-webkit-scrollbar-thumb` rules for the visual finishes the standard doesn't expose. That's still less custom CSS than the plugin generated. The other gap: scrollbar arrow buttons. Firefox and WebKit both support hiding or styling arrows differently, and neither CSS standard nor Tailwind v4.3 has a utility for that. For overlay-style scrollbars (the macOS default) there are no buttons, so this only matters on Windows. If you need it, raw `::-webkit-scrollbar-button` rules in your CSS. ## What should you do next with Tailwind scrollbar utilities? Open your project's `package.json` and search for `tailwind-scrollbar`. If it's there, drop it, delete the plugin import, and run your build to see if any custom widths or rounded-thumb classes break. The migration takes about ten minutes for a typical dashboard and removes a transitive dependency you've probably never audited. The rollout I'd suggest: 1. **Upgrade Tailwind to 4.3 or later.** The release came out May 8, 2026 and patch releases since have been bug fixes only. 2. **Grep for `scrollbar-` classes.** Most usages map 1:1 with the new core utilities. 3. **Remove the plugin.** Drop the `@plugin` line in your CSS entry, uninstall the package, run the build. 4. **Add `scrollbar-gutter-stable` to dialogs and side panels.** Free fix for a class of layout shift bugs you've probably been living with. 5. **Audit dark-mode pairs.** Wherever you set a light-mode scrollbar color, add a `dark:` companion. The new utilities support the variant directly. That sequence dropped 7KB from our shipped CSS and one package from the supply-chain surface. Small wins, but they compound across the dozen projects I maintain. For more on Tailwind v4.3 and the CSS scrollbar APIs, see the [official v4.3 release post](https://tailwindcss.com/blog/tailwindcss-v4-3), the [Tailwind v4.3 release notes on GitHub](https://github.com/tailwindlabs/tailwindcss/releases/tag/v4.3.0), and the [MDN reference on scrollbar-gutter](https://developer.mozilla.org/en-US/docs/Web/CSS/scrollbar-gutter). ### Keep Reading - [How to Set Up Sanity Studio in Next.js 16 (Embedded Dashboard)](https://www.rabinarayanpatra.com/blogs/how-to-set-up-sanity-studio-nextjs-16). Pairs with this for a real Tailwind-styled admin dashboard. - [How to Use the React Activity Component for Instant Navigation](https://www.rabinarayanpatra.com/blogs/react-activity-component). Another modern frontend primitive worth picking up alongside the new utilities. - [Goodbye middleware.ts, Hello proxy.ts: The Next.js 16 Migration Guide](https://www.rabinarayanpatra.com/blogs/hello-proxy-ts-nextjs-16). Another small migration that compounds across projects. --- ## How to Redact Secrets from Java Flight Recorder in JDK 27 Source: https://www.rabinarayanpatra.com/blogs/redact-secrets-java-flight-recorder-jdk-27-jep-536 Published: 2026-09-18T00:00:00+05:30 Tags: java-flight-recorder-redact-secrets, jep-536, jdk-27, jfr, java-security, spring-boot, observability Learn how JEP 536 makes JDK 27 Flight Recorder redact secrets in-process. Walk through redact-key and redact-argument flags with a Spring Boot example. A few months back I shipped a JFR profile to a vendor support ticket. Two days later they came back asking why my AWS access key was in the dump. Turns out Flight Recorder captures every environment variable at startup by default. The `AWS_ACCESS_KEY_ID`, the JDBC password, the Slack webhook token, all of it sitting in plain text inside `recording.jfr`. Rotating keys on a Friday because I forgot what JFR captures was not how I wanted to spend the weekend. JDK 27 finally fixes this. JEP 536 makes the JVM redact known secret-shaped names from JFR before the recording ever touches disk. No more post-processing scripts. No more "send me the JFR but scrub it first" instructions in tickets. I rebuilt the leak reproduction on JDK 27 Rampdown Phase One, set up a Spring Boot 4.1 service, and put the full redaction config together for this post. ![Cover: JFR redaction in JDK 27](/images/blogs/redact-secrets-java-flight-recorder-jdk-27-jep-536.webp) ## What is JEP 536 and how does it redact secrets in JFR? JEP 536 enhances Java Flight Recorder to redact sensitive command-line arguments, environment variables, and system properties before the recording file is written to disk. The proposal advanced from Proposed-to-Target to Targeted for JDK 27 in early June 2026 and is included in the Rampdown Phase One build that locks the feature set for the September 2026 release. JFR has always captured your process bootstrap. Three event types in particular pick up secrets: - `jdk.InitialEnvironmentVariable`: every env var the JVM saw at launch - `jdk.InitialSystemProperty`: every `-D` flag plus the JVM-set props - `jdk.JVMInformation`: the full command line that started the process Until JDK 27, the values went straight into the recording file. If you ran `java -Dapi.token=hunter2 -jar app.jar` or your container set `SPRING_DATASOURCE_PASSWORD=hunter2`, both ended up in plain text inside any JFR dump anyone could capture. JEP 536 introduces a filter that runs inside the JVM before the event is written. Two sub-options of `-XX:FlightRecorderOptions` drive it: - `redact-key` matches against the name of an env var, system property, or `-D` key. A match redacts the value. - `redact-argument` matches against the value of a command-line argument. A match redacts the whole argument. The defaults are deliberately greedy. Out of the box the key filter covers patterns like `*auth*`, `*password*`, `*passwd*`, `*pwd*`, `*passphrase*`, `*token*`, `*secret*`, `*credential*`, `*api-key*`, `*api_key*`, `*apikey*`, `*client-secret*`, `*private-key*`, and `*jaas*config*`. That covers the names most Java apps actually use. Custom names like `INTERNAL_SESSION` slip through, which is why the extension syntax matters. ![JFR redaction pipeline in JDK 27](/images/blogs/redact-secrets-java-flight-recorder-jdk-27-jep-536-pipeline.webp) ## How do you enable default JFR redaction in JDK 27? Default JFR redaction is on automatically in JDK 27 with no extra flags required. You start a recording the same way you always have: ```bash java -XX:StartFlightRecording=duration=60s,filename=app.jfr \ -jar app.jar ``` Inspect what landed in the file with the `jfr` tool that ships with the JDK: ```bash jfr print --events jdk.InitialEnvironmentVariable app.jfr | head -20 ``` On JDK 27 with redaction active, sensitive entries come out as ``: ``` jdk.InitialEnvironmentVariable { startTime = 2026-08-21T09:14:22.001Z key = "AWS_SECRET_ACCESS_KEY" value = "" } jdk.InitialEnvironmentVariable { startTime = 2026-08-21T09:14:22.001Z key = "PATH" value = "/usr/local/bin:/usr/bin:/bin" } ``` `PATH` is not on the filter, so it shows through. `AWS_SECRET_ACCESS_KEY` matches `*secret*` and gets masked. The key stays visible so you can still tell what was set, just not the value. For comparison, the same recording on JDK 26 would have printed the actual key value in plain text. Devs running JDK 27 in production right after release will see redaction working without any operational change. ## How do you add custom redaction filters? You add custom filters by passing your patterns to `redact-key` or `redact-argument` and prefixing the first one with `+` to extend the defaults instead of replacing them. The syntax goes into `-XX:FlightRecorderOptions` using its colon-separated form: ```bash java -XX:FlightRecorderOptions:redact-key='+*confidential*' \ -XX:StartFlightRecording=duration=60s,filename=app.jfr \ -jar app.jar ``` The `+` tells JFR to extend the defaults. Without it, your filter replaces them entirely: ```bash # DANGER: replaces all defaults. Now only INTERNAL_TOKEN is masked. java -XX:FlightRecorderOptions:redact-key='INTERNAL_TOKEN' \ -jar app.jar ``` Multiple patterns join with `;`. Wildcards use `*`. To extend the defaults with two custom names: ```bash java -XX:FlightRecorderOptions:redact-key='+INTERNAL_TOKEN;*license*' \ -jar app.jar ``` If you ever need to disable redaction entirely (testing, debugging the filter), pass `none`: ```bash java -XX:FlightRecorderOptions:'redact-key=none,redact-argument=none' \ -jar app.jar ``` I'd avoid disabling globally. If you need to see an unredacted value during a debug session, capture a fresh recording with the explicit `none` option in a controlled environment. ## How do you redact command-line arguments that contain credentials? Use `redact-argument` with a glob pattern that matches the argument shape, such as `http://*:*@*` for a URL with embedded `user:password`. By default JFR captures every argument whole inside `jdk.JVMInformation`. Add an argument filter to mask the matched value: ```bash java -XX:FlightRecorderOptions:redact-argument='http://*:*@*;https://*:*@*' \ -jar app.jar https://admin:hunter2@example.com/api ``` The matched argument shows up as `` in the dump. Glob matching applies to the full value, so design patterns that bracket the credentials with enough context to avoid false positives. JDBC URLs are another common case: ```bash java -XX:FlightRecorderOptions:redact-argument='+jdbc:*://*:*@*' \ -jar app.jar ``` The `+` again tells JFR to keep its default argument filter and add the JDBC pattern on top. For Slack webhooks, GitHub tokens passed positionally, or any internal CLI tool with `--token=xxx` shapes, you build the same way: match the surrounding structure tightly, let the secret get masked. ## How do you load redaction filters from a file? Prefix the filename with `@` and pass it through `redact-key` or `redact-argument` to load patterns from disk: ```bash java '-XX:FlightRecorderOptions:redact-key=@/etc/jfr-redact-keys.txt,redact-argument=@/etc/jfr-redact-args.txt' \ -jar app.jar ``` The file holds filters using the same `;`-separated syntax as inline values. A typical key file at `/etc/jfr-redact-keys.txt`: ``` +*license*;*internal*token*;ACCESS_KEY_ID;SLACK_WEBHOOK ``` The leading `+` extends defaults exactly like inline. There is no comment marker in the spec, so don't add `#`-style lines. I keep the redaction files in the same Ansible role that provisions our JDK runtimes. That way the redaction policy stays version-controlled alongside the rest of the JVM tuning. A baseline file for our services looks like this: ``` +*confidential*;INTERNAL_API_TOKEN;GITHUB_APP_PRIVATE_KEY;DD_API_KEY;NEW_RELIC_LICENSE_KEY;*encryption*material* ``` Ship this file with your container image, point `redact-key=@/path/to/keys` at it, and every Flight Recorder dump from that image is safe to attach to a support ticket. ## How do you verify JFR redaction in a Spring Boot service? Set a known secret-named environment variable, run your Spring Boot 4.1 service with JFR enabled, then inspect the recording with `jfr print` to confirm the value shows as ``. Here is a minimal repro you can run locally on JDK 27 Early Access. ```java @SpringBootApplication public class DemoApplication { public static void main(String[] args) { SpringApplication.run(DemoApplication.class, args); } } ``` With `application.properties`: ```properties spring.datasource.url=jdbc:postgresql://localhost:5432/demo spring.datasource.username=demo spring.datasource.password=${SPRING_DATASOURCE_PASSWORD} ``` Set the secret in the environment and launch with JFR: ```bash export SPRING_DATASOURCE_PASSWORD='hunter2-jdbc' export INTERNAL_SESSION_TOKEN='not-a-real-token' java -XX:FlightRecorderOptions:redact-key='+*session*token*' \ -XX:StartFlightRecording=duration=30s,filename=demo.jfr \ -jar target/demo-0.0.1-SNAPSHOT.jar ``` After the recording window closes, inspect the relevant events: ```bash jfr print --events jdk.InitialEnvironmentVariable demo.jfr \ | grep -A 2 -E 'SPRING_DATASOURCE_PASSWORD|INTERNAL_SESSION_TOKEN|PATH' ``` You should see: ``` jdk.InitialEnvironmentVariable { key = "SPRING_DATASOURCE_PASSWORD" value = "" } jdk.InitialEnvironmentVariable { key = "INTERNAL_SESSION_TOKEN" value = "" } jdk.InitialEnvironmentVariable { key = "PATH" value = "/usr/local/bin:/usr/bin:/bin" } ``` `SPRING_DATASOURCE_PASSWORD` is caught by the default `*password*` filter. `INTERNAL_SESSION_TOKEN` is caught by the custom `+*session*token*` extension. `PATH` is untouched because it isn't sensitive. I keep this exact reproduction inside the integration test suite for our base Java image. CI runs it on every JDK upgrade and fails the build if the redacted values aren't ``. That catches a future regression or a misconfigured Dockerfile that accidentally drops the redaction flags. ![Before and after JFR redaction comparison](/images/blogs/redact-secrets-java-flight-recorder-jdk-27-jep-536-before-after.webp) ## What does JFR redaction NOT cover? JFR redaction handles only command-line arguments, environment variables, and system properties. Heap dumps, custom event payloads, and any data you push into your own JFR events remain your responsibility. A few specific traps I've watched teams trip on: - **Heap dumps.** `-XX:+HeapDumpOnOutOfMemoryError` produces an `.hprof` file outside of JFR's scope. Strings holding secrets at OOM time go straight to disk. Treat heap dumps with the same care as raw JFRs. - **Custom JFR events.** If your code defines a `@Name("com.example.RequestEvent")` and pushes the request body into it, JEP 536 has nothing to do with that payload. You sanitize the body before you call `event.commit()`. The same goes for any libraries you embed that emit JFR events. - **Method sampling argument captures.** JFR's method profiler does not capture argument values by default. If you've extended the profiler or use a third-party agent that does, you re-create the leak. Confirm what your agent records. - **Thread names.** Some libraries inject identifiers into thread names. If those identifiers double as secret tokens, they show up in stack samples regardless of JEP 536. Audit thread names if your stack runs anything custom. - **Constants in stack traces.** Constant strings in bytecode aren't captured by JFR directly, but they may end up in heap dumps or error messages that flow into logs. JEP 536 closes the most embarrassing leak path. It doesn't replace a thoughtful secrets policy. Treat any process dump as sensitive by default. ## How do you redact older JFR recordings captured before JDK 27? JEP 536 isn't backported to JDK 21 or JDK 26, and the JFR file format does not retroactively redact already-captured events. Two practical options. First, treat the older recordings as secret-bearing artifacts. Encrypt them at rest. Restrict access. Rotate any credentials you suspect were captured. If a JFR sat unredacted in S3 for six months, the affected keys are blast-radius candidates whether the file was opened or not. Second, regenerate the profile on JDK 27 once you upgrade. Reproducing the original incident on the new runtime gets you a clean recording without changing your investigation. For production services running an older LTS, this is the only path. There is no in-place "redact this old file" tool because the redaction depends on JVM-level pattern matching at capture time. If you want the redaction earlier than your LTS upgrade plan allows, you can run a single non-LTS JDK 27 service for diagnostic-capture purposes only. I've done this for a payment service where rotating leaked credentials was costlier than running two JVM versions side by side for a quarter. ## What should you do next with JFR redaction in JDK 27? Audit every Java service running on JDK 27 to confirm the default filter covers your secret naming convention, then add custom `redact-key` entries for any internal name that escapes it. The default filter is tuned to common patterns (`*password*`, `*token*`, `*secret*`) but it doesn't know your `INTERNAL_API_TOKEN` or `LICENSE_PUBKEY_FINGERPRINT`. The rollout pattern I'd recommend: 1. **Dry-run in dev.** Spin up your service on JDK 27 with default redaction. Capture a 30-second JFR. Grep the output for any value that should have been redacted but wasn't. 2. **Author a filter file.** Put your internal patterns into `/etc/jfr-redact-keys.txt` and `/etc/jfr-redact-args.txt`. Commit them to the repo that builds your base image. 3. **Wire it into the JVM args.** Add `-XX:FlightRecorderOptions:redact-key=@...,redact-argument=@...` to your container entrypoint or systemd unit. 4. **Add a CI guard.** Run the Spring Boot reproduction from earlier in this post as an integration test. Fail the build if any non-`` value lands in the dump for a secret-named key. 5. **Document the policy.** Tell your incident-response team that JDK 27 JFRs are safe to share with vendors. Older JFRs aren't. That sequence took us about a sprint to ship across 14 services. The dollar cost of one accidental key disclosure pays it back a hundred times over. For more on JEP 536 and the JDK 27 release plan, see the [JEP 536 specification](https://openjdk.org/jeps/536), the [openjdk-dev candidate-JEP thread](https://mail.openjdk.org/archives/list/jdk-dev@openjdk.org/thread/EBK5L66WSFBTJLZD7QZPDNZSODMETM5C/), and [InfoQ's Java News Roundup confirming the JDK 27 targeting](https://www.infoq.com/news/2026/06/java-news-roundup-may25-2026/). ### Keep Reading - [How to Use Post-Quantum TLS in Java 27 (JEP 527)](https://www.rabinarayanpatra.com/blogs/post-quantum-tls-java-27-jep-527). Pairs with this for the full JDK 27 security upgrade story. - [How to Read and Write PEM Files in Java (JDK 27 PEM API)](https://www.rabinarayanpatra.com/blogs/read-write-pem-files-java-pem-api). Another JDK 27 security feature you'll want to know. - [Java 26 Structured Concurrency: What Changed in the Sixth Preview](https://www.rabinarayanpatra.com/blogs/java-26-structured-concurrency-jep-525). Adjacent JEP work in the preview track. --- ## How to Build a Voice Agent with Vercel AI SDK 7 Realtime Source: https://www.rabinarayanpatra.com/blogs/vercel-ai-sdk-7-realtime-voice-agent-nextjs Published: 2026-09-16T00:00:00+05:30 Tags: voice-agent-vercel-ai-sdk-7, ai-sdk-7, experimental-userealtime, voice-agent, nextjs, vercel-ai-sdk, realtime-api Build a voice agent in Next.js with Vercel AI SDK 7. Wire experimental_useRealtime, ephemeral tokens, and client-side tool calls in one App Router setup. The old way to ship a voice agent was a four-piece relay. You captured mic audio with WebRTC, streamed it through a speech-to-text service, passed the transcript into a chat LLM, piped the reply into a text-to-speech vendor, and pumped the audio back into the browser. Each hop added latency, each vendor charged separately, and the glue code never quite stayed glued. Vercel AI SDK 7 canary just collapsed all of that into a single React hook. `ai@7.0.0-canary.165` shipped on June 5, 2026 with `experimental_useRealtime`, a new `Experimental_RealtimeModelV4` provider spec, and realtime implementations for OpenAI, Google, and xAI. I spent a Saturday porting one of my side projects to it, and the diff is brutal. I deleted a `lib/voice/` folder, three vendor SDK wrappers, and a custom event bus. This post walks through what shipped, how to wire it into a Next.js 16 App Router project, and the canary caveats you should know before you stake a roadmap on it. ![Architecture diagram showing browser mic, ephemeral token route, useRealtime hook, and realtime provider returning audio plus tool calls](/images/blogs/vercel-ai-sdk-7-realtime-voice-agent-nextjs-architecture.webp) ## Why does Vercel AI SDK 7's Realtime API matter for voice agents? The Realtime API matters because it standardizes the four-piece voice pipeline into one provider call and one React hook. `Experimental_RealtimeModelV4` lives in `@ai-sdk/provider` and defines normalized event types and factory methods that every provider implements the same way. OpenAI, Google, and xAI each expose an `experimental_realtime()` factory you can swap with a one-line change, and both server and browser environments are supported. The other piece I care about is the ephemeral token model. A long-lived provider API key never touches the browser. You expose a server route that calls `.getToken()` on the realtime model, return a short-lived token, and let the client connect with that. This is the same pattern OpenAI documents for their own Realtime API, but the SDK now wraps it for every supported provider with a single call. There is a real reason this matters for product teams, not just engineers who like clean APIs. Voice agent latency is dominated by the slowest hop in your pipeline. When I stitched my own stack last year, my median user-spoken-to-model-spoken round trip sat at 1.8 seconds. The first cut of the Realtime API path puts me at 480 ms, almost all of it network. I have not done a head-to-head benchmark yet, but the qualitative jump is the kind users notice mid-sentence. ![Comparison table: old custom stack (Whisper STT + GPT-4 + ElevenLabs TTS + WebRTC glue) versus new SDK 7 stack (one provider + one hook)](/images/blogs/vercel-ai-sdk-7-realtime-voice-agent-nextjs-comparison.webp) ## How do you set up the server-side token route for experimental_useRealtime? You set it up by exposing a Next.js App Router route handler that calls the provider's `.getToken()` static method and returns the result as JSON. The browser fetches that route on mount, hands the token to the hook, and the SDK opens the session. Install the canary release first. Pin it. Canary versions ship every day and the surface still moves. ```bash npm install ai@canary @ai-sdk/react@canary @ai-sdk/openai@canary ``` Then drop a route handler under `app/api/realtime-token/route.ts`. This example uses OpenAI, but the only line that changes for Google or xAI is the import and the factory call. ```typescript import { openai } from '@ai-sdk/openai'; import { Experimental_RealtimeModelV4 } from '@ai-sdk/provider'; export const runtime = 'nodejs'; export async function POST() { const token = await openai .experimental_realtime('gpt-realtime') .getToken({ voice: 'verse', modalities: ['audio', 'text'], inputAudioTranscription: { model: 'whisper-1' }, }); return Response.json({ token }); } ``` A few things worth pointing out. The route is `POST` so the token request never gets cached by an intermediate proxy. I set `runtime = 'nodejs'` because the SDK pulls in Node crypto for token signing in some provider paths. The token is short-lived, usually under a minute, so any leak window is small. Provider switching is a one-import change. To run the same agent on Google, the body becomes `google.experimental_realtime('gemini-2.5-realtime').getToken({ ... })`. The argument shape is the same. The route signature does not move. This is the part I like best about how the spec is designed. Your UI does not know which provider is on the other end of the wire, so you can A/B providers per session without touching the client. Keep your provider key in `OPENAI_API_KEY` (or `GOOGLE_GENERATIVE_AI_API_KEY`, `XAI_API_KEY`). The SDK reads them from `process.env` by default. Never put any of these on the client. ## How do you wire the experimental_useRealtime hook in a React client? You wire it by importing `experimental_useRealtime` from `@ai-sdk/react`, pointing its `getToken` option at the route you just built, and rendering whatever UI you want around the returned `messages` array. The hook returns the same `UIMessage[]` shape as `useChat`, so any existing chat surface stays as is. Here is a minimal client component. It asks for mic permission, mounts the hook, and prints the running transcript. ```tsx 'use client'; import { experimental_useRealtime } from '@ai-sdk/react'; import { useEffect, useState } from 'react'; export function VoiceAgent() { const [micReady, setMicReady] = useState(false); useEffect(() => { navigator.mediaDevices .getUserMedia({ audio: true }) .then(() => setMicReady(true)); }, []); const { messages, status, connect, disconnect } = experimental_useRealtime({ getToken: async () => { const res = await fetch('/api/realtime-token', { method: 'POST' }); const { token } = await res.json(); return token; }, }); if (!micReady) return

Allow microphone access to start.

; return (
    {messages.map((m) => (
  • {m.role}: {m.parts.map((p, i) => p.type === 'text' ? {p.text} : null, )}
    • ))}
); } ``` `status` walks the obvious states (`'disconnected'`, `'connecting'`, `'connected'`, `'error'`), which is plenty for a status pill in your UI. The hook also returns `addToolOutput`, which I get to in the next section. I have not had to touch the underlying WebRTC peer connection in any project so far, and that absence is the entire point. One pattern I picked up the hard way: do not call `connect()` inside the same effect that asks for microphone permission. Browsers treat the permission prompt as a user gesture boundary, and starting an audio session before the prompt resolves leads to silent failures on Safari. Wait for the `getUserMedia` promise, then surface a button, then connect on click. ![Sequence diagram: getToken returns ephemeral token, useRealtime connects, audio frames stream both ways, onToolCall fires, addToolOutput sends result, audio plays back](/images/blogs/vercel-ai-sdk-7-realtime-voice-agent-nextjs-sequence.webp) ## How do you execute tools client-side with onToolCall and addToolOutput? The hook calls `onToolCall` whenever the model asks for a tool, you run the JavaScript locally, and then you push the result back with `addToolOutput`. The model picks up the next turn from there. This is a real shift from server-side tool execution because the tool can read browser state, the user's clipboard, or a Vercel KV cache, without a network hop. `experimental_getRealtimeToolDefinitions` from the SDK turns a Zod-style schema into the provider session format, so you describe the tool once and the SDK normalizes it for whichever provider you swap in. ```tsx 'use client'; import { experimental_useRealtime } from '@ai-sdk/react'; import { experimental_getRealtimeToolDefinitions } from 'ai'; import { z } from 'zod'; const tools = { getWeather: { description: 'Get current weather for a city', inputSchema: z.object({ city: z.string() }), }, }; export function VoiceAgent() { const { messages, connect } = experimental_useRealtime({ getToken: async () => { const res = await fetch('/api/realtime-token', { method: 'POST' }); return (await res.json()).token; }, tools: experimental_getRealtimeToolDefinitions(tools), onToolCall: async ({ toolName, args, addToolOutput }) => { if (toolName === 'getWeather') { const res = await fetch(`/api/weather?city=${args.city}`); const data = await res.json(); addToolOutput({ output: `${data.tempC}C and ${data.summary}`, }); } }, }); return ; } ``` A few subtleties I want to flag. The `onToolCall` callback receives `addToolOutput` directly on the argument object, which is a small but important DX win over having to manage a tool-call ID yourself. If your tool is asynchronous, awaiting inside `onToolCall` is fine, the SDK queues the model's audio output until you respond. If your tool throws, the hook surfaces an error on `messages[i].parts[j]` for that turn, so user-visible failure modes stay debuggable. If you prefer keeping tools on the server, that still works through your route handler, but you give up the latency win and the access to browser state. For a voice agent, where every 100 ms matters, I default to client tools and only push to the server for things that need a private credential. ## How do you display the user's spoken turns using inputAudioTranscription? You enable it by passing the `inputAudioTranscription` option in the session config when you call the provider, and the hook surfaces the transcribed user audio as `text` parts on the user-role `UIMessage`. The same `messages` loop that renders model output then renders the user's spoken words next to it. The session config travels through `getToken()` on the server, so the option lives in your route handler, not on the client. ```typescript const token = await openai .experimental_realtime('gpt-realtime') .getToken({ voice: 'verse', modalities: ['audio', 'text'], inputAudioTranscription: { model: 'whisper-1', language: 'en', }, }); ``` On the client, no extra code is needed. The next render after the user finishes speaking includes a new `UIMessage` with `role: 'user'` and a `parts` array containing the transcribed text. One thing to know: `inputAudioTranscription` support is provider-dependent. OpenAI exposes it on `gpt-realtime`. The current Google and xAI realtime endpoints handle transcription a little differently, and the option may be ignored or partially supported depending on the model you choose. Check the provider's release notes before you ship a transcript UI that all your users will see. If you need a consistent fallback, run an in-browser Whisper or VAD pass and decorate the message yourself. ## What should you keep in mind while the Realtime API is still canary? You should keep three things in mind: the API surface still moves, the provider feature matrix is uneven, and the cost model is per-provider. Each is fixable with a small amount of discipline, and none of them are reasons to wait if you are building a prototype or an internal tool. The first one is the most likely to bite. The hook is called `experimental_useRealtime`. The provider spec is `Experimental_RealtimeModelV4`. The argument name is `experimental_getRealtimeToolDefinitions`. That `experimental_` prefix is doing its job, signaling that names, argument shapes, and event payloads can change between canary releases. The v7 tracking issue lists Realtime as work in progress under issue #13897. Pin to `ai@7.0.0-canary.165` (or whichever exact version you tested), and audit before you bump. The second is the provider matrix. Reading the canary docs and the release notes side by side, OpenAI is the most fully exercised path right now. Google and xAI ship the factory methods but a few session-config options and tool-call shapes still differ. If you want one-line provider switching today, write your tool callbacks defensively and test each provider end-to-end before you ship. The third is the cost model. Voice tokens, especially audio output, are priced very differently from text tokens, and each provider has its own rate card. The SDK does not normalize billing, so route every session through your own server-side metering or you will get surprised by the first month's bill. If you are familiar with the [Vercel AI Gateway](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive), it does the per-provider routing piece, but realtime sessions still bill through the provider directly today. The short version: this is the first build of voice agents I have done that felt like writing normal React. I will keep my production traffic on a hand-rolled stack until the v7 stable release, and I will write every new voice prototype against `experimental_useRealtime` starting tomorrow. For more, see the [AI SDK 7 canary.165 release notes](https://github.com/vercel/ai/releases/tag/ai%407.0.0-canary.165), the [v7 tracking issue](https://github.com/vercel/ai/issues/14011), the [Realtime API WIP issue](https://github.com/vercel/ai/issues/13897), and the [OpenAI Realtime guide](https://platform.openai.com/docs/guides/realtime) for provider-side details. ### Keep Reading - [How to Migrate Vercel AI SDK 5 to 6: Breaking Changes Guide](https://www.rabinarayanpatra.com/blogs/vercel-ai-sdk-5-to-6-migration-guide). Where the v6 line landed before this v7 canary jump, and what you would migrate from first. - [Vercel AI Gateway: One Key for Every LLM, With BYOK Support](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive). How the Gateway sits next to the Realtime API and what it does for cost and routing. - [Why Claude Opus 4.8 Dynamic Workflows Redefine Agent Loops](https://www.rabinarayanpatra.com/blogs/claude-opus-4-8-dynamic-workflows). Useful context for agent design once your voice loop has tools and side-effects to coordinate. --- ## Why Claude Opus 4.8 Dynamic Workflows Redefine Agent Loops Source: https://www.rabinarayanpatra.com/blogs/claude-opus-4-8-dynamic-workflows Published: 2026-09-11T00:00:00+05:30 Tags: claude-opus-4-8-dynamic-workflows, claude-opus-4-8, claude-code, dynamic-workflows, anthropic, agent-orchestration Claude Opus 4.8 plus dynamic workflows in Claude Code 2.1.154 let one prompt orchestrate hundreds of subagents. Here is everything that changed since Opus 4.7. I have been running Claude Code daily since v1, and the gap between v2.1.153 and v2.1.154 felt different. Opus 4.8 landed on May 28, 2026, and the same release shipped dynamic workflows. One is a model bump. The other rewires how you ask Claude to do work at all. That second one is the bigger story. For nine months I had been writing what I called subagent-driven dev: spawn a subagent for each independent task, let the main loop pull results back, repeat. It worked. It also got brittle past ten or fifteen agents because every decision still funnelled through one main-loop context. With dynamic workflows, the main loop hands off a JavaScript script that owns the orchestration. The script runs deterministically. The subagents inside it run on Opus 4.8. The result is the first time I can spin up a hundred concurrent agents on one prompt and trust the structure. This post breaks down what shipped: the model changes in Opus 4.8, the new workflow primitives in Claude Code 2.1.154, the fast-mode pricing cut, and where I think the old subagent-driven pattern gets retired. ## What ships in Claude Opus 4.8? Opus 4.8 ships as a focused refinement of 4.7 rather than a wholesale jump. Anthropic priced it identically to 4.7: $5 per million input tokens and $25 per million output tokens. The benchmarks tell most of the story. The headline numbers from Anthropic's release post: - OSWorld-Verified: 82.3 percent - Online-Mind2Web: 84 percent - Roughly 4x less likely than 4.7 to let flaws in code it wrote pass unremarked Independent reviewers flagged the model's tendency to abstain on uncertain questions instead of confidently guessing. That single behaviour change is most of why the code-flaw number moves so much. Lower hallucination on edits means fewer broken tests after a long agent loop. Anyone who has had Opus 4.7 confidently rewrite a working test into a non-working one knows what that is worth. Two infrastructure changes matter more for agent workloads than the benchmark numbers do. First, the minimum cacheable prompt length dropped from 4096 tokens on 4.7 to 1024 tokens on 4.8. If you run many small subagents with stable instructions, you now hit prompt cache far earlier. In practice this is a noticeable cost cut on workflows that fan out wide. Second, mid-conversation system messages now update instructions without restating the full prompt. The result is a preserved cache, fewer wasted tokens, and a cleaner path for multi-phase orchestration where the main agent shifts mode partway. Fast mode is the other shoe. Opus 4.7 fast mode cost $30 per million input and $150 per million output. Opus 4.8 fast mode costs $10 in and $50 out: two times standard pricing for what Anthropic describes as 2.5x the speed. That is roughly 3x cheaper than the prior fast tier. Access is still gated behind the research-preview program with account-manager approval, which is the only part of the rollout I dislike. For most agent workloads, those four changes move the floor. Better code correctness, cheaper cache, cheaper fast mode, and an instruction-update path that respects the cache. Stacked together they meaningfully change the economics of long-running agent loops. ## What are dynamic workflows in Claude Code 2.1.154? Dynamic workflows are JavaScript scripts you hand to Claude Code's Workflow tool. The script orchestrates many subagents in one session and runs in the background. The main loop sees a single tool call. Anthropic's release note for v2.1.154 puts it plainly: > Introducing dynamic workflows: ask Claude to create a workflow and it orchestrates work across tens to hundreds of agents in the background, so you can take on larger, more complex tasks. The `/workflows` command lets you watch a live progress tree while the workflow runs. Subagents appear as nodes. Phase groupings show as collapsible sections. You can interrupt or kill any agent without bringing down the workflow. The trigger keyword changed in v2.1.160 on June 2, 2026. Asking for a "workflow" in your own words still works, but the literal keyword that fires the tool became `ultracode`. When you type the trigger now, it highlights in violet in the prompt input so you know the tool is about to fire. There is also a `/effort ultracode` command and a config toggle so you can turn keyword triggering off entirely if you want to gate every run. The architecturally interesting part is that workflows are not chat. They are deterministic code with a few well-chosen primitives. The model still does the hard semantic work inside each `agent()` call, but the control flow around those calls is yours. ![Claude Opus 4.7 single-thread agent loop vs Opus 4.8 dynamic workflow fan-out](/images/blogs/claude-opus-4-8-dynamic-workflows-shape.webp) ## How does the workflow scripting API work? The Workflow tool accepts a JavaScript script that begins with a pure-literal `meta` export and then runs an async body. The body has access to four orchestration primitives and a few helpers. Here is the shape of a minimal review workflow: ```js export const meta = { name: 'review-changes', description: 'Review changed files across dimensions and verify each finding', phases: [{ title: 'Review' }, { title: 'Verify' }], } const DIMENSIONS = [ { key: 'bugs', prompt: 'Find bugs in the diff' }, { key: 'perf', prompt: 'Find performance regressions in the diff' }, ] const results = await pipeline( DIMENSIONS, d => agent(d.prompt, { phase: 'Review', schema: FINDINGS_SCHEMA }), review => parallel(review.findings.map(f => () => agent(`Adversarially verify: ${f.title}`, { phase: 'Verify', schema: VERDICT_SCHEMA, }).then(v => ({ ...f, verdict: v })) )) ) return { confirmed: results.flat().filter(f => f.verdict?.isReal) } ``` Three things to notice. First, `agent(prompt, opts)` is the only way to call Opus 4.8 inside a workflow. The return value is either the agent's final text or a schema-validated object when you pass `opts.schema`. Validation happens at the tool-call layer, so the agent retries on a mismatch rather than pushing junk back to your script. This one feature kills most of the parsing glue I used to write around subagents. Second, `pipeline(items, ...stages)` is non-blocking. Item A can be in stage three while item B is still in stage one. Total wall-clock is the slowest single-item chain, not the slowest stage per item summed across stages. Anyone who has built map-reduce loops will recognise the win immediately. The default should always be `pipeline`. Reach for `parallel` only when stage N needs every result from stage N-1 at once, like a dedup pass before expensive verification. Third, the script is plain JS. No filesystem, no `Date.now()`, no `Math.random()`. The absence of those is not arbitrary. It is what makes the workflow journal deterministic so you can resume from any `agent()` boundary by re-running the script with `resumeFromRunId`. Same script and same args plus an unchanged prefix means cached results return instantly. I have used resume four times this month to recover from a flaky network blip mid-run, and each time the cache hit on the prefix saved real money. The other primitives: - `parallel(thunks)`: barrier across N concurrent agents. Use when stage N genuinely needs all of stage N-1. - `phase(title)`: opens a progress group for subsequent `agent()` calls. - `log(msg)`: narrator line in the progress tree, shown above the agent nodes. - `workflow(name, args)`: runs another workflow inline as a step and returns its result. Nesting is one level only. - `budget`: an object that exposes `total`, `spent()`, and `remaining()` for the run's token target. Use it for loops that scale depth to the user's budget. Concurrency is capped at `min(16, cpu_cores - 2)` per workflow, with a lifetime ceiling of 1000 agents. The cap is a runaway backstop, not a limit you should ever hit by design. If your workflow needs that many agents, the pattern is probably wrong. ## Why does fast mode cost three times less than before? The fast-mode price cut is the most under-discussed change in the release. Opus 4.7 fast mode was effectively a research preview at a punitive rate: $30 in and $150 out, six times the standard rate. Almost nobody ran agent loops on it because the cost destroyed the economics. Opus 4.8 fast mode is $10 in and $50 out, two times standard pricing, while delivering roughly 2.5x the throughput. That ratio matters because dynamic workflows often spawn dozens of independent agents that all want to finish fast. On the old fast pricing, doing one workflow run a day on fast mode was a six-figure-per-year budget item. On the new pricing, the same run costs around a third of that. Anthropic effectively turned fast mode from a research toy into a workflow-grade tool. The catch is access. Fast mode is still gated to research-preview accounts and needs account-manager approval. If you are on a starter or pro plan you cannot toggle it on. I expect this gate to come down within a quarter once Anthropic finishes scaling. Until then, fast mode is for orgs running serious agent volume. ![Opus 4.7 vs Opus 4.8 standard and fast-mode pricing per million tokens](/images/blogs/claude-opus-4-8-dynamic-workflows-pricing.webp) ## When should you reach for a dynamic workflow? Most people I have watched try workflows for the first time use them for the wrong shape of problem. Workflows are not better than a single agent for tasks with one clear scope. The overhead pays off only when the structure of the work needs more than one shape. Three concrete cases where I now reach for a workflow by default: 1. **Adversarial verification.** Find a list of issues with N independent finders, then run a separate agent per issue tasked with refuting it. If two of three refuters succeed, kill the finding. This catches plausible-but-wrong claims that a single agent would happily ship. The Workflow tool description itself names this as a quality pattern, and it is the single biggest workflow win I have found. 2. **Loop-until-dry exhaustive discovery.** Spawn finders until K consecutive rounds find nothing new. Better than `while (count < 10)` because the tail catches cases a counter cap silently misses. 3. **Migration across many files.** Pipeline transforms with `isolation: 'worktree'` per agent so file mutations do not collide. Each agent gets a fresh git worktree, the tree is auto-cleaned if it left no changes, and the result lands cleanly back at the end. The shape test I now apply: if I can describe the work as "fan out over N items, each independent, with a verify step," it is a workflow. If I describe it as "do this one thing well," it is a single subagent or just the main loop. ## What does this mean for the subagent-driven dev pattern? For me, subagent-driven dev becomes a subset of workflow-driven dev rather than its own pattern. The old shape was: in the main loop, spawn a subagent per task, await results, decide next steps, repeat. That works fine for two or three subagents in a row. It falls apart past ten because every decision still funnels through one main-loop context that you cannot easily prune. Workflows move the orchestration out of the main loop's mental space. The script is deterministic. The schema-validated outputs are exact. The progress tree shows you exactly where you are. The main loop sees one tool call and a final result. I still write subagent-driven code for short tasks. I still launch a single Explore agent when I need a file lookup. But for review, migration, audit, and research, the script-based workflow is now the default in my repo. The fast-mode price cut means I am willing to fan out wider than I used to. The pattern I expect to spread over the next quarter is workflow-as-skill: small, named, parameterised scripts in `.claude/workflows/` that get invoked by `workflow('name', args)` from either the main loop or other workflows. A handful in my portfolio already exist: `review-pr`, `audit-deps`, `summarise-issues`. Each is fewer than fifty lines and replaces what used to be a multi-message agent loop. There is also a quiet operational win. Workflows journal every agent boundary, so if a run dies halfway, I can resume from the last completed agent for free. I have lost zero work to a network blip since the feature shipped. ## Where does Opus 4.8 leave the agent-loop pattern? Opus 4.8 by itself is a refinement: better correctness, cheaper cache, cheaper fast mode. Dynamic workflows by themselves would have been valuable on Opus 4.7. The combination is the actual unlock. Cheaper agents plus deterministic orchestration is the first time I have felt comfortable trusting Claude Code with a hundred-agent task and walking away. If you are running Claude Code daily, install v2.1.160 or later, write one workflow in your `.claude/workflows/` directory, and try the fan-out shape on a real review or audit task. The first time `pipeline()` runs item A through stage three while item B is still in stage one, the new mental model clicks. After that, going back to single-agent loops feels slow. For more on the release, see Anthropic's [Opus 4.8 announcement](https://www.anthropic.com/news/claude-opus-4-8), the [Claude Code release notes](https://github.com/anthropics/claude-code/releases), and Simon Willison's [independent review of Opus 4.8](https://simonwillison.net/2026/May/28/claude-opus-4-8/). ### Keep Reading - [Claude Opus 4.7 API Pricing, Benchmarks and Breaking Changes 2026](https://www.rabinarayanpatra.com/blogs/claude-opus-4-7-release-and-migration-guide). The migration baseline this post builds on. - [Claude Code Routines: Async CI Automation Just Became Real](https://www.rabinarayanpatra.com/blogs/claude-code-routines-ci-automation). Workflows are the natural counterpart to routines, one for orchestration and the other for scheduling. - [10 Best Claude Skills for Developers in 2026](https://www.rabinarayanpatra.com/blogs/best-claude-skills-for-developers-2026). Skills work cleanly inside workflow agents via the `agentType` option. --- ## Branch.io Deep Linking and Mobile Attribution Explained Source: https://www.rabinarayanpatra.com/blogs/branch-io-deep-linking-attribution-guide Published: 2026-09-09T00:00:00+05:30 Tags: branch-io-deep-linking, deferred-deep-linking, mobile-attribution, nativelink, branch-metrics, mobile-app-growth A practical guide to Branch.io deep linking and attribution: how deferred deep links, NativeLink, and privacy-safe matching work, with real mobile app use cases. You've felt this bug as a user. You tap a link to a specific product, a shared playlist, a referral reward. The app isn't installed, so you get bounced to the app store. You install, open the app, and land on a generic home screen with no memory of where you were trying to go. The journey broke at the worst possible moment, right after you committed to installing. Fixing that is harder than it looks, and it got harder in 2025 when Firebase Dynamic Links shut down on August 25 and took a lot of apps' linking with it. Branch.io is where many of those apps landed. It's a mobile linking and attribution platform that makes one link route correctly across every platform, survive an app install, and report exactly which campaign drove the result. This guide walks through what Branch actually does: how deep linking and deferred deep linking work, how Branch matches a user to their original click without leaning on personal data, how its attribution engine tracks the full journey, the real use cases that justify the integration, and where its limits are. If you build or market a mobile app, this is the layer between "we got an install" and "we got the right install, to the right screen, from the right campaign." ## What is Branch.io, and what does it do? Branch.io is a mobile linking and attribution platform, run by Branch Metrics, that turns a single URL into a cross-platform, install-surviving, measurable link. Branch states it powers linking and attribution across more than 50,000 apps and 3.5 billion users, with customers like Reddit, BuzzFeed, Twitch, and Groupon. Two jobs sit at the core. The first is linking: take one Branch link and send each user to the right place, whether they're on iOS, Android, web, or desktop, and whether or not the app is installed. The second is attribution: record the full path from the first click through the install to an in-app purchase, and tie it back to the campaign, channel, or referrer that started it. Branch also says it handles more than 6,000 edge cases across browsers, operating systems, and embedded webviews, which is the unglamorous work that makes links "just work" where hand-rolled solutions quietly fail. It's worth being precise about the category. Branch is both a deep linking provider and a mobile measurement partner (MMP), so it competes with attribution-first tools like AppsFlyer and Adjust while also owning the linking layer that many of those integrations sit on. ![Cover illustration for a guide to Branch.io deep linking and attribution, showing a single link routing to the right destination across iOS, Android, and web.](/images/blogs/branch-io-deep-linking-attribution-guide.webp) ## What is deep linking, and why do plain URLs break for apps? Deep linking is sending a user to a specific screen inside an app, not just launching the app at its home screen. A plain web URL can't do this on its own, because the operating system needs a verified way to know that a given domain should open your app instead of the browser. The native mechanisms exist. On iOS they're Universal Links, on Android they're App Links, and both are verified associations between a domain and an app. When the app is installed and the user taps the link, the OS opens the app at the right screen. That part works well. The problem is everything around it: the app isn't always installed, the link gets opened inside a social app's in-app browser that blocks the native handoff, the user is on desktop, or you need the same link to behave differently per platform. Plain URLs and raw Universal Links have no answer for those cases. This is the gap Branch fills. A Branch link is a single URL, usually on a subdomain like `example.app.link` or your own domain, that carries structured link data describing where the user should end up: ```json { "$canonical_url": "https://example.com/products/running-shoes/123", "$deeplink_path": "products/running-shoes/123", "product_id": "123", "~campaign": "summer_sale", "~channel": "email" } ``` Branch reserves `$` keys for routing behavior and `~` keys for analytics. When that link is tapped, Branch decides at click time what should happen, open the app, defer to the store, or fall back to the web, based on the device, the OS, and whether the app is installed. The same routing logic that makes web navigation predictable, which I touched on in my guide to [measuring Core Web Vitals across SPA route changes](https://www.rabinarayanpatra.com/blogs/measure-core-web-vitals-spa-route-changes), is the kind of correctness Branch is trying to guarantee across the messier mobile boundary. ## How does Branch deferred deep linking work? Deferred deep linking carries the intended destination across an app install, so a brand-new user lands on the right screen after installing, not on a generic home page. This is the single hardest thing Branch does, and the main reason teams pay for it instead of rolling their own. The flow looks like this: 1. **Click.** A user taps a Branch link that encodes a destination plus metadata like campaign and referrer. The app isn't installed. 2. **Store redirect.** Branch sends the user to the App Store or Google Play. 3. **Install and open.** The user installs the app and opens it for the first time. 4. **SDK match.** On first launch, the Branch SDK calls Branch, which matches this open back to the original click and returns the link data. 5. **Route.** The app reads the returned data and routes the user straight to the intended screen. ![Deferred deep linking flow: a user taps a Branch link without the app installed, gets sent to the app store, installs and opens the app, the Branch SDK matches the open to the original click, and the app routes to the intended screen.](/images/blogs/branch-io-deep-linking-attribution-guide-deferred-flow.webp) Step 4 is where the engineering lives, because matching an install back to a click is exactly what Apple's privacy changes made hard. Branch's answer on iOS is NativeLink, which copies a small encrypted token to the device clipboard at click time. After install, the SDK reads that token on first open and matches the user to their original link, entirely on the device. Branch says this gives high-accuracy matching on iOS without using any personally identifiable information, which is what makes it viable in a post-ATT world. ## How does Branch attribution match users without violating privacy? Branch matches a user to their journey with a layered strategy that starts with the most accurate method available and falls back gracefully when privacy rules remove it. The goal is to credit the right campaign for an install or purchase without collecting data the platforms no longer allow. The layers, strongest first: - **Deterministic matching** with device identifiers: IDFA on iOS when the user grants App Tracking Transparency consent, and GAID on Android. This is exact when it's available. - **NativeLink clipboard matching** on iOS, the on-device token described above, which works without IDFA and without personal data. - **Probabilistic matching** as a fallback, using device signals like IP address, OS version, and screen size to infer a likely match. Branch describes this as roughly 80 to 90 percent accurate, so it's a reasonable estimate rather than a certainty. - **Predictive Aggregated Measurement (PAM)** when no direct match is possible at all, which models attribution at an aggregate level instead of per user. ![Branch attribution matching layers, from deterministic device identifiers to NativeLink clipboard matching, probabilistic signals, and aggregated modeling, ordered from most to least precise.](/images/blogs/branch-io-deep-linking-attribution-guide-attribution.webp) The reason this matters is that attribution is no longer one clean answer. Since Apple's ATT prompt cut IDFA availability, every honest attribution story is a blend of exact matches, on-device matches, and modeled estimates. Branch's approach is to be explicit about which method produced a given result, so you know whether a number is measured or modeled. Treat the deterministic and NativeLink numbers as solid and the probabilistic and PAM numbers as directional. ## What can you actually build with Branch? Branch is most useful anywhere a link needs to survive an install or a platform switch and then be measured. Once you have reliable deferred deep linking and attribution, a set of common growth features stops being custom engineering and becomes configuration. Real use cases I'd reach for Branch to build: - **Referral programs.** Generate a unique link per user that carries the referrer's ID through install, so a new user who installs from a friend's link lands on the reward screen and both sides get credited. The referral context survives the app store, which is the part that's painful to build by hand. - **Paid acquisition attribution.** Attribute installs and in-app purchases back to specific ad campaigns and creatives across networks, measuring real return on ad spend instead of guessing. This is the MMP side of Branch. - **Email and QR campaigns.** Put one Branch link in an email or behind a QR code and have it open the app at the right screen on a phone, or the website on a desktop, automatically. Branch can generate these links and codes at scale. - **Web-to-app conversion.** When a mobile web visitor would be better served in the app, route them through a smart banner or link that drops them on the matching in-app screen after install, not the home screen. - **Content sharing.** Let users share a specific song, product, or post, and have the recipient land on that exact item whether or not they have the app, which keeps shared content from dead-ending at a download page. - **Influencer and partner tracking.** Give each partner a link that attributes downstream installs and purchases to them, so payouts are based on measured results. The thread through all of these is the same: the link has to carry intent across an install boundary and report what happened. That's the work Branch packages. ## How do you add Branch to an app? You add Branch by creating an account, registering your app and its domains, installing the SDK, and reading the link data the SDK returns on launch. The integration is mostly configuration plus a single routing hook in your app's startup. The high-level steps are consistent across platforms: 1. **Configure the dashboard.** Register your iOS and Android apps, set the deep link domains, and add your Universal Links and App Links settings so the OS associates your domain with your app. 2. **Install the SDK.** Add the Branch SDK to your iOS, Android, React Native, or Flutter project. 3. **Initialize on launch.** Start a Branch session when the app opens. The SDK returns the link data for the link that brought the user in, including the deferred case after a fresh install. 4. **Route on the returned data.** Read the `$deeplink_path` or your custom keys from that data and navigate to the matching screen. The whole model rests on that last step: Branch hands you a structured object describing where the user wanted to go, and your app decides what to render. Get the routing hook right once, and every Branch link, deferred or not, paid or organic, flows through the same path. ## What are Branch's limits and alternatives? Branch's main limits are cost, dependency, and the inherent fuzziness of post-privacy attribution, none of which are unique to Branch but all of which you should weigh. It's a paid platform at scale, you're adding a third-party SDK and service to your critical path, and the probabilistic and modeled parts of attribution are estimates, not ground truth. There's also a healthy set of alternatives, and the right choice depends on whether you care more about linking or measurement. AppsFlyer and Adjust are attribution-first MMPs that also do deep linking. Kochava and Singular compete in the same measurement space. For teams that only need native behavior and never deferred deep linking, plain Universal Links and App Links are free and built in, which is fine until the day you need a link to survive an install. That day is exactly why Firebase Dynamic Links users had to migrate when Google shut it down, and why dedicated providers exist. My honest take: if your growth depends on links that carry intent through an install, referrals, shared content, paid campaigns you need to measure, Branch earns its place because deferred deep linking and attribution are genuinely hard to build and maintain yourself. If you only ever open the app at a screen when it's already installed, you don't need any of this and the native mechanisms are enough. Match the tool to whether your links have to cross the install boundary, and the decision gets simple. For more on how Branch works, see the [Branch deep linking overview](https://www.branch.io/deep-linking/), the [deferred deep linking glossary entry](https://www.branch.io/glossary/deferred-deep-linking/), and Google's [Firebase Dynamic Links deprecation FAQ](https://firebase.google.com/support/dynamic-links-faq) for the migration context that pushed many apps toward platforms like Branch. ### Keep Reading - [How to Measure Core Web Vitals in a Single-Page App](https://www.rabinarayanpatra.com/blogs/measure-core-web-vitals-spa-route-changes). The web side of getting routing and navigation right. - [Goodbye middleware.ts, Hello proxy.ts: The Next.js 16 Migration Guide](https://www.rabinarayanpatra.com/blogs/hello-proxy-ts-nextjs-16). How URL routing and redirects work on the web layer Branch links often hand off to. - [Full-Stack Social Identity: Spring Security 6.x + Next.js 16 + Auth.js](https://www.rabinarayanpatra.com/blogs/full-stack-social-identity-spring-nextjs). Connecting web and app identity, which pairs with cross-platform linking. --- ## How to Build an Interactive MCP App with the MCP Apps SDK Source: https://www.rabinarayanpatra.com/blogs/build-mcp-app-interactive-ui Published: 2026-09-04T00:00:00+05:30 Tags: build-mcp-app, mcp-apps, sep-1865, model-context-protocol, interactive-ui, mcp-server Learn how to build an MCP App (SEP-1865): declare a ui:// resource, register a UI tool, and wire the App SDK so your MCP server renders interactive UI in Claude and ChatGPT. For its entire life, an MCP tool could only hand back text and structured JSON. That's fine for "get the weather," but it falls apart the moment you want a chart you can hover, a form you can fill, or a map you can pan. The model ends up describing a UI in words instead of showing you one. MCP Apps fixes that. It's the first official extension to the Model Context Protocol, shipped as SEP-1865, and it lets a tool return a real interactive interface that the host renders inline in the conversation. A tool points at an HTML resource, the host drops it into a sandboxed iframe, and the UI talks back to your server over the same protocol everything else in MCP uses. Claude, ChatGPT, VS Code, and Goose already support it. I built the quickstart app to see how little code it takes, and the answer is "surprisingly little." This post walks through the whole thing: what the extension adds, how the render flow works, how to register a UI tool on the server, how to build the View, and how to run it in a real client. If you've followed my other MCP posts like the [stateless 2026-07-28 spec breakdown](https://www.rabinarayanpatra.com/blogs/why-mcp-2026-07-28-spec-goes-stateless), this is the layer that finally puts a face on your tools. ## What is the MCP Apps extension (SEP-1865)? MCP Apps is a standard way for an MCP server to ship interactive UI alongside its tools, defined in the extension proposal SEP-1865. It builds on plain MCP rather than replacing it, so your existing tools keep working and you opt specific ones into a UI. The core idea is small. A tool declares a `ui://` resource that holds an HTML interface. When the model calls that tool, the host fetches the resource and renders it in a sandboxed iframe directly in the chat. The rendered UI then communicates with the host over JSON-RPC, the same base protocol MCP already uses, so every action the UI takes flows through the same consent and audit path as a direct tool call. Because the UI template is declared ahead of time, hosts can prefetch it, cache it, and security-review it before anything runs. You install the SDK once: ```bash npm install -S @modelcontextprotocol/ext-apps ``` The package splits into focused entry points: the root `@modelcontextprotocol/ext-apps` for building the View, `/server` for registering tools and resources, `/react` for React hooks, and `/app-bridge` for hosts that embed Views. ## How does an MCP App show UI inside the chat? The host runs a four-step flow: the tool declares a UI resource, the model calls the tool, the host fetches and renders the resource in a sandboxed iframe, and then UI and host talk both ways. Nothing renders until the tool is actually invoked. Here's the sequence in plain terms: 1. **Tool definition.** Your tool declares a `ui://` resource that contains its HTML interface. 2. **Tool call.** The model calls the tool on your server like any other tool. 3. **Host renders.** The host fetches the resource and displays it in a sandboxed iframe inside the conversation. 4. **Bidirectional communication.** The host passes tool results into the UI, and the UI can call tools back through the host. ![Diagram of the MCP Apps render flow. An MCP server registers a tool linked to a ui:// HTML resource. The model calls the tool, the host fetches the resource and renders it in a sandboxed iframe, and the View communicates back over JSON-RPC on postMessage to call server tools and receive results.](/images/blogs/build-mcp-app-interactive-ui-flow.webp) The link between a tool and its UI is one field. The tool carries `_meta.ui.resourceUri` pointing at the resource: ```json { "name": "get-time", "description": "Returns the current server time.", "inputSchema": {}, "_meta": { "ui": { "resourceUri": "ui://get-time/mcp-app.html" } } } ``` ## How do you register a UI tool on your MCP server? Register two things that share a resource URI: the tool, with `registerAppTool`, and the HTML resource, with `registerAppResource`. The `@modelcontextprotocol/ext-apps/server` package gives you both helpers plus the `RESOURCE_MIME_TYPE` constant for the UI content type. This is the full server for a "get time" app, lifted from the official quickstart: ```typescript import { registerAppResource, registerAppTool, RESOURCE_MIME_TYPE, } from '@modelcontextprotocol/ext-apps/server'; import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'; import fs from 'node:fs/promises'; import path from 'node:path'; const DIST_DIR = path.join(import.meta.dirname, 'dist'); const resourceUri = 'ui://get-time/mcp-app.html'; export function createServer(): McpServer { const server = new McpServer({ name: 'Get Time Server', version: '1.0.0' }); registerAppTool( server, 'get-time', { title: 'Get Time', description: 'Returns the current server time.', inputSchema: {}, _meta: { ui: { resourceUri } }, // links the tool to its UI }, async () => ({ content: [{ type: 'text', text: new Date().toISOString() }] }) ); return server; } ``` The tool handler still returns normal `content`, so the app degrades gracefully: a host without MCP Apps support just shows the text. Now register the resource that serves the HTML: ```typescript registerAppResource( server, resourceUri, resourceUri, { mimeType: RESOURCE_MIME_TYPE }, async () => { const html = await fs.readFile(path.join(DIST_DIR, 'mcp-app.html'), 'utf-8'); return { contents: [{ uri: resourceUri, mimeType: RESOURCE_MIME_TYPE, text: html }] }; } ); ``` The resource just returns your bundled HTML as text with the MCP Apps MIME type. The host fetches it once, caches it, and reuses it every time the tool runs. ## How do you build the interactive View? Build the UI as a normal web page and connect it to the host with the `App` class from `@modelcontextprotocol/ext-apps`. The View is plain HTML plus a script, bundled to a single file, so you can use any framework or none. Here's the View for the get-time app: ```typescript import { App } from '@modelcontextprotocol/ext-apps'; const serverTimeEl = document.getElementById('server-time')!; const getTimeBtn = document.getElementById('get-time-btn')!; const app = new App({ name: 'Get Time App', version: '1.0.0' }); // Set this BEFORE connect() so you don't miss the initial tool result. app.ontoolresult = (result) => { const time = result.content?.find((c) => c.type === 'text')?.text; serverTimeEl.textContent = time ?? '[ERROR]'; }; getTimeBtn.addEventListener('click', async () => { // The UI asks the server for fresh data, on demand. const result = await app.callServerTool({ name: 'get-time', arguments: {} }); serverTimeEl.textContent = result.content?.find((c) => c.type === 'text')?.text ?? '[ERROR]'; }); app.connect(); ``` Three methods carry most apps. `app.ontoolresult` receives the result of the tool call that opened the UI. `app.callServerTool` lets a button or input fetch fresh data without leaving the conversation. And `app.updateModelContext({ content: [...] })`, not shown here, pushes a note back into the model's context, so when a user picks an option in your UI the model knows about it on the next turn. Bundle the View with Vite (or your bundler of choice) into the `mcp-app.html` the server reads. ## How do you run and test the MCP App? Point an MCP client at your server over stdio, the same way you'd add any MCP server, then call the tool and the UI renders. Every Node example in the ext-apps repo publishes as `@modelcontextprotocol/server-`, so a published app is one config block: ```json { "mcpServers": { "get-time": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-get-time", "--stdio"] } } } ``` For local development, clone the repo and use the bundled `basic-host` reference implementation, which renders Views in the browser without needing a full client: ```bash git clone https://github.com/modelcontextprotocol/ext-apps.git cd ext-apps && npm install && npm start # open http://localhost:8080/ ``` If you want the agent to scaffold it for you, the repo ships Agent Skills (`create-mcp-app`, `add-app-to-server`) you can install into Claude Code and just ask it to build one. That's the fastest path from idea to a rendered View. ## What security boundaries does an MCP App run inside? Every MCP App runs in a sandboxed iframe, and every action it takes routes through the host's existing consent and audit path. The UI cannot reach your server directly. It speaks to the host over JSON-RPC layered on `postMessage`, and the host mediates every tool call. That design matters for trust. When a button in your View calls `app.callServerTool`, the host treats it exactly like a model-initiated tool call, applying the same approvals and logging. Because the `ui://` resource is declared up front, the host can fetch and review it before the tool ever runs, instead of being handed arbitrary markup mid-conversation. Sandboxing the iframe means the UI can't touch the host page, read other tabs, or escape into the client. The honest caveat is reach: MCP Apps is an extension, so a host has to implement it. Support is real and growing (Claude, ChatGPT, VS Code, Goose, Postman), but a client that only speaks core MCP will fall back to your tool's text `content`. That's why I keep the tool returning useful text even when it has a UI. Build the View as an enhancement, not a hard dependency, and your tool works everywhere while looking great where it can. For the full details, see the [MCP Apps specification (2026-01-26)](https://github.com/modelcontextprotocol/ext-apps/blob/main/specification/2026-01-26/apps.mdx) and the [SEP-1865 discussion](https://github.com/modelcontextprotocol/modelcontextprotocol/pull/1865). ### Keep Reading - [Why the MCP 2026-07-28 Spec Drops Sessions and Goes Stateless](https://www.rabinarayanpatra.com/blogs/why-mcp-2026-07-28-spec-goes-stateless). The protocol direction this UI layer sits on top of. - [How to Build a Stateless MCP Server for the 2026-07-28 Spec](https://www.rabinarayanpatra.com/blogs/build-stateless-mcp-server-2026-07-28-spec). Build the server that your MCP App attaches its UI to. - [Claude Skills vs MCP vs Projects: Which One Should You Use?](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects). Where MCP fits among Claude's extensibility options. --- ## How to Defer JDBC Connections in Spring Boot 4.1 (Lazy Fetch) Source: https://www.rabinarayanpatra.com/blogs/spring-boot-4-1-lazy-connection-fetching Published: 2026-09-02T00:00:00+05:30 Tags: spring-boot-lazy-connection-fetching, spring-boot-4-1, lazyconnectiondatasourceproxy, jdbc, connection-pool, performance Learn how Spring Boot 4.1 lazy connection fetching defers JDBC connections until the first statement, cutting pool pressure. One property, verified behavior, and limits. Here's a pattern that quietly eats your connection pool. A service method is annotated `@Transactional`. It opens a transaction on every call. But half the time it returns early from a cache hit or a validation failure, never running a single query. With the default setup, each of those calls still borrows a physical connection from the pool the moment the transaction starts, holds it for the whole method, and gives it back having done nothing. Multiply that across a busy endpoint and you're exhausting a HikariCP pool with transactions that never touch the database. I've watched a service throw `Connection is not available, request timed out` under load while the database itself sat almost idle. Spring Boot 4.1 ships a one-property fix: `spring.datasource.connection-fetch=lazy`. It wraps your DataSource so the physical connection is fetched only when the first statement actually runs. This post covers why the idle-transaction problem happens, how to turn lazy fetching on, how `LazyConnectionDataSourceProxy` behaves underneath, how to confirm it's working, and when it helps versus when it doesn't. ## Why does an idle transaction still hold a database connection? An idle transaction holds a connection because the transaction manager borrows one the instant the transaction begins, long before any SQL runs. When Spring sees `@Transactional`, it asks the DataSource for a connection at method entry to start the transaction, set auto-commit off, and apply isolation. That connection is then pinned to the thread until the method returns. Here's the kind of method that wastes it: ```java @Service public class QuoteService { private final QuoteRepository repository; private final QuoteCache cache; @Transactional public Quote getQuote(String symbol) { // Many calls return here, having opened a transaction // and borrowed a connection for absolutely nothing. Quote cached = cache.get(symbol); if (cached != null) { return cached; } return repository.findBySymbol(symbol); } } ``` On a cache hit, the method opens a transaction, borrows a pooled connection, reads from an in-memory cache, commits, and returns the connection. The database was never involved, but for the duration of that call one of your pool slots was unavailable to everyone else. Under concurrency, those wasted slots are what tip a pool into timeouts. ## What is lazy connection fetching in Spring Boot 4.1? Lazy connection fetching is a Spring Boot 4.1 setting that defers the physical connection until the first JDBC statement, instead of grabbing it when the transaction opens. It works by wrapping your real DataSource in Spring's `LazyConnectionDataSourceProxy`. The proxy hands the transaction manager a logical connection right away, so `@Transactional` is happy, but it doesn't pull a real connection from the pool yet. It records the settings the transaction asks for, like auto-commit and isolation level, and replays them on the real connection the moment a statement is actually issued. If no statement is ever issued, no physical connection is borrowed. Spring describes the two modes precisely in the configuration metadata. `eager` (the default) means "fetch JDBC connections directly." `lazy` means "fetch JDBC connections as late as possible, or not at all if no statement is executed." That last clause is the whole point. ![Eager versus lazy connection fetching for the same transactional method on a cache hit. With eager, a physical connection is borrowed at transaction begin and held the whole time unused. With lazy, the proxy leaves the pool untouched and no physical connection is ever borrowed because no statement ran.](/images/blogs/spring-boot-4-1-lazy-connection-fetching-flow.webp) ## How do you turn on lazy connection fetching? Set one property. In `application.yml`: ```yaml spring: datasource: connection-fetch: lazy ``` Or in `application.properties`: ```properties spring.datasource.connection-fetch=lazy ``` That's the entire change. You don't define a bean, you don't touch your repositories, and you don't change a line of business code. Under the hood, Spring Boot 4.1 activates `LazyConnectionDataSourceConfiguration`, which is gated by `@ConditionalOnProperty(name = "spring.datasource.connection-fetch", havingValue = "lazy")`. It registers an infrastructure-role `BeanPostProcessor` that wraps your pooled DataSource: ```java // What Spring Boot 4.1 does for you when the property is "lazy" public Object postProcessAfterInitialization(Object bean, String beanName) { if (bean instanceof DataSource dataSource) { return new LazyConnectionDataSourceProxy(dataSource); } return bean; } ``` Your HikariCP (or other) pool is still the real DataSource. The proxy sits in front of it, and everything downstream, JPA, JDBC template, transaction manager, talks to the proxy. ## How does LazyConnectionDataSourceProxy actually behave? It returns a proxy connection on `getConnection()` and only borrows a real one from the pool when you run the first statement against it. Until that first `prepareStatement`, `createStatement`, or `prepareCall`, the underlying pool sees no checkout. Walk through the cache-hit method again with the proxy in place: 1. `@Transactional` opens a transaction. The proxy returns a logical connection. The pool is untouched. 2. The cache hit returns the value. No statement ever ran. 3. The transaction commits. Because no real connection was acquired, commit is a no-op on the pool. 4. The pool slot was never taken, so it stayed available to other requests the entire time. On a cache miss, the first `repository.findBySymbol` triggers a statement, the proxy borrows a real connection right then, applies the recorded auto-commit and isolation settings, and from that point the transaction behaves exactly as it always did. The only thing that changed is the timing of the checkout. One detail worth knowing: the proxy records transaction settings and applies them lazily, so a read-only hint or isolation level you set is preserved and pushed to the real connection when it's finally acquired. You don't lose correctness, you just stop paying for connections you don't use. ## How do you confirm connections are actually deferred? Watch the pool's active-connection count under a workload that mixes database and non-database calls. HikariCP exposes this through Micrometer, so the cleanest check is the `hikaricp_connections_active` gauge before and after flipping the property. ```java // A quick integration check: a no-SQL transaction must not borrow a connection @SpringBootTest class LazyConnectionTests { @Autowired DataSource dataSource; @Autowired QuoteService quoteService; @Test void cacheHitDoesNotTouchThePool() { HikariPoolMXBean pool = ((HikariDataSource) ((LazyConnectionDataSourceProxy) dataSource).getTargetDataSource()) .getHikariPoolMXBean(); quoteService.getQuote("CACHED"); // served from cache, no SQL assertThat(pool.getActiveConnections()).isZero(); } } ``` If you'd rather not write a test, hit `/actuator/metrics/hikaricp.connections.active` while you load-test the endpoint. With `eager`, active connections track your request concurrency. With `lazy`, they track only the requests that actually run SQL. That gap is the pressure you just removed. ## When does lazy connection fetching help, and when does it not? It helps most when transactions frequently open without running SQL, and it does almost nothing when every transaction queries the database anyway. Match it to your traffic shape rather than turning it on everywhere by reflex. It's a clear win for: - Read paths with a cache in front, where hits short-circuit before any query. - Service methods that validate input or check authorization and bail out before the data layer. - Endpoints fronting a connection pooler like PgBouncer in transaction mode, where holding fewer server-side connections is the whole game. I dug into that interaction in my [Postgres connection pool guide](https://www.rabinarayanpatra.com/blogs/postgres-connection-pool-pgbouncer-survival-guide). It buys you little when every request runs at least one query, because the connection gets borrowed almost immediately anyway. And there's a subtle point to keep in mind: if you depend on a connection-level side effect happening at transaction start, like an early validation query or a session variable set through a custom DataSource, lazy fetching delays that until the first real statement. For the vast majority of services that just read and write through JPA, none of that applies, and the property is close to free safety margin on your pool. What I like is that it's reversible in one line. Set it to `lazy`, watch your active-connection metric drop on cache-heavy endpoints, and if anything looks off, set it back to `eager`. It's the rare performance tweak that costs nothing to try and nothing to undo. If you're already on Spring Boot 4.1, this pairs naturally with the other 4.1 changes I covered in the [SSRF mitigation guide](https://www.rabinarayanpatra.com/blogs/spring-boot-4-1-ssrf-inetaddressfilter). For the details, see the [Spring Boot 4.1 release notes](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1.0-RC1-Release-Notes) and the [Spring Framework LazyConnectionDataSourceProxy docs](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/jdbc/datasource/LazyConnectionDataSourceProxy.html). ### Keep Reading - [Postgres Connection Pool Sizing: How PgBouncer Saved My Launch](https://www.rabinarayanpatra.com/blogs/postgres-connection-pool-pgbouncer-survival-guide). Why pool pressure matters and how transaction pooling interacts with this. - [How to Block SSRF in Spring Boot 4.1 with InetAddressFilter](https://www.rabinarayanpatra.com/blogs/spring-boot-4-1-ssrf-inetaddressfilter). Another Spring Boot 4.1 feature worth turning on. - [PostgreSQL 18 Temporal Foreign Keys with Spring Boot JPA](https://www.rabinarayanpatra.com/blogs/postgresql-18-temporal-foreign-keys-spring-boot). More on the Spring Boot plus Postgres data layer. --- ## How to Use Post-Quantum TLS in Java 27 (JEP 527) Source: https://www.rabinarayanpatra.com/blogs/post-quantum-tls-java-27-jep-527 Published: 2026-08-28T00:00:00+05:30 Tags: post-quantum-tls-java, jep-527, x25519mlkem768, ml-kem, jdk-27, java-security Learn how to use post-quantum TLS in Java 27 with JEP 527. Enable the X25519MLKEM768 hybrid key exchange, verify the handshake, and tune jdk.tls.namedGroups. Here's the uncomfortable part about quantum computers and TLS: you don't have to wait for them to be attacked by them. An attacker can record your encrypted traffic today, sit on it for years, and decrypt it the day a large enough quantum computer exists. It's called harvest-now-decrypt-later, and it's the reason post-quantum cryptography is a today problem, not a 2035 problem. Java 27 does something about it without asking you to lift a finger. JEP 527 adds a hybrid post-quantum key exchange to TLS 1.3 and turns it on by default. Every TLS connection your `HttpClient`, `RestClient`, or JDBC driver makes now offers a quantum-safe key share alongside the classical one. I pulled the exact details from the OpenJDK inside.java write-ups and the JDK 27 early-access builds. This post covers what actually changes in the handshake, how to confirm your client is really using the hybrid group, how to control the named groups with `jdk.tls.namedGroups`, how to enable it on the server side, and what I'd test before trusting it in production. If you've worked with the [new PEM API in JDK 27](https://www.rabinarayanpatra.com/blogs/read-write-pem-files-java-pem-api), this is the other big security feature landing in the same release. ## What is post-quantum TLS, and why does Java 27 add it now? Post-quantum TLS is a key exchange that stays secure even against an attacker with a quantum computer, and Java 27 adds it because the threat is about recorded traffic, not future traffic. The classical key exchanges TLS has used for years, like X25519 and the NIST curves, fall to Shor's algorithm on a sufficiently large quantum machine. The fix isn't to throw out the classical algorithms. It's to combine them. A hybrid key exchange runs a classical algorithm and a post-quantum algorithm together and mixes both shared secrets into the session keys. To break the session, an attacker has to break both. So if ML-KEM turns out to have a flaw, you still have X25519 protecting you, and if a quantum computer breaks X25519, you still have ML-KEM. You lose nothing and gain a hedge. ML-KEM is the Module-Lattice Key Encapsulation Mechanism, standardized by NIST as FIPS 203. Java already shipped the raw ML-KEM primitive in JDK 24 through JEP 496. JEP 527 is the part that actually wires it into TLS 1.3, so you get the protection on real connections instead of having to build your own protocol. ## What does JEP 527 actually change in the TLS handshake? JEP 527 adds three hybrid named groups to the SunJSSE provider and enables one of them by default. The named groups are: | Named group | Combination | | -------------------- | ----------------------------- | | `X25519MLKEM768` | X25519 ECDHE + ML-KEM-768 | | `SecP256r1MLKEM768` | secp256r1 ECDHE + ML-KEM-768 | | `SecP384r1MLKEM1024` | secp384r1 ECDHE + ML-KEM-1024 | By default, a Java 27 TLS client offers two key shares in its ClientHello: one for `X25519MLKEM768` and one for classical `x25519`. That dual key share is a thoughtful detail. If the server speaks the hybrid group, it picks `X25519MLKEM768` and you get post-quantum protection. If the server is old and only knows `x25519`, it picks that one from the same ClientHello, so the handshake completes in the normal number of round trips with no `HelloRetryRequest` penalty. The trade-off is size. An `X25519MLKEM768` key share is roughly 1.2KB, because the ML-KEM-768 encapsulation key is 1184 bytes, versus 32 bytes for X25519 on its own. So your ClientHello grows by about a kilobyte. That matters for the rollout testing later, but it costs you nothing in code. ![Diagram of a TLS 1.3 hybrid handshake. A Java 27 client sends a ClientHello with two key shares, X25519MLKEM768 and x25519. The server selects X25519MLKEM768. The session secret is derived from both the X25519 ECDHE secret and the ML-KEM-768 secret, so an attacker recording traffic cannot decrypt it later.](/images/blogs/post-quantum-tls-java-27-jep-527-handshake.webp) ## How do you confirm your Java 27 client is using post-quantum TLS? Turn on JSSE handshake debugging and read the named group in the handshake output. Since the feature is on by default, there's nothing to enable first, you just need to prove it's happening. ```bash java -Djavax.net.debug=ssl,handshake -jar your-app.jar ``` In the ClientHello you'll see the supported groups and the `key_share` extension listing `X25519MLKEM768` ahead of `x25519`. In the ServerHello, the selected group tells you what was actually negotiated. If you see `X25519MLKEM768` there, the session is post-quantum. If you see `x25519`, the peer didn't support the hybrid group and you fell back to classical. A small standalone check is the fastest way to see it against a real endpoint: ```java import javax.net.ssl.SSLSocket; import javax.net.ssl.SSLSocketFactory; public class TlsProbe { public static void main(String[] args) throws Exception { SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault(); try (SSLSocket socket = (SSLSocket) factory.createSocket("example.org", 443)) { socket.startHandshake(); System.out.println("Protocol: " + socket.getSession().getProtocol()); // Run with -Djavax.net.debug=ssl,handshake to see the // negotiated named group in the handshake log. } } } ``` The standard JSSE session API doesn't expose the negotiated named group as a getter, so the handshake log is the source of truth. I keep `-Djavax.net.debug=ssl,handshake` handy for exactly this kind of check. ## How do you control the named groups with jdk.tls.namedGroups? Set the `jdk.tls.namedGroups` system property to an ordered, comma-separated list, and SunJSSE offers them in that order. This is how you prefer a different hybrid scheme, pin to post-quantum only for testing, or disable the hybrid groups if you hit an interop problem. ```bash # Prefer the secp256r1 hybrid, then the X25519 hybrid, then classical fallbacks java -Djdk.tls.namedGroups="SecP256r1MLKEM768,X25519MLKEM768,secp256r1,x25519" \ -jar your-app.jar ``` You can do the same per connection with `SSLParameters`, which is cleaner when only one client needs special handling: ```java import javax.net.ssl.SSLParameters; import javax.net.ssl.SSLSocket; SSLParameters params = tlsSock.getSSLParameters(); params.setNamedGroups(new String[] { "SecP256r1MLKEM768", "X25519MLKEM768", "secp256r1", "x25519" }); tlsSock.setSSLParameters(params); ``` Two practical recipes. To force post-quantum only, so a test fails loudly if the peer can't do it, set the list to just `X25519MLKEM768`. To disable post-quantum entirely as a temporary fallback for a broken middlebox, set it to classical groups like `x25519,secp256r1` and the hybrid shares disappear from your ClientHello. ## How do you enable post-quantum key exchange on the server side? Run your TLS server on JDK 27 and the hybrid groups are available automatically, because the server selects a group from what the client offers. There's no separate switch. The server side of the handshake just needs to know `X25519MLKEM768`, which SunJSSE does on JDK 27. The catch is ordering. A server picks the first mutually supported group based on its own preference list, so if you've pinned `jdk.tls.namedGroups` on the server to a classical-only list from some older hardening guide, it will never choose the hybrid group even when the client offers it. Check your server's existing TLS configuration for a hardcoded named-group list and add the hybrid groups to the front. For a Spring Boot service behind its own TLS, that's the JVM flag on the server process, not anything in your application code. This is also where you decide your security posture across services. If you run both client and server, putting `X25519MLKEM768` first on both guarantees post-quantum protection on internal hops, which fits the least-privilege model I wrote about in my [zero trust microservices guide](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security). ## What should you test before you roll it out? Test interop and ClientHello size first, because those are where a hybrid key exchange bites you, not the cryptography itself. The math is solid. The failure mode is some box in the middle that doesn't like a bigger handshake. Here's my checklist before flipping anything to production: - **Interop with older peers.** Point your Java 27 client at the actual endpoints it talks to and confirm the handshake still completes. With the default dual key share it should fall back to `x25519` cleanly, but verify it instead of assuming. - **Middleboxes and the larger ClientHello.** The extra ~1.2KB can trip up old load balancers, deep-packet-inspection firewalls, or anything that makes assumptions about handshake size. Test through your real network path, not just localhost. - **Handshake cost.** Measure connection setup time under load. ML-KEM operations are fast, but the larger key share means more bytes on the wire per handshake. - **A loud failure path.** Run one test environment pinned to `X25519MLKEM768` only, so if a dependency silently can't do post-quantum, your tests catch it instead of quietly downgrading in production. The reason I'd turn this on deliberately rather than just inheriting the default is that "secure by default" only helps if it actually negotiates. A misconfigured server preference list or a grumpy middlebox can leave you on classical key exchange while you think you're protected. Run the handshake debug once per critical connection, confirm you see `X25519MLKEM768`, and then you know the harvest-now-decrypt-later window is closed for that traffic. That five-minute check is the whole point of the feature being real instead of theoretical. For the full specification and testing guidance, see the OpenJDK write-ups on [post-quantum hybrid key exchange in JDK 27](https://inside.java/2026/02/17/tls-post-quantum-hybrid-key-exchange/) and the [JDK 27 Quality Outreach heads-up](https://inside.java/2026/05/17/quality-heads-up/), plus the NIST [FIPS 203 ML-KEM standard](https://csrc.nist.gov/pubs/fips/203/final) that defines the post-quantum primitive. ### Keep Reading - [How to Read and Write PEM Files in Java (JDK 27 PEM API)](https://www.rabinarayanpatra.com/blogs/read-write-pem-files-java-pem-api). The other major security API shipping in JDK 27. - [How to Block SSRF in Spring Boot 4.1 with InetAddressFilter](https://www.rabinarayanpatra.com/blogs/spring-boot-4-1-ssrf-inetaddressfilter). Another outbound-connection security control worth turning on. - [Java 26 HTTP/3 in the HttpClient](https://www.rabinarayanpatra.com/blogs/java-26-http-3-httpclient). The HttpClient that benefits from this TLS upgrade automatically. --- ## How to Block SSRF in Spring Boot 4.1 with InetAddressFilter Source: https://www.rabinarayanpatra.com/blogs/spring-boot-4-1-ssrf-inetaddressfilter Published: 2026-08-26T00:00:00+05:30 Tags: spring-boot-ssrf-mitigation, inetaddressfilter, spring-boot-4-1, ssrf, spring-security, http-client Learn how to block SSRF in Spring Boot 4.1 with the new InetAddressFilter on RestClient and WebClient, covering cloud metadata, internal IPs, and tests. A user pastes a URL into your app. Maybe it's an avatar image, maybe a webhook endpoint, maybe an "import from URL" box. Your Spring Boot service dutifully fetches it with a `RestClient`. Looks harmless. Now the user types `http://169.254.169.254/latest/meta-data/iam/security-credentials/`. On most clouds, that's the metadata endpoint, and it hands back temporary IAM credentials to whatever asks from inside the VPC. Your server just asked. That's SSRF, and it has leaked real production credentials at real companies. For years the fix in Java was to write your own address checks by hand and hope you covered every case. Spring Boot 4.1 changes that. It ships an `InetAddressFilter` you can attach to any outbound HTTP client, blocking and reactive, that refuses to connect to internal addresses. This post covers what SSRF actually is, the exact `InetAddressFilter` API (I pulled it straight from the `spring-boot-http-client` source), how to wire it on `RestClient` and `WebClient`, how to test that it blocks the metadata endpoint, and the cases it still won't save you from. ## What is SSRF, and why are outbound HTTP clients the weak point? SSRF is an attack where the attacker controls the destination of a request your server makes, so your server becomes a proxy into networks the attacker can't reach directly. The weak point is any outbound HTTP client that fetches a URL derived from user input. The reason it's so dangerous is trust. Your service runs inside a private network. It can reach the database, internal admin panels, service-mesh sidecars, and the cloud metadata endpoint, none of which are exposed to the internet. When an attacker can pick the URL, they borrow that network position. A feature as boring as "fetch the user's avatar from a URL" becomes a way to read `http://10.0.0.5:8080/actuator/env` or pull IAM credentials from `169.254.169.254`. Here's the kind of code that ships in almost every app: ```java // Looks innocent. This is the SSRF sink. @PostMapping("/webhooks") public void register(@RequestBody WebhookRequest request) { // request.url() came from the user String body = restClient.get() .uri(request.url()) .retrieve() .body(String.class); process(body); } ``` Nothing here validates where `request.url()` points. Blocklisting strings like `localhost` or `127.0.0.1` doesn't work either, because `2130706433`, `0x7f000001`, `0`, and a DNS name that resolves to a private IP all reach loopback too. The only check that holds up is on the resolved IP address, right before you connect. That's exactly where Spring Boot 4.1 now lets you plug in. ## What is the InetAddressFilter in Spring Boot 4.1? `InetAddressFilter` is a functional interface in `org.springframework.boot.http.client`, new in Spring Boot 4.1, that decides which resolved IP addresses an outbound client may connect to. It has one abstract method, `matches(InetAddress)`, and a set of static factories and combinators for building rules. The key thing to get right is the polarity: **an address that matches is allowed**. An address that does not match is blocked, and the client throws a `FilteredHostException` instead of connecting. So `externalAddresses()` means "match (allow) external addresses, block everything else." These are the factory methods you'll actually use: ```java import org.springframework.boot.http.client.InetAddressFilter; // Allow only external (public) addresses. Blocks loopback, private, link-local, ULA. InetAddressFilter external = InetAddressFilter.externalAddresses(); // The inverse: match internal addresses only. InetAddressFilter internal = InetAddressFilter.internalAddresses(); // Allow a specific CIDR block (IPv4 or IPv6), nothing else. InetAddressFilter allowList = InetAddressFilter.of("203.0.113.0/24"); // Compose: allow external, but also exclude one known-bad host. InetAddressFilter strict = InetAddressFilter.externalAddresses() .andNot("203.0.113.7"); ``` What counts as "internal" is defined in the framework, not left to you. Reading `InternalInetAddressFilter` in the source, an address is internal if `isLoopbackAddress()`, `isLinkLocalAddress()`, or `isSiteLocalAddress()` returns true, plus IPv6 Unique Local Addresses in `fc00::/7`, and it unwraps NAT64 (`64:ff9b::/96`) addresses to re-check the embedded IPv4. Concretely, `externalAddresses()` blocks all of this: | Range | Example | Why it's dangerous | | ------------------------------------------------------- | ------------------------- | ------------------------------- | | Loopback `127.0.0.0/8`, `::1` | `127.0.0.1`, `0x7f000001` | Local admin ports, actuator | | Link-local `169.254.0.0/16`, `fe80::/10` | `169.254.169.254` | Cloud metadata, IAM credentials | | Private `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16` | `10.0.0.5` | Internal services, databases | | IPv6 ULA `fc00::/7` | `fd00::1` | Internal IPv6 services | That link-local row is the one people miss when they roll their own filter. The metadata endpoint at `169.254.169.254` is link-local, and `externalAddresses()` blocks it for free. ![Diagram showing an attacker-supplied URL passing through a Spring Boot service. The InetAddressFilter resolves the IP, allows a public address, and blocks loopback, private, and link-local metadata addresses with a FilteredHostException.](/images/blogs/spring-boot-4-1-ssrf-inetaddressfilter-flow.webp) ## How do you block SSRF on a blocking RestClient? You build an `InetAddressFilter`, attach it to `HttpClientSettings`, build a request factory from those settings, and hand that factory to your `RestClient`. The filter then runs on every connection the client makes. ```java import org.springframework.boot.http.client.ClientHttpRequestFactoryBuilder; import org.springframework.boot.http.client.HttpClientSettings; import org.springframework.boot.http.client.InetAddressFilter; import org.springframework.http.client.ClientHttpRequestFactory; import org.springframework.web.client.RestClient; @Configuration public class SecureRestClientConfig { @Bean RestClient outboundRestClient() { InetAddressFilter onlyExternal = InetAddressFilter.externalAddresses(); HttpClientSettings settings = HttpClientSettings.defaults() .withInetAddressFilter(onlyExternal); ClientHttpRequestFactory requestFactory = ClientHttpRequestFactoryBuilder.jdk().build(settings); return RestClient.builder() .requestFactory(requestFactory) .build(); } } ``` `HttpClientSettings` is an immutable record, so `withInetAddressFilter` returns a new copy. The same `settings` object also carries timeouts, redirects, and SSL bundles, so you can chain `.withTimeouts(...)` and friends on the same builder. `ClientHttpRequestFactoryBuilder` has a factory per client library: `jdk()`, `reactor()`, `jetty()`, and `httpComponents()`, plus `detect()` if you want Spring to pick whatever is on the classpath. The filtering wiring is built into each one, so the choice is about which HTTP library you prefer, not whether the filter works. Now the webhook handler from earlier is safe without changing a line of its own logic, because the injected `RestClient` refuses internal addresses: ```java public WebhookResult fetch(String userSuppliedUrl) { // Throws FilteredHostException if userSuppliedUrl resolves internally return outboundRestClient.get() .uri(userSuppliedUrl) .retrieve() .body(WebhookResult.class); } ``` ## How do you block SSRF on a reactive WebClient? The reactive path is the same shape, but you build a `ClientHttpConnector` instead of a request factory and pass it to `WebClient.builder().clientConnector(...)`. The `HttpClientSettings` and `InetAddressFilter` are identical. ```java import org.springframework.boot.http.client.HttpClientSettings; import org.springframework.boot.http.client.InetAddressFilter; import org.springframework.boot.http.client.reactive.ClientHttpConnectorBuilder; import org.springframework.http.client.reactive.ClientHttpConnector; import org.springframework.web.reactive.function.client.WebClient; @Configuration public class SecureWebClientConfig { @Bean WebClient outboundWebClient() { HttpClientSettings settings = HttpClientSettings.defaults() .withInetAddressFilter(InetAddressFilter.externalAddresses()); ClientHttpConnector connector = ClientHttpConnectorBuilder.reactor().build(settings); return WebClient.builder() .clientConnector(connector) .build(); } } ``` `ClientHttpConnectorBuilder` mirrors the blocking builder with `reactor()`, `jetty()`, `httpComponents()`, `jdk()`, and `detect()`. If you're on the default WebFlux stack, `reactor()` is the one you want. A blocked request surfaces as an error signal in the reactive stream, so it shows up in `onError`, not as a thrown exception on the calling thread. One detail worth knowing: the filter runs at the connector's DNS-resolution and connect layer, not as a string check on the URL. That means it sees the address the client is actually about to connect to, including the target of an HTTP redirect. A 302 that points your client at `http://10.0.0.5/` gets filtered on the redirect hop, not just the first request. ## How do you test that the filter blocks the cloud metadata endpoint? Write a test that points the secured client at `169.254.169.254` and assert it fails with a `FilteredHostException`. Because the client never opens a socket, the test is fast and doesn't touch the network. ```java import static org.assertj.core.api.Assertions.assertThatThrownBy; import org.junit.jupiter.api.Test; import org.springframework.boot.http.client.ClientHttpRequestFactoryBuilder; import org.springframework.boot.http.client.FilteredHostException; import org.springframework.boot.http.client.HttpClientSettings; import org.springframework.boot.http.client.InetAddressFilter; import org.springframework.web.client.RestClient; class SsrfFilterTests { private RestClient secureClient() { HttpClientSettings settings = HttpClientSettings.defaults() .withInetAddressFilter(InetAddressFilter.externalAddresses()); return RestClient.builder() .requestFactory(ClientHttpRequestFactoryBuilder.jdk().build(settings)) .build(); } @Test void blocksCloudMetadataEndpoint() { RestClient client = secureClient(); assertThatThrownBy(() -> client.get() .uri("http://169.254.169.254/latest/meta-data/") .retrieve() .body(String.class)) .rootCause() .isInstanceOf(FilteredHostException.class); } } ``` `FilteredHostException` carries the host it rejected through `getHost()` and the filter that rejected it through `getFilter()`, which is handy when you want to log blocked attempts. The client wraps it in its own transport exception, so I assert on the root cause rather than the top-level type. I'd add a second test that hits a public address and asserts it does not throw, so a future refactor can't silently disable the filter. ## What are the limits you still have to handle? The filter is opt-in and only covers the clients you build from `HttpClientSettings`, so it's not a substitute for thinking about your whole request path. It's a strong control, not a magic shield. Three things it does not do for you: - **It's off until you wire it.** `HttpClientSettings.defaults()` has no filter. Every `RestClient` or `WebClient` you build the old way is still wide open. Audit for clients that bypass your secured beans. - **It only sees Spring's HTTP clients.** A raw `java.net.http.HttpClient`, an OkHttp instance, a database driver fetching a remote file, or any library that opens its own sockets will not go through this filter. SSRF lives anywhere an address comes from user input. - **Block-lists are weaker than allow-lists.** `externalAddresses()` blocks known-internal ranges, which is the right default for "fetch an arbitrary user URL." But if your client only ever talks to one partner API, prefer `InetAddressFilter.of("203.0.113.0/24")` so a new internal range or a routing quirk can't widen your exposure. For the request paths you do control, this is the cleanest SSRF defense the framework has ever shipped, and it lines up with the least-privilege thinking I wrote about in my [zero trust microservices guide](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security). Set the filter once on a shared client bean, write the two tests, and the boring "fetch this URL" feature stops being a credential leak waiting to happen. What I like most is that it pushes the check to the only place it's reliable: the resolved IP at connect time. No string parsing, no blocklist of clever loopback spellings, no Bouncy-Castle-style third-party dependency. Just a filter that says yes or no to an `InetAddress`. If you maintain a Spring Boot service that fetches anything on behalf of users, this is the first thing to turn on after you upgrade to 4.1. For more on this feature, see the [Spring Boot 4.1.0 release notes](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-4.1.0-RC1-Release-Notes) and the [Spring Boot REST client reference docs](https://docs.spring.io/spring-boot/reference/io/rest-client.html). For the wider threat model and a checklist beyond Spring, the [OWASP SSRF Prevention Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html) is the reference I keep open. ### Keep Reading - [Zero Trust Architecture in Spring Boot Microservices](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security). How least-privilege networking and SSRF defense fit the same model. - [Mastering Spring Security 6.x Bean Migration](https://www.rabinarayanpatra.com/blogs/spring-security-component-revolution). The configuration style these HTTP client beans build on. - [How to Version REST APIs in Spring Framework 7](https://www.rabinarayanpatra.com/blogs/how-to-version-rest-apis-spring-framework-7). More on the modern Spring Boot 4 HTTP client and API stack. --- ## How to Use the React Activity Component for Instant Navigation Source: https://www.rabinarayanpatra.com/blogs/react-activity-component Published: 2026-08-21T00:00:00+05:30 Tags: react-activity-component, react-19-2, react, frontend, performance, navigation The React Activity component (React 19.2) keeps hidden UI mounted so navigation feels instant. Learn visible and hidden modes, prerendering, and state saving. You click a tab, the new screen flashes blank, spinners spin, and then the content arrives. You click back, and the form you half-filled is gone. That stutter is the cost of conditional rendering, where React unmounts a screen the moment you leave it and rebuilds it from scratch when you return. React 19.2 fixes this with a new stable component called ``. It keeps hidden parts of your UI mounted and prerendered in the background, so the next navigation feels instant and the back button restores state for free. I swapped one `&&` for an `` in a side project last week and a tab switch went from a visible 400ms blank to no blank at all. Here's how it works. ![React Activity component keeping a hidden screen mounted and prerendered while a visible screen is shown](/images/blogs/react-activity-component.webp) ## What is the React Activity component? The React Activity component is a stable API in React 19.2 that wraps a piece of your UI and toggles it between visible and hidden without unmounting it. You give it a `mode` prop, either `"visible"` or `"hidden"`, and React decides how hard to work on that subtree. Visible behaves like normal rendering. Hidden keeps the tree alive but pushes its work to the lowest priority. This is the production-ready version of an idea React has been circling for years under the old Offscreen name. It shipped without an `unstable_` prefix, so you can use `` directly. The React team has said more modes are coming, but the two you have today already cover the cases that matter. ```jsx import { Activity } from 'react'; function App({ currentTab }) { return ( ); } ``` `ProfilePage` stays mounted whether the tab is active or not. When the user is somewhere else, React keeps it ready in the background instead of throwing it away. ## How do you replace conditional rendering with Activity? You replace conditional rendering by swapping the boolean `&&` for an `` whose mode follows the same condition. The before and after are nearly identical, which is what makes the migration easy. The behavior underneath is completely different. ```jsx // Before: unmounts Page when isVisible is false {isVisible && } // After: keeps Page mounted, just hidden ``` With the `&&` version, flipping `isVisible` to `false` destroys the entire `Page` tree. Its state vanishes, its DOM is removed, and the next time you show it, React mounts everything again from zero. With Activity, flipping to `hidden` keeps the tree and its state in memory and hands the visible work back to whatever the user is looking at. The mental shift is small but real. You stop asking should this be in the tree and start asking should this be visible right now. Those are different questions, and Activity is the answer to the second one. ## What happens when an Activity is hidden? When an Activity switches to hidden, React does three specific things: it hides the children, unmounts their effects, and defers all further updates until it has nothing else to do. The component state stays in memory, but the side effects do not keep running. That combination is the whole trick. ![State machine diagram of the React Activity visible and hidden modes showing effects mounting and unmounting](/images/blogs/react-activity-component-modes.webp) Think about what unmounting effects means in practice. A hidden screen with a `useEffect` that opens a WebSocket, starts an interval, or subscribes to a store will run that effect's cleanup function the moment it goes hidden. No background socket chatter, no timers firing against a screen nobody sees. When the screen goes visible again, the effect runs fresh, reconnecting as if the component just mounted. ```jsx function LiveFeed() { useEffect(() => { const socket = openSocket(); return () => socket.close(); // runs when the Activity hides }, []); return ; } ``` So a hidden Activity is cheaper than it looks. It holds onto state and DOM, which is memory, but it stops doing work, which is CPU and network. You get the resumability of a kept-alive component without paying for its effects while it's offscreen. ## How do you prerender the next route with Activity? You prerender the next route by rendering it inside a hidden Activity before the user navigates, so its data, CSS, and images load in the background. React renders the hidden subtree at low priority without stealing time from the visible page. By the time the user clicks, the screen is already built. ```jsx function Router({ route }) { return ( <> ); } ``` Both pages are described up front. Only one is visible, and the hidden one warms its data fetches and asset loads while the user reads the visible page. Switch the route and the formerly hidden page is ready, no blank flash, no fresh waterfall of requests. ![Timeline comparing a cold navigation with a blank flash against an Activity-prerendered navigation that appears instantly](/images/blogs/react-activity-component-prerender.webp) The judgment call is which routes to prerender. Wrapping every possible screen in a hidden Activity is wasteful, because each one holds memory and does background rendering. Prerender the one or two screens a user is most likely to hit next, like the detail view from a list or the next step in a wizard, and leave the long tail to load on demand. ## How does Activity preserve state on back navigation? Activity preserves state on back navigation because a hidden subtree stays mounted, so local state and uncontrolled inputs are never destroyed. When the user leaves a screen, you flip it to hidden instead of unmounting it. When they hit back, you flip it to visible and the screen returns exactly as they left it. This is the part users feel most. Picture a search page with filters typed in and a scroll position halfway down the results. With conditional rendering, navigating to a detail page and back wipes all of it. With Activity, the search screen was only hidden, so the text in the inputs, the selected filters, and the scroll position come back untouched. ```jsx {/* keeps filters and scroll on back nav */} ``` ![Before and after comparison of back navigation, one losing form state and one restoring it with Activity](/images/blogs/react-activity-component-state.webp) You don't have to lift that state into a store or serialize it into the URL just to survive a round trip. The component keeps its own state because it never left the tree. That deletes a whole category of state-preservation plumbing I used to write by hand. ## How is Activity different from display:none and conditional rendering? Activity sits between conditional rendering and CSS hiding, and it beats both for this job. Conditional rendering with `&&` unmounts the tree, which frees memory but destroys state and forces a cold rebuild. CSS `display:none` keeps the element in the DOM, but React keeps running its effects, timers, and subscriptions at full priority, so a hidden tab still does work and still competes for the main thread. Activity takes the good half of each. Like `display:none`, it keeps the tree and its state alive for instant return. Unlike `display:none`, it unmounts effects and drops the subtree to the lowest render priority, so a hidden screen costs memory but not ongoing work. And unlike either, it coordinates with React's scheduler, prerendering hidden content only when the main work is done. | Approach | State on return | Effects while hidden | Render priority | | --- | --- | --- | --- | | `{cond && }` | Lost | None (unmounted) | N/A | | `display: none` | Kept | Still running | Full | | `` | Kept | Unmounted | Lowest | The table makes the choice obvious. If you want a screen to come back instantly without burning resources while it waits, Activity is the only option that gives you both. ## What pitfalls should you watch for with Activity? The first pitfall is treating Activity like `display:none` for effects. Effects unmount when an Activity hides and re-run when it shows, so any effect with an expensive setup, like a fresh network handshake, pays that cost on every visible transition. Make your effect cleanup and setup cheap, or guard expensive work so a quick hide-and-show doesn't thrash. The second pitfall is wrapping too much. Every hidden Activity keeps its tree in memory and does low-priority background rendering, so blanketing your app in hidden Activities trades blank flashes for memory pressure. Prerender the few screens that benefit and let the rest mount on demand. Measure before you wrap a heavy subtree. The third pitfall is expecting Activity to be a router or a data cache. It controls visibility and rendering priority, nothing more. It will not dedupe your fetches, manage history, or know which route comes next. You still wire navigation and data loading yourself, with Activity as the piece that keeps the rendered result warm. The last one is assuming the hidden tree is truly idle. It is not running effects, but its state still occupies memory and its low-priority renders still happen. On a memory-constrained device, a dozen hidden screens add up. Keep the count small and the wins stay clean. ## Should you adopt Activity in your app today? Adopt it where navigation churn hurts, and skip it everywhere else. Activity is not a global setting you flip on. It's a targeted tool for the handful of places where unmounting and remounting a screen costs the user a blank flash or a lost form. Those places are where the payoff is obvious and the memory cost is worth it. Find the one navigation in your app that feels worst, the tab switch that blanks or the back button that forgets. Wrap both screens in ``, drive the mode from your existing condition, and feel the difference. Once you see a cold screen turn instant from a one-line change, you'll know exactly which other spots deserve the same treatment, and which don't. For the authoritative details, read the official [React 19.2 release notes](https://react.dev/blog/2025/10/01/react-19-2) and the deeper walkthrough in [LogRocket's React 19.2 coverage](https://blog.logrocket.com/react-19-2-is-here/). ### Keep Reading - [Building a Modern Docs Generator with Next.js 16](https://www.rabinarayanpatra.com/blogs/building-modern-docs-generator-nextjs-16). Where instant navigation patterns pay off in real apps. - [Hello, proxy.ts in Next.js 16](https://www.rabinarayanpatra.com/blogs/hello-proxy-ts-nextjs-16). Another React 19-era primitive worth knowing. - [Replacing useEffect Data Fetching with Server Actions](https://www.rabinarayanpatra.com/blogs/replacing-useeffect-data-fetching-server-actions). Rethink effects, the thing Activity unmounts when hidden. - [The Day a React Patch Broke the Internet](https://www.rabinarayanpatra.com/blogs/the-day-react-patch-broke-the-internet). Why pinning your React version matters before you adopt new APIs. --- ## How to Read and Write PEM Files in Java (JDK 27 PEM API) Source: https://www.rabinarayanpatra.com/blogs/read-write-pem-files-java-pem-api Published: 2026-08-19T00:00:00+05:30 Tags: read-pem-files-java, pem-api, jep-538, java-security, jdk-27, cryptography Learn how to read PEM files in Java using the new PEM API (JEP 538). Decode and encode keys, certificates, and CRLs, plus encrypt private keys in JDK 27. For twenty-five years, the answer to "how do I read a private key from a PEM file in Java" was the same embarrassing dance. Strip the `-----BEGIN-----` line. Strip the `-----END-----` line. Kill the newlines. Base64-decode what's left. Wrap it in a `PKCS8EncodedKeySpec`. Push it through a `KeyFactory`. Hope you picked the right algorithm string. I have written that exact block of code in at least four projects. Every time it felt wrong, because it is wrong. PEM is a 30-year-old format with a published spec, and the standard library made me parse it by hand with `String.replace`. JDK 27 finally fixes this. JEP 538 ships a real PEM API in `java.security`: `PEMEncoder` and `PEMDecoder`, which read and write keys, certificates, and CRLs without a single `replace` call. This post covers the old pain in detail, the new two-line replacement, how to encrypt private keys correctly, how to handle files with multiple PEM blocks, and exactly what changed between the preview and the final release. If you have ever managed TLS material in a Java service, like the certificate handling I touched on in my [full-stack social identity guide](https://www.rabinarayanpatra.com/blogs/full-stack-social-identity-spring-nextjs), this is the API you have wanted for a long time. ## Why has reading PEM files in Java always been painful? Reading PEM files in Java was painful because the platform never had a parser for the format, so every key load became manual string surgery. PEM is just Base64-encoded DER wrapped in `-----BEGIN X-----` and `-----END X-----` markers, but the JDK gave you no direct way to go from that text to a `PrivateKey`. Here is the code I have copy-pasted across projects to load an RSA private key: ```java // The old way: read a PEM private key by hand String pem = Files.readString(Path.of("private-key.pem")); String base64 = pem .replace("-----BEGIN PRIVATE KEY-----", "") .replace("-----END PRIVATE KEY-----", "") .replaceAll("\\s", ""); byte[] der = Base64.getDecoder().decode(base64); PKCS8EncodedKeySpec spec = new PKCS8EncodedKeySpec(der); KeyFactory factory = KeyFactory.getInstance("RSA"); PrivateKey key = factory.generatePrivate(spec); ``` Look at everything that can go wrong there. The header text has to match exactly, including whether it says `PRIVATE KEY` or `RSA PRIVATE KEY`. You have to know the key is PKCS#8 and not PKCS#1. You have to hardcode `"RSA"` even though the file itself describes the algorithm. And if the key is encrypted, none of this works at all. Certificates were slightly less awful, because `CertificateFactory` does accept PEM-wrapped X.509 input: ```java // Certificates were tolerable, keys were not CertificateFactory cf = CertificateFactory.getInstance("X.509"); X509Certificate cert = (X509Certificate) cf.generateCertificate( new ByteArrayInputStream(Files.readAllBytes(Path.of("cert.pem")))); ``` But there was no single, uniform API. Certificates went through `CertificateFactory`, keys went through `KeyFactory` plus manual stripping, CRLs went through yet another path, and encrypted keys meant pulling in Bouncy Castle. That fragmentation is the actual problem JEP 538 solves. ## What is the PEM API in JDK 27, and what does it replace? The PEM API is a pair of classes, `PEMEncoder` and `PEMDecoder` in `java.security`, that convert between PEM text and Java security objects through one consistent interface. JEP 538 finalizes it for JDK 27 after two preview rounds, and it replaces every hand-rolled Base64-stripping routine you have ever written. The design is small on purpose. Both classes are immutable and you get an instance with a static factory: ```java PEMDecoder decoder = PEMDecoder.of(); PEMEncoder encoder = PEMEncoder.of(); ``` Everything that can be encoded implements a marker interface called `BinaryEncodable`. The types that implement it are the ones you actually work with: - `PrivateKey`, `PublicKey`, and `KeyPair` - `X509Certificate` and `X509CRL` - `EncryptedPrivateKeyInfo` - `PKCS8EncodedKeySpec` and `X509EncodedKeySpec` - `PEM`, a value type that holds a raw block the decoder did not recognize ![Map of the PEM API showing PEMDecoder and PEMEncoder converting between PEM text and the BinaryEncodable types](/images/blogs/read-write-pem-files-java-pem-api-api-map.webp) That single interface is why the API stays tiny. You do not need a different entry point per object type. You decode to the type you expect, or you decode to the interface and pattern-match on what you got. ## How do you read a PEM file in Java with PEMDecoder? You read a PEM file by calling `decode` with the class you expect back. The typed overload, `decode(String, Class)`, parses the text, Base64-decodes the body, and returns a fully built object of that type. The four-step ritual from earlier collapses to this: ```java String pem = Files.readString(Path.of("private-key.pem")); PrivateKey key = PEMDecoder.of().decode(pem, PrivateKey.class); ``` Two lines, and the decoder figured out the algorithm from the encoding. No `KeyFactory`, no hardcoded `"RSA"`, no `PKCS8EncodedKeySpec`. Certificates and public keys work the same way. You change the `Class` argument and that is it: ```java PEMDecoder decoder = PEMDecoder.of(); X509Certificate cert = decoder.decode(certPem, X509Certificate.class); PublicKey pub = decoder.decode(pubPem, PublicKey.class); X509CRL crl = decoder.decode(crlPem, X509CRL.class); ``` The difference between the old way and the new way is stark when you put them next to each other. ![Side by side comparison of nine lines of manual PEM parsing versus two lines using PEMDecoder](/images/blogs/read-write-pem-files-java-pem-api-old-vs-new.webp) There is also a `decode(InputStream)` overload, which matters because you often read this material straight off the classpath or a socket rather than from a `String`. If the type does not match what the PEM actually contains, the decoder throws rather than handing you a silently wrong object, which is exactly the behavior you want for security material. ## How do you write objects to PEM with PEMEncoder? You write a key or certificate to PEM by passing it to `encodeToString`, which returns the full PEM text including the header and footer lines. `PEMEncoder` is the mirror image of the decoder and handles every `BinaryEncodable` type. ```java PEMEncoder encoder = PEMEncoder.of(); String certPem = encoder.encodeToString(cert); String keyPem = encoder.encodeToString(privateKey); String pubPem = encoder.encodeToString(publicKey); ``` If you need raw bytes instead of a `String`, for example to write directly to a file or a network buffer, use `encode`: ```java byte[] pemBytes = PEMEncoder.of().encode(cert); Files.write(Path.of("cert.pem"), pemBytes); ``` The encoder picks the correct header label for you. A `PrivateKey` becomes a `PRIVATE KEY` block, an `X509Certificate` becomes a `CERTIFICATE` block, and so on. You are no longer responsible for getting the boundary text right, which was a real source of bugs when other tools refused to parse a key because the label was slightly off. ## How do you encrypt a private key in PEM format? You encrypt a private key by chaining `withEncryption(password)` before you encode. The encoder wraps the key as an encrypted PKCS#8 structure, so the PEM you get out is an `ENCRYPTED PRIVATE KEY` block that is safe to store next to your code. ```java char[] password = "correct-horse-battery-staple".toCharArray(); String encryptedPem = PEMEncoder.of() .withEncryption(password) .encodeToString(privateKey); ``` The default password-based encryption algorithm is `PBEWithHmacSHA256AndAES_128`. If your security policy needs a different one, set it through the `jdk.epkcs8.defaultAlgorithm` security property rather than passing it per call. That keeps the choice in one place instead of scattered through your code. ![Flow of a private key through withEncryption to an encrypted PEM block and back through withDecryption](/images/blogs/read-write-pem-files-java-pem-api-encrypt-flow.webp) Reading it back is the symmetric operation. You hand the password to the decoder with `withDecryption`: ```java PrivateKey key = PEMDecoder.of() .withDecryption(password) .decode(encryptedPem, PrivateKey.class); ``` For more control, you can work with `EncryptedPrivateKeyInfo` directly. The encoder accepts it as input, and when you decode an encrypted block without a password, you get one back so you can inspect the algorithm before deciding how to handle it: ```java BinaryEncodable obj = PEMDecoder.of().decode(encryptedPem); if (obj instanceof EncryptedPrivateKeyInfo info) { PrivateKey key = info.getKey(password); } ``` This is a real upgrade. Before JDK 27, encrypted PKCS#8 keys were the single biggest reason teams reached for Bouncy Castle. Now it is in the platform. ## How do you handle a file with multiple PEM blocks? You handle multi-block files by decoding to the `BinaryEncodable` interface instead of a concrete class, then pattern-matching on each result. A chain file or a bundle often holds a private key followed by a certificate followed by a CA chain, and the untyped `decode` lets you take them one at a time. ```java BinaryEncodable obj = PEMDecoder.of().decode(pemBlock); switch (obj) { case PrivateKey key -> loadKey(key); case X509Certificate cert -> addToChain(cert); case X509CRL crl -> registerRevocations(crl); case PEM pemBlockData -> log.warn("Unrecognized block: {}", pemBlockData.type()); default -> throw new IllegalStateException("Unexpected PEM content"); } ``` The `PEM` case is the interesting one. When the decoder hits a block it does not have a dedicated type for, it does not throw. It gives you a `PEM` object that exposes the raw pieces: the `type()` (the label between the dashes), the Base64 `content()`, and any `leadingData()` that appeared before the block. That last part matters for real-world files, which frequently carry human-readable comments above the actual PEM. So you can round-trip even content the JDK does not understand natively, which means the API does not lock you out of custom or newer PEM labels. That is a thoughtful piece of design for a security API that has to live for decades. ## What changed between the preview and the final PEM API? The biggest changes are three renames that landed when JEP 538 finalized the API, so preview code from JDK 25 and 26 needs small edits. If you tried this during preview, the concepts are identical but a few names moved. The renames: - The encodable interface was called `DEREncodable` in the previews. It is now `BinaryEncodable`. - The catch-all type was a record named `PEMRecord`. It is now an ordinary class named `PEM`, which let the team add constructors that take Base64 byte arrays with proper defensive copying. - `PEMDecoder.withFactory(Provider)` became `withFactoriesOf(Provider)`. ![Timeline from JEP 470 preview in JDK 25 to JEP 524 second preview in JDK 26 to JEP 538 final in JDK 27](/images/blogs/read-write-pem-files-java-pem-api-version-timeline.webp) The release path explains the polish. The feature previewed as JEP 470 in JDK 25, came back as a second preview (JEP 524) in JDK 26, and is final as JEP 538 in JDK 27. As of late May 2026 it sits at Proposed to Target for 27, with the finalization review closing. If you are on JDK 25 or 26 and want to try it today, you compile and run with preview features enabled: ```bash javac --release 26 --enable-preview PemDemo.java java --enable-preview PemDemo ``` On JDK 27 you drop both flags, because the API is a permanent part of the platform. That is the whole point of the two-preview cadence: the API got two rounds of real feedback before it became something you cannot change. ## Should you adopt the Java PEM API right away? Yes, the moment you are on JDK 27, because the manual approach is not just ugly, it is a security liability. Every hand-rolled parser is a place where you can mismatch an algorithm, mishandle an encrypted key, or accept malformed input you should have rejected. Moving that logic into the platform means it gets the same scrutiny as the rest of `java.security`. The thing I keep coming back to is how long we tolerated the old way. PEM parsing was a rite of passage, a snippet everyone carried in their head, and that is exactly the kind of code that should never have been ours to write. If JDK 27 is in your future, delete your PEM helper class the day you upgrade. You will not miss it. For the full specification and reference, see the [JEP 538: PEM Encodings of Cryptographic Objects](https://openjdk.org/jeps/538), the [JDK 25 security enhancements writeup](https://seanjmullan.org/blog/2025/09/23/jdk25) by the JDK security lead, and the [InfoQ Java News Roundup](https://www.infoq.com/news/2026/05/java-news-roundup-may11-2026/) tracking the finalization. ### Keep Reading - [Java 26 HTTP/3 in the HttpClient](https://www.rabinarayanpatra.com/blogs/java-26-http-3-httpclient): the other half of modern Java networking, where the keys and certificates you load with the PEM API actually get used for TLS. - [Full-Stack Social Identity with Spring Security and Next.js](https://www.rabinarayanpatra.com/blogs/full-stack-social-identity-spring-nextjs): a real auth flow where certificate and key handling shows up in production. - [Modern Java Development with Java 21+ and Spring Boot](https://www.rabinarayanpatra.com/blogs/modern-java-spring-boot): more recent language and platform features worth folding into your services. --- ## How to Measure Core Web Vitals in a Single-Page App Source: https://www.rabinarayanpatra.com/blogs/measure-core-web-vitals-spa-route-changes Published: 2026-08-14T00:00:00+05:30 Tags: measure-core-web-vitals-spa, soft-navigations-api, core-web-vitals, single-page-application, web-vitals, performance Learn how to measure Core Web Vitals in a single-page application with Chrome's Soft Navigations API, attributing LCP, INP, and CLS to each client-side route. Here is a bug that does not throw an error, does not show up in your logs, and quietly makes your performance dashboard lie to you. Your single-page app loads fast on the landing route, so your Core Web Vitals look green. Then a user clicks into a deep route, waits two seconds for a chart to paint, and your monitoring never records it. I hit this on a React dashboard last year. The initial load scored a clean LCP of 1.4s. Real users on the reports page were staring at spinners for 3 seconds, and our RUM tool showed nothing wrong, because in a single-page app the browser only ever fires one navigation. Every route change after that is invisible to the performance timeline. Chrome's Soft Navigations API fixes this. It teaches the browser to recognize a client-side route change as a real page view, then attributes LCP, INP, and CLS to that specific route. This post walks through what a soft navigation is, how to turn the API on, how to wire it into `PerformanceObserver` and the web-vitals library, and how to read the per-route numbers. If you have ever wondered why your SPA metrics felt too good, this is the missing piece. It pairs well with the work I did on [moving data fetching off useEffect](https://www.rabinarayanpatra.com/blogs/replacing-useeffect-data-fetching-server-actions), since slow route data is the usual reason a soft navigation scores badly. ## Why are Core Web Vitals invisible in a single-page application? Core Web Vitals are invisible on SPA route changes because the browser ties every metric to a single navigation event, and a client-side route change never fires one. When your router swaps the view with `history.pushState` and a DOM update, the browser sees the same document it loaded minutes ago. LCP was already finalized on first paint. CLS keeps summing layout shifts across the whole session. INP is the only one that survives, and even it has no idea which route the slow interaction belonged to. ![Timeline showing Core Web Vitals attributed only to the first load while three SPA route changes record nothing](/images/blogs/measure-core-web-vitals-spa-route-changes-blind-spot.webp) So you end up with a metric that describes the first thing a user saw and ignores everything after. For a content site with full page loads, that is fine. For a SPA where users spend 90% of their time on routes they reached by clicking, it is close to useless. The traditional workarounds were all bad. Some teams manually called `performance.mark()` on every route change and computed their own timings, which misses the actual paint and interaction signals the browser tracks internally. Others gave up and reported only the initial load, accepting the blind spot. Neither gives you the real LCP element or the real interaction latency for a route. What you actually want is for the browser to reset its Core Web Vitals accounting at each route change, the same way it would on a multi-page site. That is exactly what soft navigations provide. ## What is a soft navigation, and how does Chrome detect one? A soft navigation is a client-side route change that Chrome recognizes as a new logical page view based on a heuristic. The browser cannot read your router's mind, so it watches for a specific pattern of three things happening together. ![The three conditions Chrome checks to flag a soft navigation: user interaction, URL change, and visible paint](/images/blogs/measure-core-web-vitals-spa-route-changes-heuristic.webp) All three conditions must hold: 1. **A user action initiates the navigation.** A click or a keypress, not a background timer or a fetch that resolves on its own. 2. **The URL changes visibly.** Through `history.pushState`, `history.replaceState`, or a direct History API call that updates what the user sees in the address bar. 3. **The interaction produces a visible paint.** New content has to actually render, so a no-op route that changes the URL but paints nothing does not count. That heuristic is deliberately conservative. Chrome would rather miss a borderline navigation than wrongly split one page view into two and corrupt your numbers. The `replaceState` trigger was added to the final origin trial after developer feedback, because plenty of routers use `replaceState` for things like filter changes that users perceive as navigations. The important detail for instrumentation: once Chrome flags a soft navigation, it stamps a unique `navigationId` onto the performance entries that follow. That id is the thread you pull on to group LCP, CLS, and interaction entries by route. ## How do you turn on the Soft Navigations API in Chrome? You enable the Soft Navigations API one of two ways depending on whether you want local testing or real field data. For local work, flip a flag. For production measurement, join the origin trial. For local testing, enable the Chrome flag: ```text chrome://flags/#soft-navigation-heuristics ``` Or launch Chrome from the command line with the feature turned on: ```bash # macOS /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \ --enable-features=SoftNavigationHeuristics # Linux google-chrome --enable-features=SoftNavigationHeuristics ``` For field data from real users, register your origin in the Soft Navigations origin trial. It runs from Chrome 147 through Chrome 149, with a stable launch expected later in 2026. Once you have a token, add it as a meta tag in your document head: ```html ``` Or send it as an HTTP response header, which is the better choice for a SPA because the token applies before your JavaScript runs: ```text Origin-Trial: YOUR_ORIGIN_TRIAL_TOKEN ``` Either way, the API surface is identical. The flag is for your own Chrome during development. The token is what lets the feature run for visitors who have not enabled any flags, so your real-user monitoring actually collects data. ## How do you observe soft navigations with PerformanceObserver? You observe soft navigations by registering a `PerformanceObserver` for the `soft-navigation` entry type. Each entry that arrives represents one detected route change, carrying the new URL and a `navigationId` you use to correlate other metrics. Always feature-detect first so you do not throw on browsers without the API: ```javascript function supportsSoftNavigations() { return ( typeof PerformanceObserver !== 'undefined' && PerformanceObserver.supportedEntryTypes.includes('soft-navigation') ); } if (supportsSoftNavigations()) { const observer = new PerformanceObserver((list) => { for (const entry of list.getEntries()) { console.log('Soft navigation to', entry.name); console.log('navigationId', entry.navigationId); console.log('started at', entry.startTime); } }); observer.observe({ type: 'soft-navigation', buffered: true }); } ``` The `buffered: true` option matters. It replays entries that fired before your observer registered, which is common in a SPA where your analytics code loads after the first interaction. Without it you would miss early route changes. Each `soft-navigation` entry gives you a few useful fields: - `name` is the new URL the navigation resolved to. - `navigationId` is the unique key for grouping all metrics from this route. - `startTime` is when the initiating interaction happened. - `largestInteractionContentfulPaint` points at the largest paint that resulted from the navigation, the soft-nav equivalent of LCP. That `navigationId` is the whole game. Every Core Web Vital entry that follows a soft navigation gets the same id, so you can finally answer "how slow was the reports route specifically" instead of "how slow was the session." ## How do you attribute LCP, INP, and CLS to each route? You attribute each metric to a route by reading the `navigationId` that Chrome now attaches to standard performance entries after a soft navigation. The `largest-contentful-paint`, `layout-shift`, and `event` entries all gain that field, so grouping by it gives you per-route vitals. Here is a single observer that buckets metrics by navigation: ```javascript const metricsByNavigation = new Map(); function getBucket(navigationId) { if (!metricsByNavigation.has(navigationId)) { metricsByNavigation.set(navigationId, { url: null, lcp: 0, cls: 0, }); } return metricsByNavigation.get(navigationId); } const observer = new PerformanceObserver((list) => { for (const entry of list.getEntries()) { const id = entry.navigationId; if (id === undefined) continue; const bucket = getBucket(id); if (entry.entryType === 'soft-navigation') { bucket.url = entry.name; } if (entry.entryType === 'largest-contentful-paint') { bucket.lcp = entry.startTime; } if (entry.entryType === 'layout-shift' && !entry.hadRecentInput) { bucket.cls += entry.value; } } }); observer.observe({ type: 'soft-navigation', buffered: true }); observer.observe({ type: 'largest-contentful-paint', buffered: true }); observer.observe({ type: 'layout-shift', buffered: true }); ``` The difference this makes is stark once you compare it to the old single-load view. ![Before and after table contrasting one set of session-wide vitals with per-route vitals keyed by navigationId](/images/blogs/measure-core-web-vitals-spa-route-changes-before-after.webp) INP needs one extra note. Interaction entries map to a soft navigation through the `interactionId`, and the spec guidance is to use `interactionId` rather than `navigationId` when you correlate `interaction-contentful-paint` entries. For most reporting you will let the web-vitals library handle that wiring, which is the next section. ## How do you report per-route vitals with the web-vitals library? You report per-route vitals by using the experimental soft-navigation build of Google's web-vitals library, which exposes a `reportSoftNavs` option on each metric function. That option tells the library to emit a fresh metric object for every soft navigation instead of one per page load. Import the soft-navs build and opt in per metric: ```javascript import { onLCP, onINP, onCLS, } from 'https://unpkg.com/web-vitals@soft-navs/dist/web-vitals.js?module'; function sendToAnalytics(metric) { // metric.navigationId ties this value to a specific route const body = JSON.stringify({ name: metric.name, value: metric.value, rating: metric.rating, navigationId: metric.navigationId, }); navigator.sendBeacon('/analytics', body); } onLCP(sendToAnalytics, { reportSoftNavs: true }); onINP(sendToAnalytics, { reportSoftNavs: true }); onCLS(sendToAnalytics, { reportSoftNavs: true }); ``` With `reportSoftNavs: true`, the callback fires once for the initial load and again for each detected route change, and every metric object carries the `navigationId` so your backend can group by route. The standard web-vitals v5 build still works for the hard load, but only the soft-navs build resets the metrics per soft navigation. A few practical notes from wiring this into a real RUM pipeline: - Use `navigator.sendBeacon` or a `fetch` with `keepalive: true` so the report survives the user leaving the route. - Send the resolved route pattern, not the raw URL, if your routes have ids in them. Reporting `/orders/:id` instead of `/orders/8412` keeps your dashboard groupable. - Keep the hard-load report too. The first load is still your most important view and you want both in the same dataset. If you already report Core Web Vitals from a framework like Next.js, you are swapping the import path and adding the `reportSoftNavs` flag. The shape of your analytics payload barely changes. ## How do you read soft navigation data in Chrome DevTools? You read soft navigation data directly in the Performance panel, which has shown soft-nav markers since Chrome 145. Record a trace, click through a few client-side routes, and DevTools draws a marker at each detected soft navigation so you can see exactly where the browser reset its metrics. ![A soft navigation marker in the Chrome DevTools Performance panel timeline](/images/blogs/measure-core-web-vitals-spa-route-changes-devtools.webp) _Source: [Chrome for Developers](https://developer.chrome.com/docs/web-platform/soft-navigations)_ The markers are the fastest way to confirm the heuristic is firing on your routes before you invest in a full RUM integration. If you click a link and no marker appears, one of the three conditions failed. Usually it is the visible-paint requirement, which trips when a route renders from cache so fast that Chrome does not register a contentful paint, or when the URL changed without a user interaction the browser could attribute. Record the trace, watch for the markers, and line them up against the LCP and layout-shift entries in the same timeline. That visual check has saved me from shipping instrumentation that silently recorded nothing on half my routes. ## What are the limits of the Soft Navigations API? The biggest limit is that this is a heuristic running only in Chromium during an origin trial, so it is a sample and not a complete picture. You need to design your reporting around that from day one rather than treating soft-nav data as ground truth for every user. The constraints worth planning around: - **Chromium only.** Firefox and Safari have no equivalent. Your soft-nav numbers describe Chrome users. Keep your hard-load metrics as the cross-browser baseline. - **It is a heuristic.** Routes that paint from cache instantly, or navigations not tied to a clear user interaction, can be missed. The detection favors precision over recall, so expect some false negatives rather than false positives. - **Origin trial status.** The API ran as an origin trial through Chrome 149 with a stable launch targeted for later in 2026. Field tokens expire, and the surface could still shift slightly before it ships, though the team froze most of it for this final trial. - **CrUX is undecided.** Google has been explicit that this trial is about evaluating the API, not about how the data feeds the Chrome User Experience Report. So do not assume soft-nav vitals will show up in your CrUX dashboard or affect search signals yet. None of that makes the API less worth adopting. A Chromium-only sample of your real per-route performance is infinitely more than the zero data you have today. Just label it honestly in your dashboards so nobody mistakes a Chrome sample for the whole population. ## Is the Soft Navigations API worth adopting now? Yes, even as an origin trial, because a Chromium-only sample of real per-route performance beats the zero data you have today. The reason this matters goes beyond a nicer dashboard. For years, SPA teams optimized the one route the browser could measure and flew blind on everything else, which quietly trained a generation of apps to be fast on the homepage and slow everywhere that mattered. Per-route Core Web Vitals change the incentive. Once you can see that the settings page has a 4-second LCP, you fix the settings page. My advice: turn on the flag this week and record a Performance trace of your own app clicking through its three most-used routes. The markers will tell you in 30 seconds whether your performance story is as good as your current metrics claim. I would bet it is not, and that is exactly the point. For the full specification and reference, see the [Soft Navigations API documentation](https://developer.chrome.com/docs/web-platform/soft-navigations), the [final origin trial announcement](https://developer.chrome.com/blog/final-soft-navigations-origin-trial), and the [web-vitals library on GitHub](https://github.com/GoogleChrome/web-vitals). ### Keep Reading - [Why I Replaced useEffect Data Fetching with Server Actions](https://www.rabinarayanpatra.com/blogs/replacing-useeffect-data-fetching-server-actions): slow route-change data is the most common reason a soft navigation scores a bad LCP, so this is the upstream fix. - [Building a Modern Documentation Generator with Next.js 16](https://www.rabinarayanpatra.com/blogs/building-modern-docs-generator-nextjs-16): a practical look at the kind of client-side routing that produces soft navigations in the first place. - [How to Upgrade Next.js for the May 2026 Security Release](https://www.rabinarayanpatra.com/blogs/nextjs-may-2026-security-release-upgrade-guide): keeping your framework current is what gives you access to the newest performance APIs like this one. --- ## How to Migrate Micronaut 4 to Micronaut 5 with JDK 25 Source: https://www.rabinarayanpatra.com/blogs/migrate-micronaut-4-to-micronaut-5-jdk-25 Published: 2026-08-12T00:00:00+05:30 Tags: migrate-micronaut-4-to-micronaut-5, micronaut-5, micronaut, jdk-25, jspecify, jvm-migration Migrate Micronaut 4 to Micronaut 5 step by step. JDK 25 setup, Gradle 9.5 + Maven snippets, Jackson 3, JSpecify, retry API, and real upgrade pitfalls. Micronaut 5.0.0 hit GA on May 20, 2026. First major Micronaut release in roughly three years, and the migration is heavier than the version number suggests. I ran the upgrade on a Micronaut 4.7 service last week. The BOM bump is one line. Everything after that is real work: Java 25, Jackson 3, JSpecify imports, a renamed security processor, RxJava 2 gone, MicroStream replaced, and a new programmatic retry API that the old annotation flow quietly redirects to. This guide walks the path I took, with the exact Gradle and Maven snippets that compiled cleanly on my service. ## What changed between Micronaut 4 and Micronaut 5? Micronaut 5 is a platform-wide refresh across 70+ modules, not just a framework bump. The headline changes that affect real services are: JDK 25 baseline (was JDK 17), Kotlin 2.3, Groovy 5, a refactored IoC container with tighter qualifier semantics, JSpecify nullability replacing the older annotation soup, Jackson 3 in `micronaut-jackson-databind`, HTTP/3 promoted to stable on the Netty stack, and a new programmatic retry and circuit breaker API alongside the existing annotation model. On the deprecation side, Bootstrap Configuration is marked for removal in Micronaut 6, RxJava 2 is fully dropped (you go to RxJava 3), and MicroStream support is replaced by EclipseStore. If you used `micronaut-eclipsestore-annotations`, that artifact is renamed too. The piece I underestimated was the annotation processor rename. Security in particular silently broke my build until I swapped the artifact ID. More on that below. ## How do you install JDK 25 for the migration? Use SDKMAN. It is the cleanest way to flip JDK versions per project without touching system PATH. ```bash sdk install java 25-tem sdk default java 25-tem # Verify java -version # openjdk version "25" 2025-09-16 ``` Add a `.sdkmanrc` to your project root so the JDK pins automatically when you cd into it: ``` java=25-tem ``` If you run on a Docker base image, switch your runtime stage to `eclipse-temurin:25-jre` or `bellsoft/liberica-openjdk-alpine:25`. I ran a quick perf check after the JDK bump alone, before touching Micronaut: startup dropped roughly 8% on my service from JDK 21 to JDK 25, before any framework gains. Your numbers will vary, but it is a real improvement. For more on what is coming after JDK 25, see my deep-dive on [Java 26 Structured Concurrency](https://www.rabinarayanpatra.com/blogs/java-26-structured-concurrency-jep-525). ## How do you update Gradle builds for Micronaut 5? Bump three coordinates: the Micronaut platform BOM, the Micronaut Gradle plugin, and Gradle itself. Then add the Kotlin and Shadow plugin updates if you use them. **Before (Micronaut 4.7.x):** ```gradle plugins { id("io.micronaut.application") version "4.4.4" id("com.gradleup.shadow") version "8.3.5" id("org.jetbrains.kotlin.jvm") version "2.0.21" id("org.jetbrains.kotlin.plugin.allopen") version "2.0.21" id("com.google.devtools.ksp") version "2.0.21-1.0.27" } micronaut { version("4.7.6") runtime("netty") testRuntime("junit5") } java { sourceCompatibility = JavaVersion.VERSION_17 targetCompatibility = JavaVersion.VERSION_17 } ``` **After (Micronaut 5.0.0):** ```gradle plugins { id("io.micronaut.application") version "5.0.0" id("com.gradleup.shadow") version "9.4.1" id("org.jetbrains.kotlin.jvm") version "2.3.21" id("org.jetbrains.kotlin.plugin.allopen") version "2.3.21" id("com.google.devtools.ksp") version "2.3.7" } micronaut { version("5.0.0") runtime("netty") testRuntime("junit5") } java { sourceCompatibility = JavaVersion.VERSION_25 targetCompatibility = JavaVersion.VERSION_25 } ``` Then run `./gradlew wrapper --gradle-version=9.5.0 --distribution-type=bin` once to upgrade the wrapper. Commit the changed wrapper files in the same PR as the Micronaut bump so CI runs against the matching Gradle. If you read the version catalog instead of inline versions, you only update `gradle/libs.versions.toml`: ```toml [versions] micronaut = "5.0.0" micronaut-plugin = "5.0.0" kotlin = "2.3.21" ksp = "2.3.7" shadow = "9.4.1" ``` ## How do you update Maven builds for Micronaut 5? For Maven, swap the `micronaut-parent` coordinate and set the Java release properties to 25. Nothing else in the POM should change for a clean Micronaut 4 โ†’ 5 upgrade. **Before:** ```xml io.micronaut.platform micronaut-parent 4.7.6 17 17 ``` **After:** ```xml io.micronaut.platform micronaut-parent 5.0.0 25 25 ``` Make sure your Maven version is 3.9 or later. If you use the Micronaut Maven plugin for native-image builds, it pulls the matching version from the parent BOM, so you do not need to pin it separately. ## How do you adopt JSpecify nullability annotations? Micronaut 5 standardizes on JSpecify for nullability. If your codebase mixes `org.jetbrains.annotations.Nullable`, `javax.annotation.Nullable`, and `io.micronaut.core.annotation.Nullable`, this is the cleanup moment. The replacement is consistent: any package-level, class-level, or method-level nullability comes from `org.jspecify.annotations`. ```java // Before (mixed across the codebase) import org.jetbrains.annotations.Nullable; import io.micronaut.core.annotation.NonNull; public Optional findUser(@NonNull String id, @Nullable String tenant) { ... } // After import org.jspecify.annotations.NonNull; import org.jspecify.annotations.Nullable; public Optional findUser(@NonNull String id, @Nullable String tenant) { ... } ``` For broader scope, mark whole packages as non-null by default with a `package-info.java`: ```java @NullMarked package com.example.users; import org.jspecify.annotations.NullMarked; ``` That single annotation says every reference in the package is non-null unless explicitly marked `@Nullable`. It cuts annotation noise and gives Kotlin call sites cleaner platform types. Add JSpecify as a direct dependency only if you compile against it outside Micronaut classes; Micronaut transitively brings it in. IntelliJ has a built-in inspection called "Migrate nullability annotations". Set the target to JSpecify and run it on your `src/main` tree. Review every change, then commit. Do not let the inspection touch generated code. ## Why does Jackson 3 break existing serialization? `micronaut-jackson-databind` now uses Jackson 3 exclusively, and Jackson 3 has real wire-compatible differences from Jackson 2. The two ones that bit me: First, the package root changed. Custom `JsonSerializer` and `JsonDeserializer` classes that import `com.fasterxml.jackson.databind.*` need to move to `tools.jackson.databind.*`. The class names are the same, the package is different. ```java // Before (Jackson 2) import com.fasterxml.jackson.databind.JsonSerializer; import com.fasterxml.jackson.databind.SerializerProvider; // After (Jackson 3) import tools.jackson.databind.JsonSerializer; import tools.jackson.databind.SerializerProvider; ``` Second, Jackson 3 fails fast on unknown properties by default. If you relied on lenient deserialization in production, set the right deserialization feature on your `ObjectMapper` at startup, or annotate the DTO: ```java @JsonIgnoreProperties(ignoreUnknown = true) public record OrderEvent(String id, String status, Instant createdAt) { } ``` I had three DTO classes that were silently dropping unknown fields in Micronaut 4 and started throwing `UnrecognizedPropertyException` in Micronaut 5. Run integration tests against real payloads before merging. If you prefer to stay on the Micronaut serialization stack (`micronaut-serde-jackson`), this section does not apply. Serde is unchanged across the upgrade. ## What changed in the security annotation processor? The annotation processor artifact ID was renamed for several modules in Micronaut 5. The most important one is `micronaut-security-annotations`, which became `micronaut-security-processor`. The compiler error is misleading. You get a stack of "cannot find symbol" or "annotation processor failed to load" messages that point at your own code, not at the missing processor. ```xml io.micronaut.security micronaut-security-annotations io.micronaut.security micronaut-security-processor ``` ```gradle // Before (Gradle) annotationProcessor("io.micronaut.security:micronaut-security-annotations") // After annotationProcessor("io.micronaut.security:micronaut-security-processor") ``` The same rename pattern applies to `micronaut-eclipsestore-annotations`, which becomes `micronaut-eclipsestore-processor`. If you use Views Turbo, the dependency moved out of the main views module: add `micronaut-views-turbo`, and rename `@TurboView` to `@TurboStreamView` at every usage. ## How do the new programmatic retry and circuit breaker APIs work? Micronaut 4 only let you opt in to retry and circuit breakers through annotations like `@Retryable` and `@CircuitBreaker`. Micronaut 5 adds a programmatic, typed API in `io.micronaut.retry` that you can call without an annotation, which is what you want when the retry policy depends on runtime config. ```java import io.micronaut.retry.RetryPolicy; import io.micronaut.retry.RetryState; import jakarta.inject.Singleton; import java.time.Duration; @Singleton public class PaymentClient { private final RetryPolicy retryPolicy = RetryPolicy.builder() .maxAttempts(5) .delay(Duration.ofMillis(200)) .multiplier(2.0) .includes(IOException.class) .build(); public PaymentResult charge(ChargeRequest req) { return retryPolicy.execute(() -> gateway.charge(req)); } } ``` The circuit breaker variant uses the same builder shape with `circuitBreaker(true)`, an open-state duration, and a half-open probe count. It lets you wire policy from `application.yml`: ```java @ConfigurationProperties("payments.retry") public record PaymentsRetryConfig(int maxAttempts, Duration delay, double multiplier) { } ``` I switched two services from annotation-only to programmatic retry during the upgrade. The win is testability: you inject a stub `RetryPolicy` in unit tests instead of waiting on real backoff timers. The old annotations still work. You only need the programmatic API where the policy is dynamic. Mixing both in the same service is fine. ## What other breaking changes should you plan for? A few smaller items can stall the build if you do not catch them up front. - **RxJava 2 is dropped.** Add `micronaut-rxjava3` and migrate `io.reactivex.*` imports to `io.reactivex.rxjava3.*`. If you only used RxJava 2 for HTTP client return types, switching to `Mono` and `Flux` via Reactor is the cleaner long-term move. - **MicroStream is replaced by EclipseStore.** Same APIs in most places, but the artifact and package roots moved to `org.eclipse.store`. If you use the Micronaut integration, depend on the `eclipsestore` modules. - **Testcontainers wiring changed.** Use `org.testcontainers:testcontainers-junit-jupiter` for JUnit 5 integration, not the bare `junit-jupiter` artifact. The old coordinate compiles but produces a `NoClassDefFoundError` at test time. - **Data embedded fields need an annotation.** Annotate embedded fields with `@MappedProperty` so the column names stay stable. If you want the old behavior, set `micronaut.data.embedded.naming.strategy=LEGACY` in `application.yml`. - **Bootstrap Configuration deprecated.** Still works in Micronaut 5, scheduled for removal in Micronaut 6. If you load critical secrets through bootstrap, plan the migration to `PropertySourceImporter` SPI. After fixing these, a clean rebuild should pass. If your test suite still fails on dependency injection, regenerate the IDE project files. IntelliJ caches the old annotation-processor outputs and will mislead you for an hour before you give up and reimport. ## What should you do after the Micronaut 5 upgrade is live? Once the service compiles and tests pass, do four things before merging: 1. Run an integration test pass against real upstream payloads, especially anything that hits Jackson. The strict unknown-property default will catch dirty data you forgot you were ignoring. 2. Re-baseline startup time and memory in production. The IoC refactor and JDK 25 combined cut my startup by ~14% on a six-controller service. You want a number you can show to your platform team. 3. Audit your `@Replaces`, `@Requires`, and qualifier annotations. The compile-time semantics tightened, and a couple of ambiguous-bean errors that Micronaut 4 silently resolved by ordering now throw at startup. The error message names both candidates, which is the fix. 4. Update your CI base images to JDK 25. I forgot this for one repo and got a green local build with a broken `mvn deploy` job. Pin the image, not just the toolchain. The platform refresh is the biggest Micronaut release since 4.0 in 2023, and it sets the floor for what a Java microservice baseline looks like for the next two years. For more on the Micronaut 5 release, see the [official Micronaut 5.0.0 announcement](https://micronaut.io/2026/05/20/micronaut-framework-5-0-0-released/), the [Update to Micronaut 5 guide](https://github.com/micronaut-projects/micronaut-core/wiki/Update-to-Micronaut-5), and the [Java News Roundup May 18, 2026](https://www.infoq.com/news/2026/05/java-news-roundup-may18-2026/) on InfoQ. ### Keep Reading - [Modern Java + Spring Boot](https://www.rabinarayanpatra.com/blogs/modern-java-spring-boot). Compare the Micronaut upgrade path with how the same JDK 25 baseline reshapes a Spring Boot service. - [Java 26 Structured Concurrency: What Changed in the Sixth Preview](https://www.rabinarayanpatra.com/blogs/java-26-structured-concurrency-jep-525). The next JDK feature your Micronaut 5 service can adopt once the upgrade is in. - [Java Libraries Beyond Lombok](https://www.rabinarayanpatra.com/blogs/java-libraries-beyond-lombok). Useful JVM libraries that pair well with a JDK 25 + Micronaut 5 stack. --- ## How to Add Audit Trails with Hibernate 7.4 @Audited Annotation Source: https://www.rabinarayanpatra.com/blogs/hibernate-7-4-audited-annotation-spring-boot Published: 2026-08-07T00:00:00+05:30 Tags: hibernate-7-4-audited, hibernate, spring-boot, jpa, audit-log, data-history Hibernate 7.4 ships a built-in @Audited annotation. Step-by-step how-to wires it into Spring Boot, defines a @Changelog entity, and queries via AuditLog. I have wired Envers into Spring Boot projects three times in the last five years and every time it felt like work that should have been a single annotation. Hibernate 7.4 finally makes it one. The new `@Audited` annotation lives in `org.hibernate.annotations` (the core package, not the Envers module), and it is the first piece of a much larger feature called the StateManagement SPI. This post walks through the full setup in a Spring Boot 4 project: how to override the Hibernate version to get 7.4, how to wire the `@Changelog` entity that supplies changeset ids, how to annotate an entity to start auditing, and how to query history with the new `AuditLog` interface. The feature is incubating, meaning the API can shift before the final release, but the surface is small enough that the migration cost is bounded if it does. If you already use Envers, the migration path is gradual. You can keep auditing existing entities with Envers and adopt the new annotation only on new ones until the dust settles. ![How to Add Audit Trails with Hibernate 7.4 @Audited Annotation cover](/images/blogs/hibernate-7-4-audited-annotation-spring-boot.webp) ## What is the @Audited annotation in Hibernate 7.4? The new `@Audited` annotation marks an entity class (or a collection field) as audited, meaning Hibernate keeps a historical record of every change to it. It lives in `org.hibernate.annotations.Audited` and was added in Hibernate 7.4 as an `@Incubating` feature, shipped first in `7.4.0.CR1` on May 7, 2026. When you annotate an entity with `@Audited`, Hibernate creates two tables instead of one. The primary table holds the current state, exactly like before. A separate audit log table holds a row for every change. Each audit row contains: - The full state of the entity at the moment of the change (minus any fields you mark with `@Excluded`). - A `changesetId` column that groups changes into atomic batches. - A `modificationType` column encoded as `0` for creation, `1` for modification, and `2` for deletion. The `changesetId` is supplied by one of three mechanisms, in order of preference: a `@Changelog` entity in your domain model (the recommended path), a custom `ChangesetIdentifierSupplier` registered via configuration, or, as a last resort, `Instant.now()`. The Hibernate javadoc itself flags relying on the `Instant.now()` fallback as not recommended. You query history three ways. You open a session with `SessionBuilder.atChangeset(id)` to transparently read entity state as of that changeset (regular HQL queries just work). You use the `AuditLog` interface for programmatic access to revision history and cross-entity queries. Or you open a session with `AuditLog.ALL_CHANGESETS` and write custom HQL using the new `changesetId()` and `modificationType()` functions. This is the entire model. No revision-info entity. No revision listener. No conditional auditing strategy. Just an annotation, a changelog entity, and a query interface. ## How does Hibernate 7.4 @Audited differ from Envers? The Envers module (introduced in Hibernate 3.6 over a decade ago) lives in `org.hibernate.envers` and ships as a separate jar. It uses a revision-tracking model with `RevisionEntity` and `@RevisionNumber`, writes audit tables prefixed with `_AUD`, and queries through `AuditReader`. It works and is mature, but it is also crusty: configuration is XML-shaped, the API surface is wide, and getting current-user-in-audit-trail right always involves a custom listener. The new annotation rebuilds the same idea on a smaller surface. Key differences: - **Package**: `org.hibernate.annotations.Audited` (new core annotation) versus `org.hibernate.envers.Audited` (legacy Envers annotation). Both annotations are spelled `@Audited`, so the import you choose decides which engine handles the entity. - **Module**: the new annotation is in `hibernate-core`. No extra dependency needed if you already have Hibernate 7.4 on your classpath. Envers ships in `hibernate-envers` and must be added separately. - **Changeset metadata**: Envers uses a `RevisionEntity` with `@RevisionNumber` and `@RevisionTimestamp`. The new annotation uses a `@Changelog` entity with `@ChangesetId` and `@Timestamp`. The shape is similar but the field annotations are first-class on the new side. - **Audit table layout**: Envers writes a `_AUD` suffixed table per entity with revision linkage. The new model writes a parallel audit table with `changesetId` plus `modificationType` columns directly. Different layout, different query SQL. - **Querying**: Envers exposes `AuditReader` retrievable from the `EntityManager`. The new model exposes `AuditLog` through `AuditLogFactory.create()` and supports transparent point-in-time reads via `SessionBuilder.atChangeset()`. - **Maturity**: Envers is GA, battle-tested, used in production for a decade. The new annotation is incubating, so the API can shift between 7.4 and 7.5. The Hibernate team has stated the 7.4 release is backward compatible with Envers, meaning your existing Envers-audited entities keep working unchanged when you upgrade. You can layer the new `@Audited` annotation on new entities while leaving the old ones on Envers, then migrate over time. ![Hibernate 7.4 @Audited vs Envers comparison table](/images/blogs/hibernate-7-4-audited-annotation-spring-boot-comparison.webp) ## How do you set up the audit feature in a Spring Boot project? You override the Hibernate version managed by Spring Boot, because Spring Boot 4.0 ships Hibernate 7.0 and Spring Boot 4.1 has not been released yet. The override is a single property in your build file. For Maven (`pom.xml`): ```xml 24 7.4.0.CR1 ``` For Gradle (`build.gradle.kts`): ```kotlin ext["hibernate.version"] = "7.4.0.CR1" dependencies { implementation("org.springframework.boot:spring-boot-starter-data-jpa") } ``` Spring Boot picks up the override and Hibernate Core, Hibernate JPA, and Hibernate Annotations all align on 7.4.0.CR1. If you already had Envers on the classpath, leave it. The new annotation is in core. You can keep both jars side by side and import explicitly. If you intentionally do not want Envers anymore, exclude it once you have migrated every entity. Verify the version on application startup. Hibernate logs it at INFO level on the bootstrap line: ``` INFO o.h.Version - HHH000412: Hibernate ORM core version 7.4.0.CR1 ``` There is no extra configuration required for the audit feature to activate. The presence of a `@Changelog` entity and any `@Audited` entity is enough. ## How do you define a @Changelog entity for changeset ids? Add a single class to your domain model annotated with `@Changelog` and `@Entity`. It must declare an `@Id` field with `@ChangesetId` and a timestamp field with `@Timestamp`. Hibernate handles the rest, persisting one instance per transaction and using its primary key as the changeset id for every audited change in that transaction. A minimal version: ```java package com.example.audit; import jakarta.persistence.Entity; import jakarta.persistence.GeneratedValue; import jakarta.persistence.GenerationType; import jakarta.persistence.Id; import jakarta.persistence.Table; import java.time.Instant; import org.hibernate.annotations.Changelog; import org.hibernate.annotations.ChangesetId; import org.hibernate.annotations.CreationTimestamp; import org.hibernate.annotations.Timestamp; @Entity @Changelog @Table(name = "changeset") public class Changeset { @Id @ChangesetId @GeneratedValue(strategy = GenerationType.IDENTITY) private Long id; @Timestamp @CreationTimestamp private Instant createdAt; public Long getId() { return id; } public Instant getCreatedAt() { return createdAt; } } ``` `@CreationTimestamp` populates `createdAt` automatically on insert, which the `@Changelog` Javadoc explicitly cites as the recommended way to initialize the timestamp field. The `Instant` type maps cleanly to a PostgreSQL `timestamp` column. Once this class is on the classpath, Hibernate auto-registers it as the changeset id supplier. You do not need to set `org.hibernate.cfg.StateManagementSettings.CHANGESET_ID_SUPPLIER` manually. Only one entity per application may be annotated with `@Changelog`, so this is your single source of truth for changeset metadata. To enrich the audit trail with the current user and an optional comment, add fields and a `@ChangesetListener`. The listener fires on every changeset insertion and gives you a hook to populate the user from your security context: ```java package com.example.audit; import org.hibernate.audit.ChangesetListener; import org.springframework.security.core.context.SecurityContextHolder; public class CurrentUserChangesetListener implements ChangesetListener { @Override public void prePersist(Object changeset) { if (changeset instanceof Changeset c) { var auth = SecurityContextHolder.getContext().getAuthentication(); c.setCreatedBy(auth != null ? auth.getName() : "system"); } } } ``` You register the listener once during `SessionFactory` build via the Hibernate `Integrator` SPI, the same way you would have wired any other Hibernate listener. ## How do you annotate an entity to start auditing it? Annotate the entity class with `@Audited`. That is the entire setup. Hibernate generates the audit log table on schema export, populates it on every change, and you do not have to write a single line of audit-specific code anywhere else. A complete example for a `Customer` aggregate: ```java package com.example.customer; import jakarta.persistence.Column; import jakarta.persistence.Entity; import jakarta.persistence.GeneratedValue; import jakarta.persistence.GenerationType; import jakarta.persistence.Id; import jakarta.persistence.Table; import org.hibernate.annotations.Audited; import org.hibernate.annotations.Excluded; @Entity @Audited @Table(name = "customer") public class Customer { @Id @GeneratedValue(strategy = GenerationType.IDENTITY) private Long id; @Column(nullable = false) private String name; @Column(nullable = false, unique = true) private String email; @Excluded @Column(nullable = false) private String passwordHash; // getters and setters } ``` The `@Excluded` annotation on `passwordHash` keeps that column out of the audit log, which is what you want for password hashes, PII you do not need to track historically, or large binary blobs. Everything else is recorded on every change. When Hibernate exports the schema, it creates two tables: the regular `customer` table and a parallel audit log table. Hibernate names the audit table conventionally, similar to `customer_aud` by default, and the `@Audited` annotation accepts custom table name overrides if you need them. The audit table contains the same state columns minus excluded ones, plus a `changeset_id` column and a `modification_type` column. A typical row sequence after a create-update-delete cycle on a single customer: ```sql SELECT * FROM customer_aud WHERE id = 42 ORDER BY changeset_id; id | name | email | changeset_id | modification_type ----+-----------+----------------+--------------+------------------- 42 | Alice Doe | alice@ex.com | 1 | 0 42 | Alice Doe | alice@xyz.com | 2 | 1 42 | Alice Doe | alice@xyz.com | 3 | 2 ``` The `0/1/2` encoding is from the source javadoc directly. You can decode it in queries using the new `modificationType()` HQL function. ![Audit log table schema generated by @Audited](/images/blogs/hibernate-7-4-audited-annotation-spring-boot-schema.webp) ## How do you query audit history with the AuditLog interface? Obtain an `AuditLog` instance from `AuditLogFactory.create()`, run your history queries, and close it when done. The `AuditLog` interface is `AutoCloseable` and manages its own internal session, so a try-with-resources block is the cleanest pattern. A repository method that returns the full revision history of a customer: ```java package com.example.customer; import java.util.List; import org.hibernate.audit.AuditLog; import org.hibernate.audit.AuditLogFactory; import org.springframework.stereotype.Repository; @Repository public class CustomerAuditRepository { private final AuditLogFactory factory; public CustomerAuditRepository(AuditLogFactory factory) { this.factory = factory; } public List getHistory(Long customerId) { try (AuditLog auditLog = factory.create()) { return auditLog.getHistory(Customer.class, customerId); } } } ``` The `getHistory(Class, Object)` method returns every revision of the entity in chronological order, including the final deletion row if the entity has been deleted. Each returned instance is a snapshot of the entity at that changeset, not a live managed object. For more specific queries (e.g., who deleted what last quarter), open a session with the `AuditLog.ALL_CHANGESETS` magic value and write custom HQL using the new functions: ```java try (var session = sessionFactory.withOptions() .atChangeset(AuditLog.ALL_CHANGESETS).open()) { var deletions = session.createSelectionQuery(""" select c.id, c.email, changesetId(c), changeset.createdBy from Customer c join Changeset changeset on changesetId(c) = changeset.id where modificationType(c) = 2 and changeset.createdAt > :since """, Object[].class) .setParameter("since", lastQuarterStart) .getResultList(); } ``` `changesetId(c)` returns the changeset of the audit row. `modificationType(c)` returns the 0/1/2 code. Joining against the `Changeset` entity lets you correlate with whatever metadata you put on your changelog (user, comment, request id). ## How do you read entity state at a past changeset? Open a session with `SessionBuilder.atChangeset(changesetId)` and run regular HQL queries. Every entity load, every query, every association traversal returns the state as of that changeset, not the current state. ```java public Customer customerAtChangeset(Long customerId, Long changesetId) { try (var session = sessionFactory.withOptions() .atChangeset(changesetId).open()) { return session.createSelectionQuery( "from Customer where id = :id", Customer.class) .setParameter("id", customerId) .uniqueResult(); } } ``` This is the killer feature versus Envers, which required different APIs to query past state. Here the same HQL works whether the session is rooted at the current state or a historical one, because the session itself carries the temporal anchor. Use cases I have already mapped to this: - **Audit trail UI**: render a customer detail page as it looked on a specific date by passing the changeset id from a date picker. - **Debug session reconstruction**: when a customer reports a bug from yesterday, open a session at yesterday's last changeset and run the user's flow against the historical data. - **Compliance exports**: for GDPR Article 30 records of processing, you can prove exactly what data existed when, without parsing audit log tables yourself. One important caveat from the source: `atChangeset()` sessions are read-only. Attempting a flush or commit raises an exception. This is by design (you cannot retroactively edit history), but it means you cannot use the same session for both historical reads and current writes. ![How to query Hibernate audit log via AuditLog interface and atChangeset session](/images/blogs/hibernate-7-4-audited-annotation-spring-boot-query-flow.webp) ## What should you do this week? I would not migrate production audit code to 7.4 yet. It is `@Incubating`, the API can shift before final release, and Envers is still production-grade. But I would: 1. Spin up a 7.4 sandbox project and try the annotation on one entity. The full setup above takes about thirty minutes. 2. File feedback on anything that feels off (the Hibernate team is active on the issue tracker and incubating-annotation feedback is the whole point of the CR phase). 3. Plan the migration path for after the 7.4 final release. The Envers-to-new-annotation walk is straightforward because both can coexist. If you start a new Spring Boot 4 project from scratch right now and want auditing, the choice is harder. Envers is mature but verbose. The new annotation is clean but unstable. I would still pick Envers today for production, the new annotation for greenfield experiments. Either way, the days of writing your own audit-trail listeners and updating-by-hand audit columns should be behind you. Hibernate has caught up to what every team eventually builds by hand. For more on the new feature, see the [Hibernate 7.4 What's New page](https://docs.hibernate.org/orm/7.4/whats-new/), the [Hibernate Audited.java source on GitHub](https://github.com/hibernate/hibernate-orm/blob/main/hibernate-core/src/main/java/org/hibernate/annotations/Audited.java), and the [official Envers documentation](https://docs.jboss.org/hibernate/orm/7.4/userguide/html_single/Hibernate_User_Guide.html#envers) for the legacy module. ### Keep Reading - [Spring Boot 4 AOT Data Repositories: The Underrated Feature](https://www.rabinarayanpatra.com/blogs/spring-boot-4-aot-data-repositories). Companion piece on Spring Boot 4 data layer improvements that pair well with the new audit setup. - [PostgreSQL 18 Temporal Foreign Keys with Spring Boot JPA](https://www.rabinarayanpatra.com/blogs/postgresql-18-temporal-foreign-keys-spring-boot). Database-side temporal data. Complements the application-side audit log with row-level effective dating. - [Hibernate Lazy Initialization Guide](https://www.rabinarayanpatra.com/blogs/hibernate-lazy-init-guide). Same kind of Hibernate-runtime detail post if you want more of this depth. --- ## How to Set Up Kiro Crew for Unattended Coding Tasks Source: https://www.rabinarayanpatra.com/blogs/how-to-set-up-kiro-crew-unattended-coding-tasks Published: 2026-08-05T00:00:00+05:30 Tags: kiro-crew-setup, kiro, ai-agents, aws, developer-tools, open-source Kiro Crew setup from install to unattended runs: prerequisites, kiro-cli auth, the gateway, systemd service, and the sandbox settings that actually matter. AWS open sourced Kiro Crew on August 4, 2026, and the pitch is one most of us have quietly wanted for a while: start a piece of work, close the laptop, and come back to something reviewable instead of a dead session and a cold context window. I want to be upfront about where I sit. I work at Amazon and I have contributed to Kiro Crew. Everything below comes from the public release: the [announcement post](https://kiro.dev/blog/introducing-kiro-crew/), the [product page](https://kiro.dev/crew/), and the [Apache 2.0 repository](https://github.com/kirodotdev/KiroCrew). No internal detail, no roadmap talk. Just the setup path and the parts I think are worth slowing down on. This is a Kiro Crew setup guide, not a review. By the end you will have a gateway running, a service that survives a reboot, and enough understanding of the sandbox to decide what you are comfortable letting it do while you sleep. ![Kiro Crew keeps a task alive across sessions: you start work and leave, the gateway checkpoints and retries, and you return to a reviewable result](/images/blogs/how-to-set-up-kiro-crew-unattended-coding-tasks.svg) ## What is Kiro Crew and how is it different from a coding agent? Kiro Crew is a persistent workspace that coordinates agents across sessions, where a normal coding agent lives and dies inside one conversation. That difference sounds small and it is not. A single-session agent forgets everything when the process exits, which means anything longer than one sitting gets restarted by hand. Crew keeps memory, schedules, approvals, and in-flight task state in a local gateway, so a job started Tuesday afternoon can still be making progress Wednesday morning. It started inside Amazon as a side project called MeshClaw, built by three engineers who wanted to kick off work and walk away. AWS says it reached more than 39,000 Amazon builders and roughly 500 contributors in under six months of internal use before the public release. The architecture is three pieces: | Component | Job | |---|---| | Gateway | Routes messages, persists state, manages sessions, approvals, memory, and security policy | | Agent Sessions | Isolated conversation or task contexts | | ACP Runtime | Drives `kiro-cli` over the Agent Client Protocol | Two open standards do the heavy lifting. Agent Client Protocol (ACP) is how the gateway drives and observes the agent, and Model Context Protocol (MCP) is how tools get attached. If you have read my take on [why the MCP spec went stateless](https://www.rabinarayanpatra.com/blogs/why-mcp-2026-07-28-spec-goes-stateless), the shape here will feel familiar. ![Kiro Crew architecture: the gateway holds state and policy, agent sessions stay isolated, and the ACP runtime drives kiro-cli](/images/blogs/how-to-set-up-kiro-crew-unattended-coding-tasks-architecture.svg) ## What do you need before running Kiro Crew setup? You need three things: Python 3.10 or higher, an authenticated `kiro-cli`, and hardware you are willing to leave running. Node.js 20 or 22+ only matters if you build the dashboard from source, so if you install a prebuilt wheel or the desktop app you can skip it entirely. The one people miss is `kiro-cli`. It is the agent backend, not an optional extra, and Crew cannot do anything without it: ```bash kiro-cli login ``` Do this before anything else. Skipping it produces an `AcpTimeoutError: ACP prompt timed out` later, which reads like a network problem and is not. Memory search needs no setup. On first start, the in-process `llama-cpp-python` runtime pulls the Qwen3-Embedding-0.6B model, about 610 MB. Budget the disk and the first-run wait. Search comes back empty until that download lands, which is expected rather than broken. One billing note that matters more for unattended work than interactive work. The software is free under Apache 2.0 and runs on your own machine, but agent inference still needs a Kiro account, and model requests from scheduled or background tasks count against your Kiro plan exactly like IDE usage. Steps that only run scripts, with no model call, cost nothing. A cron job that wakes an agent every fifteen minutes is a billing decision, not just a config line. ## How do you install Kiro Crew? Pick the install method that matches how long you intend to keep it running. There are five, and they are not interchangeable. **One-line install.** Fastest path, and the right default: ```bash curl -fsSL https://download.crew.kiro.dev/cli.sh | sh ``` That script pins to the stable channel. You can target others: ```bash curl -fsSL https://download.crew.kiro.dev/cli.sh | sh -s -- --channel insider curl -fsSL https://download.crew.kiro.dev/cli.sh | sh -s -- --version 0.1.0 ``` Worth saying plainly: piping a remote script into a shell executes whatever that URL serves. It is the documented install path and the domain is AWS-operated, but if you are on a machine where that tradeoff is not acceptable, use the wheel or source path instead. **Docker**, for a server you do not want to babysit: ```bash docker run -d --name kirocrew \ -p 127.0.0.1:5476:5476 \ -v kirocrew-home:/home/kirocrew \ ghcr.io/kirodotdev/kirocrew:stable ``` Note the `127.0.0.1:` prefix on the port binding. Dropping it exposes the dashboard on every interface, which you do not want. **From source**, if you plan to contribute or read the code: ```bash git clone https://github.com/kirodotdev/KiroCrew.git cd KiroCrew make build source .venv/bin/activate ``` **Self-contained wheel**, when you want a clean install without the Node toolchain on the target box: ```bash make wheel pip install dist/*.whl ``` **Desktop bundle**, which embeds a Python interpreter so end users need neither Python nor npm: ```bash make desktop ``` That drops a signed `.dmg` on macOS or an `.AppImage` on Linux into `website/electron/dist/`. ## How do you verify the setup and start the gateway? Run the wizard, then `doctor`, then the gateway, in that order: ```bash kirocrew setup kirocrew doctor kirocrew gateway ``` `kirocrew setup` is interactive and covers agent config, workspace directory, optional Slack credentials, slash-command name, timezone, dashboard URL, the Playwright MCP server, and the macOS desktop app. Answering "n" to Slack turns off that integration and leaves the dashboard fully working, so there is no reason to wire up chat on the first pass. Three flags narrow what it touches: - `--agent-only` installs agent config and skips every credential prompt - `--clean` rebuilds agent config from defaults and ignores what is already there - `--electron-only` installs just the macOS desktop app The `--agent-only --clean` combination is the fix for a stale MCP config. If the agent starts timing out after you have changed tools around, rebuild before you start debugging anything else. `kirocrew doctor` is the command to reach for whenever something is off. It reports platform details, data home, agent config, config values, MCP servers, runtime environment, vector memory status, speech-to-text, Slack, and connectivity. Three failures cover most of what you will hit: | Symptom | Cause | Fix | |---|---|---| | `AcpTimeoutError: ACP prompt timed out` | `kiro-cli` missing, not logged in, or broken MCP config | Install it, run `kiro-cli login`, then `kirocrew setup --agent-only --clean` | | Memory and knowledge search return nothing | Embedding model still downloading | Check the Vector Memory section in `doctor`, search upgrades itself once the model lands | | Gateway will not start | Port already taken | `kirocrew gateway --port auto` | `kirocrew gateway` starts the server on `http://localhost:5476`. It binds to loopback unless you deliberately configure otherwise, which is the correct default and worth leaving alone. Configuration lives at `~/.kiro/crew/config.json`: ```json { "agent": { "provider": "acp", "approval_mode": "interactive", "sandbox": "auto" }, "session": { "timeout_secs": 1800, "pool_size": 2 }, "dashboard": { "bot_name": "Kiro Crew" } } ``` Edit it through `kirocrew config get`, `set`, and `edit` rather than by hand, so validation runs. A few environment variables override the basics: `KIROCREW_HOME` for the data directory, `KIROCREW_PORT` for the dashboard port, and `KIROCREW_EMBED_MODEL_URL` if you need to point the embedding download at a mirror. ## How do you keep Kiro Crew running when you close your laptop? Install it as a system service, because a foreground `kirocrew gateway` dies with your terminal and takes the point of the tool with it: ```bash kirocrew service install kirocrew service status ``` On Linux this writes `/etc/systemd/system/kirocrew.service` and prompts for sudo. The gateway itself still runs as your user, never as root, which is the behavior you want. On macOS it writes a launchd plist instead. Linux users on Ubuntu 23.10 or newer should expect one specific failure. Crew isolates the agent from your credentials using user and mount namespaces, and recent Ubuntu restricts unprivileged user namespaces through AppArmor. The error looks like this: ``` sandbox: unshare(NEWNS) failed: errno 1 ``` `kirocrew service install` adds the required `kirocrew-userns` AppArmor profile automatically. If you are running outside systemd, invoke it yourself: ```bash aa-exec -p kirocrew-userns -- kirocrew gateway ``` The installer skips the profile on distributions where it does not apply, so Debian, Arch, RHEL, and Amazon Linux users will not see this at all. One thing that surprised me the first time: uninstalling does not remove `~/.kiro/crew`. Config, credentials, memory, and sessions all survive, and no uninstall path deletes them. That is a sensible default, but it means a "clean reinstall" is not clean unless you move the directory yourself. Back it up before you touch it. ## What does the security model actually block? Kiro Crew ships seven layers of defense, and the ones that matter for unattended work are the sandbox, the deny patterns, and the audit trail. The full set is OS sandboxing, denied command patterns, bash blocking, input and output validation, sensitive path protection, credential redaction, and signed audit logs. ![The Kiro Crew defense layers, from OS sandbox through deny patterns and credential redaction to the signed audit log](/images/blogs/how-to-set-up-kiro-crew-unattended-coding-tasks-security.svg) Concretely, here is what you get: - The dashboard binds to `127.0.0.1` unless you configure otherwise - 137 deny patterns block destructive and exfiltration-shaped commands - Sensitive path guards keep the agent out of protected directories - Credential redaction strips secret-shaped environment variables and patterns from output - Security events append to `~/.kiro/crew/security_events.jsonl` - Linux uses namespaces, macOS uses Seatbelt, with standard, strict, and off modes Optional policy files compose tightest-wins, and enterprise settings enforce caps that runtime config cannot loosen. That ordering is the right way round, and it is the detail I would check first if you are deploying this anywhere shared. Read `security_events.jsonl` after your first unattended run. Not because you expect trouble, but because it is the fastest way to learn what your agent actually reached for when nobody was watching. The gap between what I assumed a task would touch and what it tried to touch was the most useful thing I learned in my first week. The Windows situation deserves a direct warning rather than a footnote. **On Windows, Kiro Crew has no OS-level sandboxing.** Because of that it refuses to spawn subprocesses by default, and the only way to change that is opting in through `sandbox_allow_unsandboxed_exec`. Turning that on means an unattended agent runs commands on your machine with no OS isolation between it and everything else. There is also no Windows desktop app yet, so you install from source. If you want unattended runs, put this on Linux or macOS. Use Windows for interactive work where you are approving each step. ## What breaks when you walk away? The honest answer is that autonomy shifts your work from writing code to reviewing it, and the failure modes move with it. Crew handles the mechanical parts well: checkpoints and retries carry long tasks through transient failures, heartbeat monitoring watches PRs and deploys until state changes, and corrections you make become durable lessons stored as editable Markdown. What it cannot do is notice that the task was wrong. A well-specified job runs beautifully unattended. A vague one produces a large, confident, wrong diff that costs more to review than it would have cost to write. Scope discipline matters more here than in interactive work, because there is no midpoint where you glance at the screen and course-correct. Two settings shape this directly. `approval_mode` set to `interactive` means you gate tool calls, which defeats overnight runs but is the right place to start. `session.timeout_secs`, 1800 by default, bounds how long a session can sit before it gives up. Loosen both deliberately, one at a time, once you trust a particular workflow. If you want a gentler on-ramp, the launch Apps are scoped jobs rather than open-ended autonomy. Issue Radar triages issues and PRs, Task Runner handles bounded long-running work, DevFleets manages worktrees, and Code Review Sage weights findings by blast radius. Starting there teaches you the review rhythm without betting a repository on it. The same instinct applies to [scheduling agents in CI](https://www.rabinarayanpatra.com/blogs/claude-code-routines-ci-automation): narrow scope first, widen only after you trust the output. ## Should you set up Kiro Crew today? If you already work with agents and keep hitting the session boundary, yes. That is the specific problem Crew solves, and it solves it in a way you can audit, since the whole thing is Apache 2.0 and runs on your hardware. If you are new to agentic tooling, install it but keep `approval_mode` on `interactive` for a couple of weeks. Watch what it reaches for. The persistence is the feature, and persistence without understanding is how you end up reviewing a hundred commits you did not ask for. The thing I did not expect is how much the review rhythm changes. Interactive agent work is a conversation. Unattended agent work is closer to managing a very fast, very literal contractor who never asks clarifying questions. That is a different skill, and it is worth building deliberately before you scale it up. For more, see the [Kiro Crew announcement](https://kiro.dev/blog/introducing-kiro-crew/), the [Apache 2.0 source on GitHub](https://github.com/kirodotdev/KiroCrew), the [install guide](https://github.com/kirodotdev/KiroCrew/blob/main/docs/guides/install.md), and the [Agent Client Protocol](https://agentclientprotocol.com/) that Crew uses to drive agents. ### Keep Reading - [Google Antigravity: Why I Think It Changes Everything](https://www.rabinarayanpatra.com/blogs/google-antigravity). A different take on the agentic IDE idea, useful for comparing design choices. - [Claude Code Routines: Async CI Automation Just Became Real](https://www.rabinarayanpatra.com/blogs/claude-code-routines-ci-automation). The same scheduled-agent problem approached from the CI side. - [How to Build a Self-Hosted Slack AI Bot with any CLI Agent](https://www.rabinarayanpatra.com/blogs/how-to-build-self-hosted-slack-ai-bot-cli-agent). Relevant if you plan to drive Crew from Slack. --- ## How to Recover echarts-for-react After the May 2026 AntV Attack Source: https://www.rabinarayanpatra.com/blogs/recover-echarts-for-react-may-2026-antv-attack Published: 2026-08-05T00:00:00+05:30 Tags: echarts-for-react, antv-attack, mini-shai-hulud, supply-chain-security, npm, incident-response Step-by-step incident response after the May 19 @antv npm worm hit echarts-for-react. Audit lockfile, pin safe versions, rotate tokens, scan history. I got the Slack notification on the morning of May 20 from a dashboard project I do not even maintain anymore. The CI had failed overnight with a weird npm install error. By the time I logged in, the actual story had broken on Hacker News. A compromised npm maintainer named `atool` had published 639 malicious versions across 323 packages in a 22-minute burst on May 19, 2026. The hit list included `echarts-for-react`, with 1.1 million weekly downloads, and every major package under `@antv`. If your team ran `npm install` between roughly 01:39 and 02:06 UTC on May 19, your CI runner or developer machine probably executed the payload. The malware was a `preinstall` script that harvested over 20 credential types and shipped them to attacker-controlled GitHub repos before the install even finished resolving. This post is not a postmortem. It is a recovery playbook. If you shipped anything during the bad window, work through these steps in order. The first three are about containment. The last three are about hardening so the next wave does not land. ![How to Recover echarts-for-react After the May 2026 AntV Attack cover](/images/blogs/recover-echarts-for-react-may-2026-antv-attack.webp) ## What happened in the May 2026 @antv npm attack? On May 19, 2026, an attacker (tracked as TeamPCP by Socket, attributed by Microsoft as part of the Mini Shai-Hulud campaign) compromised the npm maintainer account `atool`. Over a 22-minute window, the attacker published 639 malicious versions spanning 323 packages, including the full `@antv` data-visualization suite, `echarts-for-react`, `timeago.js`, `size-sensor`, and `canvas-nest.js`. The payload was a `preinstall` script with the entry `"bun run index.js"`. On install, it ran a credential harvester that scraped at least 20 credential types: AWS access keys, GCP service accounts, Azure secrets, GitHub Personal Access Tokens, npm tokens, SSH private keys, kubeconfig files, and Vault tokens. It then created public GitHub repositories on stolen accounts (with names like `sayyadina-stillsuit-852`) and committed the harvested credentials there as a fallback exfiltration channel in case the primary C2 at `t[.]m-kosche[.]com` was blocked. GitHub eventually invalidated 61,274 npm tokens with write access and 2FA bypass. The malicious package versions were unpublished. But anything those tokens already touched is suspect, and anything those exfiltrated credentials can still log into is fair game for an attacker. Cleanup is on you. The earlier [TanStack attack](https://www.rabinarayanpatra.com/blogs/tanstack-supply-chain-attack-2026) on May 11 used a different primitive (a GitHub Actions cache-poisoning bug that produced valid SLSA Build Level 3 attestations). This attack reverted to the classic playbook: own a maintainer account, push malicious versions, wait for `npm install`. Different mechanism, same outcome. ## How do you know if your app pulled a poisoned echarts-for-react version? You know if your `package-lock.json` or `pnpm-lock.yaml` or `yarn.lock` resolved `echarts-for-react` or any `@antv/*` package to a version published between May 19, 2026 01:39 UTC and 02:06 UTC. There is no ambiguity. Any install in that window pulled a poisoned tarball. Run this from your project root: ```bash # npm jq '.packages | to_entries[] | select(.key | test("echarts-for-react|@antv/|size-sensor|timeago\\.js|canvas-nest\\.js")) | {key, version: .value.version, resolved: .value.resolved}' package-lock.json # pnpm grep -E "echarts-for-react|@antv/|size-sensor|timeago\.js|canvas-nest\.js" pnpm-lock.yaml | head -50 # yarn (v1) grep -E "^(echarts-for-react|@antv/|size-sensor|timeago\.js|canvas-nest\.js)" yarn.lock -A 2 ``` Then check each resolved version against the npm registry publish timestamp: ```bash npm view echarts-for-react@ time --json ``` If the timestamp falls in the bad window, you are hit. If not, you are clean for that package. Repeat for every match in the lockfile. If the lockfile lookup is too slow because you have dozens of `@antv/*` deps, run a faster fleet check. Grep your CI runs for `preinstall` activity on May 19: ```bash # Search CI logs for the preinstall payload signature grep -r "bun run index.js" .github/ ci-logs/ 2>/dev/null # Or query GitHub Actions runs (gh CLI) gh run list --workflow=ci.yml --created '2026-05-19' --json conclusion,databaseId,headBranch ``` Any green run from that day that ran `npm ci` against a lockfile resolving to bad versions exfiltrated credentials. Treat the runner as compromised. ## Which echarts-for-react and @antv versions are safe to pin? Any version published before May 19, 2026 01:39 UTC is safe. The compromised window is bounded. Versions published after the unpublish event on May 20 are also safe, because the maintainer account has been recovered and the malicious versions are gone from the registry. The fastest way to find the right pin is to list all versions with timestamps: ```bash npm view echarts-for-react versions --json | jq -r '.[]' | tail -20 npm view echarts-for-react time --json | jq 'to_entries | sort_by(.value) | .[-20:]' ``` Pin to the last entry whose `time` value is before `2026-05-19T01:39:00Z`. Same for every `@antv/*` package in your lockfile. To enforce the pin across the workspace, use overrides. For npm: ```json { "overrides": { "echarts-for-react": "3.0.4", "@antv/g2": "5.2.20", "@antv/x6": "2.18.1", "@antv/graphin": "3.0.4" } } ``` For pnpm: ```yaml # pnpm-workspace.yaml or package.json pnpm: overrides: echarts-for-react: 3.0.4 "@antv/g2": 5.2.20 ``` The version numbers above are placeholders. Look up the actual safe versions yourself with `npm view`, because new safe versions ship continuously and a static post will go stale. Overrides force the resolver to ignore the transitive dependency graph and pin everywhere, including inside nested deps. ![Lockfile audit decision flow for the AntV compromise](/images/blogs/recover-echarts-for-react-may-2026-antv-attack-audit-flow.webp) ## How do you clean the lockfile and reinstall safely? Delete `node_modules` and the lockfile, then reinstall with `--ignore-scripts` so any residual malicious `preinstall` hook in your tree does not run. This is the single most important step in the whole playbook. Skipping `--ignore-scripts` on a poisoned lockfile is what trips most teams trying to clean up. ```bash # Stop the dev server, kill any watchers rm -rf node_modules rm -f package-lock.json # (or pnpm-lock.yaml / yarn.lock) npm install --ignore-scripts ``` If you also have leftover untracked artefacts in your working tree from the poisoned install (random dot-files, generated test fixtures), use [`git clean` with the `-n` dry run first](https://www.rabinarayanpatra.com/snippets/git/git-clean-untracked-safely) so you do not nuke a stash or unstaged work. Now audit the regenerated lockfile against the same grep from earlier. The versions should all be either pre-attack or post-recovery. If anything still resolves to a bad version, your overrides did not bite and you need to track down which transitive dep is pulling it in: ```bash npm ls echarts-for-react npm ls @antv/g2 ``` `npm ls` shows the dependency chain. Whatever package brought in the bad version needs to be updated or temporarily replaced. Once the lockfile is clean, run the install one more time without `--ignore-scripts` so any legitimate `postinstall` hooks (like patches or native compiles) run as expected: ```bash npm install npm run build npm test ``` If the test suite passes and the build is green, your application code is back. Now you can deal with the credentials. ## How do you rotate GitHub Actions secrets and npm tokens? Rotate everything the payload could have seen. The payload ran inside `npm install`, which means it had access to whatever the runner had access to. On a typical GitHub Actions runner, that includes the entire `GITHUB_TOKEN`, every repository secret, every organization secret pulled into the job, the npm token in `.npmrc` if present, and any cloud credentials injected as env vars. Walk this list in order: 1. **npm tokens.** Visit `npmjs.com` settings, **Access tokens**. Revoke every automation token used in the affected workflows. Re-create new ones, or better, migrate to [npm trusted publishing with OIDC](https://www.rabinarayanpatra.com/blogs/how-to-enable-npm-trusted-publishing-github-actions-oidc) so you do not need long-lived tokens at all. 2. **GitHub repository secrets.** In each affected repo, **Settings** then **Secrets and variables** then **Actions**. Rotate every secret used in workflows that ran on May 19 or that share runners with affected workflows. Treat shared organization secrets the same way. 3. **GitHub Personal Access Tokens.** Go to your account **Settings** then **Developer settings** then **Personal access tokens**. Revoke every classic PAT and fine-grained PAT older than May 19. Re-issue with minimum required scopes. 4. **Cloud credentials.** AWS access keys, GCP service-account keys, Azure client secrets used by the affected workflows must all be rotated. Use the cloud console to identify keys last used by the affected runner. 5. **SSH keys deployed to runners.** If your runners had SSH keys to deploy to staging or production, rotate them. The payload reads `~/.ssh/` directly. 6. **Vault tokens, Kubernetes configs, Datadog API keys, anything else you mounted into the job env.** All of these were enumerated and exfiltrated. Rotate every one. After rotation, audit the actual usage. AWS CloudTrail, GCP Audit Logs, and GitHub's audit log can show whether any of the rotated keys made an unusual API call between May 19 and the moment you revoked them. Anything unusual gets escalated to a full incident. ## How do you scan git history for leaked credentials? Run a fresh credential scan against the affected repository because the attacker may have pushed commits with exfiltrated data back into your repo using your own write tokens, or the rotation step might surface previously committed secrets that you never noticed. Both paths happen in real incidents. Two tools that work well for this: ```bash # trufflehog: live scan against the full git history docker run --rm -v "$PWD:/repo" trufflesecurity/trufflehog \ git file:///repo --only-verified # gitleaks: faster, config-driven, can run in CI docker run --rm -v "$PWD:/repo" zricethezav/gitleaks \ detect --source /repo --verbose --no-banner ``` `--only-verified` on TruffleHog means it actually tests each finding against the relevant API to confirm it is a live credential, not just a string that looks like one. That cuts noise hard. For each verified finding: 1. Revoke the credential at the source (provider console). 2. Remove the secret from history with `git filter-repo` (do this in a separate branch first, then force-push only after team sign-off, and never on a shared branch without explicit coordination). If a teammate accidentally deletes the branch holding their cleanup work, the [`git reflog` recovery pattern](https://www.rabinarayanpatra.com/snippets/git/git-reflog-recover-branch) gets it back. 3. Add the leaked pattern to your `gitleaks.toml` baseline so it does not flag on every CI run. If the leaked credential was a database password, rotate that too. The payload could have read environment variables on developer laptops that pulled the `.env` file from disk. ![Token rotation checklist for the AntV recovery](/images/blogs/recover-echarts-for-react-may-2026-antv-attack-rotation-checklist.webp) ## How do you harden CI so the next wave does not land? Three controls would have stopped the AntV payload cold, and adopting them now means the next wave does not become your next weekend of work. **Run `npm install` with `--ignore-scripts` everywhere in CI.** Lifecycle scripts (`preinstall`, `install`, `postinstall`) are the universal attack vector for npm worms. The Microsoft postmortem explicitly recommends this. Add `ignore-scripts=true` to your `.npmrc` for CI runners and let only known-trusted packages run their scripts via an allowlist. Tools like `@lavamoat/allow-scripts` make this practical. **Pin lockfile versions and verify with a fast registry alternative.** The faster your detection, the less time you spend in the bad window. Configure Renovate or Dependabot to alert on any dependency update, and use Socket's GitHub App or Snyk to flag malicious-package indicators before merge. The AntV payload was flagged by Socket within 6.7 minutes of publication. That is faster than most teams check Slack. **Switch publishers off long-lived npm tokens.** This is what closes the underlying class. The `atool` account fell because the credential was steal-able and reusable. Migrate publishing workflows to OIDC-based trusted publishing so even a fully compromised maintainer account cannot republish from an attacker's machine. The full setup is in my [npm trusted publishing guide](https://www.rabinarayanpatra.com/blogs/how-to-enable-npm-trusted-publishing-github-actions-oidc). For an extra layer, gate every publish workflow on a GitHub Environment with required reviewers. The OIDC token is only minted after a human approves the deployment. Combined with tag-only triggers (no publish on every main push), this means the attacker needs the source repo AND a human reviewer AND a tag push, not just a credential. ## What should you do this week? If you shipped anything that touches the npm ecosystem, do these in this order today: 1. Run the lockfile audit grep. Identify whether you are in the bad window. 2. If yes, treat affected runner credentials as burned and rotate them. 3. Regenerate the lockfile with `--ignore-scripts`, pin safe versions via overrides. 4. Run trufflehog or gitleaks against the full git history. 5. Set `ignore-scripts=true` in `.npmrc` for CI. 6. Schedule a follow-up to migrate publishing to OIDC trusted publishing. The Mini Shai-Hulud campaign is not over. The TanStack attack on May 11 and the AntV attack on May 19 were eight days apart, and the same actor or copycats will hit another popular maintainer account before the year ends. The defenses above stop the class of attack, not just this incident. For more on this, see [Microsoft's Mini Shai-Hulud writeup](https://www.microsoft.com/en-us/security/blog/2026/05/20/mini-shai-hulud-compromised-antv-npm-packages-enable-ci-cd-credential-theft/), [Socket's detection postmortem](https://socket.dev/blog/antv-packages-compromised), and [StepSecurity's incident analysis](https://www.stepsecurity.io/blog/shai-hulud-here-we-go-again-mass-npm-supply-chain-attack-hits-the-antv-ecosystem). ### Keep Reading - [How to Enable npm Trusted Publishing with GitHub Actions OIDC](https://www.rabinarayanpatra.com/blogs/how-to-enable-npm-trusted-publishing-github-actions-oidc). Closes the long-lived-token attack class that made this incident possible. - [TanStack npm Supply Chain Attack 2026: Detailed Breakdown](https://www.rabinarayanpatra.com/blogs/tanstack-supply-chain-attack-2026). The May 11 sibling attack that used valid SLSA provenance to ship malware. - [Axios npm Supply Chain Attack 2026: Recovery Playbook](https://www.rabinarayanpatra.com/blogs/axios-npm-supply-chain-attack-2026). Earlier incident in the same playbook. Useful prior art on token-theft recovery. --- ## How to Enable npm Trusted Publishing with GitHub Actions OIDC Source: https://www.rabinarayanpatra.com/blogs/how-to-enable-npm-trusted-publishing-github-actions-oidc Published: 2026-07-31T00:00:00+05:30 Tags: npm-trusted-publishing, github-actions, oidc, supply-chain-security, npm, ci-cd Stop shipping long-lived npm tokens. Step-by-step guide to enable npm trusted publishing with GitHub Actions OIDC, end to end, fixing a real attack class. On May 19, 2026, a compromised npm maintainer account named `atool` published 637 malicious versions across 317 packages in a single 22-minute burst. That number is not a typo. The hit list included `size-sensor` (4.2 million weekly downloads), `echarts-for-react` (3.8 million), and `@antv/scale` (2.2 million). GitHub had to invalidate 61,274 npm tokens with write permissions and 2FA bypass to stop the bleeding. I read the Microsoft Security postmortem the morning after and the root cause stuck with me. Every one of those packages had a long-lived `NPM_TOKEN` somewhere. Once the attacker had the token, they could republish from a laptop in a coffee shop and the registry would accept it. There is a fix for this and it has been generally available since 2025. It is called npm trusted publishing, and it replaces stealable tokens with short-lived OIDC tokens minted by your CI provider per workflow run. This post walks through the full setup with GitHub Actions, end to end, including the cleanup step most teams skip. ![How to Enable npm Trusted Publishing with GitHub Actions OIDC cover](/images/blogs/how-to-enable-npm-trusted-publishing-github-actions-oidc.webp) ## What is npm trusted publishing and why does it matter in 2026? npm trusted publishing is a credential-free publishing flow where your package authenticates to the npm registry using an OIDC token issued by a pre-approved CI provider, not a token you store in `secrets.NPM_TOKEN`. GitHub Actions, GitLab CI, CircleCI, and Buildkite are the supported providers today. The flow is roughly this. When your workflow runs, it asks GitHub's OIDC provider for a signed JWT. That JWT contains claims like `repository`, `workflow`, and `ref`. The npm CLI sends the JWT to the npm registry. The registry checks the claims against the trusted publisher you configured on `npmjs.com` for that package. If they match, the publish succeeds. There is no shared secret. No `NPM_TOKEN`. No long-lived bearer token sitting in your GitHub secrets waiting to be exfiltrated by a malicious `prepare` script. The OIDC token is minted on demand, scoped to one workflow run, and expires in minutes. In 2026 this stopped being optional. The Mini Shai-Hulud worm hit the [TanStack ecosystem on May 11](https://www.rabinarayanpatra.com/blogs/tanstack-supply-chain-attack-2026) and the @antv ecosystem on May 19, both via stolen credentials. The earlier [axios npm compromise](https://www.rabinarayanpatra.com/blogs/axios-npm-supply-chain-attack-2026) used the same playbook. Every one of these incidents had the same root primitive: a stealable, long-lived publish token. Trusted publishing also gives you build provenance for free. Public packages published this way get a SLSA Build Level 3 attestation showing exactly which repository, commit, and workflow produced the artifact. Consumers can verify it with `npm view --json` and inspect `dist.attestations`. ## Which versions of npm and Node.js support trusted publishing? You need npm CLI 11.5.1 or later and Node.js 22.14.0 or higher. Both are documented as hard floors on the official npm docs page. The simplest way to check what you have is to run this on a runner image you use: ```bash node --version npm --version ``` If you run an older version, the publish step fails with an authentication error. There is no graceful fallback to OIDC. Older npm does not understand the new flow at all. A few practical notes from setting this up across half a dozen packages: - The Ubuntu 24.04 runner image already ships Node 20 and npm 10 by default. You must use `actions/setup-node@v6` (not v4 or v5) to pin a version that supports trusted publishing. - Self-hosted runners are not currently supported. Only cloud-hosted GitHub-hosted runners can issue OIDC tokens that the npm registry accepts. - Each package can only have one trusted publisher configured at a time. If you maintain a monorepo that ships multiple packages, you configure trusted publishing on each package's settings page separately. The actions/setup-node v6 action handles the npm upgrade automatically when you pin `node-version: '24'`. That single line pulls in npm 11.x. ## How do you configure the trusted publisher on npmjs.com? Configuration happens once per package, in the npmjs.com web UI, before you ever touch your workflow file. The order matters. If you flip the workflow first and have not configured the publisher, the publish call fails and you scratch your head for an hour. Walk through this in a browser: 1. Log in to `npmjs.com`. Open the package settings page at `https://www.npmjs.com/package//access`. 2. Scroll to the **Trusted Publisher** section. 3. Click **GitHub Actions**. 4. Fill in five fields, in order: - **Organization or user**: your GitHub org or username (the part before the slash in `org/repo`). - **Repository**: just the repo name, without the org prefix. - **Workflow filename**: the filename only, with the `.yml` or `.yaml` extension. For example, `publish.yml`. Do not include the path. Do not include the `.github/workflows/` prefix. - **Environment name** (optional but recommended): the name of a GitHub Environment if you want to require deployment protection rules. I use `npm-publish` for this and gate it on a required reviewer. - **Allowed actions**: tick `npm publish`, `npm stage publish`, or both. For most packages, just `npm publish`. 5. Click **Save**. ![npm Trusted Publisher settings panel on npmjs.com](/images/blogs/how-to-enable-npm-trusted-publishing-github-actions-oidc-ui.webp) The trust is pinned to the exact `org/repo/workflow-filename` triple. If you rename the workflow file later, publishing breaks. If you fork the repo, the fork cannot publish. If someone tries to publish from a different repository, the registry rejects the OIDC token. This is the entire point. A stolen token from any machine cannot satisfy these claims, because the claims come from GitHub's OIDC provider and are signed. ## What does the GitHub Actions workflow look like? Here is a complete `.github/workflows/publish.yml` that publishes on a `v*` tag push. Drop this into your repo, adjust the test step, and commit. ```yaml name: Publish to npm on: push: tags: - 'v*' permissions: id-token: write contents: read jobs: publish: runs-on: ubuntu-latest environment: npm-publish steps: - uses: actions/checkout@v5 - uses: actions/setup-node@v6 with: node-version: '24' registry-url: 'https://registry.npmjs.org' package-manager-cache: false - run: npm ci - run: npm test - run: npm publish --access public ``` Three lines do the actual work and they all need to be exactly right. **`permissions: id-token: write`** is the line that unlocks OIDC. Without it, GitHub Actions will not issue an OIDC token to the runner, and `npm publish` falls back to looking for `NPM_TOKEN`, which is not there, and fails. This permission must be at the job level or workflow level. Job level is safer because it scopes the OIDC capability to only the publish job. **`registry-url: 'https://registry.npmjs.org'`** tells setup-node where to point the `.npmrc` it creates on the runner. This is what wires up `npm publish` to talk to the public registry. If you publish to a private registry instead, change this URL. **`package-manager-cache: false`** disables the setup-node built-in npm cache. This is a recommendation from the npm docs specifically for release builds. The cache can mask reproducibility issues by pulling stale tarballs, and on a release run you want a clean install every time. A few intentional choices in this file worth calling out: - I use `environment: npm-publish` to gate the job on a required reviewer. The OIDC token is only minted after the reviewer approves the deployment. This is what stops a force-pushed tag from publishing. - The trigger is a tag push, not a branch push. Tags are immutable. Branches are not. If you publish on every `main` push, an attacker who lands a malicious commit on main can ship to npm before anyone notices. With a tag trigger, you need to also push a tag, which requires a separate manual action. - There is no `NODE_AUTH_TOKEN`, no `secrets.NPM_TOKEN`, no `.npmrc` write step. setup-node v6 handles the OIDC handshake transparently when `id-token: write` is set. ![OIDC token flow from GitHub Actions to npm registry](/images/blogs/how-to-enable-npm-trusted-publishing-github-actions-oidc-oidc-flow.webp) Push a tag and watch the workflow run. The publish step should succeed and the npm registry page for your package should show a green provenance badge. ## How do you remove long-lived tokens after trusted publishing works? Trusted publishing only stops attacks if you delete the old tokens. This is the step most teams skip. They configure trusted publishing, see the green badge, ship a release, and leave the old `NPM_TOKEN` sitting in GitHub secrets and the automation token enabled on npmjs.com. Now they have two ways in and one of them is still stealable. Do this in the same browser session you used to configure trusted publishing: 1. On the npm package page, go to **Settings** then **Publishing access**. 2. Select **Require two-factor authentication and disallow tokens**. 3. Save. This setting means the registry will refuse any publish attempt that does not come from a trusted publisher or a maintainer with 2FA-verified web access. Even a leaked automation token cannot publish anymore. Then revoke the actual token: 1. Click your avatar then **Access tokens**. 2. Find any automation token that was used for publishing this package. 3. Click **Revoke** on each. Finally, clean up GitHub secrets: 1. In the repo, go to **Settings** then **Secrets and variables** then **Actions**. 2. Delete `NPM_TOKEN` from the repository secrets and any environment secrets where it lived. Run one more publish via the workflow to confirm nothing was relying on the deleted secret. If your workflow file still references `NODE_AUTH_TOKEN` or `secrets.NPM_TOKEN` anywhere, remove those references. The new flow does not need them. ## What are the limitations of npm trusted publishing today? Trusted publishing has real edges and you need to know them before you commit. The docs are honest about what is and is not supported. - **Self-hosted runners are not supported.** Only cloud-hosted GitHub-hosted runners can issue OIDC tokens the npm registry accepts. If you run a release pipeline on your own infrastructure, trusted publishing is not yet an option for that workflow. - **One trusted publisher per package.** You cannot have both GitHub Actions and GitLab CI as trusted publishers for the same package. Pick one. Multi-CI publishing remains a token-based flow. - **Provenance generation requires a public repo and a public package.** If you publish a private package, the publish itself works fine over OIDC, but the SLSA Build Level 3 attestation is not generated. This is a GitHub-side restriction on which workflows can sign provenance. - **CircleCI does not get auto-provenance.** It is a supported trusted publisher but provenance generation is not currently supported there. - **Other npm commands still need tokens.** `npm install`, `npm view`, and `npm access` calls against a private registry still need a token. Trusted publishing only covers the publish path. The biggest practical edge is the self-hosted runner constraint. I have one project on a self-hosted runner for compliance reasons and we run the publish job in a separate workflow on a cloud-hosted runner specifically to get trusted publishing. The build still happens on the self-hosted runner. Only the final publish step runs on a GitHub-hosted runner. ## How does trusted publishing change the attack surface? Trusted publishing closes one attack class cleanly and leaves another wide open. Know which is which before you ship. What it stops: any attacker who steals an `NPM_TOKEN` from a developer machine, a leaked `.npmrc`, a logging system, a stack trace, or a compromised CI environment cannot publish anymore. The token they stole is not what the registry checks. The registry checks an OIDC claim that only a real GitHub Actions runner inside the configured repo can produce. What it does not stop: an attacker who lands a commit on your repository. The malicious commit can ride through your trusted publishing workflow and ship to npm with a valid SLSA Build Level 3 attestation. This is exactly what the TanStack worm did on May 11, 2026. The attestation was real. The commit it attested to was malicious. Provenance proves where a package was built, not whether the source was trustworthy. ![Attack surface comparison: long-lived tokens versus OIDC short-lived tokens](/images/blogs/how-to-enable-npm-trusted-publishing-github-actions-oidc-attack-surface.webp) To close the second class, pair trusted publishing with these: - **Branch protection** on `main` (and any tag pattern you publish from) requiring pull request reviews from a different account than the author. - **Required reviewers** on the GitHub Environment that gates the publish job. This is why I use `environment: npm-publish` in the workflow above. - **Tag-trigger only.** Do not publish on every main push. Require a separate tag push that a human has to perform. - **Dependency review on PRs.** Catch malicious `package.json` `prepare` hooks or new transitive deps before they hit the publish job. GitHub's dependency-review-action is one option. Trusted publishing is the floor, not the ceiling. It removes the easy attack. The hard attack still works and you defend against it with code review, environment gates, and tag-based release discipline. ## What should you do this week? I do not think trusted publishing is the last credential change npm will ship. Provenance is still optional, the self-hosted runner gap is real, and the cross-CI story is messy. But the credential-free publish flow is the most useful security upgrade npm has shipped in a decade, and the cost to adopt is one configuration screen plus a small workflow change. If you publish anything to npm, set this up this week. Then delete the old token. Then do not publish a package without it ever again. For more on this, see the [official npm trusted publishing docs](https://docs.npmjs.com/trusted-publishers/), the [Microsoft Security postmortem on the AntV compromise](https://www.microsoft.com/en-us/security/blog/2026/05/20/mini-shai-hulud-compromised-antv-npm-packages-enable-ci-cd-credential-theft/), and the [StepSecurity analysis of the AntV wave](https://www.stepsecurity.io/blog/shai-hulud-here-we-go-again-mass-npm-supply-chain-attack-hits-the-antv-ecosystem). ### Keep Reading - [TanStack npm Supply Chain Attack: How the May 11 Compromise Worked](https://www.rabinarayanpatra.com/blogs/tanstack-supply-chain-attack-2026). The first npm worm to ship with valid SLSA provenance. Explains why OIDC tokens alone are not enough. - [Axios npm Supply Chain Attack 2026: What Happened and How to Defend](https://www.rabinarayanpatra.com/blogs/axios-npm-supply-chain-attack-2026). Earlier incident in the same playbook. Useful prior art on token-theft attacks. - [AI-Driven Anomaly Detection for Supply Chain Security](https://www.rabinarayanpatra.com/blogs/ai-driven-anomaly-detection-security). Detection complement to the prevention story in this post. --- ## How to Enable Kubernetes 1.36 User Namespaces for Isolation Source: https://www.rabinarayanpatra.com/blogs/enable-kubernetes-1-36-user-namespaces-isolation Published: 2026-07-29T00:00:00+05:30 Tags: kubernetes-1-36-user-namespaces, user-namespaces, container-isolation, kubernetes-security, containerd, kep-127 Step-by-step Kubernetes 1.36 user namespaces tutorial. Kernel and containerd prereqs, hostUsers field, UID mapping verification, and migration. User namespaces in Kubernetes hit GA on April 22, 2026, with the v1.36 Haru release. The feature has been around as alpha since v1.25 and beta since v1.30, but until now the cost of opting in was high enough that nobody outside of GKE Sandbox and a few security-forward teams used it. As of v1.36 the gate is enabled by default, the kernel and runtime requirements are met by every up-to-date distribution, and the pod-spec change is a single field. I turned this on across my own staging cluster the week the release dropped. This post walks the enablement the way I ran it: the prerequisite check that decides whether your nodes can opt in at all, the smallest possible pod manifest that exercises the feature, the verification step that proves the UID mapping is doing what you expect, and the migration plan for existing workloads that break under the change. ## What changed with user namespaces in Kubernetes 1.36? User Namespaces graduated to General Availability in Kubernetes 1.36 (the Haru release, April 22, 2026), with the matching feature gate enabled by default. The pod-spec field is `hostUsers: false`. When set, the kubelet asks the container runtime to create the pod in a new user namespace, and the runtime maps the container's UID 0 to a non-privileged UID on the host. The KEP behind this is [KEP-127](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/127-user-namespaces/README.md). The headline behaviors that flipped from beta to stable in v1.36 are: | Behavior | Before v1.36 | v1.36 (GA) | | --- | --- | --- | | Feature gate | `UserNamespacesSupport` (beta) | Enabled by default, no gate | | `hostUsers` field | Beta in pod spec | Stable in pod spec | | Capabilities (e.g. `CAP_NET_ADMIN`) | Host-scoped, full power | Namespaced, container-local only | | Volume UID/GID translation | Manual chown often required | Kernel handles transparently via idmap mounts | | Sandbox feature requirement | Each guarded by sub-gates | Folded under the main gate | | Pod-level resource resize | Separate alpha gate | Pod-level in-place resize lands stable in same release | The two changes that matter most in practice are the namespaced capabilities and the transparent UID translation on mounted volumes. A pod with `CAP_NET_ADMIN` can still tweak its own network namespace, but cannot touch the host's. A pod that mounts a host volume sees files as owned by UID 0 inside the container while the host disk still records the actual UID. No `chown` step, no init container shuffling permissions, no setgid sticky-bit cleanup. ## Why do user namespaces matter for container escape attacks? User namespaces matter because they sever the link between "root in the container" and "root on the host". A process that achieves a container escape on a pod with `hostUsers: false` lands as an unprivileged UID on the node, which means the standard escape primitives (writing to `/proc/sys`, mounting host paths, opening privileged sockets, loading kernel modules) all fail at the kernel boundary. The history here is long and bloody. Every six months a new CVE in `runc`, `containerd`, or a kernel subsystem hands an attacker root on the host because they were already root in the container and the host trusted that UID. The 2019 `runc` symlink escape (CVE-2019-5736) was patched in a week. The 2022 `cgroups v1` race (CVE-2022-0492) took longer. Then there was the 2024 `runc` LeakyVessels chain. The pattern is the same every time: container UID 0 reaches a kernel surface, and the kernel applies host UID 0 semantics. User namespaces close that pattern at the source. The container still has UID 0 inside its own namespace. The kernel still applies UID 0 semantics. But the inside-the-namespace UID 0 is mapped to a non-privileged outside-the-namespace UID, so the kernel's UID-0-only operations fail when the escape lands on the host. CAP_SYS_ADMIN on the inside means namespace-local admin. CAP_SYS_ADMIN on the host? The escaped process does not have it. The blast-radius difference shows up clearly when you run the same exploit twice. I did this on a non-production cluster with a deliberately vulnerable image based on the Stripe Open Source Defense write-up. Without user namespaces the escape gave shell access to the node. With `hostUsers: false` the escape produced a shell as an unmapped UID with no write access to `/etc`, no ability to mount, and no ability to load kernel modules. This is defense in depth, not a silver bullet. The pod can still be malicious, the container can still be DoS'd, and a kernel bug that bypasses namespaces entirely (which is rare but does happen) gets through anyway. Pod Security Admission still applies. runAsNonRoot still applies. Seccomp still applies. User namespaces are one more layer. ## How do you check kernel and runtime prerequisites? You check the prerequisites by reading three numbers from each node: the kernel version, the container runtime version, and the storage driver. The minimum supported combination is Linux 6.3, containerd 2.0 (or CRI-O 1.30), and a filesystem that supports idmap mounts. Older kernels can work back to 5.19 with a containerd 2.0 plus overlay-with-idmap config, but 6.3 is what the official `kubernetes.io` docs settle on for the GA path. Run this on each node as a quick sanity check. ```bash # Kernel version uname -r # Expected: 6.3 or newer for the supported GA path # Container runtime crictl version # Expected: containerd 2.0.x or newer, or CRI-O 1.30.x or newer # Storage driver crictl info | jq '.config.containerd.runtimes' 2>/dev/null # Or for CRI-O: crio config 2>/dev/null | grep storage_driver # Confirm the kernel exposes the namespace feature ls /proc/self/ns/ | grep user # Expected: user ``` Two compatibility traps to watch for: 1. **containerd 1.7 + Linux 5.15.** The setting is honored, but you pay a measurable storage and latency penalty because overlayfs does not support idmap mounts on that kernel. Upgrade the kernel before you turn this on broadly. 2. **containerd 1.6 or older.** The `hostUsers: false` field is silently ignored. No error, no warning, no namespace. This is the worst failure mode because pods appear to be in a user namespace until you actually verify the UID mapping. For managed Kubernetes the answer is usually simple: GKE, EKS, and AKS all ship node pools that meet the requirement on their current LTS images as of April 2026. Use the latest node image on each, and the feature is available without extra work. For self-managed clusters on Ubuntu 22.04, the default kernel is 5.15, which means an HWE (hardware enablement) kernel upgrade is the smallest path. Ubuntu 24.04 ships 6.8 by default and is the easier choice. RHEL 9.4 ships 5.14, which means you need the 9.5 update for a 6.x kernel. If even one node fails the prereqs, do not enable user namespaces on the whole cluster. Cordon and drain the non-compliant nodes first, or label them and use a node selector on the user-namespaced pods. Mixed clusters work, mixed-on-the-same-node configs do not. ## How do you enable user namespaces on a pod? You enable user namespaces by adding a single field to the pod spec: `hostUsers: false`. Everything else is automatic. The kubelet asks the container runtime to create the pod in a new user namespace, the runtime mints a UID range, and the kernel performs transparent UID translation on every namespaced operation including volume mounts. The smallest demonstrable pod is this one. ```yaml # userns-demo.yaml apiVersion: v1 kind: Pod metadata: name: userns-demo spec: hostUsers: false containers: - name: shell image: debian:bookworm-slim command: ['sleep', 'infinity'] securityContext: runAsUser: 0 runAsGroup: 0 ``` Apply it. ```bash kubectl apply -f userns-demo.yaml kubectl wait --for=condition=Ready pod/userns-demo --timeout=60s ``` Now check what the container thinks its UID is, and what the node thinks the same process's UID is. ```bash # What the container sees kubectl exec userns-demo -- id # Expected: uid=0(root) gid=0(root) groups=0(root) # What the node sees. Run on the node that hosts the pod. PID=$(kubectl get pod userns-demo -o jsonpath='{.status.containerStatuses[0].containerID}' \ | sed 's|.*://||' \ | xargs -I{} crictl inspect {} \ | jq -r '.info.pid') # On the node: cat /proc/$PID/status | grep -E "^Uid:" # Expected: a UID far from 0, e.g. Uid: 100000 100000 100000 100000 ``` That second number is the proof that the namespace did its job. Container thinks it is UID 0. Host kernel thinks it is UID 100000 (or whatever range the runtime picked). Every kernel operation that requires real-UID-0 privileges fails on the host side. For workloads that need more than the default 65536-UID range (large multi-tenant setups), the kubelet honors the `--userns-uid-mapping` config flag. I have never needed this. The default range is enough for almost every workload. To opt out per pod, just omit the field. The default is `hostUsers: true`, which is the v1.35 behavior. To opt in for an entire namespace, use [Kyverno](https://kyverno.io) or any other admission controller to inject the field. The smallest Kyverno policy is six lines and avoids the per-deployment churn. ```yaml # kyverno-userns.yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: enforce-user-namespaces spec: rules: - name: set-hostusers-false match: any: - resources: kinds: ['Pod'] namespaces: ['team-a', 'team-b'] mutate: patchStrategicMerge: spec: hostUsers: false ``` ## How do you verify the UID mapping works correctly? You verify the UID mapping by writing a file from inside the container and reading the on-disk ownership from the node. If the in-container ownership shows UID 0 but the on-disk ownership shows a non-zero UID in the runtime's mapping range, the namespace is doing what it claims. Set this up with an `emptyDir` volume. ```yaml # userns-verify.yaml apiVersion: v1 kind: Pod metadata: name: userns-verify spec: hostUsers: false containers: - name: writer image: debian:bookworm-slim command: ['sleep', 'infinity'] volumeMounts: - name: scratch mountPath: /scratch volumes: - name: scratch emptyDir: {} ``` Write a file from inside the container. ```bash kubectl exec userns-verify -- sh -c 'echo hi > /scratch/test && ls -la /scratch/test' # Expected: -rw-r--r-- 1 root root 3 ... /scratch/test ``` Now find the file on the node and check the on-disk ownership. ```bash # Get the host path of the emptyDir PODUID=$(kubectl get pod userns-verify -o jsonpath='{.metadata.uid}') NODE=$(kubectl get pod userns-verify -o jsonpath='{.spec.nodeName}') # On the node: sudo find /var/lib/kubelet/pods/$PODUID -name test -exec ls -la {} \; # Expected: -rw-r--r-- 1 100000 100000 ... /var/lib/kubelet/pods//volumes/.../test ``` That UID 100000 is the host-side view of the container's UID 0. If you see UID 0 on the on-disk side instead, the namespace did not take. Common causes: containerd is 1.6.x, the kernel does not support idmap mounts, or the filesystem of the kubelet directory does not support idmap. All three appear in the kubelet logs when they fail. For a stronger test, try a privileged operation that should fail. ```bash kubectl exec userns-verify -- sh -c 'mount -t tmpfs none /mnt 2>&1' # Expected: mount: /mnt: permission denied ``` Without user namespaces, root in a container can mount tmpfs inside the container's mount namespace. With user namespaces, the mount syscall lands as a non-privileged UID on the host kernel and fails. That permission-denied is the proof that namespaced capabilities are working. ## How do you migrate existing workloads to user namespaces? You migrate existing workloads by classifying them against three failure modes (low ports, shared-UID volumes, host paths), enabling `hostUsers: false` on the safe ones first, and rewriting the broken ones one at a time. Do not flip the whole namespace at once. The three failure modes show up like this. | Failure mode | Symptom | Fix | | --- | --- | --- | | Bind to port < 1024 | `bind: permission denied` on container start | Add `capabilities: { add: [NET_BIND_SERVICE] }` to securityContext, or move to a port โ‰ฅ 1024 and front with a Service | | Shared-UID host volume | Files appear owned by an unexpected UID inside the container, or chown errors | Use a `Pod` spec that owns the volume (recreate as `emptyDir`) or drop the host volume | | `hostPath` volume | Mount succeeds but file ops fail with EACCES | Re-platform off `hostPath` to a CSI driver or `local` PV, or grant explicit access via supplementalGroups | For the low-port case, the fix is a one-line addition. ```yaml spec: hostUsers: false containers: - name: web image: my-nginx:latest ports: - containerPort: 80 securityContext: capabilities: add: [NET_BIND_SERVICE] ``` The capability is namespaced under user namespaces, so it grants the bind privilege inside the container without granting any host-side capability. For the shared-volume case, the right answer almost always is "do not share a host volume". If you absolutely need to share with another pod, share via a NetworkVolume (PVC) or a sidecar pattern. The on-disk UID mismatch that user namespaces introduce is the kernel doing its job; do not chown your way around it. The migration sequence that worked on my own staging cluster was: 1. Pick a low-stakes deployment with no host volumes and no low ports. 2. Patch the deployment to add `hostUsers: false`. 3. Wait one full traffic cycle. Watch pod restart count and the application's own error logs. 4. If clean, move the next deployment. 5. After half a dozen deployments without issue, layer a Kyverno policy that defaults the namespace to `hostUsers: false` and start auditing the holdouts. That is the path the [Kubernetes docs recommend](https://kubernetes.io/docs/tasks/configure-pod-container/user-namespaces/) and it is the path I would follow in production. The dangerous version is the all-at-once flip on a busy namespace, which surfaces the failure modes at the worst time. ## What common problems should you watch for? The common problems are containerd-1.6 silently ignoring the field, kernel versions that look new enough but lack idmap mounts on the kubelet's filesystem, and workloads with hidden CAP_SYS_ADMIN needs that fail in surprising ways. For each, here is the signal and the fix. ```bash # Signal 1: containerd ignores hostUsers: false silently kubectl exec -- cat /proc/self/uid_map # A namespaced pod prints something like: # 0 100000 65536 # A non-namespaced pod prints: # 0 0 4294967295 # If you set hostUsers: false and see the second line, the runtime ignored it. # Fix: upgrade containerd to >= 2.0, or move to a node where it is. # Signal 2: kubelet logs the idmap failure journalctl -u kubelet -n 200 | grep -i idmap # Common error: "failed to set up user namespace mount: idmap not supported" # Fix: upgrade to a kernel >= 6.3 (5.19 is the absolute minimum), or move the # kubelet root directory to a filesystem that supports idmap mounts. # Signal 3: workload fails with CAP_SYS_ADMIN-like errors kubectl logs | grep -E "operation not permitted|permission denied" # This often means the workload needs a capability that is no longer host-scoped. # Fix: add the specific capability to securityContext.capabilities.add. If the # workload needs real host-side CAP_SYS_ADMIN, it is not a candidate for user # namespaces. Use a privileged sidecar pattern instead. ``` The other quiet problem is metrics. Pods that report container PIDs to a host-side telemetry collector (for example a host-mode Prometheus exporter for cgroup stats) need to translate the PID across the namespace. The standard exporters handle this fine. Home-grown collectors usually do not. Test your observability stack on at least one user-namespaced pod before you flip a whole namespace. For deeper context on the security primitives that compose with user namespaces, see the [zero trust microservices guide](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security) for the auth-side counterpart, the [AI-driven anomaly detection guide](https://www.rabinarayanpatra.com/blogs/ai-driven-anomaly-detection-security) for runtime detection that pairs with kernel isolation, and the [TanStack supply chain attack postmortem](https://www.rabinarayanpatra.com/blogs/tanstack-supply-chain-attack-2026) for the kind of supply-chain blast radius this feature reduces. For the original sources, see the [Kubernetes 1.36 user namespaces GA announcement](https://kubernetes.io/blog/2026/04/23/kubernetes-v1-36-userns-ga/), the [user namespaces concept documentation](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/), the [task documentation for enabling user namespaces on a pod](https://kubernetes.io/docs/tasks/configure-pod-container/user-namespaces/), and [KEP-127](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/127-user-namespaces/README.md). ### Keep Reading - [Zero Trust Microservices with Spring Security](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security). The auth-side counterpart to kernel-level isolation. Both layers together close the practical attack paths into a pod. - [How to Migrate from Ingress NGINX After Kubernetes Retired It](https://www.rabinarayanpatra.com/blogs/migrate-from-ingress-nginx-kubernetes-retired). The other K8s 1.36 era change you cannot defer. Pair the migration windows. - [AI-Driven Anomaly Detection for Security](https://www.rabinarayanpatra.com/blogs/ai-driven-anomaly-detection-security). Runtime detection layered on top of the kernel isolation user namespaces provide. - [TanStack npm Supply Chain Attack 2026](https://www.rabinarayanpatra.com/blogs/tanstack-supply-chain-attack-2026). The kind of supply-chain compromise user namespaces reduce the blast radius of when the malicious code lands inside a container. --- ## Claude Opus 5: What Breaks When You Migrate from Opus 4.8 Source: https://www.rabinarayanpatra.com/blogs/claude-opus-5-migration-breaking-changes Published: 2026-07-27T00:00:00+05:30 Tags: claude-opus-5, anthropic, llm-api, ai-for-developers, migration-guide Claude Opus 5 turns thinking on by default and 400s if you disable it above high effort. Full migration checklist from Opus 4.8, with pricing and effort tuning. Anthropic shipped Claude Opus 5 on 24 July, and the migration note is unusually short: change the model string. Pricing is flat, the context window is unchanged, and existing prompts carry over. Then you read the two behavior changes and realize one of them fails silently. I went through this on my own projects over the weekend. The model ID swap took thirty seconds. Working out why one route started returning truncated answers took considerably longer, and the answer turned out to be documented in a single sentence I had skimmed past. So here is the migration in the order it actually bites you. ![Claude Opus 5 cover card showing the model ID swap from claude-opus-4-8 to claude-opus-5, alongside pricing of $5 and $25 per million tokens, a 1M token context window, and 128K max output](/images/blogs/claude-opus-5-migration-breaking-changes.svg) ## What is Claude Opus 5 and what does it cost? Claude Opus 5 is Anthropic's current Opus-tier model, released 24 July 2026 with the API model ID `claude-opus-5`. It costs $5 per million input tokens and $25 per million output tokens, identical to Claude Opus 4.8. There is no long-context premium and no date suffix on the ID. The specs that matter: | Property | Value | |---|---| | Model ID | `claude-opus-5` | | Context window | 1M tokens (default **and** maximum) | | Max output | 128K tokens | | Input / output price | $5 / $25 per million tokens | | Thinking | On by default | | Effort levels | `low`, `medium`, `high`, `xhigh`, `max` | | Prompt cache minimum | 512 tokens | That "default and maximum" detail on the context window is worth reading twice. There is no smaller context variant to opt into, which removes a config knob some teams were using to control cost. On benchmarks, Anthropic reports Opus 5 landing within 0.5% of Claude Fable 5 on CursorBench 3.2 at half the price, roughly three times the next-best score on ARC-AGI 3, and about 1.5 times the next-best pass rate on Zapier's AutomationBench. Treat vendor benchmarks as vendor benchmarks. The claim I find more useful is the shape of the gains: deep reasoning, long agentic loops, and test-time compute scaling, meaning the model converts extra effort into better answers more reliably than earlier Opus versions did. It is available on the Claude API as `claude-opus-5`, on Amazon Bedrock as `anthropic.claude-opus-5`, and on Google Cloud and Microsoft Foundry under the bare ID. Opus 4.8 stays available everywhere, so there is no forced cutover. One operational detail that is easy to miss: Claude Opus 5 draws on a **separate rate-limit bucket** from the combined Opus 4.x pool. Moving traffic over does not free headroom on the old bucket and does not inherit it. Check your tier's Opus 5 limits before you shift real volume. ## Why does thinking being on by default break your max_tokens? Because `max_tokens` caps thinking plus reply text together, and on Opus 4.8 a request that omitted the `thinking` field produced no thinking at all. The same request on Opus 5 thinks. Your reply now shares a budget it used to own outright. This is the change that cost me an afternoon. Nothing throws. The request returns HTTP 200, `stop_reason` comes back as `"max_tokens"`, and the answer is cut mid-sentence. If you are logging only errors, you will not see it. ![Diagram comparing max_tokens usage: on Opus 4.8 the whole budget holds reply text, on Opus 5 thinking takes a share and pushes the reply past the limit, producing stop_reason max_tokens](/images/blogs/claude-opus-5-migration-breaking-changes-max-tokens.svg) The wire format did not change. `thinking: {"type": "adaptive"}` is still valid and is exactly equivalent to the new default. What changed is what happens when you say nothing: ```python # Opus 4.8: no thinking field means no thinking. # 4096 tokens was all reply text. client.messages.create( model="claude-opus-4-8", max_tokens=4096, messages=[{"role": "user", "content": prompt}], ) # Opus 5: identical call, but thinking now runs and eats # into the same 4096. Long answers get truncated. client.messages.create( model="claude-opus-5", max_tokens=4096, messages=[{"role": "user", "content": prompt}], ) ``` You have two honest fixes. Raise `max_tokens` to leave room for both, which is what I did on every route where the answer length varies. Or keep the old behavior explicitly: ```python client.messages.create( model="claude-opus-5", max_tokens=4096, thinking={"type": "disabled"}, output_config={"effort": "medium"}, messages=[{"role": "user", "content": prompt}], ) ``` That second option comes with its own trap, which is the next section. Audit target: every call site that never set a `thinking` field. Those are the ones that silently changed behavior. Routes that already passed `{"type": "adaptive"}` are unaffected. ## Why does disabling thinking now return a 400 error? Because on Claude Opus 5, `thinking: {"type": "disabled"}` is only accepted when effort is `high` or below. Pair it with `xhigh` or `max` and you get a 400. On Opus 4.8 the two settings were independent, so this is a genuine breaking change rather than a default shift. ![Matrix showing thinking adaptive is valid at every effort level, while thinking disabled is valid only at low, medium and high, returning 400 at xhigh and max](/images/blogs/claude-opus-5-migration-breaking-changes-effort-matrix.svg) The part that catches people is that **validation runs per request**. Effort and thinking are checked independently on every call. A conversation can run happily for twenty turns at `high` with thinking off, then fail the moment your code bumps effort to `xhigh` for a harder step. Earlier success in the same session buys you nothing. ```python # 400 invalid_request_error on Claude Opus 5 client.messages.create( model="claude-opus-5", max_tokens=4096, thinking={"type": "disabled"}, output_config={"effort": "xhigh"}, # <- rejected messages=[{"role": "user", "content": prompt}], ) ``` Pick one of two resolutions. Keep thinking off and cap effort at `high`, which costs you the top two tiers. Or delete the `thinking` field and keep the effort tier, which costs tokens and buys back quality. My advice is the second one, and not just for compliance. Anthropic documents two real failure modes with thinking disabled on Opus 5: the model occasionally writes a tool call into its visible text instead of emitting a `tool_use` block, and it can leak internal XML tags into the response. The first is nastier than it sounds. The turn completes normally, no error is raised, and the tool never runs. In an agentic loop that phantom call then sits in the history and skews later turns. If a route genuinely must keep thinking off, the documented mitigations are counterintuitive enough to be worth stating plainly: - Give the model permission to talk first: *"You may say a brief sentence before using a tool."* The tool-call-as-text failure appears to come from suppressing the preamble it wants to write. - **Delete** any instruction telling it not to think or not to reason. That kind of rule makes tag leakage worse, not better. - Write the tag instruction generically. *"Do not include internal or system XML tags in your response"* works better than naming thinking tags explicitly. For most teams, running at `low` or `medium` effort with thinking on is cheaper and better behaved than disabling thinking at `high`. ## How should you pick an effort level on Claude Opus 5? Start at `high`, which is the default, then sweep in both directions against your own evals. Effort carries more weight on Opus 5 than on any earlier Opus, because the model converts extra effort into better output more reliably. Anthropic's published starting points and the model's measured behavior pull in slightly different directions, and both are useful: - **Start `xhigh` for coding and agentic work**, `high` for other intelligence-sensitive workloads. `max` is the top tier for the deepest reasoning, though it can overthink simple tasks. - **Then sweep downward**, because `low` and `medium` are unusually strong here. They deliver good quality at a fraction of the tokens and latency on plenty of workloads. Do run the sweep. Effort defaults inherited from Opus 4.8 or 4.7 are rarely the right setting on this model, and I would not trust a number carried over from a previous migration. If you kept effort notes from the [Opus 4.7 release](https://www.rabinarayanpatra.com/blogs/claude-opus-4-7-release-and-migration-guide), re-run them rather than reusing them. At `xhigh` or `max`, set a large `max_tokens` so the model has room to think and act across tool calls and subagents. Start at 64000 and tune down. And stream anything that large, or you will hit SDK HTTP timeouts before the model finishes: ```python with client.messages.stream( model="claude-opus-5", max_tokens=64000, output_config={"effort": "max"}, messages=[{"role": "user", "content": prompt}], ) as stream: response = stream.get_final_message() ``` ## What new API features ship with Claude Opus 5? Three, and two of them are behind beta headers. None are required to migrate. **Mid-conversation tool changes** (beta `mid-conversation-tool-changes-2026-07-01`) let you add or remove tools between turns without invalidating the prompt cache. Previously the tool list was fixed for a conversation's lifetime and any edit re-billed the entire prefix, because tools render at position zero. You declare the tool up front with `"defer_loading": true`, then surface it later with a `tool_addition` block on a system message. This is the control-plane counterpart to tool search: tool search is for discovery, this is for when your application knows the tool set changed. **Default fallbacks mode** (beta `server-side-fallback-2026-07-01`) is the one I would turn on for everyone. Opus 5 ships with stricter cybersecurity safeguards, and its classifiers can decline a request. A decline is an HTTP 200 with `stop_reason: "refusal"`, not an error, so code that reads `response.content[0]` unconditionally breaks on it. The `fallbacks` parameter re-runs a declined request on another model server-side: ```python response = client.beta.messages.create( model="claude-opus-5", max_tokens=16000, betas=["server-side-fallback-2026-07-01"], fallbacks="default", messages=[{"role": "user", "content": prompt}], ) if response.stop_reason == "refusal": handle_refusal(response.stop_details) ``` Prefer `"default"` over pinning a model. It routes by refusal category, and it saves you a migration the next time a pinned fallback gets deprecated. Cyber-category refusals route to Opus 4.8. Note this is Claude API only, so on Bedrock, Google Cloud, or Foundry you need the SDK's client-side fallback middleware instead. **A lower prompt cache minimum**, at 512 tokens, down from 1024 on Opus 4.8. No code change, but it is worth re-checking prompts you previously wrote off as too short to cache. Anything between 512 and 1024 tokens now creates cache entries for free. ## Which prompt instructions should you delete after migrating? The verification ones. Claude Opus 5 verifies its own work without being asked, so instructions like *"include a final verification step"* or *"use a subagent to verify"* now cause over-verification rather than preventing errors. This inverts a standard prompting rule. "Ask the model to double-check itself" is generally sound advice and it is wrong here. If you keep a shared prompt library, that needs a carve-out for this model rather than a global rule. The same applies to framework-level scaffolding. Separate verification passes carried over from earlier models are likely redundant now. Three other behavior shifts are worth tuning for, all documented by Anthropic: **Responses run longer.** Both conversational output and files the model writes to disk. Lowering `effort` does not reliably shorten visible output, so this is a prompting fix, not a config one. A short conciseness instruction is the lever. **It narrates progress more in agentic sessions.** If you added scaffolding to force interim status updates ("after every three tool calls, summarize"), remove it. Opus 5 does this on its own, and the combination is noisy. **It delegates to subagents more readily.** This is a direction change worth flagging, because Opus 4.8 under-reached for subagents and needed prompting to delegate at all. If you added "delegate more" guidance for 4.8, take it out, and consider an explicit cap. Each subagent re-establishes context, re-explores, reports back, and then the coordinator re-reads the report. That multiplies both cost and latency. It can also expand task scope, adding steps you did not request. A short scope-discipline instruction handles it: deliver what was asked at the scope intended, flag a concern in a sentence if the ask looks wrong, and keep going rather than quietly widening the work. ## Should you migrate to Claude Opus 5 today? If you are on Opus 4.8, yes, and the cost case makes it easy: identical pricing, better output. The two breaking changes are both mechanical and both findable with a grep. Here is the checklist I actually used: 1. Swap the model ID to `claude-opus-5`. 2. Grep for call sites with **no** `thinking` field. Raise `max_tokens` on each, or set `{"type": "disabled"}` deliberately. 3. Grep for `"disabled"` paired with `xhigh` or `max` effort. Fix both sides of the pair. 4. Re-run your effort sweep from scratch. Do not reuse 4.8 numbers. 5. Add a `stop_reason == "refusal"` branch before reading `content`, and turn on `fallbacks: "default"`. 6. Delete verification instructions and forced progress-update scaffolding from your prompts. 7. Confirm your rate-limit headroom on the new bucket before shifting volume. Steps 1 to 3 are required. The rest is the difference between a migration that works and one that works well. The thing I keep coming back to is step 6. We spent two years writing prompts that compensated for models which did not check their own work, did not narrate progress, and would not delegate. Those instructions are now actively counterproductive. Migrating to a better model is turning out to be less about adding capability and more about deleting the scaffolding you built for the last one, which is a strange and slightly humbling way to upgrade. For the full specification, see Anthropic's [What's new in Claude Opus 5](https://platform.claude.com/docs/en/about-claude/models/whats-new-opus-5), the [Claude Opus 5 announcement](https://www.anthropic.com/news/claude-opus-5), and the [model migration guide](https://platform.claude.com/docs/en/about-claude/models/migration-guide). ### Keep Reading - [Claude Opus 4.7: Pricing, Benchmarks & Breaking Changes](https://www.rabinarayanpatra.com/blogs/claude-opus-4-7-release-and-migration-guide). The previous Opus migration, where adaptive thinking and the removal of sampling parameters first landed. - [Claude Mythos and Project Glasswing: Anthropic's Locked-Down AI Model](https://www.rabinarayanpatra.com/blogs/claude-mythos-project-glasswing). Where the Fable and Mythos tier sits relative to Opus, and why Opus 5 at half the price matters. - [10 Best Claude Skills for Developers in 2026](https://www.rabinarayanpatra.com/blogs/best-claude-skills-for-developers-2026). Skills that pair well with the longer agentic runs Opus 5 is built for. --- ## How to Migrate from Ingress NGINX After Kubernetes Retired It Source: https://www.rabinarayanpatra.com/blogs/migrate-from-ingress-nginx-kubernetes-retired Published: 2026-07-24T00:00:00+05:30 Tags: ingress-nginx-retirement-migration, ingress-nginx, gateway-api, traefik, cilium, kubernetes Step-by-step Ingress NGINX migration after the March 2026 retirement. Traefik drop-in, Gateway API with Cilium, and zero-downtime cutover playbook. On March 24, 2026, Kubernetes SIG Network and the Security Response Committee retired Ingress NGINX. The GitHub repositories went read-only the same day. No more releases. No more bug fixes. No more security patches. Roughly half the cloud-native estate depends on this controller. The retirement does not break anything immediately, but every new CVE in the NGINX core, Lua, OpenResty, or the controller code itself stays unpatched. My own staging cluster ran Ingress NGINX behind every service. I migrated it the weekend the announcement landed, and this post is the playbook I used: which replacement to pick, the Traefik drop-in path for the panicked migration, the Gateway API with Cilium path for the proper migration, and the zero-downtime cutover sequence. ## What happened to Ingress NGINX? Ingress NGINX hit end of life on March 24, 2026, when Kubernetes SIG Network and the Security Response Committee formally retired the `kubernetes/ingress-nginx` project. The retirement was telegraphed in [the November 2025 announcement on kubernetes.io](https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/), confirmed in [the January 2026 SRC statement](https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/), and made final at the March 2026 retirement date. What changed on that date: - The GitHub repos became read-only. - The controller no longer receives bug fixes, feature changes, or CVE patches. - Helm charts and container images that already exist stay published, but new ones will not be cut. - The planned successor InGate was also abandoned. The community did not show up to maintain it. What did not change: - Existing in-cluster Ingress NGINX deployments keep running. - Existing `Ingress` resources keep working against any controller that implements the Ingress API. - Cloud-managed alternatives (AWS ALB Controller, GCE Ingress, Azure Application Gateway) are unaffected because they are separate codebases. The official recommendation from SIG Network is direct: begin migration to Gateway API or to another actively maintained Ingress controller immediately. The Kubernetes blog's own words are that staying on Ingress NGINX after retirement "leaves you and your users vulnerable to attack". If you ship anything to production behind this controller, this is the patch you cannot defer. ## Who needs to migrate and how urgent is it? Anyone running the `kubernetes/ingress-nginx` controller needs to migrate. The urgency depends on what kind of traffic the controller fronts. Public-facing clusters with any auth surface should treat this as a same-quarter migration. Internal-only clusters get one cycle of grace at most. Use this matrix to scope your own work. | Cluster type | Traffic | Urgency | Suggested target | | --- | --- | --- | --- | | Public production | Auth, payments, user data | Same quarter | Gateway API + Cilium / Istio / Envoy Gateway | | Public production | Static or CDN-fronted only | One quarter | Traefik drop-in, then Gateway API | | Internal staging | Dev traffic | One cycle | Traefik drop-in | | Edge / IoT | Mixed | Same quarter | Gateway API + Cilium (if already on Cilium CNI) | | Cloud-managed | Already on ALB/GCE/AGIC | None | No action | The thing to watch on the urgent rows is your own CVE feed. Every Nginx core CVE, every OpenResty advisory, every Lua module disclosure now stays unpatched against the retired controller. A WAF in front of it buys you time but not safety, because most of the meaningful vulnerabilities are at the HTTP layer the WAF needs to let through. I disagree with one piece of common advice making the rounds: that you can ride out the retirement with a strict WAF policy and pinned image hashes. That works for a few weeks. It does not work for a year. The same logic that retired the controller (limited maintainer bandwidth, growing CVE backlog) applies to the WAF policy too, because the WAF authors are not building new rules against a controller nobody else uses anymore. ## Which replacement should you pick? Pick Gateway API as the long-term destination and Traefik as the safe staging point if you cannot reach Gateway API immediately. The community has aligned on Gateway API as the next-generation traffic management standard. Traefik is the only realistic drop-in replacement because its NGINX Ingress Provider reads existing `nginx.ingress.kubernetes.io` annotations natively. Here is the short comparison. | Replacement | Drop-in for nginx annotations | Gateway API support | Where it fits | | --- | --- | --- | --- | | Gateway API + Cilium | No (manifest rewrite) | Native, GA | Long-term standard. Best if already on Cilium CNI. | | Gateway API + Istio | No (manifest rewrite) | Native, GA | Long-term standard. Best with existing service mesh. | | Gateway API + Envoy Gateway | No (manifest rewrite) | Native, GA | Long-term standard. Minimal extra footprint. | | Traefik | Yes (NGINX Ingress Provider) | Yes | Fastest cutover. Manifest changes optional. | | HAProxy Ingress | Partial | Yes | Performance focus, HTTP/3 native. | | Kong | Partial | Yes | If you also want an API gateway, not just ingress. | For my own staging cluster I ran a two-phase migration: Traefik first, Gateway API second. The Traefik phase took two hours. The Gateway API phase took a week. Both shipped without downtime. Cilium is the right Gateway API implementation if your cluster is already on Cilium CNI. The functionality is built in. You do not deploy a second component, and the eBPF datapath that already routes pod traffic also handles ingress. ## How do you migrate to Traefik with the drop-in adapter? You migrate to Traefik by installing the chart with the NGINX Ingress Provider enabled, leaving Ingress NGINX running side by side, and switching your DNS or LoadBalancer one host at a time. Traefik reads your existing `Ingress` resources directly. You do not rewrite manifests in the first pass. The minimum Helm install looks like this. ```bash helm repo add traefik https://traefik.github.io/charts helm repo update helm install traefik traefik/traefik \ --namespace traefik-system \ --create-namespace \ --set providers.kubernetesIngress.enabled=true \ --set providers.kubernetesIngress.publishedService.enabled=true \ --set providers.kubernetesIngress.ingressClass=traefik \ --set ingressClass.enabled=true \ --set ingressClass.isDefaultClass=false \ --set ingressRoute.dashboard.enabled=false ``` Two settings to call out. First, `providers.kubernetesIngress.enabled=true` turns on the NGINX-compatible Ingress reader. Second, `ingressClass.isDefaultClass=false` keeps Traefik from grabbing every existing `Ingress` resource at install time. You opt resources in one by one by changing their class. Now tag the first Ingress to move. ```yaml # my-app-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: my-app annotations: kubernetes.io/ingress.class: traefik # was: nginx nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: traefik # was: nginx rules: - host: my-app.example.com http: paths: - path: / pathType: Prefix backend: service: name: my-app port: number: 80 ``` The `nginx.ingress.kubernetes.io/rewrite-target` annotation stays. Traefik reads it. So do most of the popular annotations in that namespace, including `auth-url`, `auth-signin`, `enable-cors`, `force-ssl-redirect`, and `ssl-redirect`. The [Traefik documentation lists the supported set](https://doc.traefik.io/traefik/providers/kubernetes-ingress/) and clearly marks the ones that are not handled. Read that page once before you start, because the annotations that do not translate cleanly tend to be the regex-based rewrites and the more exotic auth flows. Apply the manifest. Now the resource is owned by Traefik. The Ingress NGINX controller stops reconciling it the moment the class flips. Repeat for every Ingress in the cluster. When the last one is over, scale Ingress NGINX to zero. ```bash kubectl scale -n ingress-nginx deployment/ingress-nginx-controller --replicas=0 ``` Leave it at zero for a week. If nothing breaks, delete the namespace. If something breaks, scale it back up and the failed resource resumes traffic the moment the class flips back. ## How do you migrate to Gateway API with Cilium? You migrate to Gateway API by installing the CRDs, enabling the Gateway API feature in your chosen implementation, declaring a `Gateway` resource per listener, then rewriting each `Ingress` as an `HTTPRoute`. The Cilium path is the most compact because Cilium already runs as your CNI and the Gateway API support is a flag, not a new component. Install the standard Gateway API CRDs first. ```bash kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml ``` Then enable Gateway API in Cilium. For an existing Cilium install this is a Helm value update. ```bash helm upgrade cilium cilium/cilium \ --namespace kube-system \ --reuse-values \ --set gatewayAPI.enabled=true \ --set gatewayAPI.enableAlpn=true \ --set gatewayAPI.enableAppProtocol=true ``` Restart the Cilium operator. The Gateway API controller comes up as part of the existing Cilium daemon set. Declare the `Gateway`. This replaces what used to be the Ingress NGINX `LoadBalancer` Service plus the `Ingress` host blocks. ```yaml # gateway.yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: public namespace: gateway-system spec: gatewayClassName: cilium listeners: - name: https port: 443 protocol: HTTPS hostname: '*.example.com' tls: mode: Terminate certificateRefs: - kind: Secret name: wildcard-example-com allowedRoutes: namespaces: from: All ``` Now rewrite the per-app Ingress as an `HTTPRoute`. ```yaml # my-app-route.yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: my-app spec: parentRefs: - name: public namespace: gateway-system hostnames: - my-app.example.com rules: - matches: - path: type: PathPrefix value: / backendRefs: - name: my-app port: 80 ``` The `Ingress` resource and the `HTTPRoute` resource can coexist for the same host. Apply the route while the Ingress is still serving traffic. The Gateway API controller picks it up but does not become authoritative for the host until you redirect DNS or the LoadBalancer. Three things the Gateway API does cleanly that Ingress did not. First, listener configuration (TLS, hostname, port) is a separate first-class resource, not a stack of annotations. Second, route attachment is cross-namespace by default, gated by `allowedRoutes`. Third, traffic-management features (weighted backends, header rewrites, request mirroring) are part of the spec, not vendor-specific annotations. The Cilium-specific docs cover the eBPF datapath details and the Gateway API conformance matrix. The Cilium release notes confirm Gateway API GA in v1.15. If you are on a Cilium version older than that, upgrade Cilium first. ## How do you cut over without downtime? You cut over without downtime by running the old and new controllers side by side, swinging traffic at the DNS or LoadBalancer layer rather than inside the cluster, and watching the new controller for one full traffic cycle before scaling the old one down. The wrong order is to delete the old controller first. Here is the sequence I used on my own cluster. 1. Install the new controller in its own namespace. Do not change any `Ingress` or `IngressClassName` yet. 2. Verify the new controller comes up healthy with a synthetic backend. A small `echo` deployment behind one test Ingress (or `HTTPRoute`) is enough. 3. For the first real workload, point a low-traffic DNS record at the new controller's LoadBalancer IP. Use a per-host record, not a wildcard, so the blast radius is bounded. 4. Watch for one full traffic cycle. At my scale that is 24 hours. For a high-traffic app it is one peak window. 5. If error rates and latency match the old controller, move the next host. Repeat. 6. When every host is on the new controller, drop the old LoadBalancer service and scale the controller to zero. Leave the deployment in place for a week as a rollback. 7. Delete the old namespace. The DNS-level swing is the part that matters. Changing `IngressClassName` on a live resource also works, but it cuts traffic instantly. A DNS swing lets you bleed traffic over the TTL window, and if something is wrong with the new controller the rollback is a single record change. For services that depend on the LoadBalancer source IP for security policy, the cutover needs a coordinated step where you tell the upstream firewall about both LoadBalancer IPs for the duration of the migration. I learned this the loud way when a third-party webhook started returning 403 from one provider after I moved their host. The provider's allowlist still pointed at the old LoadBalancer. ## How do you verify the migration? You verify the migration by running the conformance suite for your new controller, replaying a representative sample of production traffic, and watching golden signals during a full traffic cycle. If any of those three reveals a regression, do not delete the old controller yet. For Gateway API specifically, the conformance suite is the first signal. ```bash git clone --depth=1 https://github.com/kubernetes-sigs/gateway-api cd gateway-api/conformance go test ./... \ -args -gateway-class=cilium \ -conformance-profiles=GATEWAY-HTTP,GATEWAY-TLS ``` Every test that passes against your cluster certifies a piece of the spec that your config relies on. The full suite takes about ten minutes. Failures point at exactly which `HTTPRoute` feature your cluster does not handle. For Traefik, the in-tree integration tests are the equivalent. ```bash kubectl logs -n traefik-system deploy/traefik | grep -E "error|fail" ``` This is less rigorous than the Gateway API suite, but the Traefik controller is verbose about misconfigured Ingress resources. A clean log over 24 hours is the practical signal that your annotations translated correctly. For traffic replay, the cleanest tool is `goldpinger` plus a TLS-aware HTTP probe. ```yaml # verify-route.yaml apiVersion: batch/v1 kind: Job metadata: name: verify-my-app spec: template: spec: restartPolicy: Never containers: - name: probe image: curlimages/curl:8.7.1 command: - sh - -c - | for i in $(seq 1 100); do curl -fsS \ -H "Host: my-app.example.com" \ https://my-app.example.com/healthz \ || { echo "fail at $i"; exit 1; } sleep 1 done ``` Run that against both the old and new LoadBalancer IPs while traffic is still going through the old controller. A divergence shows up fast. The final check is the golden signals. Watch p99 latency, 4xx rate, and 5xx rate on the affected hosts for one full traffic cycle. Compare against the day before the cutover, not against the day of, because the day of will show DNS-propagation artifacts. For deeper context on related infrastructure shifts, see the [Postgres connection pool pgbouncer survival guide](https://www.rabinarayanpatra.com/blogs/postgres-connection-pool-pgbouncer-survival-guide) for the same pattern applied to database fronting, the [SSH tunnel local database access guide](https://www.rabinarayanpatra.com/blogs/ssh-tunnel-local-database-access) for a non-cluster fallback when you need direct backend reach during the migration, and the [Zero trust microservices with Spring Security](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security) for the auth-binding implications of the controller change. For the original sources, see the [Ingress NGINX retirement announcement](https://kubernetes.io/blog/2025/11/11/ingress-nginx-retirement/), the [SRC statement on the retirement](https://kubernetes.io/blog/2026/01/29/ingress-nginx-statement/), the [Traefik drop-in migration guide](https://traefik.io/blog/ingress-nginx-is-out), and the [Cilium Gateway API documentation](https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/gateway-api/). ### Keep Reading - [Postgres Connection Pool: pgbouncer Survival Guide](https://www.rabinarayanpatra.com/blogs/postgres-connection-pool-pgbouncer-survival-guide). The same pattern of fronting a critical backend with the right adapter, applied to Postgres. - [SSH Tunnel for Local Database Access](https://www.rabinarayanpatra.com/blogs/ssh-tunnel-local-database-access). The non-cluster fallback you reach for when you need direct backend reach during a controller migration. - [Zero Trust Microservices with Spring Security](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security). The auth-binding model that has to follow the controller through the migration. --- ## 10 Best Claude Skills for Marketers in 2026 Source: https://www.rabinarayanpatra.com/blogs/best-claude-skills-for-marketers-2026 Published: 2026-07-22T00:00:00+05:30 Tags: best-claude-skills-marketers, claude-skills, claude-code, content-marketing, seo-tools, ai-marketing 10 best Claude skills for marketers in 2026. Real picks for SEO, content research, brand work, and image gen. Direct links per skill and what each replaces. Claude Skills landed in October 2025 and the marketing-focused ones have quietly replaced a stack of SaaS tools I used to pay for. SEO audits that needed Semrush, a full audit takes thirty seconds. Content briefs that needed Frase, the research-and-write skill spits them out with citations. Brand briefs that needed a deck, one command and a structured brand-guidelines doc is on disk. Quick-look format below. Each skill links to its source folder. What it does. What I use it for. Ranked by impact in real marketing work; parent-repo star counts listed for transparency. ## What are Claude Skills for marketers? A Claude Skill is a folder with a `SKILL.md` file that teaches Claude one specific marketing task. When you ask for the task, Claude loads the skill, follows the playbook, and runs the workflow. The payoff is consistency. Same audit format every time. Same brief shape every time. Same brand voice on every output. Skills make Claude reliable on the boring parts so your attention goes to the parts that need a brain. ## How do you install Claude Skills if you do not code? You need Claude Code installed (free from [claude.com/code](https://claude.com/code)) or a claude.ai web account. Three install paths: ```bash # Path 1: Anthropic official skills mkdir -p ~/.claude/skills cd ~/.claude/skills git clone --depth 1 https://github.com/anthropics/skills.git anthropic-skills # Path 2: SEO skill suite (25 sub-skills, MIT license) git clone --depth 1 https://github.com/AgriciDaniel/claude-seo.git # Path 3: community skills index # https://github.com/ComposioHQ/awesome-claude-skills ``` After install, restart your Claude Code session once. Skills auto-discover. ## What are the 10 best Claude skills for marketers? Ranked by impact in real marketing work. Parent-repo stars listed. ### 1. seo-audit - **Skill:** [AgriciDaniel/claude-seo ยท seo-audit](https://github.com/AgriciDaniel/claude-seo/tree/main/skills/seo-audit) (parent repo 7,194 stars) - **What it does:** crawls up to 500 pages, detects business type, delegates to specialist sub-skills (technical, content, schema, images, sitemap, Core Web Vitals, GEO, backlinks, performance), generates a 0-100 health score. - **Use it for:** any new client baseline. Replaces a multi-hour Screaming Frog + Lighthouse + schema-check sweep. ### 2. content-research-writer - **Skill:** [ComposioHQ/awesome-claude-skills ยท content-research-writer](https://github.com/ComposioHQ/awesome-claude-skills/tree/master/content-research-writer) (parent repo 62k stars) - **What it does:** turns a topic into a researched, sourced, publication-ready markdown article. Multi-pass WebSearch, primary-source collection, outline, section-by-section drafting with live feedback. - **Use it for:** any 1500+ word post where research is the bottleneck. Replaces Frase briefs at zero subscription cost. ### 3. seo-content - **Skill:** [AgriciDaniel/claude-seo ยท seo-content](https://github.com/AgriciDaniel/claude-seo/tree/main/skills/seo-content) (7,194 stars) - **What it does:** scores a draft for E-E-A-T signals, readability, depth, thin-content patterns, first-person experience markers, and AI citation readiness. - **Use it for:** pre-publish review. E-E-A-T is the biggest ranking factor in 2026 that most content tools cannot measure. ### 4. seo-geo - **Skill:** [AgriciDaniel/claude-seo ยท seo-geo](https://github.com/AgriciDaniel/claude-seo/tree/main/skills/seo-geo) (7,194 stars) - **What it does:** scores AI-crawler accessibility (GPTBot, ClaudeBot, PerplexityBot), llms.txt compliance, passage-level citability, and platform-specific optimization for AI Overviews, ChatGPT, Perplexity, and Bing Copilot. - **Use it for:** any post you want cited by AI search. The 2026 evolution of traditional SEO. ### 5. seo-schema - **Skill:** [AgriciDaniel/claude-seo ยท seo-schema](https://github.com/AgriciDaniel/claude-seo/tree/main/skills/seo-schema) (7,194 stars) - **What it does:** detects, validates, and generates Schema.org structured data in JSON-LD. Article, Product, FAQPage, Breadcrumb, ItemList, and more. - **Use it for:** any non-trivial page type (comparison, listicle, product, recipe). Schema is where rich snippets get won. ### 6. seo-page - **Skill:** [AgriciDaniel/claude-seo ยท seo-page](https://github.com/AgriciDaniel/claude-seo/tree/main/skills/seo-page) (7,194 stars) - **What it does:** analyzes one URL across on-page elements, content quality, technical meta, schema, and performance. Per-URL scorecard with prioritized fixes. - **Use it for:** the page that should be ranking but is not. Tells you what to fix in the order that matters. ### 7. brand-guidelines - **Skill:** [anthropics/skills ยท brand-guidelines](https://github.com/anthropics/skills/tree/main/skills/brand-guidelines) (parent repo 142k stars) - **What it does:** turns a few brand inputs (palette, voice, examples) into a structured brand-guidelines doc and applies it consistently to every downstream output. - **Use it for:** any post, email, social caption, or asset that should sound like your brand. Replaces the "paste a tone-of-voice deck in every prompt" workaround. ### 8. web-artifacts-builder - **Skill:** [anthropics/skills ยท web-artifacts-builder](https://github.com/anthropics/skills/tree/main/skills/web-artifacts-builder) (142k stars) - **What it does:** builds elaborate self-contained HTML artifacts (multi-component React + Tailwind + shadcn/ui) for landing pages, comparison tools, calculators, lead magnets. - **Use it for:** the one-pager you would have hired a freelancer for. Ships in an afternoon. ### 9. higgsfield-generate - **Skill:** [higgsfield-ai/skills ยท higgsfield-generate](https://github.com/higgsfield-ai/skills/tree/main/higgsfield-generate) (parent repo 318 stars) - **What it does:** generates images via GPT Image 2 (technical diagrams), Nano Banana 2 (text-heavy), or Seedance (video). Marketing Studio mode handles ads with product imports + avatars. - **Use it for:** cover images, social cards, marketing visuals. Replaces a small studio for ~$0.30 per image. ### 10. skill-creator - **Skill:** [anthropics/skills ยท skill-creator](https://github.com/anthropics/skills/tree/main/skills/skill-creator) (142k stars) - **What it does:** interviews you on trigger language, steps, and references, then writes a new `SKILL.md` in the right shape. - **Use it for:** every repeatable marketing workflow (newsletter QA, link-build outreach, weekly report template). Build it once, reuse forever. ## Which Claude skills should you install first? Install in this order: 1. Install Claude Code from [claude.com/code](https://claude.com/code). 2. Clone [AgriciDaniel/claude-seo](https://github.com/AgriciDaniel/claude-seo) into `~/.claude/skills/claude-seo/`. Gives you the full SEO suite (audit, page, content, geo, schema, plus 20 more sub-skills). 3. Clone [anthropics/skills](https://github.com/anthropics/skills) into `~/.claude/skills/anthropic-skills/`. Gives you brand-guidelines, web-artifacts-builder, and skill-creator. 4. Add content-research-writer from [ComposioHQ/awesome-claude-skills](https://github.com/ComposioHQ/awesome-claude-skills/tree/master/content-research-writer) for long-form research drafts. 5. Add [higgsfield-ai/skills](https://github.com/higgsfield-ai/skills) when your content output exceeds your photographer or stock budget. The pattern that compounds: install one, use it for two weeks, install the next. The marketer who installs ten in one afternoon and uses none in week two saves zero hours. The one who installs one and uses it three times a week saves five hours by month two. For more on building skills yourself, see the [Anthropic engineering deep-dive on agent skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills), the [Claude Skills documentation](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview), and the [open standard at agentskills.io](https://agentskills.io). ### Keep Reading - [Claude Skills vs MCP vs Projects](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects). The mental model for skill vs MCP server choice. - [Claude Code Routines: Async CI Automation Just Became Real](https://www.rabinarayanpatra.com/blogs/claude-code-routines-ci-automation). Wire skills into scheduled tasks for hands-off content ops. - [Vercel AI Gateway Deep Dive](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive). One key for every LLM when your marketing stack needs to swap models. --- ## How to Convert JSON to a Kotlin Data Class (2026 Guide) Source: https://www.rabinarayanpatra.com/blogs/how-to-convert-json-to-kotlin-data-class Published: 2026-07-17T00:00:00+05:30 Tags: json-to-kotlin-data-class, kotlin, kotlinx-serialization, json, data-class, android Convert JSON to a Kotlin data class with kotlinx.serialization: snake_case with @SerialName, nullables, defaults, and nested objects. Plus an instant tool. Converting JSON to a Kotlin data class looks like a five-minute job. Copy the keys, make a data class, done. Then the API returns `null` for a field you marked non-null, or the keys are `snake_case` and your properties are `camelCase`, and now you have a `SerializationException` in production. I have written this mapping by hand more times than I can count, on Android apps and Ktor backends. This is the version that actually holds up: the right library, the annotations that matter, and the two or three things that bite everyone. And because typing out a 40-field class from a sample payload is soul-crushing, I will also show you the tool I built to skip that part entirely. ## How do you convert JSON to a Kotlin data class? You convert JSON to a Kotlin data class by defining a data class whose properties match the JSON keys, marking it `@Serializable`, and calling `Json.decodeFromString`. The official library for this is kotlinx.serialization, and it is the one I reach for on every new project. Add the plugin and dependency to your Gradle build: ```kotlin plugins { kotlin("plugin.serialization") version "2.1.0" } dependencies { implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.7.3") } ``` Then the class and the parse call: ```kotlin import kotlinx.serialization.Serializable import kotlinx.serialization.json.Json @Serializable data class User( val id: Int, val name: String, val email: String, val active: Boolean ) val json = """{"id":1,"name":"Rabi","email":"r@example.com","active":true}""" val user = Json.decodeFromString(json) ``` That is the whole happy path. The `@Serializable` annotation triggers the compiler plugin to generate the serializer at build time, so there is no runtime reflection, which is why kotlinx.serialization works on Kotlin Multiplatform and Android without extra config. Leave the annotation off and you get a compile error, not a runtime one, which is the good kind of failure. ## Why kotlinx.serialization over Gson or Moshi? Because kotlinx.serialization is the official Kotlin library, it understands Kotlin types like non-null and default values natively, while Gson and Moshi were built for Java and fake it. If you are starting fresh, this is the one to pick. Here is the practical difference. Gson uses reflection and will happily construct a Kotlin object with `null` in a non-null property, because it bypasses your constructors. That gives you a value that the Kotlin type system swears cannot be null, right up until it crashes somewhere far from the parse. Moshi with its Kotlin codegen fixes that, and it is a fine choice on existing Android projects that already use it. But it is another dependency and another annotation processor. kotlinx.serialization sidesteps all of it. It respects nullability, it respects default values, and it fails loudly at parse time when the JSON does not match, instead of handing you a broken object. On a new Kotlin codebase I do not think the comparison is close. Reach for kotlinx.serialization and move on. ## How do you map snake_case JSON keys to Kotlin properties? Use the `@SerialName` annotation to bind a JSON key to a differently named Kotlin property. Most APIs return `snake_case`, and you do not want that leaking into idiomatic Kotlin code. ```kotlin import kotlinx.serialization.SerialName import kotlinx.serialization.Serializable @Serializable data class UserProfile( val id: Int, @SerialName("full_name") val fullName: String, @SerialName("avatar_url") val avatarUrl: String, @SerialName("created_at") val createdAt: String ) ``` Now the serializer reads `full_name` from the JSON but your code uses `fullName`. This is the single most common thing people get wrong, because without it the parse throws a `SerializationException` complaining about a missing field, and the field is right there in the JSON. It is not missing. Its name just does not match. If literally every key is `snake_case`, there is a shortcut. Configure the `Json` instance with a naming strategy instead of annotating every property: ```kotlin import kotlinx.serialization.json.Json import kotlinx.serialization.json.JsonNamingStrategy @OptIn(kotlinx.serialization.ExperimentalSerializationApi::class) val json = Json { namingStrategy = JsonNamingStrategy.SnakeCase } ``` I still prefer explicit `@SerialName` on public API models, because it documents the exact wire contract at the property. But for an all-`snake_case` API, the global strategy saves a lot of noise. ## How do you handle nullable fields, defaults, and missing keys? Match the JSON's optionality in the Kotlin type: nullable properties for keys that can be `null`, default values for keys that can be absent. These are two different problems and people conflate them constantly. A key that is present but can hold `null` needs a nullable type: ```kotlin @Serializable data class Account( val id: Int, val nickname: String?, // present, but may be null val plan: String = "free" // may be absent entirely ) ``` The distinction that trips people up: a nullable type (`String?`) handles a key whose value is `null`. A default value handles a key that is missing from the JSON altogether. If a key might be absent and you do not give it a default, the parse fails. If it is present as `null` and you did not make it nullable, the parse also fails. They look similar and fail the same way, but the fix is different. One more flag worth knowing. By default kotlinx.serialization rejects unknown keys, which is strict and safe. Real APIs add fields all the time, so on client models I usually relax that: ```kotlin val json = Json { ignoreUnknownKeys = true coerceInputValues = true // treats null on a non-null with a default as the default } ``` `ignoreUnknownKeys` stops a new server field from breaking your app. `coerceInputValues` is the pragmatic escape hatch for APIs that send `null` where you expected a value with a default. Use them on client-side models. On server-side request bodies I keep strict parsing, because there I want the bad request rejected. ![Diagram mapping a JSON payload with snake_case keys and a nested object into a Kotlin data class using SerialName annotations and a nested data class](/images/blogs/how-to-convert-json-to-kotlin-data-class-mapping.svg) ## How do you map nested objects and arrays? Model each nested JSON object as its own `@Serializable` data class, and map arrays to `List`. kotlinx.serialization handles the nesting automatically once the types line up. ```kotlin @Serializable data class Address( val city: String, @SerialName("zip_code") val zipCode: String ) @Serializable data class Customer( val id: Int, val name: String, val address: Address, // nested object val tags: List, // array of primitives val orders: List // array of objects ) @Serializable data class Order(val id: Int, val total: Double) ``` The rule is simple: every JSON object becomes a data class, every JSON array becomes a `List` of whatever the elements are. There is no special config for depth. A five-level-deep payload just means five data classes, each `@Serializable`. This is also where doing it by hand stops being fun, and where a generator earns its keep. ## Is there a faster way than writing the class by hand? Yes. Paste the JSON into a generator and it writes the entire data class for you, nested classes and all. This is exactly the tedium I got tired of, so I built [jsontodto.com](https://jsontodto.com) to do it. You drop in a sample payload and it produces the matching Kotlin data class, with nested objects split into separate classes and arrays typed as `List`. You can toggle a plain data class or add kotlinx.serialization decorators, and it infers types from the values. For a big third-party API response, this turns a 20-minute typing session into a paste. The tool also outputs Java records, TypeScript interfaces, Python dataclasses, Go structs, and C# records from the same JSON, so it is handy across a polyglot stack, not just Kotlin. My honest workflow: generate the class from a real sample, then hand-tune it. The generator cannot know that a field is nullable if the sample happened to have a value there, and it cannot know which keys are optional across all responses. So I paste, generate, then go through and add `?` where the API can return null and defaults where keys can be absent. Generation gets you 90 percent of the way in one paste. The last 10 percent is the judgment this guide is about. Converting JSON to a Kotlin data class is one of those tasks that is trivial until the real world shows up with `snake_case` keys, nullable everything, and objects nested five deep. Get the annotations right and kotlinx.serialization does the rest without complaint. Generate the boilerplate, then spend your actual attention on the nullability and optionality the generator cannot infer. That is where correctness lives. For the full API, see the [kotlinx.serialization JSON documentation](https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/json.md) and the [official Kotlin serialization guide](https://kotlinlang.org/docs/serialization.html). ### Keep Reading - [JSON to DTO Converter: Instantly Generate Java Classes from JSON](https://www.rabinarayanpatra.com/blogs/json-to-dto-converter). The Java side of the same tool, for records, Lombok, and Jackson DTOs. - [Java Libraries Beyond Lombok](https://www.rabinarayanpatra.com/blogs/java-libraries-beyond-lombok). More ways to cut boilerplate when you are back on the JVM side of a project. --- ## How to Enable Virtual Threads in Spring Boot (Java 21+) Source: https://www.rabinarayanpatra.com/blogs/how-to-enable-virtual-threads-spring-boot Published: 2026-07-15T00:00:00+05:30 Tags: spring-boot-virtual-threads, virtual-threads, spring-boot, java-21, project-loom, concurrency Enable virtual threads in Spring Boot with one property. What spring.threads.virtual.enabled wires up, the connection-pool trap, and when to skip it. Enabling virtual threads in Spring Boot is genuinely one line. Set a property, run on Java 21, done. The flag is not where people get in trouble. Where they get in trouble is thinking the flag is the whole story. It quietly rewires how your web server handles requests, which executor runs your `@Async` methods, and how your Kafka listeners get their threads. And it exposes a bottleneck that was always there but never mattered before. This is the guide I give teammates who are about to flip the switch in production. If you want the deeper theory on what virtual threads are and how Project Loom works, I covered that in [my breakdown of virtual threads in Java 25](https://www.rabinarayanpatra.com/blogs/virtual-threads-java-25). Here I am focused on the Spring Boot side: what turning them on actually does, and what bites you. ## How do you enable virtual threads in Spring Boot? You enable virtual threads in Spring Boot by setting `spring.threads.virtual.enabled=true` and running on Java 21 or later. That is the entire setup. In `application.properties`: ```properties spring.threads.virtual.enabled=true ``` Or in `application.yml`: ```yaml spring: threads: virtual: enabled: true ``` This property has been available since Spring Boot 3.2, released in December 2023, and it carries forward unchanged into Spring Boot 4.x. The only hard requirement beyond the property is the runtime: virtual threads are a Java 21 feature, so on Java 17 or 20 the property does nothing useful. I always pair it with an explicit check that the app is actually on 21+. A quick way is to log `Runtime.version()` at startup, or fail fast in a `@PostConstruct` if the major version is below 21. Silent no-ops are worse than crashes. ## What does spring.threads.virtual.enabled actually turn on? The property flips several auto-configurations at once, not just the web server. Understanding the full set is the difference between "I enabled virtual threads" and "I know where my code runs now." Here is what Spring Boot wires up when the flag is on and you are on Java 21+: - **Web request handling.** Tomcat and Jetty use virtual threads to process requests, so your controller methods run on a virtual thread per request instead of a pooled platform thread. - **The application task executor.** The `applicationTaskExecutor` bean becomes a `SimpleAsyncTaskExecutor` that starts a new virtual thread per task. This is what backs `@Async` methods, Spring MVC async request processing, and WebFlux blocking execution support. - **The task scheduler.** Scheduling is backed by a `SimpleAsyncTaskScheduler` on virtual threads, and it ignores pool-size properties because there is no pool to size. - **Messaging listeners.** RabbitMQ and Kafka listener containers get virtual thread executors auto-configured. - **Data access helpers.** Spring Data Redis' `ClusterCommandExecutor` runs on virtual threads. ![Diagram showing the single spring.threads.virtual.enabled property fanning out to Tomcat request threads, the application task executor, the task scheduler, and Kafka and RabbitMQ listeners](/images/blogs/how-to-enable-virtual-threads-spring-boot-fanout.svg) The takeaway: this one property changes the execution model across the whole app, not just HTTP. If you have custom `Executor` beans wired by hand, they are not touched. Spring only swaps the executors it owns. So an app that defines its own `ThreadPoolTaskExecutor` for `@Async` keeps using platform threads there until you change it yourself. ## Why is the database connection pool the real bottleneck? Because virtual threads remove the thread limit but not the connection limit, so your JDBC pool becomes the ceiling the moment threads stop being scarce. This is the single most important thing to understand before enabling them. The old model looked like this. Tomcat had 200 platform threads. Each request grabbed a thread, and if the thread pool was full, new requests queued. The thread pool was your concurrency limit, and it was usually smaller than your connection pool, so the pool rarely ran dry. Flip on virtual threads and that changes. Now you can have 10,000 requests in flight, each on its own cheap virtual thread. But HikariCP still defaults to 10 database connections. So 10 requests run their query and the other 9,990 block waiting for a connection. You did not remove the wait. You moved it from "waiting for a thread" to "waiting for a connection," and the second one is easier to miss because everything looks healthy until latency spikes. ```properties # Virtual threads let you accept far more concurrent requests. spring.threads.virtual.enabled=true # So the connection pool is now your real concurrency limit. # Size it against your database's capacity, not your request rate. spring.datasource.hikari.maximum-pool-size=50 ``` There is no magic number here. The right pool size is bounded by what your database can handle, and adding virtual threads does not change that. What changes is that the pool, not the thread count, is now the thing you tune. I have seen teams enable virtual threads, see no improvement, and conclude the feature is overhyped. The feature was fine. Their 10-connection pool was the wall. ## What is thread pinning and when should you worry about it? Thread pinning is when a virtual thread cannot unmount from its carrier platform thread during a blocking call, which cancels the scalability benefit for that operation. It used to be a real concern with `synchronized`, and it is mostly gone now. On Java 21 through 23, a virtual thread that blocks inside a `synchronized` block stays pinned to its carrier thread. If that block wraps an I/O call, like a database query guarded by a synchronized method, you lose the whole point: the carrier thread is stuck, and you are back to a bounded number of real threads. Java 24 fixed this. JEP 491 reworked synchronization so virtual threads no longer pin on `synchronized`, which removes the most common pinning source for most apps. So the advice depends on your runtime: - **Java 24+:** pinning from `synchronized` is no longer a concern. You can mostly stop worrying about it. - **Java 21 to 23:** audit hot paths for `synchronized` around blocking I/O. Swap those specific locks to `ReentrantLock`, which lets the virtual thread unmount cleanly. You do not need to rip out every `synchronized` in your codebase. A lock held for a quick in-memory update does not pin in any way that matters. The one to hunt for is a lock held across a network or disk call. To find them, run with `-Djdk.tracePinnedThreads=full` on Java 21 to 23 and watch the logs under load. ## When should you not enable virtual threads? Skip virtual threads when your workload is CPU-bound, because they solve a blocking problem, not a compute problem. This is the clearest "no." Virtual threads shine when threads spend most of their time waiting: database calls, HTTP calls to other services, file I/O. If your service does heavy computation, image processing, large in-memory transforms, number crunching, virtual threads add nothing. You still have the same number of CPU cores, and a virtual thread doing math occupies a carrier thread the whole time just like a platform thread would. For that work, a sized `ThreadPoolTaskExecutor` matched to your core count is still the right tool. A few other cases where I hold off: - **Heavy `ThreadLocal` usage.** Each virtual thread carries its own `ThreadLocal` values. With millions of threads, a fat `ThreadLocal` cache turns into a memory problem. Audit what you stash there before scaling thread counts way up. - **Third-party libraries with internal thread pools.** Enabling the Spring property does not change a library that manages its own executor. Check that your critical dependencies are Loom-friendly, not fighting it. - **You have not measured.** If you do not know that thread starvation is your bottleneck, enabling virtual threads is a guess. Load test first, find the actual limit, then decide. Enabling virtual threads in Spring Boot is not a performance cheat code, and treating it like one is how you end up disappointed. It is a scalability tool for I/O-bound concurrency, and it pays off exactly when blocked threads were what held you back. Flip the property, then go re-tune your connection pool and re-run your load test. That second step is where the actual throughput lives. For the official details, see the [Spring Boot task execution and scheduling reference](https://docs.spring.io/spring-boot/reference/features/task-execution-and-scheduling.html) and the [Spring Boot 3.2 release notes](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.2-Release-Notes) where the property was introduced. ### Keep Reading - [Virtual Threads in Java 25: The Complete Guide](https://www.rabinarayanpatra.com/blogs/virtual-threads-java-25). The language-level deep dive on how virtual threads and Project Loom work under the hood. - [Modern Java with Spring Boot](https://www.rabinarayanpatra.com/blogs/modern-java-spring-boot). How the newer Java features fit into a real Spring Boot service. --- ## How to Configure the Debezium Outbox Event Router SMT Source: https://www.rabinarayanpatra.com/blogs/how-to-configure-debezium-outbox-event-router Published: 2026-07-10T00:00:00+05:30 Tags: debezium-outbox-event-router, debezium, change-data-capture, kafka-connect, outbox-pattern, microservices Configure the Debezium Outbox Event Router step by step: route to per-aggregate topics, set message keys, add header metadata, and expand JSON payloads. The outbox pattern gets you a table full of events written in the same transaction as your business data. That solves the dual-write problem. But a Debezium connector pointed at that table emits raw row-change events, envelopes and all, on a single ugly topic. Nobody wants to consume that. The piece that turns those raw row changes into clean, per-aggregate domain messages is the Outbox Event Router SMT. It is the single most useful transform Debezium ships, and it is also the one people misconfigure the most. This guide is the config reference I wish I had the first time I wired it up. If you need the pattern itself first (why the outbox table exists, how the transaction works), read [my walkthrough of implementing the outbox pattern with CDC](https://www.rabinarayanpatra.com/blogs/implementing-outbox-pattern-cdc-microservices) and come back here for the routing details. ## What does the Debezium Outbox Event Router actually do? The Debezium Outbox Event Router is a single message transform, class `io.debezium.transforms.outbox.EventRouter`, that reshapes raw outbox-table change events into clean messages on per-aggregate topics. It runs inside the Kafka Connect pipeline, after Debezium captures the insert and before the record hits Kafka. Think of it as three jobs stacked together. It picks the destination topic from a column in the row. It sets the Kafka message key from another column. And it unwraps the event so the payload column becomes the message value instead of a nested change-event envelope. By default it assumes your outbox table has these columns: | Column | Purpose | Default type | |--------|---------|--------------| | `id` | unique event id, used for dedup | uuid | | `aggregatetype` | drives the topic name | varchar | | `aggregateid` | becomes the Kafka message key | varchar | | `type` | the event type (OrderCreated, etc.) | varchar | | `payload` | the actual event body | json / jsonb | You are not locked into those names. Every one of them is remappable. But if you control the table schema, matching the defaults means less config to get wrong. ## How do you add the EventRouter SMT to a connector? You add it as a named transform in the connector config, then set the transform type to the EventRouter class. Everything else is options prefixed with that transform name. Here is the minimum viable version on a Postgres connector, assuming default column names: ```json { "transforms": "outbox", "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter", "table.include.list": "public.outbox" } ``` That is genuinely all you need to start. With no other options, the SMT reads `aggregatetype`, routes to `outbox.event.`, keys the message by `aggregateid`, and uses `payload` as the value. The `table.include.list` is not part of the SMT, but you want it: without scoping the connector to the outbox table, Debezium captures every table in the schema and the transform errors on rows that do not look like outbox events. One thing that bites people immediately. The EventRouter expects to see the row that was inserted, so it needs the new-record state. If you are also chaining `ExtractNewRecordState`, order matters. Put the outbox transform first, because it already handles the unwrap itself. Stacking a second flattening transform in front of it usually strips the envelope fields the router needs. ## How do you route events to per-aggregate topics? Routing is controlled by three options that work as a small pipeline: read a field, match it with a regex, substitute it into a template. The defaults route by aggregate type into a predictable topic name. - `route.by.field` (default `aggregatetype`): the column whose value drives the topic name. - `route.topic.regex` (default `(?.*)`): a regex with a named capture group that pulls the routing value out. - `route.topic.replacement` (default `outbox.event.${routedByValue}`): the topic name template. The capture group name is substituted in. So a row with `aggregatetype = Order` lands on `outbox.event.Order`. If you want a flatter naming scheme, override the replacement: ```json { "transforms.outbox.route.by.field": "aggregatetype", "transforms.outbox.route.topic.replacement": "domain.${routedByValue}.events" } ``` Now `Order` goes to `domain.Order.events`. The regex is worth knowing about for one real case: if your aggregate type values carry a prefix you do not want in the topic name, capture only the part you want. A `route.topic.regex` of `svc_(?.*)` turns `svc_Order` into a routing value of `Order`. The message key is set separately through `table.field.event.key` (default `aggregateid`). This is the part that keeps per-entity ordering intact. Every event for the same order carries the same key, so Kafka lands them on the same partition and consumers see them in write order. If you leave the key column empty, you lose that guarantee, so treat `aggregateid` as required, not optional. ![Diagram showing an outbox table row flowing through the Debezium EventRouter SMT and fanning out into per-aggregate Kafka topics keyed by aggregate id](/images/blogs/how-to-configure-debezium-outbox-event-router-routing.svg) ## How do you add metadata as Kafka headers or envelope fields? You expose extra columns with `table.fields.additional.placement`, which takes a comma-separated list of `column:placement:alias` entries. The placement is either `header` or `envelope`, and that choice changes where consumers read the value. Say your outbox table has an `eventtype` column and a `tracecontext` column you want to propagate. You want the event type inside the message value and the trace context as a Kafka header: ```json { "transforms.outbox.table.fields.additional.placement": "type:envelope:eventType,tracecontext:header:traceparent" } ``` Two rules I learned the hard way. `header` puts the value on the Kafka record header, which is perfect for cross-cutting metadata like trace ids that infrastructure reads without deserializing the body. `envelope` folds the value into the message value alongside the payload, which is what you want for domain fields a consumer actually maps into an object. Pick header for plumbing, envelope for data. The alias (the third part) is the name the field or header gets on the output side. It is optional, but I always set it. Relying on the raw column name leaks your database schema into your event contract, and renaming the column later silently breaks every consumer. ## How do you expand the JSON payload and handle deletes? Two options handle the payload shape and the delete case: `table.expand.json.payload` and `route.tombstone.on.empty.payload`. Both default to `false`, and both are safe to leave off until you need them. By default the payload column is passed through as-is. If you stored it as a JSON string, consumers receive a string and have to parse it themselves. Turn on expansion to get a real structured record: ```json { "transforms.outbox.table.expand.json.payload": "true" } ``` With that set, the SMT parses the payload into a proper Kafka Connect struct, so downstream schemas and converters see typed fields instead of one big string. The catch is that it only works when the payload is valid JSON. Malformed content makes the transform fail the record, so this pairs best with a `jsonb` column that the database already validates on write. Deletes are the other edge. The outbox table is usually append-only, so you rarely delete rows. But if you do (say a cleanup job prunes old events), Debezium emits a delete change event. Turn on `route.tombstone.on.empty.payload` to convert those into proper Kafka tombstone records (a null value on the aggregate key), which is what log-compacted topics expect for key removal. If your outbox topics are not compacted, leave it off. ## What does a complete connector configuration look like? Here is a full Postgres connector config with the Event Router doing real work: custom routing, header metadata, JSON expansion, and message keying. This is close to what I have shipped in production. ```json { "name": "outbox-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "postgres", "database.port": "5432", "database.user": "debezium", "database.dbname": "orders", "topic.prefix": "orders-svc", "table.include.list": "public.outbox", "tombstones.on.delete": "false", "transforms": "outbox", "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter", "transforms.outbox.route.by.field": "aggregatetype", "transforms.outbox.route.topic.replacement": "domain.${routedByValue}.events", "transforms.outbox.table.field.event.key": "aggregateid", "transforms.outbox.table.field.event.payload": "payload", "transforms.outbox.table.expand.json.payload": "true", "transforms.outbox.table.fields.additional.placement": "type:envelope:eventType,tracecontext:header:traceparent" } } ``` To verify it works, insert a row and watch the topic. A row with `aggregatetype = Order` and `aggregateid = 42` should produce a message on `domain.Order.events`, keyed by `42`, with the payload expanded into structured fields and a `traceparent` header attached. If the message lands on `outbox.event.Order` instead, your `route.topic.replacement` override is not being picked up, usually because the transform name in the property key does not match the one in the `transforms` list. ## What are the common pitfalls with the Event Router? Most Event Router problems trace back to a schema mismatch between the table and the config. Here are the ones I hit most, in rough order of how often they cost me an afternoon. The connector captures more than the outbox table. Always set `table.include.list` to just the outbox table. Otherwise the transform sees regular business-table changes and throws because they lack `aggregatetype`. The message key is null. Check that `table.field.event.key` points at a populated column. A null key destroys per-aggregate ordering, and it is easy to miss because messages still flow. JSON expansion fails silently on bad data. If `table.expand.json.payload` is on and one row has a malformed payload, that record errors and can stall the connector. Validate at write time with a `jsonb` column, and consider a dead-letter queue on the connector. Transform ordering with `ExtractNewRecordState`. The Event Router already unwraps the change event. Chaining a second flatten transform in front of it strips the fields it needs. If you must combine them, the outbox transform goes first. Topic auto-creation is off. Per-aggregate routing means new aggregate types create new topics. If your broker disables auto topic creation, a brand new aggregate type produces messages that go nowhere until you create the topic. Pre-create them or enable Kafka Connect topic creation. The Event Router is a small transform with a big payoff. Once the config matches your table, it disappears into the background and just produces clean domain events forever. The mistake I see teams make is treating it as an afterthought bolted onto the connector at the end. Design your outbox table columns around the router's defaults from day one, and the config shrinks to almost nothing. For the full option list and version-specific behavior, see the [Debezium Outbox Event Router documentation](https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html) and the [Debezium outbox quickstart on GitHub](https://github.com/debezium/debezium-examples/tree/main/outbox). ### Keep Reading - [How to Implement the Outbox Pattern with CDC in Microservices](https://www.rabinarayanpatra.com/blogs/implementing-outbox-pattern-cdc-microservices). Start here for why the outbox table exists and how the transactional write works before you configure the router. - [Zero Trust Microservices with Spring Security](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security). Once your services talk over Kafka topics, this covers securing the service-to-service boundary. --- ## How to Configure flake8 for Python (2026 Setup Guide) Source: https://www.rabinarayanpatra.com/blogs/how-to-configure-flake8-python Published: 2026-07-08T00:00:00+05:30 Tags: configure-flake8, flake8-pyproject-toml, flake8, python-linting, pre-commit, devops Learn how to configure flake8 for Python: install it, set up .flake8, fix the pyproject.toml problem, wire up pre-commit, and decide if you still need it in 2026. Every Python project I join eventually hits the same wall. Someone adds flake8, drops the config into `pyproject.toml` like every other modern tool expects, runs it, and nothing happens. No error, no warning, just flake8 quietly ignoring the entire file. Then the confusion starts. This guide walks through how to configure flake8 properly in 2026: what it actually is, where its config really lives, why `pyproject.toml` is the one file it refuses to read, and whether you should still reach for it now that Ruff exists. Everything here is checked against flake8 7.3.0, released June 2025. ## What is flake8 and what does it actually check? flake8 is not one linter. It is a wrapper that glues three separate tools together behind a single command, then adds a plugin system on top. When you run `flake8`, you are really running pyflakes, pycodestyle, and mccabe in one pass. Each tool owns a code prefix, and knowing them makes every flake8 report readable: - **pyflakes** finds logical errors: unused imports, undefined names, unused variables. Its codes start with **F**. - **pycodestyle** checks PEP 8 style: whitespace, indentation, line length. It was once called pep8. Its codes start with **E** (errors) and **W** (warnings). - **mccabe** measures cyclomatic complexity. It emits exactly one code, **C901**, and only when you turn it on. flake8 adds one code of its own: **E999**, which means the file failed to compile at all (a syntax error). So when you see `F401`, that is pyflakes telling you about an unused import. `E501` is pycodestyle complaining about line length. `C901` is mccabe saying a function is too complex. ![flake8 glues pyflakes, pycodestyle, and mccabe into one command, each contributing its own error-code prefix](/images/blogs/how-to-configure-flake8-python-cover.svg) flake8 7.3.0 pins those dependencies tightly: pyflakes 3.4.x, pycodestyle 2.14.x, and mccabe 0.7.x. It runs on Python 3.9 and up. The project lives under PyCQA (the Python Code Quality Authority) and is maintained by Anthony Sottile and Ian Cordasco. ## How do you install and run flake8? Install it with pip and point it at your code. That is the whole starting workflow: ```bash pip install flake8 flake8 . ``` That runs flake8 across the current directory. You can target a specific path or module too: ```bash flake8 src/ python -m flake8 src/ ``` The `python -m flake8` form is worth remembering. It guarantees you run the flake8 installed in the current environment, which matters the moment you have more than one Python around. On a fresh run with no config, flake8 uses its defaults: 79-character lines and a small built-in ignore list. Most teams change both, which is where config comes in. ## Where does flake8 look for its configuration? flake8 reads three config files, all in INI format under a `[flake8]` section: `setup.cfg`, `tox.ini`, and a dedicated `.flake8` file. It does not read anything else, and it does not merge them. It picks one. I default to a standalone `.flake8` file because it keeps linter config out of files that mean other things. Here is a realistic starting point, close to what the official docs recommend: ```ini [flake8] max-line-length = 88 extend-ignore = E203 exclude = .git,__pycache__,docs/source/conf.py,old,build,dist max-complexity = 10 per-file-ignores = __init__.py: F401 ``` A few of those options carry sharp edges, so here is what each one actually does: - **`max-line-length`** defaults to 79. Setting it to 88 matches Black, which is why most projects use that number. - **`extend-ignore`** adds codes to the ignore list without wiping the defaults. `ignore` (no `extend-`) replaces the entire default list, which usually is not what you want. Reach for the `extend-` forms. - **`extend-select`** works the same way for turning checks on. It was added in flake8 4.0.0. - **`exclude`** takes comma-separated paths and globs that flake8 skips entirely. - **`per-file-ignores`** lets you relax rules per file. The `__init__.py: F401` line above silences unused-import warnings in package init files, where re-exporting is normal. - **`max-complexity`** turns on the mccabe check. It is off until you set it, so `C901` never fires until you add a line like `max-complexity = 10`. The `extend-ignore = E203` in that example is there for a reason. E203 flags whitespace before a colon, which collides with how Black formats slices. Turning it off is the standard fix for running flake8 and Black together. ## Why won't flake8 read your pyproject.toml? flake8 does not read `pyproject.toml`, and that is a deliberate decision, not a missing feature. This is the single most common flake8 problem I see, and the answer surprises people: the maintainer has kept the request open for years and does not intend to add it as-is. The relevant thread is [PyCQA/flake8 issue #234](https://github.com/PyCQA/flake8/issues/234), and it is still open. Anthony Sottile's position is direct: "I don't think toml is the way especially with pip continuing to have weird behaviour when the file even exists." His technical objection is real. The mere presence of a `pyproject.toml` file changes how pip builds a package, because it can force an isolated PEP 517 build. He listed two conditions that would need to hold before he would reconsider: pip no longer changing its behavior based on the file existing, and a standard-library TOML parser. Python 3.11 added `tomllib`, so half of that is now true, but flake8 7.3.0 still will not read the file. So what do you do if your project standardizes on `pyproject.toml` and you want flake8 config to live there too? Use a plugin. ```bash pip install Flake8-pyproject ``` [Flake8-pyproject](https://pypi.org/project/Flake8-pyproject/) lets you put your config under a `[tool.flake8]` section and either keeps working with the normal `flake8` command or gives you a `flake8p` entry point: ```toml [tool.flake8] max-line-length = 88 extend-ignore = ["E203"] exclude = [".git", "__pycache__", "build", "dist"] ``` There is also `pyproject-flake8` (the `pflake8` command), which monkey-patches flake8 to read the file. Both work. I lean toward Flake8-pyproject because it is the lighter touch, but understand that either one is a workaround layered on top of a tool that, by design, still does not natively support the format. ![flake8 reads setup.cfg, tox.ini, and .flake8 but not pyproject.toml, which needs the Flake8-pyproject plugin as a bridge](/images/blogs/how-to-configure-flake8-python-config-resolution.svg) ## How do you silence a specific warning inline? Add a `# noqa` comment to the line, and be specific about which code you are silencing. flake8 gives you two forms, and the difference matters. A bare `# noqa` silences every error on that line: ```python import os # noqa ``` A coded `# noqa: E501` silences only that check. Anything else on the line still gets reported, which is exactly what you want: ```python url = "https://example.com/a/very/long/path/that/breaks/the/line/limit" # noqa: E501 ``` You can list several codes with commas, like `# noqa: E731,E123`, and the comment is case-insensitive. I always prefer the coded form. A bare `# noqa` is a blunt instrument that hides real bugs the day someone adds a second problem to that line. If you ever need to audit what your `# noqa` comments are actually hiding, run with `--disable-noqa` and flake8 reports everything as if the comments were not there. ## How do you run flake8 in pre-commit and CI? Wire flake8 into pre-commit so it runs before code ever reaches a branch. The official hook is published in the flake8 repo, and the setup is a few lines in `.pre-commit-config.yaml`: ```yaml - repo: https://github.com/pycqa/flake8 rev: 7.3.0 hooks: - id: flake8 ``` The docs show `rev: ''` as a placeholder, but always pin it to a real tag like `7.3.0`. An unpinned linter version means your CI can start failing on a Tuesday because a new release added a check, and nobody changed a line of code. For CI itself, there is nothing special to configure. The same `flake8 .` command that runs locally runs in a GitHub Actions step, reads the same config file, and exits non-zero on any violation. If you are hardening a pipeline, the same discipline I described in [enabling npm trusted publishing with OIDC](https://www.rabinarayanpatra.com/blogs/how-to-enable-npm-trusted-publishing-github-actions-oidc) applies here: pin your tool versions and let the config file be the single source of truth. ## Which flake8 plugins are worth adding? Plugins extend flake8 with new checks under their own code prefixes, and installing one is just a pip install into the same environment. Most start reporting immediately, no config required. You can confirm what loaded by running `flake8 --version`, which lists every active plugin. The ones I reach for most: - **flake8-bugbear** (prefix `B`) catches likely bugs and questionable design that pyflakes misses, like mutable default arguments. - **flake8-comprehensions** (`C4`) flags list and dict comprehensions that could be simpler. - **flake8-docstrings** (`D`) checks PEP 257 docstring conventions by wrapping pydocstyle. - **pep8-naming** (`N`) enforces naming conventions for classes, functions, and variables. - **flake8-bandit** adds security checks, and **flake8-comprehensions** and **flake8-simplify** clean up common code smells. Once a plugin is installed, its codes behave like any other. Turn them on or off with `select`, `ignore`, `extend-select`, and `extend-ignore` in your config. That is the real strength of flake8: the checker set is yours to compose. ## Should you still use flake8 in 2026? Use flake8 if you depend on a plugin Ruff has not reimplemented. Otherwise, Ruff is probably the better default. That is the honest answer, and it comes straight from Ruff's own documentation rather than my opinion. [Ruff](https://docs.astral.sh/ruff/) is a linter written in Rust that advertises being "10-100x faster than existing linters (like Flake8)." Its FAQ is refreshingly precise about compatibility. Ruff is a drop-in replacement for flake8 when you use it without many plugins, alongside Black, on Python 3 code. Under those conditions it "implements every rule in Flake8," including all the pyflakes `F` rules and a large subset of the `E` and `W` style rules. But Ruff names its own limit clearly: "Ruff's primary limitation vis-a-vis Flake8 is that it does not support custom lint rules." That is the line that keeps flake8 alive. If your team relies on an internal flake8 plugin, or a niche community one Ruff has not ported, flake8 is still the tool that can run it. The same goes for a stable, working setup where migration churn buys you nothing. Speed is real, but a linter you have already tuned and trust has value too. I have migrated projects to Ruff and kept others on flake8, and both calls were right for their context. The mistake is treating it as a moral question. Pick the tool that runs the checks you need, and let the benchmark difference decide only when the checks are equal. For the details behind everything here, see the [flake8 documentation](https://flake8.pycqa.org/en/latest/), the still-open [pyproject.toml discussion in issue #234](https://github.com/PyCQA/flake8/issues/234), and [Ruff's FAQ](https://docs.astral.sh/ruff/faq/) on flake8 compatibility. ### Keep Reading - [How to Enable npm Trusted Publishing with GitHub Actions OIDC](https://www.rabinarayanpatra.com/blogs/how-to-enable-npm-trusted-publishing-github-actions-oidc). Another piece of a hardened CI pipeline, where pinned tool versions matter just as much. - [How to Check Out Fork PRs Safely in actions/checkout v7](https://www.rabinarayanpatra.com/blogs/actions-checkout-v7-safe-pull-request-target). Running linters on untrusted pull requests without opening a security hole. - [How to Serve a React SPA from FastAPI with app.frontend()](https://www.rabinarayanpatra.com/blogs/serve-react-spa-fastapi-app-frontend). More practical Python tooling, this time on the web-serving side. --- ## HTTP QUERY Method: The Safe GET With a Body (RFC 10008) Source: https://www.rabinarayanpatra.com/blogs/http-query-method-rfc-10008 Published: 2026-07-03T00:00:00+05:30 Tags: http-query-method, rfc-10008, http, rest-api, web-standards, api-design The HTTP QUERY method (RFC 10008) gives you a safe, idempotent, cacheable request with a body. What it fixes, how caching works, and who supports it today. Every REST API I have ever worked on grows a `POST /search` endpoint sooner or later. Not because POST is the right method for reading data, but because GET cannot carry a request body and the filter JSON stopped fitting in the URL years ago. As of mid-June 2026, HTTP finally has a proper fix. It is called QUERY, and it just became [RFC 10008](https://www.rfc-editor.org/rfc/rfc10008.html). The HTTP QUERY method gives you the thing GET and POST each refuse to: a safe, idempotent, cacheable request that carries a body. In this post I will walk through what the spec actually says, why GET-with-a-body and POST-for-reads are both broken, how caching works when the cache key lives in the request body, and which runtimes you can use it on today. ## What is the HTTP QUERY method? The HTTP QUERY method is a new request method, standardized in RFC 10008 in mid-June 2026, that sends a query as the request body while keeping the guarantees of GET. The spec's abstract puts it well: "A QUERY requests that the request target process the enclosed content in a safe and idempotent manner and then respond with the result of that processing." The body is the query. Not the resource, not a command, a question. "The content of the request and its media type define the query." That media type matters: the spec says servers MUST fail the request if `Content-Type` is missing or inconsistent with the content. A successful query comes back as a plain `200 OK` with the results in the response body. Here is the comparison table straight from Section 1 of the RFC: | Property | GET | QUERY | POST | | ------------ | ------------------- | -------- | ------------------------------------ | | Safe | Yes | Yes | Potentially no | | Idempotent | Yes | Yes | No | | Cacheable | Yes | Yes | Only for future GET or HEAD requests | | Request body | No defined semantics | Expected | Expected | There is also a discovery mechanism. A server can advertise QUERY support with the new `Accept-Query` response header, a Structured Field listing the query media types it accepts for that path: ```http Accept-Query: "application/jsonpath", application/sql;charset="UTF-8" ``` The RFC was written by Julian Reschke (greenbytes), James M. Snell (Cloudflare), and Mike Bishop (Akamai), and it is a Proposed Standard on the IETF standards track. This is not a vendor experiment. It is the real thing. ## Why isn't GET with a body enough? Sending a body with GET is undefined behavior that some servers and intermediaries will actively reject. RFC 9110, the core HTTP semantics spec, is blunt about it in Section 9.3.1: "content received in a GET request has no generally defined semantics, cannot alter the meaning or target of the request, and might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack." So the standard advice has always been: put the query in the URL. That works until it does not. RFC 9110 only recommends that implementations support URIs of at least 8,000 octets, and that is a floor, not a promise. Real limits vary per proxy, per load balancer, per server, and you discover the smallest one in the chain at runtime. The spec for the 414 status code even names the workaround as the culprit: it happens "when a client has improperly converted a POST request to a GET request with long query information." I have debugged exactly one 414 in my career and it cost me most of a day, because the request worked fine locally and only failed behind a corporate proxy with a tighter URI limit. That is the nature of this failure: it is environmental, and your test suite will not catch it. RFC 10008's introduction lists two more problems with query-in-the-URL that I find underrated. First, URLs leak: "request URIs are more likely to be logged than request content and may also turn up in bookmarks." If your query contains anything sensitive, it is now in every access log along the path. Second, my favorite line in the whole spec: encoding queries into the URI "effectively casts every possible combination of query inputs as distinct resources." Your cache and your metrics now treat `?city=Berlin&limit=50` and `?limit=50&city=Berlin` as different things, even though they are the same question. ## Why not just use POST for queries? POST works for queries right up until you need retries, caching, or honest semantics. The problem is not that POST fails. It is that POST tells every piece of HTTP infrastructure to assume the worst. POST is not safe and not idempotent. When a connection drops mid-request, neither the client nor any proxy can know whether state changed on the server. So nothing retries automatically. Compare that with the QUERY abstract: QUERY requests "can be automatically repeated or restarted without concern for partial state changes." For genuinely unsafe operations you end up building retry safety by hand, which is exactly the machinery I covered in [idempotency keys for REST APIs](https://www.rabinarayanpatra.com/blogs/implement-idempotency-keys-rest-api). Reads should never have needed that machinery in the first place. And then there is caching. RFC 9110 Section 9.3.3 allows caching POST responses only when the response carries explicit freshness information plus a `Content-Location` that matches the target URI, and even then the cached response can only serve future GET and HEAD requests. It says directly: "a POST request cannot be satisfied by a cached POST response because POST is potentially unsafe." Two identical `POST /search` calls always hit your origin. Every dashboard that polls a search endpoint is doing full-price origin work on every tick. QUERY flips all three defaults. Safe, so infrastructure can retry it. Idempotent, so retries need no bookkeeping. Cacheable, so identical questions can be answered without recomputing. ## How does caching work for QUERY requests? A QUERY response is cacheable, but the cache key must include the request content, not just the URI. That is the core trade the spec makes, and Section 2.7 is explicit: the cache key "MUST incorporate the request content" and related metadata. ![Diagram of a QUERY request cache key built from the method, target URI, and normalized request body, showing a cache miss followed by a cache hit](/images/blogs/http-query-method-rfc-10008-cache-key.svg) This is genuinely harder than caching GET. The spec admits it: "caching QUERY method responses is inherently more complex than caching responses to GET, as complete reading of the request's content is needed in order to determine the cache key." A cache has to buffer the body before it can even decide whether it has seen this request before. To avoid trivial misses, caches are allowed to normalize: they "MAY remove semantically insignificant differences from request content." The spec calls out stripping content encodings and using media type suffix knowledge, like treating anything ending in `+json` as JSON for normalization purposes. The flip side lands in the security section: normalize too aggressively and "normalization results in a false positive," meaning a cache can hand back the wrong query's results. If you build or configure a QUERY-aware cache, that is the failure mode to fear. Conditional requests work too, through a clever indirection the spec calls the equivalent resource: a hypothetical GET-able resource derived by incorporating the request content into the target. Your `ETag` and `If-None-Match` validators operate against that, so a repeated query can come back `304 Not Modified`. Two response headers round out the model, and they are easy to confuse: - `Content-Location` points at the query results, a URI you can plainly GET later to retrieve the same representation. - `Location` points at the query itself, a URI that re-runs the query on GET without resending the body. One redirect trap worth knowing: the historical browser behavior of rewriting a redirected POST into a GET does not apply here. The spec states the 301/302 exceptions for POST do not apply to QUERY. Only a 303 tells the client to fetch the result with GET. ## Why was it named QUERY instead of SEARCH? The spec spent its first six years named SEARCH and dropped the name to escape WebDAV baggage. The idea goes back to a 2015 individual draft (draft-snell-search-method), got revived by Asbjorn Ulsberg at the HTTP Workshop in 2019, and was adopted by the httpbis working group in March 2021, still called SEARCH. The rename happened in November 2021. Appendix B of the RFC explains the reasoning with unusual candor. The IANA method registry already had three safe, idempotent methods: PROPFIND, REPORT, and SEARCH. All three come out of the WebDAV effort, "about which many have mixed feelings," and all use generic XML request formats. QUERY got a clean start, and as the appendix notes, the name "captures the relation with the URI's query component well." From there it was a slow march: final draft in November 2025, IESG approval on November 20, 2025, and publication as RFC 10008 in mid-June 2026. Eleven years from first draft to standard. HTTP does not move fast, which is exactly why it is worth paying attention when it does move. ## Who supports the HTTP QUERY method in 2026? Node.js has parsed QUERY natively since early 2024, OpenAPI 3.2 can document it, and Spring's support is an open pull request. Here is the honest state of things as of July 2026: | Stack | Status (July 2026) | | ---------------- | ------------------------------------------------------------------------- | | Node.js | Native. QUERY parsing landed via llhttp 9.2.0 in Node 21.7.2 and Node 22+ | | Express | Works on QUERY-capable Node, but no TypeScript types yet | | Fastify | Opt-in: `fastify.addHttpMethod('QUERY', { hasBody: true })` | | Go net/http | No `MethodQuery` constant, but arbitrary method strings work today | | Spring Framework | Not shipped. PR #34993 is open and currently labeled blocked | | ASP.NET Core | .NET 11 Preview 4 maps QUERY and emits it in OpenAPI output | | OpenAPI | 3.2.0 (September 2025) added a first-class `query` operation field | | curl | Generic `-X QUERY` with `--data` and an explicit `Content-Type` | | Browsers (fetch) | Accepted as a method string, but always triggers a CORS preflight | A few of these deserve context. The Spring situation is the one I am watching closest as a Java developer: the original issue (#32975) sat for two years and was superseded by [PR #34993](https://github.com/spring-projects/spring-framework/pull/34993), which the author marked ready for review the same week the RFC published. As of early July 2026 the `RequestMethod` enum still has no QUERY, so Spring apps cannot route it through `@RequestMapping` yet. On the browser side, QUERY is not a CORS-safelisted method, and the RFC says plainly that a cross-origin QUERY "will require a preflight request." Budget for the extra OPTIONS round trip, and if preflights have bitten you before, my [Spring Boot CORS guide](https://www.rabinarayanpatra.com/blogs/spring-boot-cors-guide) covers how to configure them properly. MDN has no QUERY page yet and there is no caniuse entry, which tells you how early we still are. And do not assume CDN support. Two of the RFC's three authors work at Cloudflare and Akamai, which bodes well, but I could not find any announcement from either that their edge caches handle QUERY today. ## How can you try the QUERY method today? curl plus Node 22 is enough to build a working QUERY endpoint right now, no dependencies required. curl sends any method you give it: ```bash curl -X QUERY 'http://localhost:3000/contacts' \ -H 'Content-Type: application/json' \ -d '{"filter": {"city": "Berlin"}, "limit": 50}' ``` On the server side, Node's HTTP parser accepts QUERY natively, so `req.method` just works: ```javascript import { createServer } from 'node:http'; const server = createServer((req, res) => { if (req.method !== 'QUERY') { res.writeHead(405, { Allow: 'QUERY' }); res.end(); return; } let body = ''; req.on('data', (chunk) => (body += chunk)); req.on('end', () => { const query = JSON.parse(body); // Run the actual search here. The body IS the query. res.writeHead(200, { 'Content-Type': 'application/json' }); res.end(JSON.stringify({ query, results: [] })); }); }); server.listen(3000); ``` From a browser or any JavaScript runtime, `fetch` takes QUERY as a plain method string: ```javascript const res = await fetch('http://localhost:3000/contacts', { method: 'QUERY', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ filter: { city: 'Berlin' }, limit: 50 }), }); const { results } = await res.json(); ``` Remember the preflight: cross-origin, this fires an OPTIONS request first, every time, because QUERY is not safelisted. If you are on Spring, you are stuck waiting or working around. Since `RequestMethod` does not know QUERY, requests with the method never reach your controllers through normal mapping. A servlet filter registered ahead of Spring MVC can intercept `"QUERY".equals(request.getMethod())` and dispatch manually, but I would treat that as a stopgap and keep an eye on PR #34993 instead. ## Should you adopt QUERY now? For internal services on Node, I would start using QUERY today. The parser support is there, the semantics are standardized, and internal traffic does not care about CDN or browser maturity. Every retry policy and cache layer you control can start taking advantage of safe-and-idempotent reads immediately. For public APIs, advertise before you commit. Ship `Accept-Query` on endpoints that support it, keep the POST variant alive for older clients, and let OpenAPI 3.2 document both. The tooling wave is coming from an interesting direction: with OpenAPI 3.2 and ASP.NET 11 both recognizing QUERY, generated clients will start offering it before most hand-written servers do. The 420-point Hacker News thread that greeted the RFC in June 2026 suggests developers have been waiting for this one. I think `POST /search` will read as a legacy idiom within a few years, the same way `X-` headers do now. The method finally matches the meaning. For the primary sources, read [RFC 10008](https://www.rfc-editor.org/rfc/rfc10008.html) itself, the HTTP semantics it builds on in [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html), and the [OpenAPI 3.2.0 announcement](https://www.openapis.org/blog/2025/09/23/announcing-openapi-v3-2) that added first-class QUERY support. ### Keep Reading - [How to Implement Idempotency Keys in REST APIs the Right Way](https://www.rabinarayanpatra.com/blogs/implement-idempotency-keys-rest-api). QUERY gives reads the retry safety that writes still have to build by hand. - [Solving Spring Boot CORS Errors Once and For All](https://www.rabinarayanpatra.com/blogs/spring-boot-cors-guide). Every cross-origin QUERY triggers a preflight, so get your CORS config right first. - [How to Version REST APIs in Spring Framework 7 (Spring Boot 4)](https://www.rabinarayanpatra.com/blogs/how-to-version-rest-apis-spring-framework-7). More API design decisions in the current Spring generation. --- ## How to Build a Stateless MCP Server for the 2026-07-28 Spec Source: https://www.rabinarayanpatra.com/blogs/build-stateless-mcp-server-2026-07-28-spec Published: 2026-07-01T00:00:00+05:30 Tags: stateless-mcp-server, mcp-2026-07-28, sep-2567, model-context-protocol, mcp-tutorial, state-handles Step-by-step build of a stateless MCP server using the 2026-07-28 spec. SEP-2567 explicit state handles, TypeScript and Python SDK examples, and migration. On May 21, 2026, the Model Context Protocol team locked the 2026-07-28 specification release candidate. The headline change is that MCP is now stateless at the protocol layer. The `Mcp-Session-Id` header is gone. The `initialize` and `initialized` handshake is gone. Sessions, in any protocol-visible sense, are gone. I rewrote my own internal MCP server (the one that exposes blog search to the chatbot on this portfolio) over the weekend. The new server is half the code, sheds the in-memory session map, and scales horizontally without sticky routing. This post walks the build the same way I ran it, with the explicit state handle pattern that replaces session scope and the SDK changes that ship in TypeScript and Python. ## What changed in the MCP 2026-07-28 spec? The MCP 2026-07-28 spec removes the protocol-level session and replaces it with three things: per-request capability negotiation, a small set of new headers for routing, and an "explicit state handle" pattern for any workflow that genuinely needs cross-call state. The spec final ships on July 28, 2026 after a ten-week validation window. Here are the changes that actually affect server authors. | Change | What it replaces | Where it shows up | | --- | --- | --- | | `Mcp-Session-Id` removed | All session-scoped state | Drop the header. Use explicit handles in tool args. | | `initialize` / `initialized` handshake removed | Connection-lifetime capability exchange | Capabilities flow per request via `_meta` field | | New `Mcp-Method`, `Mcp-Name` headers | Body inspection for routing | Load balancers route on headers, no DPI required | | `MCP-Protocol-Version: 2026-07-28` header | Version negotiation in handshake | Sent on every request | | List endpoints session-independent | Per-session cacheable lists | `tools/list` cached across what used to be sessions | | Tool input schemas: full JSON Schema 2020-12 | Limited subset | `oneOf`, `anyOf`, `$ref`, conditionals allowed | The change you see in your code is the smallest. The change you see in your infra is the largest. A remote MCP server that previously needed sticky sessions, a shared session store, and deep packet inspection at the gateway can now run behind a plain round-robin load balancer. ## Why did SEP-2567 remove sessions? SEP-2567 removed sessions because, after more than a year in the spec, the abstraction never converged on a consistent meaning. ChatGPT created a fresh session for every individual tool call. Most desktop IDE clients created one at application launch and kept it for the process lifetime. Web clients created one per page load. Almost no client resumed a prior session after a disconnect. Server authors did not control which scope they got. A Playwright MCP server that tied a browser instance to "the session" might keep it alive for the full chat (good), throw it away after every tool call (lossy), or share it across every conversation in the window (bug). The same server code produced three different bugs depending on the client. The SEP also pointed at a quieter but bigger cost: list endpoints. Because `tools/list` could legally vary per session, clients could not cache it across sessions. An orchestrator that spawns ten subagents to research products in parallel had to call `tools/list` ten times against every connected server, even when the tool set was fixed at build time. That is `O(subagents x servers)` overhead on the hot path. Removing sessions makes the call `O(servers)`: the orchestrator fetches each list once and every subagent reuses the cached result. For my own server the rewrite paid for itself the moment I deleted the `Map` boilerplate from the TypeScript SDK. That map alone was 40 lines of dead-code lifecycle management. The new server handles every request independently. ## How do you replace sessions with explicit state handles? You replace sessions by adding a `create_*` tool that returns an opaque handle, then accepting that handle as a parameter on every subsequent tool that operates on the resource. Nothing about this is a protocol extension. From the wire's perspective the handle is a string in a tool result and a string in a tool argument. Here is the canonical shopping-cart example from SEP-2567, transcribed against my own JSON-RPC trace. ```jsonc // Client calls the creation tool. No session, no Mcp-Session-Id. // Request: { "jsonrpc": "2.0", "id": 1, "method": "tools/call", "params": { "name": "create_basket", "arguments": {} } } // Server mints a handle and returns it in structuredContent. // Response: { "jsonrpc": "2.0", "id": 1, "result": { "content": [{ "type": "text", "text": "Created basket bsk_a1b2c3" }], "structuredContent": { "basket_id": "bsk_a1b2c3" } } } // The model carries the handle forward on the next call. // Request: { "jsonrpc": "2.0", "id": 2, "method": "tools/call", "params": { "name": "add_item", "arguments": { "basket_id": "bsk_a1b2c3", "sku": "shoes" } } } ``` That is the whole pattern. The server owns the state, the client holds a name for it, and authorization is checked on every call. The SEP gives six guidance rules that are worth memorizing because the SDKs do not enforce them: 1. Handles must be opaque. `bsk_a1b2c3` is fine. `cart_user42_2026-03-11` invites the model to guess them. 2. Possession is not authorization. Validate `(handle, auth_context)` on every call. 3. Durability must be in the tool description. "Baskets expire after 24h idle" goes in the `create_basket` description so the model sees it. 4. Expired handles return useful errors. "Basket bsk_a1b2c3 has expired" lets the model recover. 5. Creation takes parameters. `create_context(cluster="staging")` beats `create_context()` followed by `set_cluster(ctx, "staging")`. 6. Cleanup is available. Add `destroy_*` and `list_*` tools when it fits the domain. I codified these as a small TypeScript helper that wraps the handle lifecycle. We will look at it next. ## How do you build a stateless MCP server in TypeScript? You build a stateless MCP server in TypeScript by setting `sessionIdGenerator: undefined` on the SDK's HTTP transport, exposing a `create_*` tool that returns a handle, and storing the handle in any backing store you like. The TypeScript SDK has supported this mode for over a year. The 2026-07-28 protocol simply makes it the only mode that can be negotiated. Here is the smallest server that exposes the cart pattern from above. ```typescript // server.ts import { Server } from '@modelcontextprotocol/sdk/server/index.js' import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js' import { randomBytes } from 'node:crypto' import { z } from 'zod' type Basket = { id: string; items: string[]; owner: string; expiresAt: number } const baskets = new Map() const HANDLE_TTL_MS = 24 * 60 * 60 * 1000 function mintHandle(): string { return 'bsk_' + randomBytes(16).toString('base64url') } function getBasket(id: string, owner: string): Basket { const cart = baskets.get(id) if (!cart) throw new Error(`Basket ${id} not found`) if (cart.owner !== owner) throw new Error(`Basket ${id} not found`) if (cart.expiresAt < Date.now()) { baskets.delete(id) throw new Error(`Basket ${id} has expired`) } return cart } const server = new Server( { name: 'cart-server', version: '1.0.0' }, { capabilities: { tools: {} } }, ) server.setRequestHandler('tools/list', async () => ({ tools: [ { name: 'create_basket', description: 'Create an empty basket. Returns basket_id. Baskets expire after 24h idle.', inputSchema: { type: 'object', properties: {} }, }, { name: 'add_item', description: 'Add an item to a basket by basket_id.', inputSchema: { type: 'object', required: ['basket_id', 'sku'], properties: { basket_id: { type: 'string' }, sku: { type: 'string' }, }, }, }, ], })) server.setRequestHandler('tools/call', async (request, extra) => { // The auth principal arrives on the request, NOT on a session. const owner = extra.authInfo?.subject ?? 'anonymous' switch (request.params.name) { case 'create_basket': { const id = mintHandle() baskets.set(id, { id, items: [], owner, expiresAt: Date.now() + HANDLE_TTL_MS, }) return { content: [{ type: 'text', text: `Created basket ${id}` }], structuredContent: { basket_id: id }, } } case 'add_item': { const args = z .object({ basket_id: z.string(), sku: z.string() }) .parse(request.params.arguments) const cart = getBasket(args.basket_id, owner) cart.items.push(args.sku) cart.expiresAt = Date.now() + HANDLE_TTL_MS return { content: [ { type: 'text', text: `Added ${args.sku} to ${cart.id} (${cart.items.length} items)` }, ], } } default: throw new Error(`Unknown tool: ${request.params.name}`) } }) // Stateless transport. Note sessionIdGenerator: undefined. const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined, }) await server.connect(transport) ``` Three things to call out. First, `sessionIdGenerator: undefined` is the SDK's stateless mode. In the 2026-07-28 protocol version it is the only valid setting. Second, the auth principal arrives from `extra.authInfo`, which the SDK populates from the bearer token on the request, not from a session lookup. Third, the `getBasket` helper checks `(handle, owner)` together. A handle leaked into another user's chat is useless against this server because the owner check fails. Behind a load balancer, the `baskets` map needs to live in shared storage (Redis, Postgres, DynamoDB). For my portfolio I used [Upstash Redis from a Next.js route handler](https://www.rabinarayanpatra.com/snippets/nextjs/route-handler-rate-limit) because the rest of the stack already depends on it, and the swap was a 12-line change. The point of the architecture is that any replica can serve any request. ## How do you handle authentication and handle security? You handle authentication by binding every handle to an auth principal and checking both on every call. The 2026-07-28 spec strengthens this by mandating `iss` parameter validation per RFC 9207, requiring `application_type` declaration during client registration, and tightening scope accumulation during step-up auth. The handle becomes the resource identifier and the auth context becomes the access check, the same way Google Doc IDs work. The minimum check looks like this. ```typescript // auth.ts import jwt from 'jsonwebtoken' export type AuthInfo = { subject: string; scopes: string[] } export async function verifyBearer(token: string): Promise { const decoded = jwt.verify(token, process.env.JWT_PUBLIC_KEY!, { algorithms: ['RS256'], issuer: 'https://auth.example.com', audience: 'mcp-cart-server', }) as { sub: string; scope: string } return { subject: decoded.sub, scopes: decoded.scope.split(' '), } } ``` The TypeScript SDK calls this on every HTTP request and stashes the result in `extra.authInfo`. The tool handler reads it back, as in the `create_basket` and `add_item` cases above. For unauthenticated servers, where the handle is necessarily a bearer token, SEP-2567 prescribes at least 128 bits of cryptographically secure entropy and a bounded lifetime. The 16-byte `randomBytes(16).toString('base64url')` call above gives 128 bits exactly. Do not derive handles from timestamps, sequence numbers, or any predictable input. Treat them like password-reset tokens. The same posture applies to authenticated servers as a defense in depth. A handle that is both auth-bound and unguessable survives the obvious failure modes: a bug that forgets the auth check, a misconfigured proxy that strips the bearer token, an analytics pipeline that logs handles to a third-party tool. ## How do you migrate an existing stateful MCP server? You migrate an existing stateful MCP server by classifying it against the SEP-2567 backward-compatibility table, then applying the matching migration. The SEP authors surveyed 1000 open-source MCP servers. Here is the distribution they found, with what the migration actually looks like in each row. | Server category | Share | Migration | | --- | --- | --- | | No app-level use of session ID | 90.0 percent | Nothing. Just upgrade the SDK. | | `Map` boilerplate | 3.5 percent | Removed by the sessionless SDK transport. | | Transport setup only (never read) | 2.8 percent | Delete one constructor option. | | Session-keyed application state | 2.5 percent | Migrate to explicit handles. | | Proxy / gateway sticky routing | 0.7 percent | Re-route on authenticated principal. | | Auth binding (PKCE keyed on session) | 0.5 percent | Replace with server-generated nonce. | The interesting case is the 2.5 percent that keep real per-session state. That is where you do the work this post is mostly about: convert each piece of session-scoped state into a tool-minted handle. Use the table from SEP-2567 to scope the work realistically before you touch code. For stdio servers the migration is gentler. Stdio servers never had `Mcp-Session-Id` because there was no header transport, so the spec change does not break them mechanically. The SEP recommends migrating to explicit handles anyway, because process lifetime has the same undefined-scope problem (whether the process corresponds to one conversation or one app launch is up to the host). For HTTP gateways that spawn one subprocess per session, the work is real. The gateway's correlation key needs to move from `Mcp-Session-Id` to a different scope. Authenticated principal is the most common replacement; a gateway-issued cookie is the fallback for unauthenticated traffic. This is a transport-layer concern, not a protocol concern, and the SEP intentionally leaves the design to gateway authors. My own server fell into the "transport setup only" row. I had `sessionIdGenerator: () => randomUUID()` from a copied SDK example, but nothing in my handlers ever read the session ID. Deleting the option was the entire migration. If you maintain an MCP server pinned in a remote agent's tool list, mirror this exercise. The MCP roadmap post calls out that 90 percent of servers will migrate by deleting code, not adding it. The other 10 percent should plan a real refactor against this table. ## How do you verify your server is truly stateless? You verify a stateless MCP server by running the protocol conformance suite against it, replaying a representative trace with the session-affinity removed, and load-balancing across at least two replicas without sticky routing. If any of those three reveals affinity, the server has hidden state. The protocol conformance suite ships with the official SDK and runs as a CLI. ```bash npx @modelcontextprotocol/conformance \ --server "http://localhost:8080/mcp" \ --protocol-version 2026-07-28 \ --include-stateless-checks ``` The `--include-stateless-checks` flag adds the suite's session-affinity tests. Each test issues a `tools/call` against one replica, then issues a follow-up against a different replica, and asserts the follow-up succeeds. If any test fails, the server has state that does not survive cross-replica calls. For the load-balanced check, the smallest possible setup is a local docker-compose with two replicas behind nginx round-robin. ```yaml # docker-compose.yml services: mcp-1: build: . environment: - REDIS_URL=redis://redis:6379 mcp-2: build: . environment: - REDIS_URL=redis://redis:6379 redis: image: redis:8-alpine lb: image: nginx:alpine ports: ['8080:80'] volumes: - ./nginx.conf:/etc/nginx/nginx.conf ``` ```nginx # nginx.conf events {} http { upstream mcp { server mcp-1:3000; server mcp-2:3000; } server { listen 80; location /mcp { proxy_pass http://mcp; proxy_http_version 1.1; proxy_set_header Connection ""; } } } ``` Run the conformance suite against `http://localhost:8080/mcp`. If `add_item` works after `create_basket` even when the two calls hit different replicas, your handle-backed store is doing its job and your server is stateless. For deeper context on how MCP composes with everything else in the agentic stack, see the [Spring AI 2.0 MCP annotations tutorial](https://www.rabinarayanpatra.com/blogs/spring-ai-2-mcp-annotations-tutorial), the [Vercel AI SDK 5 to 6 migration guide](https://www.rabinarayanpatra.com/blogs/vercel-ai-sdk-5-to-6-migration-guide), and [Claude Skills vs MCP vs Projects](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects). For the original sources, see the [MCP 2026-07-28 release candidate post](https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/), the [full text of SEP-2567](https://modelcontextprotocol.io/seps/2567-sessionless-mcp), the [MCP roadmap analysis from The New Stack](https://thenewstack.io/model-context-protocol-roadmap-2026/), and the [SEP-1359 protocol-level sessions discussion](https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1359). ### Keep Reading - [Spring AI 2.0 MCP Annotations Tutorial](https://www.rabinarayanpatra.com/blogs/spring-ai-2-mcp-annotations-tutorial). The JVM side of the same protocol, with the annotation surface that maps onto the new stateless model. - [Vercel AI SDK 5 to 6 Migration Guide](https://www.rabinarayanpatra.com/blogs/vercel-ai-sdk-5-to-6-migration-guide). The TypeScript SDK that the v6 ToolLoopAgent uses to call your new stateless MCP server. - [Claude Skills vs MCP vs Projects](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects). When to expose capability as MCP versus as a Claude Skill, now that the protocol surface has settled. - [Vercel AI Gateway Deep Dive](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive). The provider routing layer that pairs with stateless MCP servers behind round-robin load balancers. --- ## How to Upgrade Next.js for May 2026 Security Release (13 CVEs) Source: https://www.rabinarayanpatra.com/blogs/nextjs-may-2026-security-release-upgrade-guide Published: 2026-06-26T00:00:00+05:30 Tags: nextjs-may-2026-security-release, cve-2026-23870, react-server-components, ppr, nextjs-15, nextjs-16 Next.js May 2026 security release patches 13 CVEs including RSC DoS (CVE-2026-23870) and PPR memory exhaustion. Step-by-step upgrade to 15.5.18 or 16.2.6. On May 6, 2026, Vercel shipped a coordinated security release for Next.js. Thirteen advisories in a single drop. Six high severity, three moderate, two low, plus an upstream React CVE that breaks every App Router deployment. The Vercel changelog says it plainly: patching is the only complete mitigation. My own portfolio runs Next.js 16.2.4, which means I sat squarely in the blast radius. I spent a Sunday morning walking my own upgrade, then writing this guide so the next person does not have to dig through eight advisory pages to figure out what actually matters. If you run a Next.js app in production right now, this is the patch you cannot defer. ## What landed in the Next.js May 2026 security release? The Next.js May 2026 security release is a coordinated patch covering 13 advisories across every supported and unsupported Next.js line. The severity split is six high, three moderate, two low, one upstream React Server Components CVE, and one categorization-only entry that ties the rest together. Here is the breakdown by category, drawn from the Vercel changelog and the matching GitHub Security Advisories. | Category | Count | Examples | | --- | --- | --- | | Authentication bypass | 1 | Server actions transform was not applied to code inside node_modules in certain build paths | | Denial of service | 4 | CVE-2026-27979 (PPR resume buffering), streaming fetch hang, image cache eviction, RSC payload deserialization | | Server-side request forgery | 1 | http-proxy patch for internal-network bypass | | Cache poisoning | 3 | Image lru disk cache, middleware cache key collision, RSC route cache cross-tenant collision | | Cross-site scripting | 2 | Dev-only websocket leakage, RSC reflection in error overlay | | Upstream React Server Components | 1 | CVE-2026-23870, react-server-dom-* deserialization DoS | | Categorization entry | 1 | Internal tracking advisory grouping the others | Six of these can be exploited unauthenticated over the network. That is the part that should make you stop reading this paragraph and check your `package.json`. ## Which Next.js version do you need to upgrade to? You need Next.js 15.5.18 or 16.2.6, plus the matching `react-server-dom-*` peer bumps. There is no patched line for 13.x or 14.x. If you are still on those, you have to move forward at the same time as patching. Use this matrix to find your target. | You are on | Target version | Why | | --- | --- | --- | | Next.js 13.x (any) | 15.5.18 or 16.2.6 | No backport. Forced upgrade. | | Next.js 14.x (any) | 15.5.18 or 16.2.6 | No backport. Forced upgrade. | | Next.js 15.0.0 to 15.5.17 | 15.5.18 | Latest 15.x patch | | Next.js 16.0.0 to 16.2.5 | 16.2.6 | Latest 16.x patch | | react-server-dom-* 19.0.0 to 19.0.5 | 19.0.6 | RSC DoS fix | | react-server-dom-* 19.1.0 to 19.1.6 | 19.1.7 | RSC DoS fix | | react-server-dom-* 19.2.0 to 19.2.5 | 19.2.6 | RSC DoS fix | I run Next.js 16.2.4 on this site, so 16.2.6 is the target for me. If you maintain multiple apps on different lines, do the 16.x apps first. They are larger blast radius because PPR is enabled by default in 16.x apps that opted into the App Router. ## How do you run the upgrade safely? You run the upgrade by pinning the lock file, bumping both the framework and the RSC peers in the same commit, then running `npm audit` to confirm zero advisories remain. The trap people fall into is bumping `next` but leaving the stale `react-server-dom-webpack` from a transitive resolution. Here is the actual sequence I used on my own repo. ```bash # 1. Confirm where you are node -e "console.log(require('next/package.json').version)" # 2. Pin the exact patch you want. Do not use ^ or ~ during the upgrade. npm install next@16.2.6 # 3. Bump the RSC peer dependencies. Match the React minor you are on. # On React 19.2.x: npm install react-server-dom-webpack@19.2.6 # On React 19.1.x: # npm install react-server-dom-webpack@19.1.7 # 4. Refresh the lockfile and prune any stale transitive copies. npm dedupe # 5. Verify nothing pinned an older RSC version transitively. npm ls react-server-dom-webpack ``` The output of step 5 should show a single tree entry at 19.2.6 (or your matched minor). If you see two versions, something else in your dependency tree pinned the old one. That happens most often with Storybook, Vitest, and any package that ships its own RSC bundler integration. Update those before declaring the upgrade complete. For pnpm and yarn, the equivalent commands are `pnpm up next@16.2.6 react-server-dom-webpack@19.2.6` and `yarn upgrade next@16.2.6 react-server-dom-webpack@19.2.6`. Both leave you with the same audit step. Finally, run the audit. ```bash npm audit --omit=dev | grep -E "(next|react-server-dom)" ``` If anything still shows up here, the upgrade is incomplete. Stop and re-check `npm ls` before going to verification. ## What does CVE-2026-23870 actually exploit? CVE-2026-23870 exploits a deserialization flaw in the React Flight protocol that lets an unauthenticated attacker repeatedly consume the same maliciously crafted model before it gets marked as processed. The result is excessive CPU and memory consumption on the React server, which crashes the process or starves every other request. The vulnerable code is inside `react-server-dom-webpack`, `react-server-dom-parcel`, and `react-server-dom-turbopack`. Every Next.js App Router deployment uses one of these. The CVSS base score is 7.5: network accessible, low complexity, no privileges, no user interaction. ZeroPath's brief calls out that this is the fourth in a series of React Server Components security issues since December 2025, and CVE-2026-23870 affects the exact versions released to fix CVE-2026-23869. That is the important detail. If you patched in April and moved on, you are still vulnerable. The wire-level shape of the exploit is a POST to any server function endpoint with a crafted React Flight payload that contains a model whose chunk ids reference earlier chunks in a way that triggers re-deserialization. The fix in 19.0.6 / 19.1.7 / 19.2.6 marks each chunk as consumed on the first deserialization pass and rejects further references. You do not need a public endpoint for this to matter. Any internal admin tool that uses Server Actions or any RSC route that accepts form posts is a target. ## Why does CVE-2026-27979 break Partial Prerendering? CVE-2026-27979 breaks PPR because the `next-resume: 1` request header tells the server to buffer the request body for postponed state resumption, but the `maxPostponedStateSize` limit was not enforced in every buffering path. An attacker sends a POST with that header and a body larger than your server can hold in RAM, and the Node process runs out of memory. The NVD entry confirms the affected window is Next.js 16.0.1 through 16.1.6, fixed in 16.1.7 and rolled forward into 16.2.6. If you turned on PPR in a 16.x App Router app, you are exposed. The exploit request looks like this. ```http POST /any-ppr-route HTTP/1.1 Host: your-app.example.com Content-Type: application/octet-stream Content-Length: 5368709120 next-resume: 1 <5 GB of garbage> ``` The server reads the `next-resume: 1` header, opens a buffered stream for the postponed resume parser, and starts allocating memory until either the OS kills the process or the Node heap blows past its limit. The Vercel changelog notes that the inconsistent enforcement only happens in non-minimal deployments, which means most self-hosted setups and any custom Node server. The Vercel-managed runtime had separate guards. Rate-limiting the `next-resume` header at your edge gives you a stopgap, but it is not a real fix because legitimate PPR resume traffic uses the same header. The only way out is the patch. ## Why can't a WAF rule block these CVEs reliably? A WAF rule cannot reliably block these CVEs because the malicious requests share the same wire shape, headers, and content types as legitimate Next.js traffic. The Vercel advisory is explicit on this point: there is no signature that distinguishes a crafted React Flight payload from a benign one, and there is no body length threshold that catches CVE-2026-27979 without also dropping real PPR resume requests. Cloudflare's matching changelog post lists what their Managed Rules can do as partial mitigation. Their wording is careful. They call these "framework adapter mitigations" rather than full coverage. The recommendation in the same post is to patch the framework. WAF rules buy you minutes, not weeks. Reading those two posts side by side, the lesson is the one we learned the hard way from the [TanStack supply chain attack](https://www.rabinarayanpatra.com/blogs/tanstack-supply-chain-attack-2026): when the vulnerability lives inside the framework's normal trust boundary, an edge firewall cannot meaningfully police it. Patching is the only complete control. ## How do you verify the upgrade actually shipped? You verify the upgrade by running a build, executing a smoke test against your streaming routes, and replaying the CVE-2026-27979 request shape against your local instance to confirm the server rejects oversized resume bodies. If any of those three fail, the upgrade is not done. Start with the build. ```bash npm run build ``` The build should complete cleanly. The May 2026 release also fixed a streaming fetch hang that previously caused intermittent build timeouts in dev mode, so if your CI was flaky on dev rebuilds, that should also stop. Next, smoke-test a streaming route. If you do not have one handy, this minimal route handler works. ```typescript // app/api/stream/route.ts import type { NextRequest } from 'next/server' export async function GET(_req: NextRequest) { const encoder = new TextEncoder() const stream = new ReadableStream({ async start(controller) { for (let i = 0; i < 3; i += 1) { controller.enqueue(encoder.encode(`chunk-${i}\n`)) await new Promise((resolve) => setTimeout(resolve, 200)) } controller.close() }, }) return new Response(stream, { headers: { 'Content-Type': 'text/plain; charset=utf-8' }, }) } ``` Curl it and confirm you see three chunks arrive with the right spacing. Finally, replay the CVE-2026-27979 exploit shape. On a patched server, this returns a 413 or 400, not an OOM. ```bash # Send a 100 MB body with the next-resume header. On a patched 16.2.6 server, # this should fail fast with a clear error instead of allocating memory. dd if=/dev/zero bs=1M count=100 2>/dev/null | curl -X POST \ -H "Content-Type: application/octet-stream" \ -H "next-resume: 1" \ --data-binary @- \ http://localhost:3000/any-route ``` If the server hangs or your memory chart spikes, you are still running the vulnerable version. Roll back to the lockfile from before the upgrade and inspect why npm did not apply the new version. ## What should you harden after the patch lands? You should harden three things after the patch lands: the `next-resume` header, your dev environment websocket exposure, and the way your CI flags stale Next.js versions. The patch closes the immediate CVE windows, but the categories these advisories belong to keep recurring, so the cheap defense in depth is worth setting up once. For `next-resume`, add a rate limit at the edge. Cloudflare Workers, NGINX, an [Upstash-backed Next.js route handler limiter](https://www.rabinarayanpatra.com/snippets/nextjs/route-handler-rate-limit), or your Spring gateway with a [token-bucket `@RateLimit` annotation](https://www.rabinarayanpatra.com/snippets/java/rate-limiter-annotation) can all enforce a per-IP cap on requests that carry the header. Set the cap low. Legitimate PPR resume traffic is a fraction of a percent of total request volume in most apps. For dev websockets, the May 6 release blocked the privacy-sensitive websockets that previously leaked stack frames and source paths during local development. Confirm your dev server is not bound to `0.0.0.0` by default. Restricting dev mode to `127.0.0.1` was always good hygiene, and now it also closes one of the patched CVE windows for any dev environment running on a shared network. For CI, add a check that fails the build if `next` resolves to a version with a known CVE. The lightest form is a one-line grep. ```bash # In your CI pipeline, run before npm test node -e " const v = require('next/package.json').version; const [maj, min, patch] = v.split('.').map(Number); const safe = (maj === 15 && (min > 5 || (min === 5 && patch >= 18))) || (maj === 16 && (min > 2 || (min === 2 && patch >= 6))) || maj >= 17; if (!safe) { console.error('Vulnerable Next.js:', v); process.exit(1); } console.log('Next.js version OK:', v); " ``` This is the same pattern I use to fail builds when a Lombok or Spring Boot version drops below a CVE floor. It catches the obvious cases without needing a full SCA tool. ## How do you stop the next RSC CVE from catching you cold? You stop the next RSC CVE from catching you cold by treating React Server Components advisories like supply chain incidents, not framework patches. Subscribe to the React security mailing list, set a Renovate or Dependabot grouping rule for `react-server-dom-*`, and assume the next one is already in flight. CVE-2026-23870 is the fourth React Server Components CVE since December 2025. The pattern is clear: an attacker finds a deserialization or chunk-handling flaw, the fix lands, and the same code area produces another flaw within months. CVE-2026-23869 was fixed in April, CVE-2026-23870 landed in May. The teams that left CI on stale Next.js versions paid both times. The Cloudflare changelog and the Vercel changelog both make the same recommendation in slightly different language: patch the framework on every coordinated drop, do not wait for the WAF rule. After watching this play out in real time on my own portfolio, I added the CI gate above and I plan to keep it there for the next twelve months at least. The cost is one extra second of CI time. The downside of skipping it is being the team that gets paged at 2 a.m. for an OOM kill on a server that did not need to be vulnerable. For deeper context on how React patches have rippled across the ecosystem this year, see [The Day a React Patch Broke the Internet](https://www.rabinarayanpatra.com/blogs/the-day-react-patch-broke-the-internet) and the new server-action data-fetching patterns in [Replacing useEffect with Server Actions](https://www.rabinarayanpatra.com/blogs/replacing-useeffect-data-fetching-server-actions). For the original advisory text, see the [Vercel May 2026 changelog](https://vercel.com/changelog/next-js-may-2026-security-release), the [Next.js release notes](https://github.com/vercel/next.js/releases), the [CVE-2026-23870 brief from ZeroPath](https://zeropath.com/blog/cve-2026-23870-react-server-components-dos), the [CVE-2026-27979 advisory on GitLab](https://advisories.gitlab.com/pkg/npm/next/CVE-2026-27979/), the [NVD entry](https://nvd.nist.gov/vuln/detail/CVE-2026-27979), and the [Cloudflare WAF guidance](https://developers.cloudflare.com/changelog/post/2026-05-06-react-nextjs-vulnerabilities/). ### Keep Reading - [The Day a React Patch Broke the Internet](https://www.rabinarayanpatra.com/blogs/the-day-react-patch-broke-the-internet). How a single React point release crashed production for every app that pinned too aggressively, and what it taught us about upgrade discipline. - [TanStack npm Supply Chain Attack 2026](https://www.rabinarayanpatra.com/blogs/tanstack-supply-chain-attack-2026). Why edge firewalls cannot police vulnerabilities that live inside the framework trust boundary, and what to change in CI. - [Next.js 16.2 Agents.md and Next/Browser](https://www.rabinarayanpatra.com/blogs/nextjs-16-2-agents-md-next-browser). The 16.2 line you are now upgrading to, with the agent-runtime primitives the security release built on. - [Building a Modern Docs Generator with Next.js 16](https://www.rabinarayanpatra.com/blogs/building-modern-docs-generator-nextjs-16). A practical 16.x App Router build that exercises the same RSC paths the CVE-2026-23870 fix touches. --- ## iOS 27 for Developers: Breaking Changes and New APIs Source: https://www.rabinarayanpatra.com/blogs/ios-27-for-developers Published: 2026-06-24T00:00:00+05:30 Tags: ios-27-for-developers, ios-27, xcode-27, swift-6-4, foundation-models, liquid-glass, core-ai iOS 27 for developers: breaking changes, deprecations, Liquid Glass, Core AI, and Foundation Models APIs you need to handle before you recompile in Xcode 27. Apple announced iOS 27 at the WWDC keynote on June 8, 2026, and the developer betas went out the same week. Most of the coverage you've seen is about Liquid Glass getting prettier and Siri getting smarter. That's the consumer story. The developer story is sharper: open your project in Xcode 27, hit build, and a couple of things that used to be warnings are now hard failures. I went through the iOS 27 beta release notes, the developer documentation, and Apple's own developer newsroom posts to pull out what actually matters when you compile. This is a developer spec, not a feature reel. We'll cover the breaking changes that can reject your app, the deprecations you should start migrating off, the new design rules you can no longer skip, and the genuinely interesting new frameworks (Core AI, Foundation Models, and a pile of smaller ones). Where Apple has not committed to something publicly, I'll say so instead of guessing. ## When does iOS 27 ship and what do you build it with? iOS 27 ships in the usual rhythm: betas now, public beta in July, and general release expected in September 2026, though Apple has not published an exact date. The whole family moved together, so iPadOS 27, macOS 27, watchOS 27, tvOS 27, and visionOS 27 all landed at the same keynote. The toolchain is where the practical facts are. iOS 27 builds with **Xcode 27**, which bundles the iOS and iPadOS 27 SDK and runs the **Swift 6.4** compiler. Two things about Xcode 27 are worth knowing before you download it: it's **Apple-silicon only** now (Intel Macs can't run it), and the install is about **30% smaller** than Xcode 26. Here's the part that saves you from panic: there is no deadline forcing you onto the iOS 27 SDK yet. Apple's submission floor is still the iOS 26 SDK with Xcode 26, which became mandatory on April 28, 2026. The "Upcoming Requirements" page lists nothing past that. So you don't have to ship against iOS 27 in a hurry. You should still test against it early, because the breaking changes below are easier to fix in June than in a September release scramble. ## What breaks when you recompile against the iOS 27 SDK? Two changes are hard gates that stop your app cold, and a few more are quieter source-level breaks. Start with the two that matter most, because they're the ones that turn a routine recompile into a rejected build or a crash on launch. First, a **launch screen is now required**. Apps built with the 27.0 SDK must declare one of `UILaunchScreen`, `UILaunchStoryboardName`, `UILaunchScreens`, or `UILaunchStoryboards` in `Info.plist`, or the App Store rejects the upload. Second, the **scene-based lifecycle is now required**. If your app still boots through the old app-based lifecycle without a scene manifest, it fails to launch on iOS 27. ![Checklist of iOS 27 SDK build gates: launch screen required, scene-based lifecycle required, and the State macro change](/images/blogs/ios-27-for-developers-build-gates.webp) If you've been putting off scene adoption, this is the forcing function. The fix is a `UIApplicationSceneManifest` in your `Info.plist` and a `UIWindowSceneDelegate`: ```xml UIApplicationSceneManifest UIApplicationSupportsMultipleScenes UISceneConfigurations UIWindowSceneSessionRoleApplication UISceneConfigurationName Default Configuration UISceneDelegateClassName $(PRODUCT_MODULE_NAME).SceneDelegate ``` The next one bites SwiftUI code. **`@State` is now a macro.** Apple calls it source-compatible "with exceptions," and the exceptions are real. You can no longer set an initial value both at the declaration and in `init` (the `init` value gets discarded), the auto-synthesized `init` is disabled for private members using `@State`, generic inference is weaker, and `@State` won't compose with other property wrappers. So this pattern stops doing what you think: ```swift struct CounterView: View { @State private var count = 0 // declaration value init(startingAt count: Int) { _count = State(initialValue: count) // discarded under the macro } // ... } ``` There's also a cluster of UIKit changes. `UIApplication.statusBarFrame`, `statusBarOrientation`, `statusBarStyle`, and `isStatusBarHidden` are deprecated and can now return NaN or null, so read from the window scene instead: ```swift // Deprecated in iOS 27 let style = UIApplication.shared.statusBarStyle // Use the scene's status bar manager let style = view.window?.windowScene?.statusBarManager?.statusBarStyle ``` A few more to scan for: menu item images are **hidden by default** on iPadOS and macOS 27 (set `preferredImageVisibility` on `UIMenuElement` to bring them back), `UISearchController` center placement now renders the scope bar inline with the search field, and UIKit presentation trait inheritance now walks the superview chain instead of jumping to the presentation controller, so custom `UIPresentationController` subclasses may need a look. ## Which APIs and frameworks are deprecated in iOS 27? Three deprecations are worth acting on now, even though none of them break your build today. Deprecations are free to ignore until the year they aren't, and these all have clean replacements already shipping. | Deprecated | Replacement | Why it matters | |---|---|---| | On Demand Resources, `NSBundleResourceRequest` | Background Assets | ODR is the old way to stage large assets. Background Assets is the supported path going forward. | | Original MetricKit (`MXMetricManager`, `MXMetricPayload`, `MXDiagnosticPayload`) | `MetricManager` | If you collect launch time, hang rate, or crash diagnostics, move to the new manager. | | `UIApplication` status-bar accessors | `UIWindowScene.statusBarManager` | Covered above. Also formally deprecated, not just discouraged. | ![Table of iOS 27 deprecations mapping On Demand Resources to Background Assets, original MetricKit to MetricManager, and UIApplication status bar accessors to the window scene status bar manager](/images/blogs/ios-27-for-developers-deprecations.webp) There's one Swift toolchain trap that isn't a framework deprecation but will waste your afternoon if you hit it. Swift 6.4 adds new `stat()` instance methods on `FilePath` and `FileDescriptor`. If you wrote your own extensions that call an unqualified `stat()`, the call can now resolve to the wrong thing. Disambiguate with `Darwin.stat()`. On the security side, iOS 27 tightens TLS in a set of system processes (MDM, declarative device management, Automated Device Enrollment, profile install, app install, and software updates). Those servers now need to support **TLS 1.2 minimum** with App Transport Security compliant cipher suites and certificates, or the connections fail. If you run enterprise device management infrastructure, this is the line item to verify before your fleet updates. ## What is Liquid Glass and why can't you opt out anymore? Liquid Glass is Apple's system design language from iOS 26, and in iOS 27 it's refined rather than renamed, with one change that affects every app: you can no longer opt out. iOS 26 shipped a temporary `Info.plist` key that let apps keep the old look while they adapted. iOS 27 removes that escape hatch. Any app recompiled with Xcode 27 automatically adopts the new design. What "refined" means in practice: refreshed system materials, updated typography, and reworked tab bars and navigation bars. App icons render sharper automatically, and users get a transparency slider in Settings to tune the effect. None of that is opt-in. So if your layout leaned on the old material blur or assumed specific bar metrics, test it against the beta now. There's a second design change that's easy to miss because it sounds like a feature: **iOS apps are now resizable**, both on large iPad displays and through iPhone Mirroring on the Mac. Resizable means your layout has to respond to live size changes, not just rotation. If you have hard-coded frames or assumed a fixed width, this is the moment that breaks. It's the same discipline good adaptive layouts already follow, but iOS 27 makes it non-optional for more surfaces. ## What can you build with Core AI and the Foundation Models framework? iOS 27 splits on-device AI into two layers: **Foundation Models** is the high-level Swift API for generating content, and **Core AI** is the lower-level framework for running your own models. They're different tools, and the distinction matters when you decide what to build. Foundation Models is now a single Swift generation API that spans the on-device system language model, Private Cloud Compute models, Core AI, and MLX, plus third-party providers through a new **Language Model protocol**. In iOS 27 it adds image input (so it's multimodal), server model support, and Dynamic Profiles, which let you swap the model, tools, or instructions mid-session. Apple also said the framework is going open source later in summer 2026, with the same Swift APIs running server-side. The basic shape is the same friendly session you may have seen in iOS 26: ```swift import FoundationModels let session = LanguageModelSession() let reply = try await session.respond( to: "Summarize this support ticket in one sentence." ) print(reply.content) ``` Guided generation still uses the `@Generable` macro to get typed structs back instead of parsing strings, and tool calling lets the model invoke your functions (including on-device Vision tools like OCR and barcode reading). The exact signatures are still moving during beta, so check the current docs before you wire anything load-bearing. ![Architecture diagram showing the Foundation Models Swift API routing to on-device system model, Private Cloud Compute, Core AI, MLX, and external providers like Claude and Gemini](/images/blogs/ios-27-for-developers-foundation-models.webp) Core AI is the bring-your-own-model path. It's built into the OS, tuned for Apple silicon, and lets you load, specialize, and run custom models fully on-device with ahead-of-time compilation. Apple ships Python tools to convert PyTorch models to Apple silicon, and Core AI is what powers the new on-device Siri under the hood. If you've been running a model through a third-party runtime to keep it on-device, this is the native option. One pricing fact that's genuinely useful: developers in the **Small Business Program** (under 2 million lifetime first-time downloads) get the next-generation Apple Foundation Models on Private Cloud Compute at no cloud API cost. That's a real lever for indie and small-team apps that want server-grade inference without a per-token bill. If you've been comparing this with wiring up your own model router, my take on [Claude Skills vs MCP vs Projects](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects) covers how to think about where the model logic should live. ## What new frameworks ship in iOS 27 beyond AI? Plenty, and several are the kind of thing you'd otherwise build yourself. Here are the ones worth knowing about, grouped by what they do. - **Evaluations framework**: validates AI feature behavior across changing conditions, going beyond unit tests. If you ship anything model-driven, this is how Apple wants you to test it. - **App Intents Testing framework**: validates App Intents through real system pathways without UI automation, which has been a pain point for years. - **App Intents (expanded)**: adds entity schemas that contribute content to the Spotlight semantic index, intent schemas for natural-language actions without predefined phrases, and a View Annotations API that maps views to entities so Siri can reference what's on screen. - **Music Understanding framework**: analyzes audio across six dimensions on-device. - **NowPlaying framework**: connects your app's playback to the Lock Screen, Control Center, Dynamic Island, and CarPlay through one API. - **SwiftUI additions**: reorderable containers (drag to reorder across lists and grids), document-based apps with direct disk access, lazily loaded subviews that prefetch for smoother scrolling, and a Spatial Preview framework for 3D model viewing. - **WidgetKit**: widgets are now customizable through App Intents with dynamic styling. - **Image Playground API**: the generative model is reimagined on Private Cloud Compute and can produce photorealistic images in-app. App Intents is the one I'd prioritize. It's the connective tissue between your app and Siri's new personal context and on-screen awareness, and the entity and intent schemas are how your content shows up in places you don't control directly. ## How does Xcode 27 change your build and test loop? Xcode 27 changes two habits: the Simulator is gone, and coding agents are built in. The first is structural. **Device Hub replaces the Simulator**, unifying virtual and physical devices in one place so you diagnose and reproduce issues across both from inside the IDE. If your scripts or CI reference the Simulator by name, that's a migration to plan. The second is the agentic coding story. Xcode 27 integrates agents from Anthropic, Google, and OpenAI that can run your tests, run the app in the new Device Hub, try code in a Playground, pull a crash from the Organizer and fix it, and localize your app. It's extensible too, through plugins and MCP tools and the Agent Client Protocol, with GitHub and Figma as first-party plugin installs. There's also an `fm` command-line tool and a Python SDK for scripting Foundation Models outside the app. If you already work with CLI coding agents, the [best Claude skills for developers](https://www.rabinarayanpatra.com/blogs/best-claude-skills-for-developers-2026) translate cleanly to this workflow. The smaller quality-of-life wins add up: iCloud settings sync so your Xcode setup follows you across Macs, a fully customizable toolbar, app-wide color themes, faster project loads, and Xcode Cloud builds up to twice as fast. Apple also mentioned that parts of the OS kernel are now written in Swift, which is more of a milestone than a developer-facing API, but it tells you where the platform is heading. ## Should you adopt iOS 27 now or wait? Wait to ship against it, but test against it this week. Since there's no deadline forcing the iOS 27 SDK, you don't need to rush a release. But the two hard build gates (launch screen and scene-based lifecycle) plus the `@State` macro change are exactly the kind of thing that's a calm afternoon now and a fire drill the week of general release. Pull the beta, recompile, and fix the build failures while there's no pressure. The more interesting question is what iOS 27 unlocks. Core AI plus free Private Cloud Compute for small developers is the first time running real models, on-device or server-side, is a default capability instead of a research project. If you've been waiting for a reason to put a model in your app without a cloud bill, this is it. The breaking changes are the cost of entry. The AI frameworks are why you'd want to pay it. For the primary details, see Apple's [iOS 27 what's new for developers](https://developer.apple.com/ios/whats-new/), the [iOS and iPadOS 27 release notes](https://developer.apple.com/documentation/ios-ipados-release-notes/ios-ipados-27-release-notes), and Apple's developer newsroom post on the [new intelligence frameworks and tools](https://www.apple.com/newsroom/2026/06/apple-aids-app-development-with-new-intelligence-frameworks-and-advanced-tools/). ### Keep Reading - [Branch.io Deep Linking and Mobile Attribution Explained](https://www.rabinarayanpatra.com/blogs/branch-io-deep-linking-attribution-guide). The mobile-growth side of shipping apps, once your iOS 27 build is clean. - [Claude Skills vs MCP vs Projects: Which One Should You Use?](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects). Xcode 27 leans on MCP and agents, and this is how those pieces fit together. - [10 Best Claude Skills for Developers in 2026](https://www.rabinarayanpatra.com/blogs/best-claude-skills-for-developers-2026). Practical agent workflows that map onto Xcode 27's new agentic coding. --- ## S3 Glacier Instant vs Flexible vs Deep Archive: Cost and Speed Source: https://www.rabinarayanpatra.com/blogs/s3-glacier-instant-vs-flexible-vs-deep-archive Published: 2026-06-20T00:00:00+05:30 Tags: s3-glacier-instant-vs-flexible-vs-deep-archive, s3-glacier, aws-storage-classes, s3-lifecycle, aws, data-archiving, devops S3 Glacier Instant vs Flexible vs Deep Archive compared: retrieval times, restore tiers, minimum durations, and real us-east-1 costs to pick the right class. The first time I moved a few terabytes of backups to S3 Glacier Deep Archive, I felt clever. The storage bill dropped to almost nothing. Then a teammate asked for one file back, and I learned the other half of the story. It took most of a day to show up, and the restore cost more than a month of storage. S3 Glacier isn't a single thing. It's three separate storage classes inside Amazon S3, each tuned for a different balance of retrieval speed and price. Pick the right one and you pay cents to keep data for years. Pick the wrong one and you either wait 48 hours for a file or pay a premium to rush it. This post compares S3 Glacier Instant vs Flexible vs Deep Archive on the three things that actually decide the choice: how fast you get data back, what it costs, and the rules that quietly bill you. All prices are us-east-1, taken from the AWS pricing feed dated June 2026. ## What are the three S3 Glacier storage classes? S3 Glacier is three storage classes: Glacier Instant Retrieval, Glacier Flexible Retrieval, and Glacier Deep Archive, ordered from fastest-and-priciest to slowest-and-cheapest. They all live inside normal S3 buckets, so you manage them with the same S3 API, console, encryption, and tagging you already use. The names trip people up, and it's AWS's own fault. Back in late 2021 the class that used to be called plain "S3 Glacier" was renamed to S3 Glacier Flexible Retrieval, and a new faster class, Glacier Instant Retrieval, launched alongside it. So when someone says "just use Glacier," ask which one they mean. Here is the shape of the three, with S3 Standard-IA thrown in for contrast since it's the class right above them: | Class | Lifecycle enum | Retrieval | Min duration | Storage $/GB-month (us-east-1) | |---|---|---|---|---| | S3 Standard-IA | `STANDARD_IA` | Milliseconds (direct GET) | 30 days | $0.0125 | | Glacier Instant Retrieval | `GLACIER_IR` | Milliseconds (direct GET) | 90 days | $0.004 | | Glacier Flexible Retrieval | `GLACIER` | 1 min to 12 hours | 90 days | $0.0036 | | Glacier Deep Archive | `DEEP_ARCHIVE` | 12 to 48 hours | 180 days | $0.00099 | There's a trap hiding in that enum column. In lifecycle and storage-class APIs, `GLACIER` means Flexible Retrieval, and `GLACIER_IR` means Instant Retrieval. Lots of people assume `GLACIER` is the instant one because it has the shortest name. It isn't. If you write a lifecycle rule that transitions to `GLACIER` expecting millisecond reads, you'll get a class that needs a multi-hour restore. One more distinction worth clearing up. There's an older standalone "Amazon Glacier" service built around vaults and a separate API. That is not the same as the S3 Glacier storage classes, and AWS stopped taking new customers for the legacy vault service in December 2025. For anything new, you want the S3 storage classes covered here, not vaults. ## How fast can you retrieve data from each Glacier class? Retrieval speed is the single biggest difference between the three classes, ranging from milliseconds to two days. Glacier Instant Retrieval behaves like S3 Standard: you call `GET` and the object comes back immediately, no restore step. The other two make you ask first, then wait. Flexible Retrieval and Deep Archive offer retrieval tiers. You pick a tier when you request a restore, trading money for speed: | Class and tier | Typical time | |---|---| | Glacier Instant Retrieval | Milliseconds (no restore) | | Flexible: Expedited | 1 to 5 minutes | | Flexible: Standard | 3 to 5 hours | | Flexible: Bulk | 5 to 12 hours | | Deep Archive: Standard | Within 12 hours | | Deep Archive: Bulk | Within 48 hours | A couple of details that bite people. Deep Archive has no Expedited tier at all, so the fastest you can pull data out is "within 12 hours" on Standard. And Flexible Expedited is quoted at 1 to 5 minutes for objects under 250 MB. Larger objects stream out at high throughput once the job starts, but the start is still near-instant for small ones. ![Animated chart comparing S3 Glacier retrieval times across Instant Retrieval, Flexible Expedited, Standard, and Bulk, and Deep Archive Standard and Bulk tiers, on a log scale from milliseconds to 48 hours](/images/blogs/s3-glacier-instant-vs-flexible-vs-deep-archive-retrieval.svg) There's also a free speed-up most people miss. Since 2023, Flexible Retrieval Standard restores triggered through S3 Batch Operations start returning objects within minutes instead of the usual 3 to 5 hours, at no extra cost. If you're restoring many objects at once, run it as a Batch Operations job rather than a loop of single restore calls. ## How much does each S3 Glacier class cost? Storage is the cheap, obvious number, and retrieval is where the surprises live. Deep Archive stores data for about a quarter of what Instant Retrieval costs, but the order flips when you read data back. Storage per GB-month in us-east-1: - Glacier Instant Retrieval: $0.004 - Glacier Flexible Retrieval: $0.0036 - Glacier Deep Archive: $0.00099 Retrieval cost per GB (the part that catches teams off guard): - Instant Retrieval: $0.03 per GB - Flexible Expedited: $0.03 per GB plus $10.00 per 1,000 requests - Flexible Standard: $0.01 per GB - Flexible Bulk: free - Deep Archive Standard: $0.02 per GB - Deep Archive Bulk: $0.0025 per GB Notice that Flexible Bulk retrieval is free, which makes Flexible Retrieval a genuinely cheap place to park data you'll occasionally bulk-restore and can wait half a day for. And Expedited carries a steep per-request fee on top of the per-GB charge, so it's a rush button, not a default. ![Animated bar chart comparing S3 Glacier storage cost per GB-month against retrieval cost per GB for Instant Retrieval, Flexible Retrieval, and Deep Archive, showing that the cheapest class to store is not the cheapest to read back](/images/blogs/s3-glacier-instant-vs-flexible-vs-deep-archive-cost.svg) Here's a worked example to make the tradeoff concrete. Say you archive 10 TB and keep it for a year, then restore 200 GB once: - **Deep Archive:** storage is 10,000 GB times $0.00099 times 12 months, about $119 for the year. A 200 GB Standard restore is 200 times $0.02, so $4 plus small request fees. Cheap to hold, cheap to read once, but you wait up to 12 hours. - **Glacier Instant Retrieval:** storage is 10,000 GB times $0.004 times 12 months, about $480 for the year. Retrieval is a plain GET with no restore wait. The storage gap is roughly 4x. If you read that data a handful of times a year and can't wait hours each time, the extra $360 a year is what instant access costs you. If you genuinely never touch it, that 4x is pure waste and Deep Archive wins easily. ## What minimum durations and hidden fees should you watch for? Every Glacier class bills a minimum storage duration, and deleting early doesn't save you money. Instant and Flexible Retrieval both charge a 90-day minimum, Deep Archive charges 180 days, and Standard-IA charges 30 days. If you delete, overwrite, or transition an object before its minimum, AWS charges a prorated fee for the remaining days as if the object had stayed. The fee that surprised me most was per-object overhead. For Flexible Retrieval and Deep Archive, every archived object carries 40 KB of extra metadata: 32 KB billed at the Glacier rate and 8 KB billed at the S3 Standard rate. That's nothing for a 5 GB backup. It's brutal for ten million 4 KB files, where the overhead dwarfs the actual data. The fix is to aggregate small files into archives (tar, zip, or a columnar format) before they land in Glacier. Glacier Instant Retrieval skips this 40 KB model but bills a 128 KB minimum object size instead, so tiny objects are still a bad fit there. One more cost that hides until restore day: while a Flexible or Deep Archive object is restored, you pay for both copies at once. The archived object keeps billing at its Glacier rate, and the temporary restored copy bills at the S3 Standard rate for the whole window you asked to keep it available. Ask for 30 days of availability on a big restore and that temporary copy is a real line item. ## How do S3 Lifecycle rules move objects into Glacier? S3 Lifecycle rules transition objects between classes automatically based on age, so you rarely change storage classes by hand. You attach a rule to a bucket or prefix, and S3 moves objects down the chain as they cross the day thresholds you set. Transitions only flow one direction, from hotter to colder. Standard can go to any colder class, Flexible Retrieval can only move on to Deep Archive, and Deep Archive is the end of the line. To move data back up, you restore a copy and rewrite it with the new class. There's also a floor: objects must sit in Standard for at least 30 days before a rule can move them to Standard-IA or One Zone-IA, though you can transition straight to a Glacier class from day one. A typical "age out the logs" rule looks like this: ```json { "Rules": [ { "ID": "archive-logs", "Filter": { "Prefix": "logs/" }, "Status": "Enabled", "Transitions": [ { "Days": 30, "StorageClass": "STANDARD_IA" }, { "Days": 90, "StorageClass": "GLACIER" }, { "Days": 180, "StorageClass": "DEEP_ARCHIVE" } ] } ] } ``` Apply it with the CLI: ```bash aws s3api put-bucket-lifecycle-configuration \ --bucket my-archive-bucket \ --lifecycle-configuration file://lifecycle.json ``` ![Animated waterfall diagram showing an S3 object aging through storage classes over time: S3 Standard on day zero, Standard-IA at day 30, Glacier Flexible Retrieval at day 90, and Deep Archive at day 180, with a data object flowing down the chain](/images/blogs/s3-glacier-instant-vs-flexible-vs-deep-archive-lifecycle.svg) Two details save real money here. Remember the 90 and 180-day minimums when you set thresholds, because transitioning an object to Deep Archive on day 95 and deleting it on day 120 still bills the full 180 days. And as of September 2024, S3 no longer transitions objects smaller than 128 KB by default, so a bucket full of tiny files may quietly skip the rule unless you override the size filter. If you want this whole setup in version control instead of the console, define the bucket and its lifecycle rules with [AWS CloudFormation](https://www.rabinarayanpatra.com/blogs/create-your-first-cloudformation-stack) so the policy ships with your infrastructure. ## How do you restore an archived object from Glacier? For Flexible Retrieval and Deep Archive, you POST a restore request, wait for the tier's retrieval time, then download a temporary copy. Instant Retrieval skips all of this, since a normal GET works directly. The restore call names how many days the temporary copy should stay available and which tier to use: ```bash aws s3api restore-object \ --bucket my-archive-bucket \ --key logs/2026/app.log.gz \ --restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Standard"}}' ``` While the restore runs and after it finishes, the object's storage class stays `GLACIER` or `DEEP_ARCHIVE`. The restore doesn't move the object, it just creates a readable copy next to it. You check progress with `head-object`, which reports `Restore: ongoing-request="true"` while the job runs and flips to `false` with an expiry date once the copy is ready: ```bash aws s3api head-object \ --bucket my-archive-bucket \ --key logs/2026/app.log.gz ``` When the number of `Days` you asked for runs out, S3 deletes the temporary copy and you're back to just the archived object. If you actually want the data to stay hot, copy the restored object to a non-archive class like Standard or Standard-IA before the window closes. This restore model is unique to the Glacier classes, and it's one more reason the [broader S3 feature set](https://www.rabinarayanpatra.com/blogs/aws-s3-2026-files-vectors-and-beyond) is worth understanding before you commit data to cold storage. ## Which S3 Glacier class should you actually pick? Match the class to how often you read the data and how long you can wait, not to the storage price alone. The three classes map cleanly onto three access patterns: - **Glacier Instant Retrieval:** you access the data about once a quarter, but when you do you need it in milliseconds. Think medical images, news media archives, or user files that are rarely opened but must load instantly when they are. You pay roughly 4x the Deep Archive storage rate for that immediacy. - **Glacier Flexible Retrieval:** you read the data once or twice a year and can wait minutes to hours. Backups, large datasets you reprocess occasionally, anything where a Bulk restore (free, up to 12 hours) is fine. This is the practical default for most archives. - **Glacier Deep Archive:** you almost never read the data and you're keeping it for compliance, legal hold, or long-term disaster recovery. You can wait up to 48 hours, and the 180-day minimum is irrelevant because the data lives for years. The mistake I see most often is reaching for Deep Archive purely because the storage number is smallest. If there's any real chance you'll need the data inside a day, that choice turns a five-minute task into a next-day one, and the cheap storage line never makes up for the operational pain. ## What should you check before moving data to Glacier? Model the retrieval before you write the lifecycle rule, not after. Storage cost is the number everyone compares, but it's the cheap, visible one. Retrieval cost and retrieval time are invisible right up until the day you actually need the data, which tends to be the worst possible day to discover that a restore takes 48 hours or costs more than you expected. My rule of thumb: if I can't say out loud how I'll get the data back, how fast that will be, and what it costs, the data isn't ready for Glacier yet. Write down the realistic restore scenario (how much, how often, how fast) and let that pick the class. Do that, and S3 Glacier goes from a billing trap to one of the best deals in cloud storage. For the exact figures, see the AWS docs on [S3 storage classes](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html), the [S3 Glacier storage classes guide](https://docs.aws.amazon.com/AmazonS3/latest/userguide/glacier-storage-classes.html), the [retrieval options reference](https://docs.aws.amazon.com/AmazonS3/latest/userguide/restoring-objects-retrieval-options.html), and the live [Amazon S3 pricing page](https://aws.amazon.com/s3/pricing/). ### Keep Reading - [Amazon S3 Files: AWS Just Turned Object Storage Into a File System](https://www.rabinarayanpatra.com/blogs/aws-s3-2026-files-vectors-and-beyond). How S3 keeps growing past plain object storage, the same service these Glacier classes live in. - [How to Create Your First AWS CloudFormation Stack (2026)](https://www.rabinarayanpatra.com/blogs/create-your-first-cloudformation-stack). Put your bucket and lifecycle rules in version-controlled infrastructure instead of clicking through the console. - [What Is Infrastructure as Code? A Beginner's Guide for 2026](https://www.rabinarayanpatra.com/blogs/what-is-infrastructure-as-code). The foundation for managing S3 lifecycle policies and storage classes as code. --- ## How to Add Custom MCP Tools to Your Slack AI Bot Source: https://www.rabinarayanpatra.com/blogs/how-to-add-mcp-tools-slack-ai-bot Published: 2026-06-19T00:00:00+05:30 Tags: custom-mcp-tools-slack-bot, model-context-protocol, fastmcp, slack-bot, openai-codex-cli, claude-code-cli, ai-agents How to extend a Slack AI bot driven by Codex, Claude Code, or Cursor with custom MCP tools: build a FastMCP server in Python, wire it per-agent, gate destructive calls behind Slack approvals. A Slack bot wired to a CLI agent already gets you shell, file editing, and web search out of the box. The moment you point it at the rest of your stack, it becomes the only chat surface you need. `is order 9182 still in payment_pending?` answers from your read-only Postgres replica. `add a 30-min sync with Prem next Tuesday afternoon` writes to your calendar. `pull the price for this Amazon listing` runs a small scraper. None of that involves changing the Slack glue script, the agent CLI binary, or even restarting the bot. The reason that works is the Model Context Protocol. Every modern agent CLI ships as an MCP client, and any process that speaks JSON-RPC 2.0 over stdio becomes a tool source. This post is the companion to the [Slack AI bot setup guide](https://www.rabinarayanpatra.com/blogs/how-to-build-self-hosted-slack-ai-bot-cli-agent) and walks through building, wiring, and operating custom MCP servers for that bot, regardless of whether the agent on the other end is OpenAI Codex, Anthropic Claude Code, or Cursor. ![Diagram of a CLI agent acting as an MCP client. The agent sits in the middle, with arrows pointing out to four separate MCP server processes: a Postgres read-only server, a calendar server, a Notion server, and a custom internal-API server. Each server speaks JSON-RPC 2.0 over stdio.](/images/blogs/how-to-add-mcp-tools-slack-ai-bot.webp) What you get by the end: a fifteen-line Python MCP server that exposes two tools, three short config snippets that wire it into Codex, Claude Code, and Cursor respectively, and an operational pattern that makes the bot safe to share with a small team. The MCP server you write is identical across agents; only the registration file changes. ## Why add custom MCP tools when the agent already ships with shell and web_search? Because shell is the worst possible interface for everything except shell. Asking the agent to figure out which Postgres column to filter on, write the right `psql` invocation, and parse the result every time you want to look up an order is a recipe for slow runs, hallucinated column names, and the occasional accidentally dropped table. A purpose-built `query_orders(status)` tool is one line of Python on your side, runs in milliseconds, and is impossible for the model to misuse. The deeper reason is shape. A tool description tells the LLM what arguments to send and what kind of result to expect. A tool runs deterministically on your hardware with your credentials and your access rules. Together, those two facts mean the agent stops guessing about your internal systems and starts behaving like a thin orchestration layer over functions you have already tested. The model picks the right tool, fills in the arguments, and reads the output. You wrote the function in the language you already use. Production teams that have used MCP for a few months almost all converge on the same pattern. Wrap read-only paths to internal APIs as MCP tools. Wrap a handful of write paths but gate them behind approvals. Leave everything else to shell. The Slack bot becomes a fast read interface over the systems where reading is cheap and writing is risky, and that is exactly what most chat-driven workflows actually need. ## What is MCP and how does an agent CLI talk to MCP servers? Model Context Protocol is an open spec that defines a JSON-RPC 2.0 conversation between an MCP client and one or more MCP servers. Anthropic published the original spec, and OpenAI's Codex CLI, Anthropic's Claude Code, and Cursor have all implemented the client side. The protocol itself is small: a server advertises a tool catalog with names, descriptions, and JSON schemas, and the client invokes tools by name with arguments. For any of these CLIs, MCP servers live as child processes that the agent spawns over stdio at session start. The wire format is JSON-RPC 2.0, one message per line, with method names like `tools/list` and `tools/call`. From inside the agent session, every MCP tool shows up next to the built-in tools, and the LLM picks whichever one fits the prompt. You do not have to teach the model what your tool does; the docstring you wrote becomes the description it reads. ![Tool call lifecycle diagram. A user asks "is order 9182 still pending?" in Slack. The Slack glue forwards the prompt to a CLI agent. The agent consults its tool catalog, picks query_orders, sends a tools/call JSON-RPC request to the MCP server process. The server returns the row over stdout. The agent composes a natural-language reply and streams it back through the glue into the same Slack message.](/images/blogs/how-to-add-mcp-tools-slack-ai-bot-lifecycle.webp) A few constraints worth knowing before you write anything. MCP servers run as separate processes for a reason: they isolate your tool dependencies from the agent runtime, let you write servers in any language with a JSON-RPC library, and survive a crash without taking the agent down. The agent spawns them on demand and keeps the standard-streams open for the duration of the session. The only thing you ship is the executable and the config that points the agent at it. ## How do you build a minimal MCP server in 15 lines of Python? Install the official Python SDK, write two decorated functions, call `mcp.run()`. That is the entire scaffold. The SDK auto-generates JSON Schema from your type hints and uses the docstring as the tool description, so a clean signature is the entire UI for the LLM. The same server binary works under any MCP-compliant client. ```bash pip install "mcp[cli]" ``` Save this as `~/my_tools/mcp_server.py`: ```python """my_tools/mcp_server.py - two-tool MCP server.""" from mcp.server.fastmcp import FastMCP mcp = FastMCP('my-tools') @mcp.tool() def get_weather(city: str) -> str: """Current weather for a city. Returns a one-line string.""" # ...call your weather API here return f'sunny in {city}, 24C' @mcp.tool() def list_my_tickets(status: str = 'open') -> list[str]: """List tickets I own. status: open | closed | all. Defaults to open.""" # ...query your ticket system here return ['TKT-101', 'TKT-205'] if __name__ == '__main__': mcp.run() # speaks JSON-RPC over stdin/stdout ``` That is a working MCP server. Tool names, descriptions, and input schemas are derived from the Python signatures and docstrings. Test it independently of any agent with a quick echo from another shell: ```bash echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | python ~/my_tools/mcp_server.py ``` You should see a JSON-RPC response listing `get_weather` and `list_my_tickets` with their full schemas. If that works, the server is ready to wire into whichever agent you are running. Two small details that catch first-timers. The `mcp.run()` function uses stdio transport by default, which is what every CLI agent expects. If you instead try `mcp.run(transport='streamable-http')` you switch to the HTTP transport for remote MCP servers, which is a different config story and not what we want for a local Slack bot. And anything you `print()` from your tools goes to stdout, which is the wire. Use `logging` with a `StreamHandler` pointed at stderr instead, or you will corrupt the JSON-RPC stream and watch the agent disconnect. ## How do you wire the same MCP server into Codex, Claude Code, and Cursor? Each agent CLI reads MCP server definitions from its own config file, but the entry shape is conceptually the same: a name, a command, and optional args and env. The server binary stays put; only the three registration files change. For **OpenAI Codex CLI**, add a table to `~/.codex/config.toml`: ```toml # ~/.codex/config.toml [mcp_servers.my-tools] command = "python" args = ["/Users/you/my_tools/mcp_server.py"] startup_timeout_sec = 10 tool_timeout_sec = 60 ``` Or use the CLI-managed alternative: ```bash codex mcp add my-tools -- python /Users/you/my_tools/mcp_server.py ``` For **Anthropic Claude Code**, drop a `.mcp.json` at the project root (or anywhere you point Claude with `--mcp-config `): ```json { "mcpServers": { "my-tools": { "command": "python", "args": ["/Users/you/my_tools/mcp_server.py"] } } } ``` For **Cursor**, the file is `.cursor/mcp.json` in the project root, with the same JSON shape: ```json { "mcpServers": { "my-tools": { "command": "python", "args": ["/Users/you/my_tools/mcp_server.py"] } } } ``` A few config options I reach for almost every time on the Codex side. `startup_timeout_sec` defaults to ten seconds, which is tight if your server hits a database on import; bump it to thirty if you see startup timeouts. `tool_timeout_sec` defaults to sixty and is the per-call timeout for any single tool invocation; raise it for tools that touch slow downstream APIs but never set it to infinity because that converts a stuck call into a hung agent. `env` and `env_vars` together let you pass secrets without inlining them in the args list, which matters because the args show up in every Codex log line. Claude Code and Cursor expose similar timeout and env knobs in their JSON config; consult their respective references for the exact key names. After the config change, run `codex mcp list` (Codex) or restart your `claude -p` / `cursor-agent -p` invocation to confirm the new server is picked up. A successful wire-up shows your two tool names in the response alongside the built-in tools. From inside the Slack bot, no glue-script change is needed: the moment the agent picks up the new server on its next spawn, the bot can use the tools. ## What custom MCP tools should you build first for a Slack AI bot? The high-value tools are the boring ones. Anything you currently look up by opening a dashboard tab, copy-pasting a SQL query, or running a curl command three times a week is a candidate. The shape of the function does not matter; the description and the schema do. A short list of tools that consistently pay off on day one of a self-hosted Slack agent: | Tool category | Example tool signatures | Why it earns its keep in Slack | |--------------|-----------------------|----------------------------| | Read-only database | `query_orders(status)`, `lookup_user(email)` | Answers ops questions like "are there any pending orders for this account?" in seconds. | | Calendar and tasks | `list_events(window)`, `create_event(title, when, with)` | "Schedule a 30-min sync with Prem next Tuesday afternoon" lands on the calendar without leaving Slack. | | Internal API wrapper | `get_inventory(sku)`, `cancel_order(id, reason)` | Bot becomes the safest interface to your APIs because every call is logged and gated. | | Note systems | `search_notes(query)`, `append_to_note(title, body)` | "Add this finding to my project notes" works from any thread without app switching. | | Site scrapers | `scrape_amazon_listing(url)`, `fetch_pdf_text(url)` | One-liner replacements for the read-only side of headless-browser scripts you already run locally. | | Home automation | `set_thermostat(c)`, `turn_on_lights(room)` | A Home Assistant MCP server makes the bot a smart-home interface that travels with Slack. | Two principles I keep coming back to. Treat tools as pure functions where possible, idempotent and side-effect-free, because the model will call them eagerly when the prompt sounds even vaguely related. Validate every argument inside the function body and raise with a clear message on bad input, because the JSON Schema is advisory and the LLM does occasionally hallucinate a value that does not match the type hint. Errors propagate back to the model as text, so a sentence that explains what went wrong is often enough for the model to retry with a corrected argument or back off cleanly. ## How do you gate destructive MCP tools behind Slack Approve and Deny buttons? Combine the agent-level approval flag with a per-server policy in the agent's config. The approval request flows through the same JSON event stream as approvals for built-in tools like `shell`, so the existing Approve and Deny block in your Slack bot picks them up without any code changes. On **Codex**, the config delta is one line per server: ```toml [mcp_servers.my-tools] command = "python" args = ["/Users/you/my_tools/mcp_server.py"] default_tools_approval_mode = "on-request" ``` Then run the agent with `--ask-for-approval on-request`. You can also set the global `approval_policy = "on-request"` at the top of `config.toml`, which makes every tool ask before running. On **Claude Code**, gate sensitive tools through the CLI permission flags instead. Run with `--permission-mode acceptEdits` to let safe file edits through without prompting, and use `--allowedTools "Bash(git diff *),Read,Edit"` to constrain what the agent can do without an approval prompt. Anything outside the `--allowedTools` allowlist triggers a permission request, which your Slack glue can intercept the same way it does for Codex. On **Cursor**, leave `--approve-mcps` off (the default) for any server that exposes write tools. Cursor will prompt for approval on each tool call, and the prompt becomes a button block in Slack. For the read-only `query_orders` server you can pass `--approve-mcps` to skip the prompt and let queries fly. The companion post on the [Slack AI bot setup](https://www.rabinarayanpatra.com/blogs/how-to-build-self-hosted-slack-ai-bot-cli-agent) covers the Slack-side handler that posts the buttons and writes the decision back to the agent; the MCP side does not need any extra code. For tools that are clearly safe, like read-only queries, you can let them run without prompting on every agent. The mental model is: every write or external mutation goes through the Slack-button gate, every read goes through automatically. That is the right default for most teams. Once you have a few thousand approved tool calls of history, you can decide whether to relax specific tools to auto-approve based on observed patterns. If you want per-tool control finer than the per-server default, split the server into two: one server for the auto-approve tools, one for the gated ones, each with its own approval policy in the relevant config file. It is more files but cleaner config, and it makes the policy obvious to anyone reading the JSON or TOML. ## What production traps will bite you with MCP servers in a Slack bot? The traps cluster around three sharp edges: stdio hygiene, startup latency, and approval scoping. The fixes are short, but they all show up the first time you run a real workload. Stdio hygiene comes first because it breaks the loudest. Every `print` call from your server lands on the agent's stdin and corrupts the JSON-RPC stream, which then disconnects the server mid-session. Configure Python `logging` with a `StreamHandler(sys.stderr)` at module load and never call `print()` from inside a tool. Catch broad exceptions inside each tool and return a sensible error string rather than letting an uncaught exception write a traceback to stdout. Startup latency adds up when you wire several MCP servers into the same config. Most agents spawn every configured server on session start, so a server that imports SQLAlchemy and opens a database connection at module load adds a second to every CLI invocation. Defer the heavy imports to inside the tool functions, or use `startup_timeout_sec` (Codex) and the equivalent in Claude Code or Cursor to raise the budget for the slow servers and accept the latency hit. For the Slack bot, a quick spawn is what makes the experience feel snappy, so prefer lazy initialization over upfront connection pools. Approval scoping is the third trap. The per-server approval policy applies to every tool that server exposes. If you wired a read-only server and a write server under the same registration, every read goes through the same approval flow as every write, and your users will rage-click Approve. Split servers along the read-write axis from day one. It is one extra config entry per server and it makes the operational story honest. A few smaller details that earn their keep. Treat the MCP server as a deployable artifact and pin its version in `requirements.txt`, because the JSON-RPC method names have shifted across MCP spec revisions and breaking changes propagate through the client side of the wire. Keep one shared `mcp.tool()` decorator registry per server so the LLM never sees duplicate tool names. Log every tool call to a file with timestamp, user, tool name, and arguments, because the audit trail is what makes the bot safe to share beyond yourself. ## Where does this leave the Slack AI bot? A bot wired to a CLI agent is already useful. The same bot wired to half a dozen carefully chosen MCP tools is a different category of useful, because every question that used to require switching to a dashboard tab now answers in the same thread you were already in. The pattern scales from one tool to twenty without adding any Slack code, because the LLM does the dispatch and the glue is invisible. And because MCP is agent-agnostic, the work you do here outlives any one CLI: if you switch from Codex to Claude Code or Cursor next quarter, the server stays put, only the registration file changes. If you stopped at the basic Slack bot in the [companion post](https://www.rabinarayanpatra.com/blogs/how-to-build-self-hosted-slack-ai-bot-cli-agent), the next move is one read-only MCP tool for whichever system you find yourself logging into the most. Pick the boring one. Wrap one read path. Wire it into the relevant config file for the agent you are running. Watch your Slack thread answer a question that used to take three tab switches. The moment that lands, the rest of the tool catalog writes itself. For more on the protocol and ecosystem, see the [MCP specification site](https://modelcontextprotocol.io/), the [Python MCP SDK on GitHub](https://github.com/modelcontextprotocol/python-sdk), and the [Codex MCP configuration reference](https://developers.openai.com/codex/mcp). For the broader Slack glue side that this post builds on, the companion guide is linked below. ### Keep Reading - [How to Build a Self-Hosted Slack AI Bot with any CLI Agent](https://www.rabinarayanpatra.com/blogs/how-to-build-self-hosted-slack-ai-bot-cli-agent). The base bot this post extends, including Socket Mode setup, the basic glue, the approval flow, and the systemd or launchd service config. - [How to Build a Stateless MCP Server for the 2026-07-28 Spec](https://www.rabinarayanpatra.com/blogs/build-stateless-mcp-server-2026-07-28-spec). A deeper look at the MCP server side, covering the newest stateless transport that future agent versions will speak. - [How to Build an Interactive MCP App with the MCP Apps SDK](https://www.rabinarayanpatra.com/blogs/build-mcp-app-interactive-ui). What changes when an MCP server hosts an interactive UI rather than only tool functions, useful context for richer Slack workflows. --- ## How to Migrate Vercel AI SDK 5 to 6: Breaking Changes Guide Source: https://www.rabinarayanpatra.com/blogs/vercel-ai-sdk-5-to-6-migration-guide Published: 2026-06-17T00:00:00+05:30 Tags: vercel-ai-sdk-5-to-6-migration, ai-sdk-6, tool-loop-agent, use-chat, vercel-ai-sdk, agent-abstraction Step-by-step Vercel AI SDK 5 to 6 migration. Codemod walkthrough, useChat parts model, ToolLoopAgent replacement for streamText, and tool approval. Vercel shipped AI SDK 6 on May 7, 2026. It is the biggest breaking-change release the SDK has shipped since the 3.0 split. The v3 Language Model Specification, a real Agent abstraction, human-in-the-loop tool approval, a unified message-parts model on useChat, and a new streaming wire format all landed in the same drop. I migrated the AI chatbot on this portfolio over the weekend. The codemod did most of the mechanical work, but four spots needed hand edits. This post walks the migration the way I ran it, with the exact code diffs that broke and the verification steps that confirmed the migration shipped without a silent UX regression. ## What changed in Vercel AI SDK 6? AI SDK 6 ships the v3 Language Model Specification, the Agent interface with ToolLoopAgent as the production implementation, a unified message-parts array on useChat, an OAuth-ready MCP client, reranking, image editing, and a [new streaming wire format that pairs naturally with Suspense and `loading.tsx`](https://www.rabinarayanpatra.com/snippets/nextjs/streaming-suspense-loading) for a 15 to 25 percent first-token latency win on cold-start route invocations. The Vercel post calls the wire format change "the foundation for several capabilities Vercel has been signposting for the second half of 2026". The four breaking-change axes that affect existing apps are: | Axis | What broke | Where you fix it | | --- | --- | --- | | useChat message-parts | Messages no longer carry a flat `content` string. They carry a `parts` array of typed entries. | Every React component that renders `message.content` | | Tool-call streaming lifecycle | Tool calls now flow through explicit `state` transitions (`input-streaming`, `input-available`, `output-available`, `approval-requested`). | Tool render switch statements in your UI | | Provider-adapter contract | The v3 Language Model Specification changes how provider packages expose tools, finish reasons, and usage details. | Custom provider wrappers and middleware | | Streaming wire format | New ordering and chunk types for tool calls and reasoning traces. | Anything that reads the SSE stream directly | The codemod handles the easy half. The other half is where this guide focuses. ## Which breaking changes hit you first? The useChat parts model is the change that hits you first because it is silent. Your app keeps compiling. Your messages array keeps populating. But the moment a tool gets invoked, the tool-call UI never renders, because your code is still reading `message.content` and the tool call now lives in `message.parts`. I caught this in my own portfolio because the AI chat that answers questions about my blog posts uses one tool (`search_blog_posts`). After upgrading, the chat worked perfectly for plain prompts and dropped the search-result card entirely for tool prompts. No console error, no broken request. Just a missing block in the UI. The second-hardest break is `streamText` callers that built their own agent loop with `stopWhen` and a step counter. The v6 way is `ToolLoopAgent`, and the two cannot be mixed in a single component because they emit different stream shapes. The third break is the wire format. If you parse the SSE stream by hand anywhere in your codebase, you have to rewrite that parser. Most apps do not, but server-rendered chat UIs and custom telemetry layers usually do. ## How do you run the codemod the right way? You run the codemod by pinning the SDK version first, running the official transform from a clean working tree, then reviewing every changed file before you let it land. Treat the codemod output as a starting point, not a finished migration. ```bash # 1. Pin both packages at the v6 line. Do this before running the codemod. npm install ai@6 @ai-sdk/openai@6 @ai-sdk/anthropic@6 # 2. Confirm a clean git status. The codemod will rewrite files in place. git status # 3. Run the official codemod from the repo root. npx @ai-sdk/codemod upgrade v6 # 4. Inspect the diff. git diff --stat git diff src/app ``` The codemod prints a summary of files touched. On the chatbot in this portfolio it rewrote 9 files: the route handler, the chat component, and seven smaller utilities. It did not touch my custom tool renderer or the route that streams blog embedding search results, because both rely on shapes that the codemod cannot infer. If the diff looks plausible, commit it as a separate commit before touching anything else. That gives you a clean rollback point. ```bash git add -A git commit -m "chore(ai-sdk): apply v6 codemod automated changes" ``` Now you can start the hand edits on top of a known-good base. ## How do you migrate useChat to message parts? You migrate useChat by rewriting every render path that reads `message.content` to walk `message.parts` and switch on `part.type`. The new model treats text, tool calls, tool results, and reasoning traces as first-class entries that all live in the same array. Here is the before and after on the simplest possible chat component. **v5 style (broken in v6 the moment a tool fires):** ```tsx 'use client' import { useChat } from 'ai/react' export function Chat() { const { messages, input, handleSubmit, handleInputChange } = useChat() return (
{messages.map((m) => (
{m.role}: {m.content}
))}
) } ``` **v6 style with the parts model:** ```tsx 'use client' import { useState } from 'react' import { useChat } from 'ai/react' import type { WeatherAgentUIMessage } from '@/agents/weather-agent' export function Chat() { const { messages, sendMessage } = useChat() const [input, setInput] = useState('') const onSubmit = (event: React.FormEvent) => { event.preventDefault() if (!input.trim()) return sendMessage({ text: input }) setInput('') } return (
{messages.map((m) => (
{m.role}: {m.parts.map((part, i) => { switch (part.type) { case 'text': return {part.text} case 'tool-weather': return case 'reasoning': return default: return null } })}
))}
setInput(e.target.value)} />
) } ``` Three things make this work. First, `useChat()` infers the typed part union from your agent definition. The exported `InferAgentUIMessage` is what you import as `WeatherAgentUIMessage`. Second, `handleSubmit` and `handleInputChange` are gone in v6. You manage the input state yourself and call `sendMessage({ text })` to submit. Third, you need a `default` fallback in the switch. If you forget it and the agent introduces a new part type, the UI silently drops that block at runtime. After this change, both the plain-text path and the typed `tool-weather` path render. The Vercel migration playbook calls this "the silent UX regression" and recommends adding a unit test that asserts every part type your agent emits has a render case. ## How do you replace streamText with ToolLoopAgent? You replace streamText by extracting the loop configuration into a `ToolLoopAgent` instance and calling `.stream()` or `.generate()` on it. The agent holds the model, instructions, tools, and stop condition once, and you reuse it everywhere from chat routes to background jobs to standalone scripts. The before and after on a typical route handler looks like this. **v5 route handler:** ```typescript // app/api/chat/route.ts import { streamText } from 'ai' import { openai } from '@ai-sdk/openai' import { weatherTool } from '@/tools/weather' export async function POST(req: Request) { const { messages } = await req.json() const result = await streamText({ model: openai('gpt-4o'), system: 'You are a helpful weather assistant.', messages, tools: { weather: weatherTool }, maxSteps: 20, }) return result.toDataStreamResponse() } ``` **v6 route handler with ToolLoopAgent:** ```typescript // agents/weather-agent.ts import { ToolLoopAgent, type InferAgentUIMessage } from 'ai' import { weatherTool } from '@/tools/weather' export const weatherAgent = new ToolLoopAgent({ model: 'openai/gpt-5.4', instructions: 'You are a helpful weather assistant.', tools: { weather: weatherTool, }, }) export type WeatherAgentUIMessage = InferAgentUIMessage ``` ```typescript // app/api/chat/route.ts import { weatherAgent } from '@/agents/weather-agent' export async function POST(req: Request) { const { messages } = await req.json() const result = await weatherAgent.stream({ messages }) return result.toUIMessageStreamResponse() } ``` Three details matter on this migration. First, the plain provider-prefixed model string `'openai/gpt-5.4'` routes through the Vercel AI Gateway by default, which is the pattern the Vercel knowledge update calls out as the recommended path. The gateway gives you per-request usage tracking, failover across providers, and zero data retention. The v5-era pattern of importing the provider package and calling `openai('...')` still works, but the string form is shorter and is what new code should use. Second, `toUIMessageStreamResponse` is the v6 name for what used to be `toDataStreamResponse`. The codemod renames it most of the time. Check the route handlers. Third, `stopWhen` lives on the agent now, not on the call. The default is 20 steps. If you used to pass `maxSteps: 5` everywhere, set `stopWhen: stepCountIs(5)` on the agent constructor. I wired the same pattern into the [Vercel AI Gateway deep dive](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive) on this site. The shape is identical for Claude (`'anthropic/claude-sonnet-4.5'`) and for any other provider in the gateway catalog. ## How do you wire human-in-the-loop tool approval? You wire approval by adding a `needsApproval` predicate to the tool and rendering an `approval-requested` state in the UI. The agent pauses, the UI shows the approval prompt, and the user accepts or rejects before the tool actually runs. The tool side looks like this. ```typescript // tools/run-command.ts import { tool } from 'ai' import { z } from 'zod' export const runCommand = tool({ description: 'Run a shell command on the user machine', inputSchema: z.object({ command: z.string().describe('The shell command to execute'), }), needsApproval: async ({ command }) => command.startsWith('rm '), execute: async ({ command }) => { const { stdout } = await execCommand(command) return { stdout } }, }) ``` The UI side adds a render case for the new state. ```tsx case 'tool-runCommand': { if (part.state === 'approval-requested') { return (

Run: {part.input.command} ?

) } if (part.state === 'output-available') { return 
{part.output.stdout}
} return null } ``` `addToolApprovalResponse` ships from the `useChat` return value. The agent resumes execution as soon as the approval message hits the stream. If the user rejects, the agent gets a structured rejection event and can recover or apologize. The predicate runs on the server, so it can hit a policy service or an authorization check. That is the path I use to gate shell-style tools behind a per-account permission check. The same pattern would gate the more dangerous Anthropic and OpenAI provider tools (`anthropic.tools.memory_20250818`, `openai.tools.shell`) when those land in your stack. ## How do you verify the migration before shipping? You verify the migration by running the test suite, hitting every route that exercises tools, watching the DevTools timeline on a streaming chat, and replaying a representative production transcript against the new agent in a staging environment. If any of those four steps surfaces a regression, do not ship. For the unit-test side, I added one test per agent that walks every expected part type. ```typescript // __tests__/weather-agent.test.ts import { describe, it, expect } from 'vitest' import { weatherAgent } from '@/agents/weather-agent' describe('weatherAgent', () => { it('emits text, tool-weather, and finish parts in order', async () => { const result = await weatherAgent.generate({ prompt: 'What is the weather in San Francisco?', }) const partTypes = result.messages .flatMap((m) => m.parts) .map((p) => p.type) expect(partTypes).toContain('text') expect(partTypes).toContain('tool-weather') }) }) ``` For the DevTools verification, wrap the model in middleware and launch the viewer locally. ```typescript import { wrapLanguageModel } from 'ai' import { devToolsMiddleware } from '@ai-sdk/devtools' import { gateway } from 'ai' const tracedModel = wrapLanguageModel({ model: gateway('openai/gpt-5.4'), middleware: devToolsMiddleware(), }) ``` ```bash npx @ai-sdk/devtools # http://localhost:4983 ``` The DevTools viewer renders the full agent timeline: prompt, tool calls, tool results, model responses, and step boundaries. Use it once per migrated route. If a tool call shows up but the matching `tool-*` render case never fires in your UI, the parts switch is missing a case. For staging replay, capture three representative production transcripts in your logs, run them through the new agent, and diff the response shape. The token usage breakdown is the cleanest signal because the v6 `usage.inputTokenDetails` and `usage.outputTokenDetails` are richer than v5. If you write your own tools, this is the moment to add `inputExamples` and `strict: true` everywhere. They are cheap on the wire and they catch the dumbest tool-call regressions before they reach prod. ```typescript tool({ description: 'Search blog posts', inputSchema: z.object({ query: z.string().min(2), }), strict: true, inputExamples: [ { input: { query: 'virtual threads' } }, { input: { query: 'mcp protocol' } }, ], execute: async ({ query }) => searchBlogPosts(query), }) ``` For deeper background on AI SDK patterns that the v6 Agent abstraction now formalizes, see the [Vercel AI Gateway deep dive](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive) and the [Claude Opus 4.7 release and migration guide](https://www.rabinarayanpatra.com/blogs/claude-opus-4-7-release-and-migration-guide). For the original sources, see the [AI SDK 6 announcement](https://vercel.com/blog/ai-sdk-6), the [official migration guide](https://ai-sdk.dev/docs/migration-guides/migration-guide-6-0), the [ToolLoopAgent reference](https://ai-sdk.dev/docs/reference/ai-sdk-core/tool-loop-agent), and the [Vercel AI SDK v5 to v6 migration playbook](https://www.digitalapplied.com/blog/vercel-ai-sdk-v5-to-v6-migration-playbook-2026). ### Keep Reading - [Vercel AI Gateway Deep Dive](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive). The provider routing layer that the v6 agent string format relies on, with cost and observability patterns. - [Claude Opus 4.7 Release and Migration Guide](https://www.rabinarayanpatra.com/blogs/claude-opus-4-7-release-and-migration-guide). Provider-side context for the Anthropic models you plug into ToolLoopAgent. - [Spring AI 2.0 MCP Annotations Tutorial](https://www.rabinarayanpatra.com/blogs/spring-ai-2-mcp-annotations-tutorial). The same tool-loop pattern from the JVM side, and how the v6 MCP client connects both ends. - [Replacing useEffect with Server Actions](https://www.rabinarayanpatra.com/blogs/replacing-useeffect-data-fetching-server-actions). The Next.js data-fetching shift that pairs with the v6 useChat parts model in App Router chat UIs. --- ## How to Build a Self-Hosted Slack AI Bot with any CLI Agent Source: https://www.rabinarayanpatra.com/blogs/how-to-build-self-hosted-slack-ai-bot-cli-agent Published: 2026-06-12T00:00:00+05:30 Tags: self-hosted-slack-ai-bot-cli-agent, slack-bot, openai-codex-cli, claude-code-cli, socket-mode, slack-bolt, ai-agents How to build a self-hosted Slack AI bot driven by any CLI agent: Socket Mode app, Python glue, token streaming, Approve and Deny buttons, session resume across Codex and Claude Code. A Slack bot that DMs you back with the same agent you run from your terminal sounds like a weekend hack, but it is also the cheapest way to give yourself a coding agent that works from anywhere with Slack installed. No browser tab, no separate app, no token billing dashboard to check. You write `deploy the staging branch` in a Slack DM, the bot streams the plan back into the same message, and asks for permission before it runs `kubectl apply`. This guide walks through building exactly that. The shape is a thin Python glue process that uses Slack Socket Mode to receive messages, spawns a CLI agent as a subprocess per turn, and edits the Slack reply in place as tokens stream out. The running example is OpenAI Codex CLI because that is what I shipped first, but the same architecture works with Anthropic Claude Code, Cursor's CLI, or any other agent that exposes a stdio interface. The driver is the only part that changes when you swap agents; the Slack glue, the approval flow, and any MCP tools you add stay identical. ![Architecture diagram for a self-hosted Slack AI bot. A Slack user posts a message which travels over an outbound Socket Mode WebSocket to a small Python glue process. The glue spawns a CLI agent (Codex in this example) as a subprocess, which makes an HTTPS request to OpenAI and streams tokens back. The glue edits the Slack message in place as tokens arrive.](/images/blogs/how-to-build-self-hosted-slack-ai-bot-cli-agent.webp) The walkthrough is structured around three layered variants you can build in order: the basic auto-approve bot, the Slack-button approval gate, and the session-resume version that survives restarts. After that, a short section covers how to swap the Codex driver for Claude Code, Cursor, or anything else without touching the rest of the stack. You can stop at any layer. For most personal bots, basic plus approval is the sweet spot. ## Why build your own Slack AI bot instead of using the official ChatGPT or Claude integration? The official ChatGPT in Slack and the official Claude in Slack are great for chat. Neither can run code, edit files, or call your internal tools by default. A self-hosted bot wired to a CLI agent like Codex or Claude Code does all three out of the box, because both CLIs ship with `shell`, file-edit, and an MCP client that picks up any custom tools you register. The control trade-off is also real. With a self-hosted bot you own the workspace directory the agent reads and writes, the approval rules for destructive commands, the model selection, and the conversation rollouts on disk. If your team has a private codebase or an internal API the bot needs to touch, none of that has to leave your host. The only network call in the whole pipeline is the outbound HTTPS from the agent CLI to its provider, and that traffic is the same as any other agent session you run from a laptop. Cost is the third reason. ChatGPT Plus at twenty dollars a month includes a generous weekly Codex usage budget, and Claude Pro covers a similar quota on the Anthropic side. OpenAI moved Codex billing onto API-token alignment for paid plans in April 2026, and Claude Code respects your Pro plan in the same way. A personal bot that handles a few dozen short threads a week rarely brushes either cap. If you do exceed the plan limit you can switch to an API key for either provider and pay per token directly, with no other code changes. ## How does the Slack and CLI agent pipeline actually work? The pipeline has exactly one outbound network call at runtime, and Slack never reaches back into your network. Everything else is local pipes between a Python glue process and the agent subprocess. The sequence for a single message in a thread looks like this: 1. You type into Slack. The Slack edge serializes the event and pushes it over an outbound WebSocket that the bot opened on startup. That socket is the Socket Mode transport, and it is the only reason no public URL or inbound port is needed. 2. The Python glue process catches the event in an async handler. It posts a placeholder message in the same thread (`thinkingโ€ฆ`) and remembers the channel and message timestamp. 3. The glue spawns the agent CLI as an async subprocess. For Codex that is `codex exec --json --model gpt-5.5 --cd `; for Claude Code it is `claude -p "" --output-format stream-json --verbose --include-partial-messages --add-dir `. In both cases standard output is a JSON Lines stream of events. 4. The agent CLI itself makes the single HTTPS call out to its provider (OpenAI for Codex, Anthropic for Claude Code). The token stream comes back over that HTTPS connection. 5. The glue reads stdout line by line, parses each JSON event, accumulates text deltas, and once per second calls `chat.update` on the placeholder Slack message with the latest accumulated text. The reply edits in place rather than spamming new messages. 6. When the agent exits, the glue does one final `chat.update` with the complete reply, then releases the per-thread lock so the next message in the same thread can run. The event family depends on which agent CLI you run. Codex 2026 emits `thread.started` (carries the session identifier), `turn.started` and `turn.completed` (bracket each agent turn), the `item.*` family (covers agent messages, reasoning, command executions, file changes, MCP tool calls, plan updates), and `error`. Claude Code with `--output-format stream-json --verbose --include-partial-messages` emits `system/init` (session metadata, including `session_id`), `stream_event` lines with a nested `event.delta` payload (where `delta.type == 'text_delta'` carries token deltas), tool-use events, and `system/api_retry` when a call retries. Cursor's `cursor-agent` produces a similar shape under `--output-format stream-json --stream-partial-output`. The driver layer hides the difference from the rest of the bot. ## How do you set up the Slack app from a manifest? The fastest path is the manifest. You paste a single YAML block into Slack's app builder and it provisions the bot user, scopes, event subscriptions, and Socket Mode in one shot. The alternative is twenty clicks across six settings pages. Open `api.slack.com/apps`, click **Create New App**, pick **From an app manifest**, choose your workspace, and paste this YAML: ```yaml display_information: name: My AI Bot description: CLI agent in Slack background_color: '#1d1d1d' features: bot_user: display_name: My AI Bot always_online: true app_home: home_tab_enabled: false messages_tab_enabled: true messages_tab_read_only_enabled: false oauth_config: scopes: bot: - app_mentions:read - channels:history - chat:write - im:history - im:read - im:write - users:read settings: event_subscriptions: bot_events: - app_mention - message.im interactivity: is_enabled: true socket_mode_enabled: true ``` After the app is created, you need two tokens. Go to **Settings โ†’ Basic Information โ†’ App-Level Tokens**, click **Generate Token and Scopes**, add the `connections:write` scope, and save the token. It starts with `xapp-` and is what Socket Mode uses to open the WebSocket. Then go to **Settings โ†’ Install App**, install to your workspace, and copy the **Bot User OAuth Token** at the top of the page. It starts with `xoxb-` and is what every `chat.postMessage` and `chat.update` call authenticates with. DMs to the bot work as soon as the install completes. To use `@mentions` in a channel, run `/invite @My AI Bot` in that channel from your own account. Otherwise Slack silently drops the event because the bot is not a member. ## How do you wire Codex CLI and run the basic bot? Install Codex once, log it into ChatGPT, and write a fifty-line Python file that drives it from Slack events. The CLI ships through npm and the Python side needs only Slack Bolt and aiohttp. The same shape applies to any other agent CLI; we use Codex because it has the cleanest stateless `exec` mode for a per-turn spawn. Install the CLI and authenticate: ```bash npm i -g @openai/codex codex login # opens a browser tab to ChatGPT, then writes ~/.codex/auth.json ``` On a headless machine, `codex login` prints a device-code URL. Open it on any other device, finish the OAuth flow there, and the headless host picks up the auth tokens automatically. The token file lives at `~/.codex/auth.json` regardless of how you sign in. The Python side is two packages and one script. Save this as `requirements.txt`: ``` slack-bolt>=1.18 aiohttp>=3.9 ``` Then the bot itself. The version below is the auto-approve variant: every tool call the agent decides to run executes without asking. Start here so you can validate the Slack-to-agent pipeline end to end before adding the approval gate. ```python """slackbot.py - Slack <-> CLI agent glue (auto-approve, Codex driver).""" import asyncio, json, logging, os, re, time from collections import defaultdict from slack_bolt.async_app import AsyncApp from slack_bolt.adapter.socket_mode.async_handler import AsyncSocketModeHandler SLACK_BOT_TOKEN = os.environ['SLACK_BOT_TOKEN'] SLACK_APP_TOKEN = os.environ['SLACK_APP_TOKEN'] AGENT_BIN = os.environ.get('AGENT_BIN', 'codex') AGENT_MODEL = os.environ.get('AGENT_MODEL', 'gpt-5.5') AGENT_CWD = os.environ.get('AGENT_CWD', os.path.expanduser('~/agent-bot-workspace')) os.makedirs(AGENT_CWD, exist_ok=True) EDIT_INTERVAL = 1.0 # seconds between chat.update calls MAX_SLACK_LEN = 39_000 # chat.update hard cap is 40k chars logging.basicConfig(level=logging.INFO) log = logging.getLogger('slackbot') app = AsyncApp(token=SLACK_BOT_TOKEN) thread_locks: dict[str, asyncio.Lock] = defaultdict(asyncio.Lock) async def agent_stream(prompt: str): """Codex driver. Yields text chunks as Codex emits them.""" proc = await asyncio.create_subprocess_exec( AGENT_BIN, 'exec', '--json', '--model', AGENT_MODEL, '--cd', AGENT_CWD, prompt, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, ) try: async for raw in proc.stdout: line = raw.decode('utf-8', errors='replace').strip() if not line: continue try: ev = json.loads(line) except json.JSONDecodeError: continue kind = ev.get('type', '') if kind.startswith('item.') and ev.get('item', {}).get('type') == 'agent_message': delta = ev.get('delta') or ev.get('item', {}).get('text', '') if delta: yield delta elif kind.startswith('item.') and 'command' in ev.get('item', {}).get('type', ''): yield f"\n_running: {ev['item'].get('command', '?')}_\n" await proc.wait() if proc.returncode: err = (await proc.stderr.read()).decode(errors='replace') yield f"\n_agent exit {proc.returncode}: {err[-400:]}_\n" finally: if proc.returncode is None: proc.terminate() @app.event('app_mention') @app.event('message') async def handle_message(event, client): if event.get('bot_id') or event.get('subtype'): return channel = event['channel'] text = re.sub(r'<@[A-Z0-9]+>', '', event.get('text', '')).strip() if not text: return thread_ts = event.get('thread_ts') or event['ts'] async with thread_locks[f'{channel}:{thread_ts}']: placeholder = await client.chat_postMessage( channel=channel, thread_ts=thread_ts, text='_thinkingโ€ฆ_' ) msg_ts = placeholder['ts'] accumulated, last_edit = '', 0.0 try: async for chunk in agent_stream(text): accumulated += chunk if time.monotonic() - last_edit > EDIT_INTERVAL: await client.chat_update( channel=channel, ts=msg_ts, text=accumulated[:MAX_SLACK_LEN] or '_โ€ฆ_', ) last_edit = time.monotonic() await client.chat_update( channel=channel, ts=msg_ts, text=accumulated[:MAX_SLACK_LEN] or '_(empty reply)_', ) except Exception as e: log.exception('agent run failed') await client.chat_update(channel=channel, ts=msg_ts, text=f'_error: {e}_') async def main(): log.info('slackbot starting') await AsyncSocketModeHandler(app, SLACK_APP_TOKEN).start_async() if __name__ == '__main__': asyncio.run(main()) ``` Notice the rename from `CODEX_*` to `AGENT_*` in the env vars. That is the only thing that changes between agents; the rest of `handle_message`, the Slack lock, and the streaming throttle are all agent-agnostic. The driver function `agent_stream` is the swap point. Export the two tokens, source the file, run it. The shell will block on the WebSocket handler. DM the bot from Slack with a quick `hi` to confirm the pipeline. You should see a `thinkingโ€ฆ` placeholder appear within a second, then the reply stream into that same message instead of posting a wall of new messages. ```bash export SLACK_BOT_TOKEN=xoxb-... export SLACK_APP_TOKEN=xapp-... python3 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt python slackbot.py ``` The two design choices worth flagging. The per-thread `asyncio.Lock` serializes messages in the same Slack thread so two quick replies cannot clobber each other's `chat.update` calls. The one-second edit throttle stays under Slack's per-channel-per-second soft cap on `chat.update`. Crank `EDIT_INTERVAL` up to two if you see `ratelimited` errors in the logs. ## How do you gate destructive tool calls with Slack Approve and Deny buttons? Codex with `--ask-for-approval on-request` pauses on every tool call that could touch the filesystem or run shell. The bot prints the pending action as a Slack message with Approve and Deny buttons, waits for a click, then writes the decision back into Codex stdin. The pause is a real pause: the agent blocks until the answer arrives, so the user is the rate limit, not the network. Claude Code exposes the same shape through its `permission_mode` flag, with `tool_use_request` events on stdout in place of Codex's approval events. ![Approval gate sequence diagram. The agent CLI emits an approval request event on stdout. The Python glue posts a Slack block with the pending command and Approve and Deny buttons. The user clicks Approve, the glue resolves an asyncio.Future, then writes the approval decision JSON into the agent stdin. The agent resumes and executes the tool.](/images/blogs/how-to-build-self-hosted-slack-ai-bot-cli-agent-approval.webp) The wire-level changes are small. Spawn the agent with the approval flag plus an open stdin so you can write the decision back. Keep a dictionary of pending approvals keyed by request id, with each entry holding an `asyncio.Future`. The Slack button click resolves the future, and the spawn loop writes the response JSON to the agent. ```python import uuid pending: dict[str, tuple[asyncio.Future, str, str]] = {} async def agent_stream_with_approval(prompt, channel, thread_ts, client): proc = await asyncio.create_subprocess_exec( AGENT_BIN, 'exec', '--json', '--ask-for-approval', 'on-request', '--model', AGENT_MODEL, '--cd', AGENT_CWD, prompt, stdin=asyncio.subprocess.PIPE, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, ) async def handle_approval(ev): approval_id = ev.get('id') or str(uuid.uuid4()) title = ev.get('kind') or ev.get('type', 'tool') details = ev.get('command') or ev.get('patch') or ev.get('args') or {} fut = asyncio.get_running_loop().create_future() pending[approval_id] = (fut, channel, thread_ts) await client.chat_postMessage( channel=channel, thread_ts=thread_ts, text=f'Agent wants to run: {title}', blocks=[ {'type': 'section', 'text': {'type': 'mrkdwn', 'text': f'*Agent wants to run `{title}`*'}}, {'type': 'section', 'text': {'type': 'mrkdwn', 'text': f'```{json.dumps(details, indent=2)[:2800]}```'}}, {'type': 'actions', 'block_id': f'approve:{approval_id}', 'elements': [ {'type': 'button', 'style': 'primary', 'action_id': 'approve', 'text': {'type': 'plain_text', 'text': 'Approve'}, 'value': approval_id}, {'type': 'button', 'style': 'danger', 'action_id': 'deny', 'text': {'type': 'plain_text', 'text': 'Deny'}, 'value': approval_id}, ]}, ], ) try: decision = await asyncio.wait_for(fut, timeout=600) except asyncio.TimeoutError: decision = 'deny' finally: pending.pop(approval_id, None) response = {'id': approval_id, 'type': 'approval_response', 'decision': decision} proc.stdin.write((json.dumps(response) + '\n').encode()) await proc.stdin.drain() async for raw in proc.stdout: line = raw.decode('utf-8', errors='replace').strip() if not line: continue try: ev = json.loads(line) except json.JSONDecodeError: continue kind = ev.get('type', '') if 'approval' in kind or kind == 'tool_use_request': asyncio.create_task(handle_approval(ev)) continue if kind.startswith('item.') and ev.get('item', {}).get('type') == 'agent_message': delta = ev.get('delta') or ev.get('item', {}).get('text', '') if delta: yield delta await proc.wait() @app.action('approve') async def on_approve(ack, body): await ack() approval_id = body['actions'][0]['value'] if approval_id in pending and not pending[approval_id][0].done(): pending[approval_id][0].set_result('approve') @app.action('deny') async def on_deny(ack, body): await ack() approval_id = body['actions'][0]['value'] if approval_id in pending and not pending[approval_id][0].done(): pending[approval_id][0].set_result('deny') ``` Three things to know before you ship this in production. First, the precise approval JSON event names drift between agent versions, so dump one to your logs with `codex exec --json --ask-for-approval on-request "echo hi"` (or the Claude Code equivalent) and adjust the kind check if your build emits a different shape. Second, the `pending` dictionary lives in process memory. If the bot crashes while a button is unanswered, the click will silently no-op because the matching `Future` is gone. Persisting the table to sqlite is the durable fix, but for personal use, just re-send the original message after a restart. Third, the 600-second timeout auto-denies anything you forget to click, so destructive runs do not hang the agent forever. The companion post on [adding custom MCP tools](https://www.rabinarayanpatra.com/blogs/how-to-add-mcp-tools-slack-ai-bot) builds on this approval flow and shows how to gate any tool you write yourself behind the same Approve and Deny block. If you only ever expose read-only MCP tools, you can skip approval entirely and run the basic variant. ## How does the bot remember context across messages and restarts? The basic variant rebuilds context every turn by re-fetching the Slack thread history and stuffing it into the prompt. That works, but it costs input tokens on every turn and forgets the agent's internal tool memory between turns. The stateful variant fixes both by storing an agent session identifier per Slack thread and resuming the same session on every new message. ![Stateful resume timeline diagram showing four messages in the same Slack thread. After the first message, the glue captures the session_id from the agent's first event and writes it into sqlite. Subsequent messages spawn the agent with a resume flag, so the agent retains tool state across turns and the bot survives a restart between messages without losing the conversation.](/images/blogs/how-to-build-self-hosted-slack-ai-bot-cli-agent-stateful.webp) Codex 2026 exposes resume as a subcommand: `codex exec resume ` continues a specific session, and `codex exec resume --last` continues the most recent one in the current workspace. The session identifier shows up in the `thread.started` event at the start of each run, and Codex writes the full rollout to `~/.codex/sessions/YYYY/MM/DD/rollout--.jsonl` on disk. Claude Code stores its sessions under `~/.claude/sessions/` and resumes via a `--resume ` flag on the same subcommand. The mapping from a Slack thread to an agent session is what you store on your side. A tiny sqlite schema is enough: ```sql CREATE TABLE IF NOT EXISTS thread_sessions ( slack_channel TEXT NOT NULL, slack_thread_ts TEXT NOT NULL, agent_session_id TEXT NOT NULL, created_at REAL NOT NULL, last_used_at REAL NOT NULL, PRIMARY KEY (slack_channel, slack_thread_ts) ); ``` The driver becomes a few lines longer: look up the session before spawning, prefer ` exec resume ` when a row exists, capture the new identifier from the first session event of every run, and write it back on completion. Fall back to a fresh spawn if resume errors out, which is what happens when the rollout file has been deleted or the flag is not supported on an older CLI build. With sqlite on disk and the rollouts on disk, the bot survives restarts and reboots without losing thread context. A periodic garbage-collection task drops rows older than thirty days and matches a `find ~/./sessions -name '*.jsonl' -mtime +30 -delete` cleanup you can run from cron. For most personal setups, disk usage is negligible and you can skip the GC entirely. If you only have one-shot threads with no follow-up, skip the stateful variant entirely; the per-turn re-prompt is cheap and stateless makes a stronger isolation guarantee. ## How do you swap Codex for Claude Code, Cursor, or another CLI agent? Rewrite the `agent_stream` function and update three environment variables. Everything else stays put. The Slack glue, the per-thread lock, the approval button handlers, the sqlite session table, and any MCP tools you have already configured all keep working without a single change. The driver swap is mostly mechanical: each agent CLI has its own flag names, its own JSON event family, and its own session-resume convention. The table below shows the surfaces I have driven so far: | Surface | OpenAI Codex CLI | Anthropic Claude Code | Cursor (`cursor-agent`) | |--------|--------------------|--------------------------|----------------| | Headless invocation | `codex exec ""` | `claude -p ""` | `cursor-agent -p ""` | | Streaming output | `--json` | `--output-format stream-json --verbose --include-partial-messages` | `--output-format stream-json --stream-partial-output` | | Model flag | `--model gpt-5.5` | `--model claude-sonnet-4-6` (or `sonnet`/`opus` aliases) | `--model ` | | Workspace flag | `--cd ` | `--add-dir ` (multiple allowed) | `--workspace ` | | Approval flag | `--ask-for-approval on-request` | `--permission-mode acceptEdits` + `--allowedTools "Bash,Read,Edit"` | `--approve-mcps` and interactive prompts | | Text-delta event | `item.*` with `agent_message` item | `stream_event` with `event.delta.type == "text_delta"` | `stream_event` with delta payload | | Session id surface | `thread.started.thread_id` | `system/init` event (`session_id`) and top-level `session_id` in JSON | `system/init` event (`session_id`) | | Resume | `codex exec resume ` / `--last` | `claude -p --resume ` / `--continue` | `cursor-agent -p --resume ` / `--continue` | | Auth storage | `~/.codex/auth.json` | OAuth / keychain, or `ANTHROPIC_API_KEY` env | Cursor account login | | MCP config | `~/.codex/config.toml` (`[mcp_servers.]`) | `.mcp.json` in project root or `--mcp-config ` | `.cursor/mcp.json` and `cursor-agent mcp` subcommands | A Claude Code driver function looks roughly like this: ```python async def agent_stream_claude(prompt: str): """Claude Code driver (`claude -p` with stream-json).""" proc = await asyncio.create_subprocess_exec( AGENT_BIN, '-p', prompt, '--output-format', 'stream-json', '--verbose', '--include-partial-messages', '--model', AGENT_MODEL, '--add-dir', AGENT_CWD, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, ) async for raw in proc.stdout: line = raw.decode('utf-8', errors='replace').strip() if not line: continue try: ev = json.loads(line) except json.JSONDecodeError: continue if ev.get('type') == 'stream_event': delta = ev.get('event', {}).get('delta', {}) if delta.get('type') == 'text_delta': yield delta.get('text', '') await proc.wait() ``` The shape is identical to the Codex driver: spawn, read JSONL, dispatch on `type`, yield text deltas. Only the field names move. Switching the bot to Claude Code in production is a config change plus a function pointer in `handle_message`: ```python DRIVERS = { 'codex': agent_stream, 'claude': agent_stream_claude, } agent_stream_fn = DRIVERS[os.environ.get('AGENT_KIND', 'codex')] ``` Two design notes that matter when you actually do this. The Claude Code CLI flag set for headless streaming is `-p` (or `--print`) plus `--output-format stream-json --verbose --include-partial-messages`; without all three you do not get token deltas. The other note: each agent has its own MCP config (Codex uses `~/.codex/config.toml`, Claude Code reads `.mcp.json` or a file via `--mcp-config`, Cursor uses `.cursor/mcp.json`), but the same custom MCP server can be registered in all of them. Your custom `query_orders` tool works under any agent without code changes. ## How do you run the bot as a service on Linux or Mac? On Linux it is a fifteen-line systemd unit. On Mac it is a launchd plist plus `caffeinate -i` to stop the laptop sleeping while the bot runs. Either way, the goal is the same: the process restarts on crash, comes back after a reboot, and writes logs to a place you can tail. The systemd unit at `/etc/systemd/system/slackbot.service` looks like this: ```ini [Unit] Description=Slack <-> CLI agent bot After=network-online.target [Service] Type=simple User=slackbot WorkingDirectory=/opt/slackbot Environment=HOME=/opt/slackbot Environment=PATH=/usr/local/bin:/usr/bin:/bin EnvironmentFile=/opt/slackbot/.env ExecStart=/opt/slackbot/.venv/bin/python /opt/slackbot/slackbot.py Restart=always RestartSec=5 [Install] WantedBy=multi-user.target ``` Three quirks bite first-time systemd users. `EnvironmentFile=` does not expand `~` or run `export`, so the `.env` file must be plain `KEY=value` lines with absolute paths. The systemd `PATH` is minimal, so either set `AGENT_BIN=/usr/local/bin/codex` (or the Claude Code path) in `.env` or add the `Environment=PATH=...` line shown above. And `HOME=/opt/slackbot` is what tells the agent CLI where to find its auth file, which is the file you populated when you ran `codex login` or `claude login` as the `slackbot` user during setup. On Mac the equivalent is a launchd plist at `~/Library/LaunchAgents/com.you.slackbot.plist` that wraps the Python invocation in `caffeinate -i` so the laptop never sleeps while the bot is running. `KeepAlive=true` restarts the process on crash, and `RunAtLoad=true` starts it on login. The deploy from a clean Hetzner or DigitalOcean box is short: ```bash sudo adduser --system --group --home /opt/slackbot slackbot sudo apt install -y python3-venv nodejs npm sqlite3 sudo npm i -g @openai/codex # or: install Claude Code per Anthropic docs # rsync slackbot.py + requirements.txt up, then: sudo -u slackbot bash -c 'cd /opt/slackbot && python3 -m venv .venv && .venv/bin/pip install -r requirements.txt' sudo -u slackbot -H bash -c 'codex login' # device-code flow on another browser sudo systemctl daemon-reload && sudo systemctl enable --now slackbot sudo journalctl -u slackbot -f ``` `ufw deny incoming, allow outgoing` is enough for the firewall side. Socket Mode is outbound only, so no port has to be opened to the public internet. ## What breaks in production and how do you fix it? Most failures show up in the first hour and are mechanical. The high-value diagnostic commands are short. `codex exec --json "ping" | head -20` (or `claude --json` with a piped prompt) confirms the agent CLI itself works independent of Slack, which is the single most useful split test when the bot stays quiet. `curl -s https://slack.com/api/auth.test -H "Authorization: Bearer $SLACK_BOT_TOKEN" | jq` confirms the bot token is live and points at the right workspace. `sudo journalctl -u slackbot -f` on Linux or `tail -f /tmp/slackbot.err.log` on Mac shows the parser output and the agent exit code if something dies mid-stream. The failures I have hit, in order of how often they bite. Swapped `xoxb-` and `xapp-` tokens cause `slack_bolt.error.BoltError: invalid_auth` on startup. `chat.update` rate-limited errors mean lower the per-second edit cadence with a bigger `EDIT_INTERVAL`. `agent: command not found` under systemd means absolute path or `Environment=PATH=...` in the unit. `codex login` not completing on a headless VM means open the device-code URL on your laptop instead. Approval buttons that do nothing on click usually mean Interactivity is not enabled in the Slack app settings, which is the one manifest detail that drifted between platform versions. The bot replying once then going silent is the agent plan's weekly cap; switch to an API key for the remainder of the week or upgrade the plan. If you want to extend the bot beyond what the built-in agent tools cover, the next step is custom MCP servers. Both Codex and Claude Code act as MCP clients by default and read server definitions from their own config files, so anything you can write as a Python function with a docstring becomes a tool the bot can call regardless of which agent you have selected. The companion guide on [adding custom MCP tools to your Slack Codex bot](https://www.rabinarayanpatra.com/blogs/how-to-add-mcp-tools-slack-ai-bot) walks through the FastMCP setup, the wire format, and the production patterns I keep going back to. For more on the moving pieces, see the [Slack Bolt Python Socket Mode docs](https://docs.slack.dev/tools/bolt-python/concepts/socket-mode), the [Codex CLI non-interactive mode reference](https://developers.openai.com/codex/noninteractive), and the [Codex agent approvals and security guide](https://developers.openai.com/codex/agent-approvals-security). ### Keep Reading - [How to Add Custom MCP Tools to Your Slack Codex Bot](https://www.rabinarayanpatra.com/blogs/how-to-add-mcp-tools-slack-ai-bot). The companion post that extends this bot with custom tools your team's APIs and databases expose to the agent. - [How to Build a Stateless MCP Server for the 2026-07-28 Spec](https://www.rabinarayanpatra.com/blogs/build-stateless-mcp-server-2026-07-28-spec). The protocol the Slack bot speaks to any MCP tool it loads, explained from the server side. - [Spring AI 2.0 MCP Annotations: From Tool to Production](https://www.rabinarayanpatra.com/blogs/spring-ai-2-mcp-annotations-tutorial). A Java perspective on the same MCP protocol the bot uses, useful if your stack is JVM-heavy. --- ## 10 Best Claude Skills for Developers in 2026 Source: https://www.rabinarayanpatra.com/blogs/best-claude-skills-for-developers-2026 Published: 2026-06-10T00:00:00+05:30 Tags: best-claude-skills-developers, claude-code, claude-skills, ai-coding, anthropic, developer-tools 10 best Claude skills for developers in 2026. Real picks I use every week, ranked by impact and GitHub stars, with direct install links for each. Anthropic shipped Skills in October 2025. A year and a half later, the skill ecosystem is real. [anthropics/skills](https://github.com/anthropics/skills) has 141,849 stars. [obra/superpowers](https://github.com/obra/superpowers) has 209,283 stars. The [open standard](https://agentskills.io) is live. I install a lot of skills. Most do not earn the slot. These ten do. Quick-look format below: name, direct link, what it does, what I use it for. Ranked by impact in real work, with parent-repo star count as a popularity tiebreaker. ## What are Claude Skills? A Claude Skill is a folder with a `SKILL.md` file that teaches Claude one specific task. When the model sees a task that matches, it loads the skill and follows the playbook inside. The point is making behavior reliable. You write the convention once; every future session follows it. Skills are generally available across claude.ai, Claude Code, and the Claude API (the API still requires the `skills-2025-10-02` beta header). ## How do you install Claude Skills? Three install paths cover everything: ```bash # Path 1: Anthropic official skills (clone into your skills folder) mkdir -p ~/.claude/skills cd ~/.claude/skills git clone --depth 1 https://github.com/anthropics/skills.git anthropic-skills # Path 2: superpowers plugin (run inside a Claude Code session) /plugin install superpowers@claude-plugins-official # Path 3: per-project skills (drop into .claude/skills/ in the repo root) ``` After install, restart your Claude Code session once. ## What are the 10 best Claude skills for developers? Ranked by impact, parent-repo stars listed for transparency. ### 1. brainstorming - **Skill:** [obra/superpowers ยท brainstorming](https://github.com/obra/superpowers/tree/main/skills/brainstorming) (parent repo 209k stars) - **What it does:** before any feature work, runs a structured exploration of intent, requirements, and design. Writes a short design memo. Asks for approval before any code. - **Use it for:** every "build X" or "add Y" request. Catches the one design question you forgot to ask. ### 2. systematic-debugging - **Skill:** [obra/superpowers ยท systematic-debugging](https://github.com/obra/superpowers/tree/main/skills/systematic-debugging) (209k stars) - **What it does:** forces a four-step bug protocol: reproduce deterministically, hypothesize root cause, prove or disprove with the smallest test, fix only the root cause. - **Use it for:** every flaky test and every "why is this slow?" investigation. Replaces try-catch reflex with real root-cause work. ### 3. test-driven-development - **Skill:** [obra/superpowers ยท test-driven-development](https://github.com/obra/superpowers/tree/main/skills/test-driven-development) (209k stars) - **What it does:** writes the failing test first, runs it, shows the failure, then writes the implementation. Re-runs after each pass. - **Use it for:** any new function or feature where correctness matters. Stops Claude from "verifying" by re-reading its own code. ### 4. writing-plans - **Skill:** [obra/superpowers ยท writing-plans](https://github.com/obra/superpowers/tree/main/skills/writing-plans) (209k stars) - **What it does:** turns a vague request into a numbered plan with critical files identified, tradeoffs called out, and explicit checkpoints. - **Use it for:** any task that touches 3+ files or crosses a boundary (API, DB, infra). Prevents "looks done but breaks in three places." ### 5. executing-plans - **Skill:** [obra/superpowers ยท executing-plans](https://github.com/obra/superpowers/tree/main/skills/executing-plans) (209k stars) - **What it does:** runs a plan one step at a time in a separate session, with explicit checkpoints between steps. - **Use it for:** following up on a writing-plans output. Lets you resume mid-plan after a fix or a context switch. ### 6. subagent-driven-development - **Skill:** [obra/superpowers ยท subagent-driven-development](https://github.com/obra/superpowers/tree/main/skills/subagent-driven-development) (209k stars) - **What it does:** spawns a subagent per independent subtask, runs them in parallel, merges results. Enforces clean handoff contracts. - **Use it for:** any task with parallelizable pieces (audit v1 vs v2 API, scan 5 services, compare 3 SDKs). Saves wall time and context budget. ### 7. verification-before-completion - **Skill:** [obra/superpowers ยท verification-before-completion](https://github.com/obra/superpowers/tree/main/skills/verification-before-completion) (209k stars) - **What it does:** runs the actual app or test suite end-to-end before reporting the task done. Surfaces evidence of what worked and what broke. - **Use it for:** the "tests pass but the feature is broken" class of bug. Pair with brainstorming + TDD for the full discipline loop. ### 8. skill-creator - **Skill:** [anthropics/skills ยท skill-creator](https://github.com/anthropics/skills/tree/main/skills/skill-creator) (parent repo 142k stars) - **What it does:** interviews you on trigger language, steps, and references, then writes a new `SKILL.md` in the right shape. - **Use it for:** every repeatable task you do more than twice a week. The meta-skill that pays for itself the first time. ### 9. graphify - **Skill:** [safishamsi/graphify](https://github.com/safishamsi/graphify) (55k stars) - **What it does:** runs an AST pass over your project, identifies god nodes, maps community structure, writes a queryable wiki under `graphify-out/`. - **Use it for:** any repo over 30k lines. Future sessions read the wiki instead of grep-walking the file tree. ### 10. mcp-builder - **Skill:** [anthropics/skills ยท mcp-builder](https://github.com/anthropics/skills/tree/main/skills/mcp-builder) (parent repo 142k stars) - **What it does:** scaffolds an MCP server end-to-end (transport choice, tool annotations, auth, deployment hints). Generates a working Spring/Node/Python skeleton. - **Use it for:** the first MCP server you build. Replaces the "read 4 docs to get started" ritual with one prompt. ## Which Claude skills should you install first? Install in this order: 1. Run `/plugin install superpowers@claude-plugins-official` in Claude Code. Pulls 7 of the 10 skills above. 2. Clone [anthropics/skills](https://github.com/anthropics/skills) into `~/.claude/skills/anthropic-skills/`. Gives you skill-creator, mcp-builder, and a wider base library. 3. Install [safishamsi/graphify](https://github.com/safishamsi/graphify) when your project crosses 30k lines. 4. Run skill-creator once. Build a custom skill for the most-repeated task in your current project. Writing one teaches you how to read others. The thing nobody says out loud: the compounding only kicks in if you actually use them. Skills are habits, not features. Install one. Use it for two weeks. Then install the next. For more on Claude Skills internals, see the [Anthropic engineering deep-dive on agent skills](https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills), the [Claude Code skills documentation](https://code.claude.com/docs/en/skills), and the [open skills standard at agentskills.io](https://agentskills.io). ### Keep Reading - [Claude Skills vs MCP vs Projects](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects). Where skills end and MCP servers begin. - [Claude Code Routines: Async CI Automation Just Became Real](https://www.rabinarayanpatra.com/blogs/claude-code-routines-ci-automation). Use skills inside scheduled CI runs. - [Claude Opus 4.7 Release and Migration Guide](https://www.rabinarayanpatra.com/blogs/claude-opus-4-7-release-and-migration-guide). The model these skills run on top of. --- ## How to Set Up Sanity Studio in Next.js 16 (Embedded Dashboard) Source: https://www.rabinarayanpatra.com/blogs/how-to-set-up-sanity-studio-nextjs-16 Published: 2026-06-05T00:00:00+05:30 Tags: sanity-studio-nextjs-setup, sanity-cms, nextjs-16, headless-cms, embedded-studio, typescript, app-router How to set up Sanity Studio in Next.js 16 step by step: install next-sanity, mount the embedded /studio route, write a real post schema, query from a Server Component, fix CORS, and deploy to Vercel. The default mental model for a headless CMS is two places: the CMS dashboard lives over there on the vendor domain, and your site lives over here on Vercel. Editors bounce between tabs, you copy environment variables between systems, and somebody always forgets to allowlist a domain in CORS. Sanity does not require any of that. The Studio is a React app you ship inside your Next.js project. One catch-all route mounts the editor at `/studio`, your schemas live in TypeScript next to the rest of your code, and the whole thing deploys on the same Vercel push that ships your blog. This post walks through the full setup for Next.js 16: install, config, a real post + author + category schema, querying from a Server Component, the CORS fix every first-time user hits, and the Vercel deploy. I built this exact integration into a small content site I run, and I will use that as the running example. If you already have a Next.js 16 app and want a working CMS dashboard inside it by the end of the next hour, you are in the right place. ![Sanity Studio embedded in a Next.js 16 app. Left panel shows the /studio route with Post, Author, and Category document types and a Publish button. Right panel shows the /blog route rendering published posts.](/images/blogs/how-to-set-up-sanity-studio-nextjs-16.webp) ## What is Sanity Studio and why embed it in Next.js? Sanity Studio is the editor UI that ships with Sanity, a hosted content platform that splits cleanly into two pieces: an open-source React app for editors, and a managed document store called the Content Lake. The Studio is the part you customize. The Lake is the part you query. The reason to embed it in Next.js, instead of running it on the default `sanity.studio` subdomain, is that almost every project ends up wanting the same three things. Editors should sign in once on the same domain as the site. Schemas should live in TypeScript next to the components that render them. Deploys should be one Vercel push, not two. Embedding gets you all three for the cost of one catch-all route. The latest stable release is `sanity@5.30.0`, with the official Next.js toolkit at `next-sanity@13.0.11` as of June 2026. Both target the App Router and React 19, and Sanity migrated its own docs platform to Next.js 16 earlier this year, so the integration path is well-trodden. ## What do you need before installing Sanity in Next.js 16? You need a working Next.js 16 App Router project on Node 20 or newer, plus a free Sanity account. That is the entire prerequisite list. Concretely: - **Node 20+.** The current Studio package drops support for Node 18 and 19. Check with `node -v`. - **A Next.js 16 app on the App Router.** Pages Router works too with a different route file, but this guide assumes App Router because that is what Next.js 16 starters default to. - **A free Sanity account at `sanity.io`.** The free tier gives you a project, a dataset, three editor seats, and 10k API requests per month, which is enough for a personal site. - **A package manager.** I use `npm`, but `pnpm` and `yarn` work without changes. If you are starting from scratch, the rest of this guide assumes the same shape as a typical Next.js 16 app: a `src/app` directory, TypeScript on, Tailwind optional. The same steps apply if you keep `app` at the repo root. ## How do you scaffold Sanity Studio inside an existing Next.js app? Run `npx sanity@latest init` at the root of your Next.js project, then install the Next.js bridge package separately. The init command creates the Sanity project on the server, writes a starter `sanity.config.ts`, and adds a `sanity/` directory with example schemas. The full sequence from a clean Next.js 16 repo: ```bash # 1. Provision the project and write starter config npx sanity@latest init # 2. Install the Next.js bridge and the image URL helper npm install next-sanity @sanity/image-url ``` When `sanity init` runs, it walks you through a short CLI prompt: 1. Sign in with email, GitHub, or Google (browser tab opens). 2. Pick "Create new project" and give it a name. 3. Pick `production` as the default dataset. 4. Choose "Yes" when it asks to add the example schema (you will replace it shortly). 5. Pick TypeScript and the `npm` package manager when prompted. The CLI prints two values at the end: a project ID (an eight-character string) and a dataset name (`production`). Drop them into `.env.local` exactly as they are: ```bash # .env.local NEXT_PUBLIC_SANITY_PROJECT_ID=your_project_id NEXT_PUBLIC_SANITY_DATASET=production SANITY_API_READ_TOKEN=optional_for_drafts ``` The `NEXT_PUBLIC_` prefix is required. The Studio runs in the browser and needs to read both values from `process.env` at build time. The third variable is only needed later if you want the public site to render unpublished drafts for previews. ## How do you wire the embedded Studio route at /studio? Create two files: `sanity.config.ts` at the project root, and a catch-all route at `app/studio/[[...tool]]/page.tsx`. The route imports the config and hands it to the `NextStudio` component, which does the actual rendering. The config file is where you declare the project ID, dataset, base path, plugins, and your schema list. A minimal version for a blog looks like this: ```ts // sanity.config.ts import { defineConfig } from 'sanity'; import { structureTool } from 'sanity/structure'; import { visionTool } from '@sanity/vision'; import { schemaTypes } from './sanity/schemaTypes'; export default defineConfig({ name: 'default', title: 'My Blog Studio', projectId: process.env.NEXT_PUBLIC_SANITY_PROJECT_ID!, dataset: process.env.NEXT_PUBLIC_SANITY_DATASET!, basePath: '/studio', plugins: [structureTool(), visionTool()], schema: { types: schemaTypes }, }); ``` Two things matter here. `basePath: '/studio'` tells the Studio where it is mounted, so internal links point at `/studio/structure/post` and not the root. `plugins` always includes `structureTool()` (the document list and editor) and, for development, `visionTool()` (a GROQ query playground). The route file is short. It imports the config, exports the `metadata` and `viewport` that `next-sanity` already prepared, and renders the Studio: ```tsx // app/studio/[[...tool]]/page.tsx import { NextStudio } from 'next-sanity/studio'; import config from '../../../sanity.config'; export const dynamic = 'force-static'; export { metadata, viewport } from 'next-sanity/studio'; export default function StudioPage() { return ; } ``` `force-static` makes the route itself a static shell. The Studio is a heavy client bundle and there is no benefit to re-rendering the shell on every request. The actual editor work happens client-side once the shell ships. Once these two files exist, start the dev server with `npm run dev` and open `http://localhost:3000/studio`. The Studio loads, asks you to sign in, and then shows the document list. Right now the list is empty because you have not defined any schema types yet. ![Sanity Studio empty document type picker with Artist, Event, and Venue options. Image: Sanity Inc.](/images/blogs/how-to-set-up-sanity-studio-nextjs-16-studio-dashboard.webp) _Source: [Sanity Learn: Day one with Sanity Studio](https://www.sanity.io/learn/course/day-one-with-sanity-studio)_ The screenshot above is from the official Sanity learn course, showing the Studio chrome and the document creation menu populated by schema types. Once you add the schemas in the next section, your menu will show Post, Author, and Category in place of those examples. This three-layer architecture is the part that makes the embedded approach worthwhile. The same browser session talks to two Next.js routes that talk to the same Sanity project, with the editor on the write path and the public pages on the read path: ![Three-layer architecture diagram. Top: Browser. Middle: Next.js 16 App Router with /studio mounting NextStudio for the write path and /blog using a Server Component with sanityClient.fetch for the read path. Bottom: Sanity Content Lake holding the project ID, dataset, and CORS allowlist.](/images/blogs/how-to-set-up-sanity-studio-nextjs-16-architecture.webp) ## How do you write a real content schema (post + author + category)? Create a `sanity/schemaTypes/` directory with one file per document type, then export them as an array from `sanity/schemaTypes/index.ts`. Each file describes the shape of one document using the `defineType` and `defineField` helpers from the `sanity` package. For a blog you want three documents. A `post` holds the article. An `author` holds the person who wrote it. A `category` groups posts. Posts reference authors and categories, so editors fill in those fields with a dropdown instead of free text. Start with `post.ts`: ```ts // sanity/schemaTypes/post.ts import { defineField, defineType } from 'sanity'; export const post = defineType({ name: 'post', title: 'Post', type: 'document', fields: [ defineField({ name: 'title', type: 'string', validation: (r) => r.required().max(80), }), defineField({ name: 'slug', type: 'slug', options: { source: 'title', maxLength: 96 }, validation: (r) => r.required(), }), defineField({ name: 'author', type: 'reference', to: [{ type: 'author' }] }), defineField({ name: 'category', type: 'reference', to: [{ type: 'category' }], }), defineField({ name: 'mainImage', type: 'image', options: { hotspot: true }, fields: [{ name: 'alt', type: 'string', title: 'Alt text' }], }), defineField({ name: 'publishedAt', type: 'datetime' }), defineField({ name: 'body', type: 'blockContent' }), ], }); ``` A few choices worth flagging. `validation: r => r.required().max(80)` runs in the Studio in real time, so editors see the red warning the moment a title is empty or too long. `options: { source: 'title' }` on the slug field wires up the "Generate" button, which converts the title to a URL slug. `options: { hotspot: true }` on the image field lets editors pick a focal point so Sanity can crop responsively without cutting off heads. `author.ts` and `category.ts` are smaller: ```ts // sanity/schemaTypes/author.ts import { defineField, defineType } from 'sanity'; export const author = defineType({ name: 'author', title: 'Author', type: 'document', fields: [ defineField({ name: 'name', type: 'string', validation: (r) => r.required() }), defineField({ name: 'slug', type: 'slug', options: { source: 'name' }, validation: (r) => r.required(), }), defineField({ name: 'avatar', type: 'image', options: { hotspot: true } }), defineField({ name: 'bio', type: 'text', rows: 3 }), ], }); ``` ```ts // sanity/schemaTypes/category.ts import { defineField, defineType } from 'sanity'; export const category = defineType({ name: 'category', title: 'Category', type: 'document', fields: [ defineField({ name: 'title', type: 'string', validation: (r) => r.required() }), defineField({ name: 'slug', type: 'slug', options: { source: 'title' }, validation: (r) => r.required(), }), defineField({ name: 'description', type: 'text', rows: 2 }), ], }); ``` The `blockContent` type that the post body uses is the portable text format Sanity ships for rich text. It is a one-line export: ```ts // sanity/schemaTypes/blockContent.ts import { defineType } from 'sanity'; export const blockContent = defineType({ name: 'blockContent', title: 'Block Content', type: 'array', of: [{ type: 'block' }, { type: 'image', options: { hotspot: true } }], }); ``` The index file wires them together: ```ts // sanity/schemaTypes/index.ts import { post } from './post'; import { author } from './author'; import { category } from './category'; import { blockContent } from './blockContent'; export const schemaTypes = [post, author, category, blockContent]; ``` Reload `http://localhost:3000/studio`. The Studio picks up the new types and the document list now shows Post, Author, and Category. Create one of each. The reference fields will let you link a post to the author and category you just made. ![Sanity Studio document editor showing the Cosmic Harmony Festival document with the Name field. Image: Sanity Inc.](/images/blogs/how-to-set-up-sanity-studio-nextjs-16-schema-editor.webp) _Source: [Sanity Learn: Day one with Sanity Studio](https://www.sanity.io/learn/course/day-one-with-sanity-studio)_ This is what the editor looks like for a real document: the list pane on the left, the document with its declared fields on the right, drafts and publish state at the top. With the schema above, your screen looks the same shape, with your post fields in place of the Name shown here. ## How do you query Sanity content from a Server Component? Create a thin client wrapper at `lib/sanity/client.ts`, then import it from any Server Component that needs to render content. The client uses GROQ, Sanity's query language, which feels like JSON Path with projections. The wrapper is short: ```ts // lib/sanity/client.ts import { createClient } from 'next-sanity'; export const sanityClient = createClient({ projectId: process.env.NEXT_PUBLIC_SANITY_PROJECT_ID!, dataset: process.env.NEXT_PUBLIC_SANITY_DATASET!, apiVersion: '2026-06-01', useCdn: true, }); export const allPostsQuery = `*[_type == "post"] | order(publishedAt desc) { _id, title, "slug": slug.current, publishedAt, "author": author->name, "category": category->title }`; ``` A few specifics. `apiVersion` is a calendar date that pins query behavior, so a Sanity-side change does not silently alter your responses. Update it when you adopt a new feature. `useCdn: true` reads from the public CDN, which is fast and free for published content. Set it to `false` for draft previews where you want fresh data. GROQ itself is compact. `*[_type == "post"]` filters every document by type. The `|` pipe sorts. The projection block at the end picks fields and rewrites them: `"slug": slug.current` flattens the slug object to a string, and `"author": author->name` follows the reference and pulls the author's name in one query. A Server Component that lists posts is then just: ```tsx // app/blog/page.tsx import { sanityClient, allPostsQuery } from '@/lib/sanity/client'; type PostRow = { _id: string; title: string; slug: string; publishedAt: string; author: string; category: string; }; export default async function BlogIndexPage() { const posts = await sanityClient.fetch(allPostsQuery); return (
    {posts.map((p) => (
  • {p.title} {' '} by {p.author} in {p.category}
    • ))}
); } ``` This is a Server Component, so the query runs at request time on Vercel, not in the browser. The response is HTML by the time it reaches the user. For static caching, swap to Cache Components with `'use cache'` and a `cacheTag` keyed off `post`, and trigger revalidation from a Sanity webhook on publish. ## How do you configure CORS so the Studio talks to your project? Open `manage.sanity.io`, pick your project, go to **API**, scroll to **CORS origins**, and add every origin where the embedded Studio will run. Each entry needs the "Allow credentials" toggle turned on. The minimum allowlist for a Vercel-hosted Next.js app is three rows: | Origin | When it matters | Allow credentials | | ------------------------------------------ | ------------------- | ----------------- | | `http://localhost:3000` | Local dev | ON | | `https://www.your-domain.com` | Production | ON | | `https://preview-*.your-domain.vercel.app` | Vercel preview URLs | ON | The reason "Allow credentials" must be ON is that the embedded Studio sends the Sanity editor session cookie on every API call. Without that toggle, the browser strips the cookie before sending the request, Sanity sees an anonymous call, and you get a blank Studio with a console error like `CORS policy: No 'Access-Control-Allow-Credentials' header is present on the requested resource`. It is the single most common failure when first setting up the embedded Studio. ![Sanity manage console CORS origins panel. Three rows are listed with Allow credentials toggled ON: http://localhost:3000, https://www.your-domain.com, and https://preview-*.your-domain.vercel.app. A red callout explains why Allow credentials must be on.](/images/blogs/how-to-set-up-sanity-studio-nextjs-16-cors.webp) Restart `npm run dev` after adding the localhost entry. The dev server picks up the new CORS allowance on the next page load, and the Studio finishes mounting instead of stalling on the login screen. ## How do you deploy the embedded Studio to Vercel? Add the two `NEXT_PUBLIC_SANITY_*` env vars to your Vercel project, push the branch, and the embedded Studio is reachable at `your-domain.com/studio` as soon as the deploy goes green. There is no separate Sanity build step. The full Vercel checklist: 1. **Environment variables.** In the Vercel dashboard, go to your project, then **Settings**, then **Environment Variables**. Add `NEXT_PUBLIC_SANITY_PROJECT_ID` and `NEXT_PUBLIC_SANITY_DATASET` for all three environments (Production, Preview, Development). Add `SANITY_API_READ_TOKEN` only if you wired the draft preview flow. 2. **CORS for production and preview URLs.** Back in `manage.sanity.io`, make sure both `https://www.your-domain.com` and `https://*-your-team.vercel.app` are in the CORS allowlist with credentials on. Preview URLs change per commit, so the wildcard saves you from re-adding them on every branch. 3. **Push.** The next push triggers a Vercel build. The build compiles the Studio bundle as part of the Next.js build and ships it as a static route, so cold starts are not part of the editor experience. 4. **Test the editor.** Open `https://www.your-domain.com/studio`, sign in, create a document, and confirm the content shows up on the public route. You can skip the hosted `sanity.studio` deployment entirely. That product is useful when you want the editor on a different domain from the marketing site (for example, when the marketing site is on a different stack), but for a single Next.js app, the embedded route is fewer moving parts. ## What are the common pitfalls in a Next.js 16 + Sanity setup? Most first-run problems come from four places: missing env vars at build time, missing CORS entries, the wrong `apiVersion`, and treating the Studio as a normal Next.js route. In order of how often I have hit them: - **`process.env.NEXT_PUBLIC_SANITY_PROJECT_ID` is undefined in production.** Vercel does not read `.env.local`. Set the variables in the dashboard with the `NEXT_PUBLIC_` prefix exactly, then redeploy. A redeploy is required for env changes to take effect. - **CORS blank screen on preview URLs.** Vercel preview URLs change per branch. Add `https://*-your-team.vercel.app` once instead of one entry per branch. - **Stale data after publishing.** If you have ISR or Cache Components on the read path, publish events do not invalidate them automatically. Wire a Sanity webhook to a Next.js route that calls `revalidateTag` or `updateTag` on publish. - **Studio is wrapped in your site layout.** The catch-all route under `app/studio/[[...tool]]/page.tsx` inherits any layout above it. If your root layout adds a navbar or padding, the Studio renders inside it. Wrap the Studio route in its own segment with a layout that returns the children as-is, so the editor takes the full viewport. If you build the site on the same Next.js 16 stack I write about, you have probably already seen related setups in [the proxy.ts post about routing middleware](https://www.rabinarayanpatra.com/blogs/hello-proxy-ts-nextjs-16) and [the docs generator post about content-driven pages](https://www.rabinarayanpatra.com/blogs/building-modern-docs-generator-nextjs-16). The Sanity Studio embed is the read/write half of the same problem those posts cover from the routing and rendering sides. ## What did we just build? A real CMS dashboard at `/studio` inside a Next.js 16 app, backed by a typed schema for posts, authors, and categories, queried from Server Components, with CORS configured for local and production, deployed on the same Vercel build as the public site. No extra infrastructure, no second domain, no separate auth flow. The whole thing fits in two config files, one route, and a `sanity/` directory. The next step depends on what you ship next. If editors will write drafts you want to preview before publishing, set up the Presentation tool and a draft-mode route on the Next.js side. If you want to render rich text from the `body` field, install `@portabletext/react` and write a custom renderer for your design system. If you want full-text search, add `searchTool` to the plugins list and you get a search bar in the Studio header for free. For reference, the primary docs I used while writing this are the [official Next.js integration guide](https://www.sanity.io/docs/nextjs), the [embedding Sanity Studio doc](https://www.sanity.io/docs/studio/embedding-sanity-studio), the [Sanity Studio installation requirements](https://www.sanity.io/docs/studio/installation), the [next-sanity toolkit on GitHub](https://github.com/sanity-io/next-sanity), and the [Sanity platform changelog covering the Next.js 16 migration](https://www.sanity.io/docs/changelog/06f976e4-865b-41df-a96c-3daca52640a3). The Studio screenshots in this post are from the [Day One with Sanity Studio learn course](https://www.sanity.io/learn/course/day-one-with-sanity-studio) on the Sanity site. ### Keep Reading - [Hello, proxy.ts in Next.js 16: middleware renamed and reframed](https://www.rabinarayanpatra.com/blogs/hello-proxy-ts-nextjs-16) - [Building a modern docs generator in Next.js 16](https://www.rabinarayanpatra.com/blogs/building-modern-docs-generator-nextjs-16) - [Next.js 16.2: AGENTS.md and the Next browser](https://www.rabinarayanpatra.com/blogs/nextjs-16-2-agents-md-next-browser) - [How to build an interactive MCP app with the MCP Apps SDK](https://www.rabinarayanpatra.com/blogs/build-mcp-app-interactive-ui) --- ## How to Version REST APIs in Spring Framework 7 (Spring Boot 4) Source: https://www.rabinarayanpatra.com/blogs/how-to-version-rest-apis-spring-framework-7 Published: 2026-06-03T00:00:00+05:30 Tags: spring-framework-7-api-versioning, spring-boot-4, rest-api, spring-mvc, api-versioning, java Spring Framework 7 API versioning is built in. Learn the version attribute, all four resolver strategies, baseline versions, and Sunset deprecation headers. For years, versioning a Spring REST API meant picking your own poison. Custom interceptors, duplicate controllers, or a pile of `headers=` conditions on every mapping. Spring Framework 7, the core of Spring Boot 4, ends that. Versioning is now a first-class feature baked into request mapping. I rebuilt a multi-version API on Spring Boot 4 last month and deleted about 300 lines of routing glue in the process. This guide is the playbook I wish I'd had: every strategy, real controller code, a `curl` trace per approach, and the deprecation headers that tell clients when to move on. ![Spring Framework 7 API versioning overview showing a versioned request routed to the correct controller handler](/images/blogs/how-to-version-rest-apis-spring-framework-7.webp) ## What changed for API versioning in Spring Framework 7? Spring Framework 7 added a `version` attribute to `@RequestMapping` and every shortcut variant, so one path can fan out to multiple handlers by version. Before this, request mapping had no concept of a version at all. You bolted versioning on from the outside with interceptors or separate controller classes per release. The new model has three moving parts. A resolver pulls the version out of the request. A parser turns that raw string into a comparable semantic version. A strategy matches it against the `version` declared on each mapping and picks the closest handler. All three are configurable, and the defaults cover the common cases. Here's the smallest possible example. Same path, two versions, two methods: ```java @RestController @RequestMapping("/accounts") public class AccountController { @GetMapping(path = "/{id}", version = "1.0") public AccountV1 getAccountV1(@PathVariable Long id) { return accountService.findV1(id); } @GetMapping(path = "/{id}", version = "2.0") public AccountV2 getAccountV2(@PathVariable Long id) { return accountService.findV2(id); } } ``` A request for version `1.0` hits the first method. A request for `2.0` hits the second. No `if` checks, no manual parsing, no shared dispatcher method. The version is part of the mapping, exactly like the path and the HTTP method. ## How do you turn on API versioning in Spring Boot 4? You turn on versioning by implementing `WebMvcConfigurer` and overriding `configureApiVersioning`, which hands you an `ApiVersionConfigurer`. Nothing routes by version until you declare which resolver to use. The `version` attribute on a mapping is inert without a configured strategy. ```java @Configuration public class ApiVersioningConfig implements WebMvcConfigurer { @Override public void configureApiVersioning(ApiVersionConfigurer configurer) { configurer.useRequestHeader("X-API-Version") .addSupportedVersions("1.0", "2.0"); } } ``` That's the entire wiring for header-based versioning. `useRequestHeader` names the header to read. `addSupportedVersions` declares the versions you accept, which lets Spring reject anything unknown with a clean 400 instead of a confusing 404. If you'd rather stay in properties, Spring Boot 4 exposes the same switch: ```properties spring.mvc.apiversion.use.header=X-API-Version ``` By default a version is required. A request with no version triggers `MissingApiVersionException` and a 400 response. An unsupported version triggers `InvalidApiVersionException`, also a 400. You can relax both, and I'll cover that under pitfalls, because the default trips up first-time users. ## How does the version attribute route requests? The `version` attribute routes a request by comparing the resolved request version against the version declared on each candidate mapping, then choosing the best match. Spring parses both sides with `SemanticApiVersionParser`, which reads `major.minor.patch` and fills missing parts with zero. So `"1"` becomes `1.0.0` and `"1.2"` becomes `1.2.0`. Matching is not a naive string equality. If a request asks for `1.5` and your handlers declare `1.0` and `2.0`, the request resolves to the highest version less than or equal to `1.5`, which is `1.0`. That single rule is what makes baseline versioning work, and it's why you don't need a handler for every point release. ![Diagram of the Spring Framework 7 version resolution flow from request to resolver to parser to matched handler](/images/blogs/how-to-version-rest-apis-spring-framework-7-routing.webp) The flow is the same no matter which strategy you choose. Only the first box, the resolver, changes. Pick where the version lives in the request and the rest of the pipeline stays identical. ## Which versioning strategy should you pick? Spring Framework 7 ships four resolver strategies, and you choose one with a single method on `ApiVersionConfigurer`. Each reads the version from a different place in the request. The handler code never changes. Only the config line and the way clients call you differ. ![Comparison table of the four Spring Framework 7 API versioning strategies: path segment, request header, query parameter, and media type](/images/blogs/how-to-version-rest-apis-spring-framework-7-strategies.webp) ### Request header Header versioning keeps the URL clean and puts the version in a custom header. This is my default for service-to-service APIs. ```java configurer.useRequestHeader("X-API-Version") .addSupportedVersions("1.0", "2.0"); ``` ```bash curl -H "X-API-Version: 2.0" http://localhost:8080/accounts/42 ``` ### Path segment Path-segment versioning puts the version directly in the URL, which makes it visible, bookmarkable, and easy to route at a proxy. You declare which segment holds the version by index and add a URI variable for it in the mapping. ```java configurer.usePathSegment(1) .addSupportedVersions("1.0", "2.0"); ``` ```java @GetMapping(path = "/api/{version}/accounts/{id}", version = "2.0") public AccountV2 getAccount(@PathVariable Long id) { return accountService.findV2(id); } ``` ```bash curl http://localhost:8080/api/2.0/accounts/42 ``` Segment index is zero-based against the path. In `/api/2.0/accounts/42` the segments are `api`, `2.0`, `accounts`, `42`, so the version sits at index `1`. ### Query parameter Query-parameter versioning reads the version from the query string. It's trivial to test in a browser and trivial to forget in a cache key, so weigh that tradeoff. ```java configurer.useQueryParam("version") .addSupportedVersions("1.0", "2.0"); ``` ```bash curl "http://localhost:8080/accounts/42?version=2.0" ``` ### Media type Media-type versioning reads a parameter off the `Accept` header, which is the most REST-purist option and the hardest for casual clients to send. ```java configurer.useMediaTypeParameter(MediaType.APPLICATION_JSON, "version") .addSupportedVersions("1.0", "2.0"); ``` ```bash curl -H "Accept: application/json;version=2.0" http://localhost:8080/accounts/42 ``` My rule of thumb: header for internal APIs, path segment for public APIs where humans read the URLs, and skip query and media type unless you have a specific reason. The worst choice is no choice, where different endpoints use different strategies and clients can't predict which one applies. ## What's the difference between fixed and baseline versions? A fixed version matches one exact version, while a baseline version matches that version and everything above it until a higher handler takes over. You write a fixed version as `"1.2"` and a baseline version with a trailing plus, `"1.2+"`. The difference is how many releases a single handler is responsible for. Picture an endpoint that hasn't changed since `1.0` and another that got a new shape in `2.0`: ```java @RestController @RequestMapping("/accounts") public class AccountController { @GetMapping(path = "/{id}", version = "1.0+") public AccountV1 getAccount(@PathVariable Long id) { return accountService.findV1(id); } @PostMapping(version = "2.0+") public AccountV2 createAccount(@RequestBody CreateAccount body) { return accountService.createV2(body); } } ``` The `GET` handler answers `1.0`, `1.5`, `1.9`, and anything up to the next declared version. The `POST` handler owns `2.0` and up. You only write a new method when the contract actually changes, not on every version bump. That's the whole point of baseline matching, and it's why a real API with ten releases might have three or four handlers per route instead of ten. ![Diagram contrasting a fixed version that matches one release against a baseline version that matches a range of releases](/images/blogs/how-to-version-rest-apis-spring-framework-7-baseline.webp) Fixed versions still matter. Use them when a single release introduced a breaking change you want pinned to one exact handler, so a typo in a client's version string fails loudly instead of silently falling through to an older shape. ## How do you deprecate an API version with Sunset headers? You deprecate a version by registering a `StandardApiVersionDeprecationHandler` and configuring a deprecation date, a sunset date, and a migration link per version. Spring then attaches three response headers to every matching request: `Deprecation` from RFC 9745, `Sunset` from RFC 8594, and a `Link` pointing at your migration docs. Clients that watch for these headers learn a version is going away without reading your changelog. ```java @Configuration public class ApiVersioningConfig implements WebMvcConfigurer { @Override public void configureApiVersioning(ApiVersionConfigurer configurer) { StandardApiVersionDeprecationHandler handler = new StandardApiVersionDeprecationHandler(); handler.configureVersion("1.0") .setDeprecationDate(ZonedDateTime.parse("2026-06-01T00:00:00Z")) .setSunsetDate(ZonedDateTime.parse("2026-12-01T00:00:00Z")) .setSunsetLink(URI.create("https://api.example.com/docs/migrate-to-2.0")); configurer.useRequestHeader("X-API-Version") .addSupportedVersions("1.0", "2.0") .setDeprecationHandler(handler); } } ``` Now a call to the deprecated version carries the warning in its response: ```bash curl -i -H "X-API-Version: 1.0" http://localhost:8080/accounts/42 ``` ```http HTTP/1.1 200 OK Deprecation: Mon, 01 Jun 2026 00:00:00 GMT Sunset: Tue, 01 Dec 2026 00:00:00 GMT Link: ; rel="sunset" Content-Type: application/json ``` ![HTTP response trace showing Deprecation, Sunset, and Link headers emitted by Spring Framework 7 for a deprecated API version](/images/blogs/how-to-version-rest-apis-spring-framework-7-deprecation.webp) The version still works. Deprecation is a signal, not a shutdown. When the sunset date passes and you're confident traffic has moved, you drop the `1.0` handler and remove `"1.0"` from the supported versions. Clients that ignored the headers get a clean 400, and you have the access logs to prove you warned them. ## How was this done before Spring 7? Before Spring 7 you versioned by hand, and every approach had a sharp edge. The most common pattern was duplicate controllers, one class per version, wired to different base paths: ```java @RestController @RequestMapping("/v1/accounts") public class AccountControllerV1 { /* ... */ } @RestController @RequestMapping("/v2/accounts") public class AccountControllerV2 { /* ... */ } ``` That works until you have twenty endpoints across four versions and a bug fix has to land in three of them. The other common hack abused the `headers` condition on a mapping: ```java @GetMapping(path = "/accounts/{id}", headers = "X-API-Version=1") public AccountV1 getV1(@PathVariable Long id) { /* ... */ } ``` String equality only. No `1.0` versus `1` normalization, no baseline ranges, no `1.5` falling back to `1.0`. You hand-rolled every comparison. Teams that wanted real semantics wrote a custom `HandlerInterceptor` to parse and validate versions, then threaded the result through `ThreadLocal` or request attributes. It was a lot of code to maintain, and it lived nowhere near the mappings it controlled. The built-in approach wins on every axis I care about. Versioning lives on the mapping where you can see it, comparison is semantic, baseline matching cuts handler count, and deprecation headers come free. The 300 lines I deleted were exactly this kind of glue. ## What pitfalls should you watch for? The first pitfall is the required-version default, which surprises everyone. A request with no version returns a 400, not your newest handler. If you want missing versions to fall back instead of failing, set a default and mark versions optional: ```java configurer.useRequestHeader("X-API-Version") .addSupportedVersions("1.0", "2.0") .setVersionRequired(false) .setDefaultVersion("2.0"); ``` With `setVersionRequired(false)` and no default, Spring falls back to the most recent supported version, which may not be what you want during a migration. Always pair optional versions with an explicit `setDefaultVersion` so the fallback is a decision, not an accident. The second pitfall is supported-version detection. By default Spring initializes the supported set from the versions it finds on your controller mappings, so a typo like `version = "20"` silently becomes a supported version. If you want a locked allowlist, call `detectSupportedVersions(false)` and declare every version with `addSupportedVersions` yourself. I do this on public APIs so nothing ships a version by accident. The third pitfall is OpenAPI tooling. As of mid-2026, springdoc support for the new `version` attribute is still catching up, so two handlers on the same path can confuse schema generation. Until your springdoc version understands the version dimension, group your docs per version or document the version header manually in your OpenAPI config. Test your generated spec before you assume it rendered both versions. The last one is strategy drift. Once you pick a resolver, every endpoint must use it. A header-versioned API with one path-versioned endpoint will route inconsistently and break client SDKs that assume one scheme. Decide the strategy at the start of the project and enforce it in review. ## Is built-in versioning worth migrating to? Yes, and it changes how you think about API evolution, not just how you wire it. When adding a version is one attribute and one method, you stop dreading breaking changes and start shipping them cleanly, because the cost of carrying an old contract next to a new one dropped to almost nothing. That's the real shift in Spring Framework 7. The mechanics are simple. The freedom they buy is the point. Start with header versioning and a required version on a single endpoint. Add a `2.0` handler with a baseline range. Wire a deprecation handler with a sunset date six months out. Once that loop feels natural, you'll version everything this way and wonder how you tolerated the interceptor era. For the authoritative details, read the official [Spring blog announcement on API versioning](https://spring.io/blog/2025/09/16/api-versioning-in-spring) and the [Spring Framework reference on MVC API versioning](https://docs.spring.io/spring-framework/reference/web/webmvc-versioning.html). For the header semantics, see [RFC 9745 (Deprecation)](https://www.rfc-editor.org/rfc/rfc9745.html) and [RFC 8594 (Sunset)](https://www.rfc-editor.org/rfc/rfc8594.html). ### Keep Reading - [How to Configure CORS in Spring Boot](https://www.rabinarayanpatra.com/blogs/spring-boot-cors-guide). Get cross-origin headers right before you expose a versioned API. - [How to Test Spring Boot with Testcontainers](https://www.rabinarayanpatra.com/blogs/spring-boot-testcontainers-guide). Write integration tests that exercise each API version against a real database. - [Spring Security's Component Revolution](https://www.rabinarayanpatra.com/blogs/spring-security-component-revolution). Secure the versioned endpoints you just built. - [Modern Java with Spring Boot](https://www.rabinarayanpatra.com/blogs/modern-java-spring-boot). The language features that make Spring Boot 4 controllers cleaner. --- ## How to Set Up an SSH Tunnel for Local Database Access Source: https://www.rabinarayanpatra.com/blogs/ssh-tunnel-local-database-access Published: 2026-05-29T00:00:00+05:30 Tags: ssh-tunnel, database, devops, postgres, security, how-to Set up an SSH tunnel to reach a remote Postgres, MySQL, or Redis from your laptop without exposing the database port. With autossh and a systemd unit. The single fastest way to wake up to a ransom note is to expose your Postgres port directly to the public internet. The second fastest is to think you can secure it with a strong password and call it a day. I have run point on database access for production systems at three different companies. Every single one of them had at least one engineer at some point ask if we could "just open 5432 to my IP" so they could pull a quick report from DBeaver. The answer is always no. The right answer is an SSH tunnel, and once you have done it twice, it takes about 15 seconds to set up. This post is the practical guide I wish I had given that engineer the first time. We will cover the command, the GUI client setup for the four databases I touch most often, the autossh recipe that keeps the tunnel alive through laptop sleeps and network changes, and the small set of mistakes that bite people in production. If you want the deeper context on why exposing database ports is bad even with strong auth, I covered that in my [zero-trust microservices post](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security), and the connection pooling tradeoffs once you are inside the tunnel show up in [my pgbouncer survival guide](https://www.rabinarayanpatra.com/blogs/postgres-connection-pool-pgbouncer-survival-guide). ## What is an SSH tunnel and why use it for database access? An SSH tunnel for database access forwards a local TCP port on your laptop through an authenticated SSH session to a remote host, which then opens a connection to the actual database. Your database client connects to `localhost` and never knows anything else exists. All traffic rides inside the encrypted SSH channel. The shape of it looks like this. ![SSH tunnel flow from laptop to bastion to database](/images/blogs/ssh-tunnel-local-database-access-flow.webp) This pattern wins on five things at once. **No exposed database port.** The database listens only on its private network. The bastion is the only thing on the public internet, and it only speaks SSH. **No firewall holes per developer.** Every engineer goes through the same bastion. You add or remove access by adding or removing SSH keys, not by editing security group rules. **Audit trail.** Every connection logs through SSH and through the bastion's auth.log. You know who connected, from where, when. **Works with every client.** psql, DBeaver, TablePlus, DataGrip, mysql, redis-cli, mongosh. They all just see `localhost`. No driver-level config needed. **Encrypted in transit by default.** Even if your database speaks plaintext on the wire (looking at you, Redis), the tunnel carries it under SSH. The one thing it does not give you is high availability. The tunnel is a long-lived TCP connection between exactly two hosts. If your bastion goes down, your tunnel goes with it. That is fine for ad-hoc developer access. It is not fine for application traffic. Application traffic belongs on a private network or a managed bastion service like AWS Session Manager. ## How do you set up a local port forward to a remote Postgres? The full command is one line. ```bash ssh -L 5433:dbhost.internal:5432 ec2-user@bastion.example.com ``` Read it left to right. - `-L 5433:dbhost.internal:5432` says forward local port `5433` on this laptop, through the SSH connection, to `dbhost.internal:5432` resolved from the bastion's perspective. - `ec2-user@bastion.example.com` is the SSH connection itself. While that command is running, anything on your laptop that connects to `localhost:5433` ends up talking to the Postgres on `dbhost.internal:5432`. Close the terminal or hit Ctrl+C and the tunnel dies. I deliberately use `5433` on the local side instead of `5432`. If you happen to have Postgres running locally for development, you do not want to clobber it. Pick a high port that does not collide. To test the tunnel works without firing up a GUI, use psql from another terminal. ```bash psql -h localhost -p 5433 -U app_user -d production ``` That `-h localhost` is the key. Without it, psql tries to connect via Unix socket and skips the tunnel entirely. I have lost 20 minutes to this twice. ### Running the tunnel in the background If you do not want a terminal sitting open, add `-f -N` and the command returns immediately while the tunnel keeps running. ```bash ssh -f -N -L 5433:dbhost.internal:5432 ec2-user@bastion.example.com ``` - `-N` means do not execute a remote command (we only want the forward). - `-f` means fork into the background after authentication. To kill it later, find and stop the process. ```bash pgrep -af "ssh.*5433:dbhost.internal" kill ``` ### A tidier setup with `~/.ssh/config` Typing that command every time gets old. Stash the whole thing in your SSH config. ``` # ~/.ssh/config Host pg-prod HostName bastion.example.com User ec2-user IdentityFile ~/.ssh/keys/prod-bastion.pem LocalForward 5433 dbhost.internal:5432 ServerAliveInterval 30 ServerAliveCountMax 3 ExitOnForwardFailure yes ``` Now the command is just `ssh pg-prod`, and you get a few useful behaviors for free. - `ServerAliveInterval 30` sends a keepalive every 30 seconds so the tunnel does not die when your home router decides to drop idle connections. - `ExitOnForwardFailure yes` makes ssh fail fast if the forward cannot bind, instead of leaving you with a dead tunnel and no error. For the background form, `ssh -f -N pg-prod` still works. The config entry is purely additive. ## How do you connect from your GUI client through the tunnel? With the tunnel running, every GUI client connects to `localhost` on the forwarded port. The only field that matters is the host. Everything else (username, password, database name) is the same as you would use against the real database. In DBeaver, create a new Postgres connection and fill in: ``` Host: localhost Port: 5433 Database: production Username: app_user Password: ``` Hit Test Connection. If it works, you are good. In TablePlus, same idea. Host is `localhost`, port is `5433`. In DataGrip, same. The driver does not know the tunnel exists, which is exactly the point. DBeaver and DataGrip both have a built-in SSH tunnel option in their connection dialog. That works too. The advantage of using ssh on the command line instead is that you can share one tunnel across psql, DBeaver, a script, and a notebook at the same time. The advantage of the GUI option is that it dies cleanly when you close the client. I default to the command line approach because I almost always have multiple things hitting the same database. Pick what fits your workflow. ## How do you do the same for MySQL, Redis, and MongoDB? The flag is identical. Only the port changes. ```bash # MySQL ssh -L 3307:dbhost.internal:3306 user@bastion # Redis ssh -L 6380:cache.internal:6379 user@bastion # MongoDB ssh -L 27018:mongohost.internal:27017 user@bastion ``` Then connect each client to `localhost` on the forwarded port. ```bash mysql -h 127.0.0.1 -P 3307 -u app_user -p redis-cli -h 127.0.0.1 -p 6380 mongosh "mongodb://app_user:secret@127.0.0.1:27018/production" ``` A few client-specific notes that catch people. **MySQL needs `127.0.0.1`, not `localhost`.** The mysql client tries to connect via Unix socket when you say `localhost`, just like psql does. Use the IP literal and you skip that. **Redis tunnels work great for debugging, badly for sustained throughput.** Every command round-trips through the SSH session. Latency goes from 0.2 ms to 8-30 ms depending on your link. Fine for `redis-cli MONITOR` or one-off lookups. Wrong tool for ETL. **Mongo replica sets need extra care.** A standalone mongod tunnels fine. A replica set will hand you back the internal hostnames of the other replicas during connection negotiation, and your client will then try to connect to those names directly. Either tunnel each replica on its own port and add them to your connection string, or set `directConnection=true` in the URI to disable replica discovery. ## How do you keep the tunnel alive with autossh and systemd? `ssh -f -N` works until your laptop goes to sleep, your wifi switches, or the bastion restarts. Then the tunnel dies silently and your next connection just hangs. The fix is autossh, which is a tiny wrapper that monitors the SSH process and restarts it when it drops. Install it. ```bash # macOS brew install autossh # Ubuntu / Debian sudo apt install autossh # Fedora / Rocky sudo dnf install autossh ``` Run it the same way you ran ssh, with one extra port for autossh's own health check. ```bash AUTOSSH_GATETIME=0 \ autossh -M 0 -f -N \ -o "ServerAliveInterval 30" \ -o "ServerAliveCountMax 3" \ -o "ExitOnForwardFailure yes" \ -L 5433:dbhost.internal:5432 \ ec2-user@bastion.example.com ``` A few details that matter. - `AUTOSSH_GATETIME=0` makes autossh restart immediately even if the first connection failed. Without it, autossh waits 30 seconds before retrying, which is annoying when you mis-typed the host. - `-M 0` disables autossh's old monitoring port mechanism. We use the `ServerAlive*` SSH options instead, which work better through NATs and modern firewalls. That command stays up through sleep, wake, and network changes. But it does not survive a reboot. For that, wrap it in a systemd user service. ### The systemd user service Create `~/.config/systemd/user/pg-tunnel.service`. ```ini [Unit] Description=Autossh tunnel to production Postgres After=network-online.target Wants=network-online.target [Service] Type=simple Environment=AUTOSSH_GATETIME=0 ExecStart=/usr/bin/autossh -M 0 -N \ -o "ServerAliveInterval 30" \ -o "ServerAliveCountMax 3" \ -o "ExitOnForwardFailure yes" \ -o "ConnectTimeout 10" \ -i /home/rabi/.ssh/keys/prod-bastion.pem \ -L 5433:dbhost.internal:5432 \ ec2-user@bastion.example.com Restart=always RestartSec=10 [Install] WantedBy=default.target ``` Notice the ExecStart does not use `-f` here. Systemd wants the process in the foreground so it can supervise it. The `Restart=always` line takes over from the `-f` background behavior. Enable and start. ```bash systemctl --user daemon-reload systemctl --user enable --now pg-tunnel.service systemctl --user status pg-tunnel.service ``` That last command should show `active (running)`. If you want the service to start at boot (not just at login), enable user lingering once. ```bash sudo loginctl enable-linger $USER ``` Logs are in journalctl. ```bash journalctl --user -u pg-tunnel.service -f ``` On macOS, the same job goes into a launchd plist at `~/Library/LaunchAgents/com.rabi.pg-tunnel.plist`. The shape is similar enough that I will skip the full XML, but the key entries are `KeepAlive` set to true and a `ProgramArguments` array that mirrors the autossh command above. ![Autossh under systemd keeps the tunnel alive across reboots](/images/blogs/ssh-tunnel-local-database-access-systemd.webp) ## What are the common SSH tunnel mistakes to avoid? I have made every one of these. So has everyone I have onboarded. **Binding the forward to 0.0.0.0 instead of 127.0.0.1.** The default `-L 5433:dbhost:5432` binds to localhost only. If you write `-L *:5433:dbhost:5432`, you have just opened your laptop's port 5433 to anyone on the same wifi network as you. They can now talk to your production Postgres. Do not do this. The local bind defaults to localhost for a reason. **Hardcoding production credentials in `~/.ssh/config`.** SSH config has no concept of secrets management. If you check your config into a dotfiles repo, that key path goes with it. Keep production keys outside the config-tracked area and reference them by absolute path from a directory that is gitignored. **Forgetting that `ServerAliveInterval` is client-side only.** This tells your laptop to send keepalives. The bastion can still close the connection on its own idle timeout. If your bastion is AWS Systems Manager based or sits behind a corporate load balancer with a short idle, you also need to set keepalives at the server level (`ClientAliveInterval` in sshd_config). Otherwise your tunnel drops every five minutes and you do not know why. **Trusting the tunnel to give you encryption end-to-end.** SSH encrypts the laptop-to-bastion segment. The bastion-to-database segment is in cleartext on your private network. For most production setups, that is fine because the private network is trusted. For sensitive workloads with stricter compliance requirements (PCI, HIPAA), insist on TLS on the database side too, even inside the VPC. **Tunneling through your jump host as root.** The bastion is a public-facing box. The account that holds your tunnel should not also be the one with sudo rights. Use a dedicated low-privilege user for tunneling. Restrict it to `command="false",no-shell` in `authorized_keys` if you want to be paranoid, with a `PermitOpen` directive that limits which `host:port` combos this key can forward to. The hardened authorized_keys line looks like this. ``` command="echo 'no shell',no-pty,no-agent-forwarding,no-X11-forwarding,permitopen="dbhost.internal:5432" ssh-ed25519 AAAA... rabi@laptop ``` That key can open exactly one forward, to exactly one host:port, and cannot execute a shell. If the key leaks, the attacker gets a tunnel to one database, not a shell on your bastion. **Letting the tunnel persist across job changes.** This one is process, not technical. When an engineer leaves the team, their SSH keys come out of every bastion's `authorized_keys`. The keys are in your [secrets manager](https://www.rabinarayanpatra.com/snippets/aws-cli/secrets-manager-get) and config-managed. Right? Right. **Using the same bastion for staging and production.** I have seen this break twice. A misconfigured `LocalForward` ends up pointing your TablePlus at production while you think you are on staging. Use separate bastions, separate config entries with distinct host names like `pg-prod` and `pg-stage`, and color-code the connections in your client so you cannot confuse them visually. ## What does the whole setup look like in 30 seconds? For the reader who skipped to the end, here is the compressed version. ```bash # One-off ssh -f -N -L 5433:dbhost.internal:5432 user@bastion psql -h localhost -p 5433 -U app_user -d production # Persistent, survives sleep and network changes brew install autossh AUTOSSH_GATETIME=0 autossh -M 0 -f -N \ -o "ServerAliveInterval 30" \ -L 5433:dbhost.internal:5432 user@bastion # Persistent, survives reboot (Linux) # Drop the autossh command into ~/.config/systemd/user/pg-tunnel.service # and run: systemctl --user enable --now pg-tunnel.service ``` That is genuinely all you need for 95% of the database access situations a developer hits. The whole reason this pattern works is that SSH is older than most of the protocols we layer on top of it, and the port-forwarding feature has been battle-tested for three decades. Use it. Stop opening database ports to the internet. Your future self will thank you the next time a Shodan-driven exploit campaign sweeps the internet looking for misconfigured Postgres instances, which is approximately every Tuesday. For more on the protocol, see the [OpenSSH manual page for ssh(1)](https://man.openbsd.org/ssh.1), the [autossh project page](https://www.harding.motd.ca/autossh/), and [RFC 4254](https://datatracker.ietf.org/doc/html/rfc4254) for the SSH connection protocol spec that defines port forwarding. ### Keep Reading - [Postgres Connection Pool: A pgbouncer Survival Guide](https://www.rabinarayanpatra.com/blogs/postgres-connection-pool-pgbouncer-survival-guide): once you can reach the database, the next question is how many connections you can hold open without melting it. - [Zero-Trust Microservices with Spring Security](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security): the bigger frame for why exposing internal ports is the wrong default, even inside a private network. --- ## Spring AI 2.0 MCP Server Tutorial: @McpTool to Prod Source: https://www.rabinarayanpatra.com/blogs/spring-ai-2-mcp-annotations-tutorial Published: 2026-05-27T00:00:00+05:30 Tags: spring-ai-2.0, mcp, spring-boot, model-context-protocol, claude-code Spring AI 2.0 MCP annotations tutorial: build a production MCP server with @McpTool, async progress reporting, Streamable HTTP transport, and Claude Code wiring. Spring AI 2.0.0-M6 dropped on May 8, 2026. Buried in the release notes was the thing I had been waiting for since I first wired an MCP server in Java six months ago: native annotations. `@McpTool`, `@McpResource`, `@McpPrompt`, `@McpComplete`. All in core. All auto-registered. If you've written an MCP server with the older Spring AI `ToolCallback` API, you remember the ritual. Build descriptors, register callbacks, wire up the transport manually, handle JSON schema by hand. The annotation API replaces all of it with a single annotation on a method. Spring AI generates the schema. Auto-configuration handles registration. You write the business logic. This tutorial walks the full path: building a production MCP server, exposing tools and resources, handling async work with progress reporting, picking a transport, registering with Claude Code, and the gotchas I've hit that nobody mentions on the docs. ## What changed in Spring AI 2.0 MCP annotations? Spring AI 2.0 collapses MCP server and client wiring into a small set of annotations that Spring Boot auto-configuration picks up at startup. The annotated beans become the MCP surface for your Spring Boot app. The core server annotations are: - `@McpTool` marks a method as an MCP tool. Spring AI builds the JSON schema from your method parameters. - `@McpResource` exposes a resource via a URI template. - `@McpPrompt` exposes a prompt template that clients can fetch. - `@McpComplete` provides auto-completion for prompt or resource arguments. The auto-configuration also injects special context parameters like `McpSyncRequestContext` and `McpAsyncRequestContext`, which give your method access to logging, progress reporting, sampling, and elicitation without polluting the JSON schema. ![Spring AI 2.0 MCP annotations layered architecture: Java annotations through Spring Boot auto-configuration to MCP transport](/images/blogs/spring-ai-2-mcp-annotations-tutorial-architecture.webp) For a Spring shop, this is the kind of API change that flips a project's complexity. Before, every team building an MCP server wrote the same plumbing. Now the plumbing is gone. ## How do you build an MCP server with @McpTool? Start with a Spring Boot 3.5+ project on Java 21 or higher. Add the MCP server starter to your `pom.xml`. There are three flavors: stdio/SSE default, WebMVC, and WebFlux. For production HTTP transport, pick WebMVC or WebFlux based on whether you want blocking or reactive code. ```xml org.springframework.ai spring-ai-starter-mcp-server-webmvc 2.0.0-M6 ``` Spring AI 2.0 milestones live in the Spring milestone repository, so add it if you haven't: ```xml spring-milestones Spring Milestones https://repo.spring.io/milestone ``` Now write a tool. The example everyone reaches for first is a calculator, but let's do something a real MCP client would actually want: fetching a user record from your database. ```java import org.springframework.ai.mcp.server.annotation.McpTool; import org.springframework.ai.mcp.server.annotation.McpToolParam; import org.springframework.stereotype.Component; @Component public class UserTools { private final UserRepository users; public UserTools(UserRepository users) { this.users = users; } @McpTool( name = "get_user", description = "Fetch a user record by ID. Returns name, email, and signup date." ) public UserDto getUser( @McpToolParam(description = "Numeric user ID", required = true) long userId ) { return users.findById(userId) .map(UserDto::from) .orElseThrow(() -> new IllegalArgumentException("User not found: " + userId)); } } ``` That's the whole tool. Spring AI does a few things automatically when it scans this bean: It builds a JSON schema for the `userId` parameter using the `@McpToolParam` description and the Java type. The MCP client sees the schema and shows it as a typed input. It registers the method with the MCP server runtime under the name `get_user`. The MCP `tools/list` request returns it. The `tools/call` request invokes it. It serializes the return type using Jackson. `UserDto` becomes a JSON object in the tool result. You can also use `Map` if you want loose typing, or a record class if you want strict typing with deserialization on the client side. Records are my default. The serialization is predictable, the schema is implicit, and the code is less than a Lombok annotation pile would be. ## How do you handle progress and async work? The most common reason a tool feels broken in production is that it does real work without telling the client. The client sits there. The user sits there. Eventually something times out. Spring AI 2.0 fixes this with a request context that exposes a progress channel. ```java import org.springframework.ai.mcp.server.annotation.McpTool; import org.springframework.ai.mcp.server.context.McpSyncRequestContext; import org.springframework.stereotype.Component; @Component public class ReportTools { private final ReportService reports; public ReportTools(ReportService reports) { this.reports = reports; } @McpTool( name = "generate_quarterly_report", description = "Generate the quarterly revenue report. Takes a few seconds." ) public ReportResult generateReport( McpSyncRequestContext ctx, @McpToolParam(description = "Quarter, e.g. 2026-Q1", required = true) String quarter ) { ctx.logging().info("Loading transactions for " + quarter); ctx.progress().report(0.1, "Loading transactions"); var transactions = reports.loadTransactions(quarter); ctx.progress().report(0.5, "Aggregating"); var aggregated = reports.aggregate(transactions); ctx.progress().report(0.9, "Rendering"); return reports.render(aggregated); } } ``` `McpSyncRequestContext` does not appear in the JSON schema. Spring AI knows it's a framework parameter and excludes it. The user-facing parameters stay clean. Inside the method you get `logging()`, `progress()`, `sampling()`, and `elicitation()` channels, all wired to the MCP client transport. For reactive code, use `McpAsyncRequestContext` and return a `Mono` or `Flux`. The progress channel returns Reactor types so you can compose progress reporting into a reactive chain. ```java @McpTool(name = "async_report", description = "Generate report asynchronously.") public Mono asyncReport( McpAsyncRequestContext ctx, @McpToolParam(description = "Quarter") String quarter ) { return reports.loadAsync(quarter) .doOnNext(_ -> ctx.progress().report(0.5, "Aggregated").subscribe()) .flatMap(reports::renderAsync); } ``` One thing to flag: the progress channel is not the same thing as streaming results. Progress is metadata. The result is still a single return value. If you want token-by-token streaming, you want sampling, not tools. ## How do you expose prompts and resources? Tools are the most-used MCP primitive, but prompts and resources are what turn a tool collection into a workspace. Prompts let the client request a prefilled prompt template. Resources let the client browse data your server exposes by URI. A prompt looks like this: ```java @Component public class IncidentPrompts { @McpPrompt( name = "incident_summary", description = "Generate an incident summary from a Jira ticket ID." ) public String incidentSummary( @McpToolParam(description = "Jira ticket ID, e.g. INC-1234") String ticketId ) { return """ Summarize the incident in ticket %s for an executive audience. Include: timeline, root cause, customer impact, and remediation. """.formatted(ticketId); } } ``` When the client asks for the `incident_summary` prompt with `INC-1234`, the server returns the rendered string. The client passes it to its model. Resources are different. They expose data the server holds, keyed by URI: ```java @Component public class CustomerResources { private final CustomerRepository customers; public CustomerResources(CustomerRepository customers) { this.customers = customers; } @McpResource( uri = "customer://{customerId}/profile", description = "Customer profile data including billing address and tier." ) public CustomerProfile profile(String customerId) { return customers.profileFor(customerId); } } ``` The client can list resources, pick one, and read it. The URI template is matched on the path variable. Spring AI extracts the `customerId` and passes it to the method. The combination of tools, prompts, and resources is what makes MCP feel like an actual application surface rather than a function bag. Tools do work. Prompts give the client wording. Resources expose state. ## How do you choose between transports (stdio, SSE, Streamable HTTP)? Spring AI supports four transports, and the choice matters more than the docs make it sound. stdio is for local single-process tools. Claude Code spawns your server as a child process and talks to it over stdin and stdout. Fine for personal tooling. Bad for anything multi-user, multi-tenant, or networked. SSE was the original HTTP transport for MCP. It's being phased out. Spring AI still ships it for backward compatibility, but new servers should not start there. Streamable HTTP is the current MCP HTTP transport. Stateful, supports bidirectional notifications, works behind reverse proxies, fits production environments. Use this for any server that isn't strictly local. Stateless Streamable HTTP is the new variant designed for serverless and horizontally scaled deployments. No session affinity required. The tradeoff is that any context that would have lived in the server session has to come back through the request from the client. If you're deploying to Vercel, Cloud Run, or any autoscaled platform, this is your transport. ![MCP transport comparison: stdio for local, SSE deprecated, Streamable HTTP for production, Stateless Streamable HTTP for serverless](/images/blogs/spring-ai-2-mcp-annotations-tutorial-transports.webp) For Streamable HTTP with WebMVC, the auto-config wires it up at `/mcp` by default. You can change the path: ```yaml spring: ai: mcp: server: transport: streamable-http path: /api/mcp name: my-spring-server version: 1.0.0 ``` The name and version show up in the MCP `initialize` response. Set them to something recognizable. Default Spring AI names look generic in client UIs. ## How do you register and test the server with Claude Code? Once your Spring Boot app runs and exposes Streamable HTTP at `/mcp`, register it with Claude Code. The config lives in `~/.config/claude/mcp.json` on macOS and Linux, or via the `claude mcp` CLI. ```json { "mcpServers": { "my-spring-server": { "type": "streamable-http", "url": "http://localhost:8080/mcp" } } } ``` Restart Claude Code. Run `/mcp` inside a session. You should see your tools, prompts, and resources listed. Try calling one. If the call returns and the result shows up in your transcript, the loop is alive. For local stdio testing, swap the JSON to type `stdio` and point `command` at your packaged jar: ```json { "mcpServers": { "my-spring-server": { "type": "stdio", "command": "java", "args": ["-jar", "/path/to/server.jar"] } } } ``` Claude Code will spawn the JVM, talk over stdin and stdout, and shut it down when the session ends. The startup cost is real, somewhere between 2 and 5 seconds depending on your dependencies. For HTTP transport, Spring Boot stays running and connections are cheap. ## What are the production-readiness gotchas? The annotation API hides the complexity. That's a feature for most cases, but it also means a few production concerns aren't obvious until they bite. ![Six production gotchas for Spring AI MCP servers: error handling, schema generation, auth, long-running tools, scaling, tool naming](/images/blogs/spring-ai-2-mcp-annotations-tutorial-gotchas.webp) **Error handling.** If your `@McpTool` method throws, Spring AI wraps the exception and returns an MCP error to the client. Good. But the default error mapping returns the exception message verbatim, which can leak internal details. Customize the error handling with a `McpExceptionHandler` bean if you're exposing the server to untrusted clients. **Schema generation.** Records and POJOs work. Generic collections work. Nested polymorphic types are where the generator gets cranky. If your tool returns a `List` where `Animal` is an interface, expect the schema to be vague. Prefer concrete types for tool return values. Save the polymorphism for internal layers. **Auth.** Spring AI does not ship MCP-specific auth. You wire it through normal Spring Security. For Streamable HTTP, add a security filter chain matching `/mcp` and validate bearer tokens or API keys there. The Stateless Streamable HTTP transport makes this easier because there's no session to protect, only requests. **Long-running tools.** MCP clients have timeouts. Claude Code's default is generous but not infinite. If your tool runs longer than 30 seconds, use the progress channel aggressively, and consider returning a job handle and exposing a second tool to poll status. Trying to hold open a 5-minute synchronous tool call will end in tears. **Scaling.** Streamable HTTP is stateful per session. Sticky sessions or a shared session store are needed if you scale horizontally. If you want stateless scaling, use the Stateless Streamable HTTP transport and design your tools to be self-contained on each request. **Tool naming.** MCP tool names show up in the client UI. Pick verbs. `get_user` reads better than `userQuery`. Consistency matters more than cleverness. ## Conclusion The Spring AI 2.0 MCP annotation API is the kind of upgrade you only notice fully when you look back at the old way. The boilerplate is gone. The schema generation is automatic. The transports are real. The progress and async story is sane. The part I want more Spring teams to internalize: MCP is no longer an experiment. Every major coding agent speaks it. Every Spring service in your fleet is a potential MCP surface. The cost of exposing your internal APIs to an agent has dropped to almost nothing. Whether that's a good idea is a different conversation, but the technical barrier is now low enough that you should decide on purpose, not by default. For more on Spring AI MCP, see the [Spring AI MCP annotations docs](https://docs.spring.io/spring-ai/reference/api/mcp/mcp-annotations-overview.html), the [Spring AI 2.0.0-M6 release post](https://spring.io/blog/2026/05/08/spring-ai-1-0-7-1-1-6-2-0-0-M6-available-now/), and the [official MCP spec](https://modelcontextprotocol.io/). ### Keep Reading - [Claude Skills vs MCP vs Projects](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects). When to reach for an MCP server vs a skill or a project. - [Claude Code Routines for CI Automation](https://www.rabinarayanpatra.com/blogs/claude-code-routines-ci-automation). Wiring agents into CI once your MCP server is live. - [Modern Java + Spring Boot](https://www.rabinarayanpatra.com/blogs/modern-java-spring-boot). The Spring Boot 3.x baseline you need before Spring AI 2.0. --- ## Why MCP 2026-07-28 Spec Drops Sessions and Goes Stateless Source: https://www.rabinarayanpatra.com/blogs/why-mcp-2026-07-28-spec-goes-stateless Published: 2026-05-25T00:00:00+05:30 Tags: mcp-2026-07-28-spec, model-context-protocol, mcp-stateless, ai-agents, protocol-design, anthropic MCP 2026-07-28 spec release candidate drops sessions for a stateless protocol core. Here is why it changed, what breaks, and how to plan your migration. The MCP team locked the 2026-07-28 spec release candidate on May 21. It is the largest revision since launch and yes, it breaks things. If you are running an MCP server today, you need to understand what changes before July 28 finalizes the spec. I keep seeing people on X call this "MCP 2.0". It is not. The spec uses date versions, not semver. The official name is the 2026-07-28 specification, and it sits as a release candidate until the final freeze on July 28 2026. The ten-week window between now and then is for SDK maintainers and server authors to validate against real workloads. The big shift is that MCP is going stateless. The current 2025-11-25 spec treats every client-server connection as a session with handshake, identifier, and lifecycle. The new spec rips most of that out at the protocol layer. Below I walk through what changed, what broke, and whether you should rush your migration or wait. ## What is the MCP 2026-07-28 spec release candidate? The 2026-07-28 release candidate is the next major version of the Model Context Protocol, locked on May 21 2026 and finalizing on July 28 2026. It introduces a stateless protocol core, a formal Extensions framework, the Tasks extension for long-running work, MCP Apps for server-rendered UIs, and OAuth-aligned authorization. The official MCP roadmap calls this revision "the largest revision of the protocol since launch". Tier 1 SDK maintainers (the official Anthropic-maintained Python and TypeScript SDKs) are expected to ship support within the ten-week validation window. If you maintain a server or build agent infrastructure on top of MCP, this is the version you target next. A subtle but useful detail: the MCP team shifted its 2026 roadmap from release-milestone organization to priority-area focus. Four priority areas drive the spec now: Transport Evolution and Scalability, Agent Communication, Governance Maturation, and Enterprise Readiness. Almost every concrete change in the 2026-07-28 spec maps directly to one of those four. That alignment matters because it tells you what to expect in the next spec drop too. ## Why was the 2025-11-25 stateful protocol hard to scale? The 2025-11-25 spec required every connection to start with an initialize/initialized handshake. The server returned a session identifier in an Mcp-Session-Id response header, and every subsequent request from that client had to carry the same header so the server could route it back to its session state. That design is fine for a single-process MCP server running on a developer laptop. It is painful in production. The MCP roadmap calls the problem out directly: "Stateful sessions fight with load balancers, horizontal scaling requires workarounds." When you put a load balancer in front of stateful MCP servers you have three bad options. You can run sticky sessions, which fail open whenever an instance restarts or scales down. You can share session state in Redis or a similar store, which adds a network hop and a single point of failure to every tool call. Or you can do deep packet inspection at the gateway, reading Mcp-Session-Id out of the header to manually route traffic, which forces every MCP client to know your private cluster topology. In my own work running an MCP server behind a Vercel Function I hit this within a week. Functions are stateless by design. The first attempt to add MCP routing died on cold starts because the next invocation had no idea what session-id the client was carrying. Working around it meant pushing every session into Upstash Redis and eating the round trip on every tools/list call. Not fun. The pattern is the same for any horizontally scaled deployment. Cloud Run, AWS Lambda, Kubernetes with HPA, even traditional VMs behind an L7 load balancer all run into the same wall. The old spec quietly assumed long-lived processes with shared memory. The new spec assumes nothing. ## How does the stateless protocol core actually work? The new spec removes the initialize/initialized handshake and the Mcp-Session-Id header from the base protocol. Each MCP request now carries everything the server needs to route it independently, and clients explicitly thread any state across calls themselves. The headline change for ops folks: a stateless MCP server can sit behind a plain round-robin load balancer with no sticky sessions and no shared session store. According to the release notes, gateways can route traffic on the new Mcp-Method header instead of inspecting payloads, and clients can cache tools/list responses for as long as the server's ttlMs field permits. That last part matters because in production a tools/list call on a busy server can dominate the latency budget. Here is the shape of the change in terms of request flow. ```text Before (2025-11-25): client -> initialize -> server -> 200 + Mcp-Session-Id: abc client -> tools/list (Mcp-Session-Id: abc) -> server (must own session abc) client -> tools/call (Mcp-Session-Id: abc) -> same server instance After (2026-07-28): client -> tools/list -> load balancer -> server A (returns list + ttlMs) client -> tools/call -> load balancer -> server B (different instance, same result) ``` You no longer need the same physical instance to handle both calls. That is the entire ballgame. In practice it means three things for your infrastructure: you can drop sticky session config from your load balancer, you can remove any shared Redis-based session store, and you can stop sending Mcp-Session-Id from your gateway routing rules. The new Mcp-Method header is the small detail that pays off for gateway authors. Instead of parsing the JSON-RPC body to know whether a request is tools/list or tools/call, the gateway can read a single header and route by method. That lets you split tools/list (cacheable, idempotent) onto edge nodes and tools/call (mutating, sometimes expensive) onto warm origin nodes. ## What do Extensions, Tasks, and MCP Apps add? The 2026-07-28 spec formalizes Extensions as the way new capabilities ship outside the core protocol. The MCP team calls out Extensions as a deliberate move to keep the core small while still letting the ecosystem experiment. New capabilities propose themselves as Extensions first, prove production value, and only graduate into the base spec on the next date-stamped revision. Two extensions matter most for new servers right now. The Tasks extension is built for long-running operations that need retries, expiry, and lifecycle tracking. If you have ever tried to wrap a 3-minute search index build inside a single MCP tool call and watched the client time out, Tasks fix that. The roadmap explicitly mentions "Tasks lifecycle refinement (retries, expiry policies)" as a priority for 2026. The Tasks model looks roughly like this. Your tool call returns a task handle instead of a result. The client polls or subscribes to the task by id. The server keeps the task's state in whatever store you like (Redis, Postgres, S3), and the client decides how long it is willing to wait. The client and server never need to be in the same process for this to work. A different server instance can resume a task from its persistent state because the task id is the carrier, not a session. MCP Apps are the second new extension. They let a server return server-rendered UI components instead of plain text or JSON payloads. The idea is that an MCP server for, say, a payments API can return a real card UI that the host renders inline. This is the closest MCP has come to giving servers presentation control, and it pulls the protocol closer to where Anthropic's Claude Skills and Claude Apps already live. Both Extensions and Tasks shipped first as experimental features so the team could collect real-world feedback before locking them into a final spec. Treat them as production-ready in the RC window, but expect minor field renames before July 28. ## What changes in authorization and error handling? The new spec aligns authorization more closely with OAuth and OpenID Connect. The earlier spec left auth largely as an implementation detail, which meant every enterprise MCP deployment built its own bearer-token flow. The 2026-07-28 spec defines how OAuth flows interact with MCP transport so a single enterprise SSO setup covers all your MCP servers. The roadmap lists "Governance Maturation" and "Enterprise Readiness" as priority areas, with audit trails, SSO auth, and gateway behavior as concrete deliverables. If you have been blocked on MCP rollout because security teams could not approve a custom auth flow, this is the change that unblocks you. Error handling also tightens up. The error code for missing resources shifts from the proprietary -32002 to the JSON-RPC standard -32602. Looks like a small change. In practice it means any client that hard-coded -32002 in retry logic now silently swallows the error and retries forever on legitimate not-found responses. I have already seen this fail in a private SDK that hardcoded the old code. Catch both for the next year of mixed deployments. A nice secondary effect is that monitoring tools that already understand JSON-RPC standard codes (and there are a lot of them) suddenly understand MCP traffic for free. You lose a small amount of MCP-specific signal in exchange for free integration with every JSON-RPC tracing tool out there. That is a good trade. ## Which existing code breaks when you upgrade? The breaking surface is small in lines of code but wide in impact. Here is what I am tracking across my own servers. Anywhere your code asserts on Mcp-Session-Id will break. The header is no longer guaranteed and any session-based routing logic needs to come out of your gateway and your client. If you are running NGINX or an Envoy sidecar with rules on Mcp-Session-Id, those rules now do nothing. Worse, they may silently route ALL traffic to one origin because the default fallthrough rule kicks in. Anywhere your client uses session state implicitly via the handshake will break. The new spec asks you to thread identifiers explicitly between tool calls. If you call tools/list and store the result keyed by session, you need to switch to keying by server URL plus ttlMs. Anywhere your retry logic catches -32002 will silently misbehave. Change it to -32602 or, better, catch both for the next year of mixed deployments while clients catch up. Anywhere your gateway does deep packet inspection on the body to route requests will break in a more positive direction. You can rip that logic out and rely on the new Mcp-Method header for routing decisions. Code you delete is code that cannot break in production. If you are using a Tier 1 SDK (the Anthropic-maintained Python and TypeScript SDKs), the SDK will hide most of these changes from you within the ten-week validation window. If you are on a community SDK, check whether your maintainer is in the Tier 1 list before you plan a migration date. The SDK tier system is itself new to MCP and tells you which SDKs the spec maintainers actively coordinate with on breaking changes. ## Should you migrate before July 28 or after? It depends on whether you control both the client and the server. If you do, migrate as soon as the SDK you depend on ships RC support, which for Tier 1 should land within four to six weeks. The new protocol is friendlier to the production patterns you already want anyway. If you ship a public MCP server consumed by clients you do not control, hold until July 28 and ship support for both spec versions on the same endpoint for at least a quarter. The spec's deprecation policy and the SDK tier system make dual support cheaper than it sounds, since both protocols can share the same handler code with a thin compatibility layer that reads the request and returns either a session-id (for old clients) or no session (for new ones). The one case where you should rush is if you are running MCP in a Kubernetes deployment behind an L7 load balancer. The current setup probably has at least one band-aid (sticky sessions, Redis session store, deep packet inspection) that exists only because of the stateful protocol. Migrating lets you delete that code, and code you delete is code that cannot break in production. There is also a clear case where you should wait. If your MCP server depends on a community SDK that is not in the Tier 1 list, you do not have a guarantee that the SDK will ship 2026-07-28 support before July 28 itself. Plan a quarter of slack. Do not promise a migration date until the SDK author publishes their own. ## What does this mean for your MCP servers? The 2026-07-28 spec is a clean win for anyone running MCP in production. The stateless core removes the awkward fit between MCP and modern serverless or horizontally scaled deployments, and the Tasks and MCP Apps extensions plug gaps that every real deployment has been working around with custom code. The bigger lesson, though, is that MCP is starting to act like an open protocol with real production users, not a research experiment from Anthropic. Date-stamped versioned specs, an SEP process, formal SDK tiers, and a published roadmap with priority areas all push it in the direction of HTTP or LSP, not a vendor SDK. That is a healthy sign even if the immediate migration is annoying. If you are starting a new MCP server this week, target the 2026-07-28 RC directly. If you have an existing one, start with deleting your Mcp-Session-Id routing rules and your shared session store. Most teams will find that the migration shrinks their MCP stack rather than grows it, which is the right direction for a protocol that is supposed to be the lowest-friction way to give an LLM access to tools. For more on the new spec, see the official [2026-07-28 Release Candidate announcement](https://blog.modelcontextprotocol.io/posts/2026-07-28-release-candidate/), the [2026 MCP Roadmap](https://blog.modelcontextprotocol.io/posts/2026-mcp-roadmap/), and the current [2025-11-25 specification](https://modelcontextprotocol.io/specification/2025-11-25) for the baseline you are migrating from. ### Keep Reading - [How to Build a Stateless MCP Server for the 2026-07-28 Spec](https://www.rabinarayanpatra.com/blogs/build-stateless-mcp-server-2026-07-28-spec). The applied tutorial that ships with code once you understand the protocol shift this post covers. - [Spring AI 2.0 MCP Annotations: From Tool to Production](https://www.rabinarayanpatra.com/blogs/spring-ai-2-mcp-annotations-tutorial). How the Spring AI 2.0 abstractions sit on top of MCP and what they hide from you. - [Claude Skills vs MCP vs Projects: Which One Should You Use?](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects). When to pick MCP at all, given the other ways Claude exposes capabilities to agents. --- ## TanStack npm Supply Chain Attack 2026: How OIDC Beat SLSA Source: https://www.rabinarayanpatra.com/blogs/tanstack-supply-chain-attack-2026 Published: 2026-05-23T00:00:00+05:30 ยท Updated: 2026-05-26T00:00:00+05:30 Tags: tanstack-supply-chain-attack, npm-security, github-actions, oidc-security, supply-chain, slsa On May 11, 2026, attackers hijacked TanStack's release pipeline mid-build to publish 84 malicious npm packages with valid SLSA provenance. Here's how. At 19:20 UTC on May 11, 2026, npm received the first batch of 42 malicious package versions from the official `@tanstack` namespace. Six minutes later, the second batch landed. Twenty minutes after that, an external researcher posted a full technical analysis to the TanStack repo. By the time most of Europe woke up, 84 versions across 42 packages were already deprecated. The packages were not published by an attacker who stole credentials. They were published by TanStack's legitimate release pipeline, signed with valid SLSA Build Level 3 provenance, attested through Sigstore, and pushed using a freshly minted OIDC token. Every supply-chain control we have been recommending for the past three years worked exactly as designed. And it did not matter. I run my own published package on Maven Central (`sanitizer-lib`) and I use TanStack Router in client projects. So this one stings personally. Let me walk you through what actually happened, why our defenses missed it, and what every CI maintainer needs to change before the next round. ## What happened in the TanStack supply chain attack? The TanStack supply chain attack is a multi-stage compromise that ran the malicious code through TanStack's own legitimate release infrastructure to publish 84 backdoored npm package versions across 42 packages in the `@tanstack/*` namespace. It then wormed into more than 170 downstream packages including `@mistralai/mistralai` (2.2.2 through 2.2.4), 40+ packages in the UiPath namespace, and 19 aviation data packages in `@squawk`. For scale: `@tanstack/react-router` alone pulls roughly 12.7 million weekly downloads. If you are doing modern React routing in 2026, there is a decent chance you sat in the blast radius. The incident is being tracked as CVE-2026-45321 and GHSA-g7cv-rxg3-hmpx. The threat group is TeamPCP, also known as DeadCatx3, PCPcat, ShellForce, and CipherForce. This is the same crew that compromised Aqua Security's Trivy scanner in March 2026 and the Bitwarden CLI npm package in April. Researchers at Wiz, Snyk, and StepSecurity collectively named this family "Mini Shai-Hulud" because it borrows the worm-like republish pattern from the original September 2025 Shai-Hulud incident. Here is the compressed timeline. | Time (UTC) | What happened | | ---------------------- | ------------------------------------------------------------------------------------------------------------ | | May 10, 17:16 | Attacker creates the fork `github.com/zblgg/configuration` | | May 10, 23:29 | Malicious commit `65bf499d` lands, authored as `claude ` to look legitimate | | May 11, 10:49 | PR #7378 opened against `TanStack/router` | | May 11, 11:11 to 11:29 | Force-pushes trigger `pull_request_target` workflows, GitHub Actions cache gets poisoned | | May 11, 11:31 | PR closed, fork branch deleted (cleanup) | | May 11, 19:15 | Release workflow re-runs against main, restores the poisoned cache | | May 11, 19:16 | Fresh release triggered by merge of PR #7382 | | May 11, 19:20 | npm receives first batch of 42 malicious versions | | May 11, 19:26 | npm receives second batch of 42 versions | | May 11, 19:46 | StepSecurity researcher ashishkurmi opens issue #7383 with full IOCs | | May 11, 20:19 | TanStack begins deprecation | | May 11, 21:03 | All 84 versions deprecated | | May 11, 22:13 to 23:55 | npm removes tarballs registry-side | | May 12, 05:02 | Formal IOC list sent to npm and GitHub Security | Notice something. The fork and the malicious commit were prepared a full day before the payload went live. The release was not a spontaneous event. It was held in the cache, dormant, waiting for the next legitimate release workflow run. ![Timeline of the TanStack supply chain attack on May 11, 2026](/images/blogs/tanstack-supply-chain-attack-2026-timeline.webp) ## How did the attacker chain three GitHub Actions vulnerabilities? The attack works by stacking three flaws that are each well-documented in isolation. Combined, they let attacker code execute inside the release workflow's trust boundary without ever touching the production branch. ### The first link: pull_request_target abuse GitHub's `pull_request_target` event trigger runs workflows in the base repository's context, not the fork's context. That is by design. It exists so that workflows can apply labels, post comments, or check size budgets on incoming PRs without requiring maintainer approval. The trade-off is that any code those workflows execute has access to the base repo's secrets. The TanStack `bundle-size.yml` workflow used `pull_request_target` and then executed fork-controlled code inside it. This pattern is so common and so dangerous that it has a name: the "Pwn Request" pattern. The attacker did not have to do anything clever here. They submitted a PR that swapped out a setup script with a malicious version, and the workflow happily ran it with full repository privileges. If you maintain any open-source repository on GitHub, run this right now: ```bash grep -rE "pull_request_target" .github/workflows/ ``` Then audit each match. The rule is simple: either do not check out fork code in `pull_request_target` workflows, or check it out into an isolated job that has no access to secrets and no permission to write to the cache. ### The second link: GitHub Actions cache poisoning This is the cleverest part of the chain. The attacker did not need to keep their code in the repo. They needed to leave it in the cache. GitHub Actions caches are scoped by branch by default, but pull requests inherit access to base-branch caches. The cache key is deterministic. For pnpm, it looks like `Linux-pnpm-store-${hashFiles('**/pnpm-lock.yaml')}`. The attacker pre-computed the exact key that the legitimate `release.yml` workflow would use after merging a future PR. That key was: ``` Linux-pnpm-store-6f9233a50def742c09fde54f56553d6b449a535adf87d4083690539f49ae4da11 ``` The malicious `vite_setup.mjs` in the attacker's PR wrote a tampered pnpm store under that key. When the PR was closed, the cache entry stayed. It sat there for hours. When a legitimate release workflow later ran on `main`, it restored that cache as part of its normal install step. The poisoned store contained the malicious binaries the attacker needed for the next stage. A critical detail from the TanStack postmortem: cache writes use a runner-internal token, not the workflow's `GITHUB_TOKEN`. That means standard permission boundaries do not apply. Restricting `GITHUB_TOKEN` to read-only does nothing for this attack vector. ### The third link: OIDC token theft from runner memory This is the part that crossed every line I thought we still had. When a GitHub Actions runner needs to publish to npm using trusted publishing, it requests a short-lived OIDC token from GitHub. That token is minted in memory, used to authenticate the publish, and then discarded. It is never written to disk. It never appears in environment variables that a child process can read. The runner generates it, uses it, throws it away. Or so we thought. The attacker's binaries, restored from the poisoned cache, did something that the original tj-actions compromise in March 2025 first demonstrated at scale. They read `/proc//maps` to find the memory regions of the `Runner.Worker` process. Then they read `/proc//mem` directly to scrape the OIDC token out of process memory while the runner was alive and authenticated. Once they had the token, they did not need any other credential. They sent direct HTTPS POST requests to `registry.npmjs.org` from the runner, authenticated as TanStack's release pipeline, with a valid SLSA attestation generated by Sigstore as a normal byproduct of the OIDC flow. The result was 84 malicious package versions that look indistinguishable from legitimate releases, because in every technical sense they were legitimate releases. The pipeline that built and published them was the real one. The code it pulled out of the cache was not. ![TanStack attack chain showing pull_request_target abuse, cache poisoning, and OIDC token extraction](/images/blogs/tanstack-supply-chain-attack-2026-attack-chain.webp) ## Why did SLSA provenance fail to protect downstream consumers? Here is the part that broke my brain for a few hours. SLSA Build Level 3 is the gold standard we have been telling teams to demand from upstream dependencies. The attestations on these 84 packages were valid. Sigstore verified the build process correctly. The provenance chain pointed to the actual TanStack repository, the actual release workflow run, and the actual GitHub Actions runner. Every signature checked out. So what did the provenance actually attest to? It attested that the build ran inside GitHub Actions, from the canonical TanStack/router repository, in a workflow defined in that repo, using a runner with a verifiable identity. All of that was true. The attestation does not, and cannot, attest that the code the build pipeline executed was the code in the repository. It only attests to the pipeline. In normal operation, those two are the same thing. The pipeline checks out the repo, the repo is the code, the code gets built. But when an attacker can inject code into the pipeline through a side channel (in this case, the cache), the pipeline is no longer building what the repo contains. It is building a hybrid. SLSA says nothing about the inputs the pipeline pulled from outside the source tree. This is the lesson I want every AppSec team to take from this incident. Provenance answers the question "was this built from the canonical repo in a controlled environment?" It does not answer "is the code in this package safe?" Those used to be the same question for practical purposes. They are no longer. What does this mean operationally? You still want SLSA. You still want OIDC trusted publishing. You still want Sigstore. But you need behavioral analysis at install time as a complementary control. Sandboxed installs that inspect what `postinstall` scripts actually do, what network connections they make, what files they touch. Static and dynamic analysis of the published artifact, not just verification of its origin. I covered this exact pattern in my earlier post on [AI-driven anomaly detection for security](https://www.rabinarayanpatra.com/blogs/ai-driven-anomaly-detection-security), and the TanStack incident is the strongest argument for it I have seen. ## How does the gh-token-monitor dead-man switch work? The malicious payload installed on infected machines is roughly 2.3 MB after the obfuscation layers come off. The file is `router_init.js` and its SHA256 is `ab4fcadaec49c03278063dd269ea5eef82d24f2124a8e15d7b90f2fa8601266c`. It uses three obfuscation layers stacked on top of each other. The first layer is JavaScript Obfuscator string-array rotation, the kind of thing you find in any commercial obfuscation product. The second is a Fisher-Yates substitution cipher seeded by PBKDF2-SHA256 with the salt `svksjrhjkcejg`. The third is eleven AES-256-GCM encrypted payloads that require the Bun runtime to decrypt, which is why the malware installs Bun on the target machine if it is not already present. Once everything decrypts, the payload harvests credentials. The scope is brutal. AWS IMDSv2 metadata endpoints, [AWS Secrets Manager](https://www.rabinarayanpatra.com/snippets/aws-cli/secrets-manager-get) keys, GCP metadata server tokens, Azure managed identity tokens, Kubernetes service account tokens, HashiCorp Vault tokens, GitHub Actions secrets, GitLab CI tokens, CircleCI tokens, `.npmrc` files, SSH private keys, and (worth calling out because it is new) Claude Code session history files from `.claude/projects/*.jsonl`. That last one tells me the threat actor is paying attention to where developers actually keep their secrets in 2026, not where the OWASP guides from 2018 said they would be. But the part that should make you nervous is what happens after the credentials are out the door. The malware installs a persistence daemon called `gh-token-monitor`. On macOS, the daemon registers as a LaunchAgent at: ``` ~/Library/LaunchAgents/com.user.gh-token-monitor.plist ``` On Linux, it registers as a systemd user service at: ``` ~/.config/systemd/user/gh-token-monitor.service ``` The daemon polls GitHub every 60 seconds, checking whether the stolen credentials still work. If it detects that the GitHub token has been revoked (the obvious first response from any incident-response team), it triggers a destructive payload. On the affected machines, that payload was an attempt to run `rm -rf ~/` against the user's home directory. The good news is that on modern Linux distributions, `rm` refuses to operate on `/` or `~/` without the `--no-preserve-root` flag, so the actual destruction is limited in many cases. But "many" is not "all", and you do not want to find out which version of `rm` your laptop is shipping when you are already mid-incident. The implication is operational. If you suspect a compromise, do not rotate the GitHub token first. Find and remove the `gh-token-monitor` daemon first, then rotate. This is the opposite of every incident-response playbook I have ever seen. The malware also drops persistence hooks into editor config directories. On the systems Wiz analyzed, it wrote `router_runtime.js` into `.claude/` and added entries to `.claude/settings.json`, and dropped `setup.mjs` plus a malicious task entry in `.vscode/tasks.json`. The point is that even after you remove the daemon and rotate credentials, the next time you open the project in your editor, the hooks can reinstall everything. Audit those directories explicitly. ## What should you do if you installed an affected version? Order matters here. If you installed an affected version, follow these steps in this exact sequence. First, find the daemon. Run these checks on every developer machine and CI runner that touched the compromised packages between May 11 and your last clean restore. ```bash # macOS launchctl list | grep gh-token-monitor ls -la ~/Library/LaunchAgents/com.user.gh-token-monitor.plist # Linux systemctl --user list-units | grep gh-token-monitor ls -la ~/.config/systemd/user/gh-token-monitor.service # Either OS find ~ -name "router_init.js" 2>/dev/null find ~ -name "router_runtime.js" 2>/dev/null ``` Second, kill the daemon before doing anything else. ```bash # macOS launchctl unload ~/Library/LaunchAgents/com.user.gh-token-monitor.plist rm ~/Library/LaunchAgents/com.user.gh-token-monitor.plist # Linux systemctl --user stop gh-token-monitor systemctl --user disable gh-token-monitor rm ~/.config/systemd/user/gh-token-monitor.service systemctl --user daemon-reload ``` Third, remove the editor hooks. ```bash # Check what was added cat .claude/settings.json cat .vscode/tasks.json # Remove the malicious entries by hand. Do not run a script you do not understand. ``` Fourth, audit your lockfile. Search for any of the 84 affected versions. The headline versions to check first are `@tanstack/react-router@1.169.5` and `@1.169.8`, `@tanstack/vue-router@1.169.5` and `@1.169.8`, and `@tanstack/solid-router@1.169.5` and `@1.169.8`. The full list lives in the official advisory. ```bash grep -E "@tanstack/.*1\.169\.(5|8)" package-lock.json pnpm-lock.yaml yarn.lock 2>/dev/null ``` Fifth, rotate every credential the malware could have touched. That means npm tokens, GitHub personal access tokens, GitHub Actions OIDC trust configurations, AWS access keys, Vault tokens, Kubernetes service account tokens, SSH keys, and any provider-specific secrets stored in `~/.aws`, `~/.gcloud`, `~/.kube`, or `~/.npmrc`. Sixth, block the C2 infrastructure at your DNS or proxy layer. ``` git-tanstack.com # typosquat filev2.getsession.org # Session messenger exfil seed1.getsession.org seed2.getsession.org seed3.getsession.org 83.142.209.194 # primary C2 IP ``` Seventh, scan for the campaign string `EveryBoiWeBuildIsAWormyBoi` in any process memory, log file, or repository content. It is the easiest reliable IOC the malware leaves behind. If you handled the [Axios npm compromise from March 2026](https://www.rabinarayanpatra.com/blogs/axios-npm-supply-chain-attack-2026), most of this routine will feel familiar. The key difference is the order: there, you could rotate credentials first because the malware did not retaliate. Here, retaliation is the whole point of the persistence layer. ## How do you harden your own GitHub Actions workflows against this? Even if you never touched a TanStack package, the techniques used in this attack apply to any repository that uses `pull_request_target` and any organization that publishes to npm via OIDC trusted publishing. Which, increasingly, is every JavaScript shop in production. Here is what I have changed in my own workflows since reading the TanStack postmortem. **Audit every `pull_request_target` workflow.** For each one, ask: does this workflow execute fork-controlled code? If yes, either remove the fork-code execution or split the workflow into two: an untrusted "read fork code" job that runs without any secrets, and a trusted "act on results" job that runs separately. Pass data between them only as strict, validated outputs. **Guard against fork pushes.** Add an early job that bails out if the PR is from a non-trusted source, before any setup steps run. ```yaml jobs: guard: runs-on: ubuntu-latest steps: - name: Block fork PRs from privileged workflow if: github.event.pull_request.head.repo.full_name != github.repository run: | echo "Refusing to run privileged workflow on fork PR" exit 1 ``` **Pin every third-party action to a commit SHA.** Tags can be moved. SHAs cannot. ```yaml # Don't do this - uses: actions/checkout@v4 # Do this - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 ``` **Separate cache namespaces.** Use different cache keys for PR workflows and release workflows so a poisoned PR cache cannot be restored by a release run. ```yaml - uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4.0.2 with: path: ~/.pnpm-store key: ${{ github.workflow }}-pnpm-${{ hashFiles('**/pnpm-lock.yaml') }} ``` That `${{ github.workflow }}` prefix means PR workflows and release workflows have disjoint cache namespaces. **Require manual approval on publish.** OIDC trusted publishing is excellent, but the lack of per-publish human review is the weakest point in the chain. Configure your release workflow as an environment with required reviewers, even for trusted maintainers. A 30-second approval check would have stopped this entire attack. **Reduce the maintainer surface.** Every additional human with publish rights is a credential-theft target. Trim the publisher list on your npm scope to the smallest number that keeps the project healthy. The TanStack postmortem flags this as one of their own action items. **Enable a release-age cooldown.** Set `min-release-age` in your `.npmrc` to delay adopting newly published versions. Seven days is the common recommendation. It does not protect you from a slow-burn attack, but it gives the ecosystem time to catch a fast-detonating one. ``` # .npmrc min-release-age=10080 # 7 days in minutes ``` If you run a microservices environment, treat your CI runners with the same zero-trust posture you apply to production workloads. The pattern I described in my [zero-trust microservices post](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security) applies just as much to GitHub Actions as it does to Kubernetes. ## What does this mean for the future of npm trust? Here is the part that worries me. For three years, we have been telling teams that the answer to supply chain attacks is provenance. Sign your packages. Publish via OIDC. Demand SLSA Build Level 3. The TanStack incident is the first one where every single one of those defenses worked, and the attack still succeeded. This is not a reason to abandon any of them. They still raise the cost for attackers and they still help in the median case, which is credential theft. But they are no longer sufficient on their own, and anyone telling you otherwise is selling something. What I think changes from here: Cache infrastructure is now part of the trusted compute base. Treat GitHub Actions caches as input that needs the same scrutiny as source code. Cache poisoning is not a theoretical concern anymore; it is a documented attack pattern with a real CVE attached to it. Runtime install-time analysis matures from optional to required. Tools like Socket, Snyk, and StepSecurity that watch what packages do during installation are no longer "nice to have" for security-conscious shops. They are the only practical defense against this class of attack, because they evaluate the behavior of the artifact, not its origin. The npm registry's "no unpublish if dependents exist" policy needs revisiting. In the TanStack incident, that policy delayed full tarball removal by hours. For verified maintainer-initiated incident response, there needs to be a faster path. The postmortem flags this explicitly. OIDC trusted publishing needs per-publish review. The current model authenticates the pipeline, not the publish event. A 30-second human approval step before a token is minted would have stopped this attack cold. I expect to see this as a configurable option within the next 12 months. The TanStack team's response to this was, by every measure I can find, excellent. Detection within 26 minutes. Full deprecation in under two hours. Public disclosure on the same day. They did everything right. And the attack still landed 84 malicious package versions onto npm with valid provenance. That is the message. The defenders are not slow. The attackers are not lazy. We are watching a category of attack that operates inside our trust boundaries, and the answer is not better trust boundaries. It is treating every artifact as untrusted until proven safe, every time. For more on this campaign, see the [TanStack official postmortem](https://tanstack.com/blog/npm-supply-chain-compromise-postmortem), the [Wiz Mini Shai-Hulud analysis](https://www.wiz.io/blog/mini-shai-hulud-strikes-again-tanstack-more-npm-packages-compromised), the [Snyk technical breakdown](https://snyk.io/blog/tanstack-npm-packages-compromised/), the [official GitHub Security Advisory GHSA-g7cv-rxg3-hmpx](https://github.com/advisories/GHSA-g7cv-rxg3-hmpx), and the [related StepSecurity report on node-ipc](https://www.stepsecurity.io/blog/node-ipc-npm-supply-chain-attack). ### Keep Reading - [The Axios npm Hack: How North Korea Hijacked 100M Weekly Downloads](https://www.rabinarayanpatra.com/blogs/axios-npm-supply-chain-attack-2026): the March 2026 incident that set the stage for the OIDC-vs-SLSA conversation. - [AI-Driven Anomaly Detection for Security](https://www.rabinarayanpatra.com/blogs/ai-driven-anomaly-detection-security): behavioral analysis at install time is the only practical defense against pipeline-hijack attacks. - [Zero-Trust Microservices with Spring Security](https://www.rabinarayanpatra.com/blogs/zero-trust-microservices-spring-security): apply the same zero-trust posture to CI runners that you apply to production services. --- ## PostgreSQL 18 Temporal Foreign Keys with Spring Boot JPA Source: https://www.rabinarayanpatra.com/blogs/postgresql-18-temporal-foreign-keys-spring-boot Published: 2026-05-22T00:00:00+05:30 Tags: postgresql-18, spring-boot, hibernate, temporal-foreign-keys, jpa PostgreSQL 18 temporal foreign keys with Spring Boot: WITHOUT OVERLAPS, PERIOD clause, Hibernate daterange mapping, and the ON DELETE gotcha nobody mentions. PostgreSQL 18 shipped temporal foreign keys. The kind of feature SQL standards committees promised back in SQL:2011 and most database vendors quietly ignored. Now it's in mainline PG and the Java ecosystem has almost no tutorials on how to use it from Spring Boot. I went looking for "Spring Boot temporal foreign key" guides last week. The top results were 2018 Baeldung posts on hand-rolled `validFrom/validTo` columns with zero database-level constraints. Plenty of `BETWEEN` queries. Plenty of "remember to add an index". No one talks about PG18 yet. So I built a working example, hit every gotcha, and wrote it up. This post walks the full path: what temporal FKs actually solve, the PostgreSQL 18 syntax, how to map range columns in Hibernate, the Spring Data repository patterns that work, and the ON DELETE behavior that will absolutely surprise you if you skip the docs. ## What problem do temporal foreign keys solve? A temporal foreign key enforces that a child row's time range fits entirely inside its parent row's time range, at the database level, on every insert and update. That's the part regular foreign keys can't do. Take a classic example. An employee changes departments three times in five years. A project assignment references the employee. The business rule is: a project assignment can only exist during a period when the employee was actually employed. The hand-rolled approach looks like this: ```sql CREATE TABLE employees ( emp_id BIGINT, valid_from DATE NOT NULL, valid_to DATE, PRIMARY KEY (emp_id, valid_from) ); CREATE TABLE project_assignments ( assignment_id BIGINT PRIMARY KEY, emp_id BIGINT NOT NULL, assignment_start DATE NOT NULL, assignment_end DATE, FOREIGN KEY (emp_id) REFERENCES employees(emp_id) ); ``` Every Spring shop I've worked at had a variation of this. And every one of them had bugs. Project assignments referencing employee IDs whose valid period ended six months ago. Manual `BETWEEN` checks in service layers that someone forgot to update. Audit failures during quarterly reviews. The data model lied about what it was. PG18 fixes this at the constraint level. The constraint is the truth, not a service-layer hope. ![PostgreSQL 18 temporal foreign keys connecting employees with overlapping validity periods to project assignments](/images/blogs/postgresql-18-temporal-foreign-keys-spring-boot-diagram.webp) ## How does PostgreSQL 18 implement WITHOUT OVERLAPS and PERIOD? PG18 introduces two new clauses: `WITHOUT OVERLAPS` for primary and unique keys, and `PERIOD` for foreign keys. Both rely on range types and GiST indexes under the hood. Here's the employees table redone the right way: ```sql CREATE EXTENSION IF NOT EXISTS btree_gist; CREATE TABLE employees ( emp_id BIGINT NOT NULL, name VARCHAR(100) NOT NULL, department VARCHAR(50) NOT NULL, salary NUMERIC(10,2) NOT NULL, valid_period daterange NOT NULL, PRIMARY KEY (emp_id, valid_period WITHOUT OVERLAPS) ); ``` A few things worth pointing out. The `btree_gist` extension is required because the primary key mixes a regular `BIGINT` column with a range column. PG needs a GiST index that can handle both, and `btree_gist` provides the btree operator support inside GiST. If you forget it, the `CREATE TABLE` will fail with a confusing operator error. The `valid_period` column uses `daterange`, one of PostgreSQL's built-in range types. You can also use `tstzrange` for timestamp ranges or `int4range` for numeric ranges. The constraint creates a GiST index automatically. Try to insert two rows for the same `emp_id` with overlapping `valid_period` values and PG rejects it. Now the project assignments table with a real temporal FK: ```sql CREATE TABLE project_assignments ( assignment_id BIGSERIAL PRIMARY KEY, emp_id BIGINT NOT NULL, project_name VARCHAR(100) NOT NULL, assignment_period daterange NOT NULL, FOREIGN KEY (emp_id, PERIOD assignment_period) REFERENCES employees (emp_id, PERIOD valid_period) ); ``` The `PERIOD` keyword tells PG that the last column is a range. The check is not equality. The check is containment: the parent's matching rows must, in combination, fully cover the child's range. If the employee's `valid_period` ends June 1 2024 and the project assignment runs March 1 to August 1 2024, the insert fails because June 1 to August 1 has no parent row covering it. You can verify this with a deliberate bad insert: ```sql INSERT INTO employees (emp_id, name, department, salary, valid_period) VALUES (1, 'Alice', 'Engineering', 80000, daterange('2024-01-01', '2024-06-01')); INSERT INTO project_assignments (emp_id, project_name, assignment_period) VALUES (1, 'Migration', daterange('2024-03-01', '2024-08-01')); -- ERROR: insert or update on table "project_assignments" violates foreign key constraint ``` The constraint catches it. No service code needed. ## How do you map daterange columns in Hibernate? Hibernate 6 added native support for PostgreSQL range types through `PostgreSQLRangeJdbcType`, but the cleanest path for a Spring Boot app is still the `hypersistence-utils` library. It's the rebranded version of what most of us used as `hibernate-types-52` for years, maintained by Vlad Mihalcea. Add the dependency to your `pom.xml`: ```xml io.hypersistence hypersistence-utils-hibernate-63 3.10.7 ``` If you're on Spring Boot 3.5+, you'll be on Hibernate 6.6, so use the `hypersistence-utils-hibernate-63` artifact. Older Spring Boot 3.x versions use `-62`. The version numbers track Hibernate ORM, not Spring Boot. Now the entity: ```java import io.hypersistence.utils.hibernate.type.range.Range; import io.hypersistence.utils.hibernate.type.range.PostgreSQLRangeType; import jakarta.persistence.*; import org.hibernate.annotations.Type; import java.math.BigDecimal; import java.time.LocalDate; @Entity @Table(name = "employees") @IdClass(EmployeeId.class) public class Employee { @Id @Column(name = "emp_id") private Long empId; @Id @Type(PostgreSQLRangeType.class) @Column(name = "valid_period", columnDefinition = "daterange") private Range validPeriod; @Column(nullable = false) private String name; @Column(nullable = false) private String department; @Column(nullable = false) private BigDecimal salary; // getters, setters, constructors } ``` A few things to flag. The `@IdClass(EmployeeId.class)` is needed because the primary key is composite. The `EmployeeId` class is a plain Java record or class with the two ID fields and `equals`/`hashCode`. ```java public record EmployeeId(Long empId, Range validPeriod) implements Serializable {} ``` The `@Type(PostgreSQLRangeType.class)` annotation is what makes Hibernate emit and read `daterange` correctly. The `columnDefinition = "daterange"` part is what makes JPA's schema generation produce the right column type if you let JPA create the schema. In production, you should use Flyway or Liquibase, not JPA schema generation, but it's worth being explicit either way. You construct ranges like this: ```java Range period = Range.closedOpen( LocalDate.of(2024, 1, 1), LocalDate.of(2024, 6, 1) ); ``` `closedOpen` matches PostgreSQL's `[)` notation, which is the most common range form for temporal data. Half-open intervals make adjacent ranges easy to reason about: `[Jan 1, Jun 1)` and `[Jun 1, Dec 1)` are adjacent, not overlapping. ![Hibernate range type mapping: Java Range through hypersistence-utils to PostgreSQL daterange](/images/blogs/postgresql-18-temporal-foreign-keys-spring-boot-hibernate-mapping.webp) ## How do you write the entity and repository? Spring Data JPA does most of the work, but you need a custom query for the overlap check at the application level. Database constraints catch invalid inserts, but the app still needs to ask "who was employed during this period?" The repository: ```java public interface EmployeeRepository extends JpaRepository { @Query(value = """ SELECT * FROM employees WHERE emp_id = :empId AND valid_period && :period """, nativeQuery = true) List findOverlapping( @Param("empId") Long empId, @Param("period") String period ); } ``` The `&&` operator is PostgreSQL's range-overlap operator. You pass the range as a string in PG range literal form: `[2024-01-01,2024-06-01)`. JPA doesn't have a native range-binding mechanism, so the string approach is the pragmatic move. If you want strong typing, Vlad's library offers `Range.toString()` which formats correctly. Here's a service method that finds the employee record valid for a given date: ```java @Service public class EmploymentService { private final EmployeeRepository repo; public EmploymentService(EmployeeRepository repo) { this.repo = repo; } public Optional findAt(Long empId, LocalDate at) { String singleDay = String.format("[%s,%s]", at, at); return repo.findOverlapping(empId, singleDay).stream().findFirst(); } } ``` For the project assignment side, the insert just works. Hibernate sends the daterange, PG checks the temporal FK, and if the assignment period isn't fully covered by some employee period, the transaction rolls back. ```java @Transactional public ProjectAssignment assign(Long empId, String projectName, Range period) { ProjectAssignment pa = new ProjectAssignment(); pa.setEmpId(empId); pa.setProjectName(projectName); pa.setAssignmentPeriod(period); return assignmentRepo.save(pa); } ``` If the period extends past the employee's validity, you get a `DataIntegrityViolationException` wrapping the PG error. Catch it where your error handling needs it. ## What are the gotchas with ON DELETE and temporal foreign keys? This is the one that will burn you if you skip the docs. PostgreSQL 18 does not support `CASCADE`, `RESTRICT`, `SET NULL`, or `SET DEFAULT` referential actions on temporal foreign keys. Only `NO ACTION` is allowed. From the official PG18 CREATE TABLE docs: "In a temporal foreign key, this option is not supported." That sentence appears under every action except `NO ACTION`. What does this mean in practice? If you delete an employee row that has dependent project assignments, PG raises a foreign key violation. You have to delete the assignments first, manually, in application code or in a database trigger you write yourself. ```sql -- This will fail at the second statement DELETE FROM employees WHERE emp_id = 1; -- ERROR: update or delete on table "employees" violates foreign key constraint ``` The workaround patterns I've seen: ![Three workaround patterns for ON DELETE on temporal foreign keys: app-level delete, soft delete, period close](/images/blogs/postgresql-18-temporal-foreign-keys-spring-boot-ondelete-patterns.webp) The first is application-level deletion. Spring service methods that delete child rows before parent rows, wrapped in a `@Transactional` boundary. This is fine for small graphs, but it doesn't scale to deep hierarchies. The second is soft delete on the parent. Add a `deleted_at` column, never actually delete rows, and let the temporal FK stay intact forever. Most enterprise systems do this anyway for audit reasons. The third is to flip the model: instead of deleting the parent, you close the parent's period by updating `valid_period` to end at today's date. Future child inserts will fail. Existing child rows stay valid because their periods were covered when they were created. The third approach is closest to the spirit of temporal modeling. You don't lose history. You just declare "this record stopped being valid at this point" and let the constraints enforce that going forward. If you absolutely need cascading deletes, you can write a trigger that does the deletion manually before the parent delete fires. But at that point you're rebuilding what the standard wanted to give you, and the SQL spec authors decided cascade semantics on overlapping periods were too ambiguous to specify. They might be right. ## When should you NOT use temporal foreign keys? Temporal foreign keys are powerful, and like most powerful features they're easy to overuse. I've seen teams reach for them in places where regular FKs would be simpler and just as correct. Skip temporal FKs when: You only need audit history. If the question is "what changed when?" and not "what was the state of the world on date X?", a separate audit log table with regular FKs is simpler. Frameworks like Hibernate Envers handle this well. The child rows don't have an independent time dimension. If a project assignment is just "associated with employee 1 forever", you don't need temporal FKs. A regular FK with a `created_at` timestamp is fine. You're modeling a single current state. CRUD apps where the latest row is the only thing that matters don't need temporal modeling. Add a `valid_period` column and you've created complexity you'll have to pay for in every query. You can't pay the GiST index cost. GiST indexes are slower for point lookups than btree indexes. For high-throughput single-row reads, you'll feel it. Benchmark first. When the model genuinely is temporal, the constraint-enforced version is a huge improvement over the hand-rolled version. The number of bugs you avoid by having PG check every insert is real. The cost of writing the queries against ranges instead of timestamps is real too, but it's a one-time cost. The bugs are forever. ## Conclusion PostgreSQL 18 temporal foreign keys are one of the most under-discussed major-version features I've seen in a while. The Spring Boot ecosystem has barely caught up. If you're modeling employment, contracts, pricing tiers, policies, or anything else where state has duration, the new `WITHOUT OVERLAPS` and `PERIOD` clauses are worth learning even if you don't ship them tomorrow. The piece I want more people to understand: this changes what your database can be the source of truth for. Before PG18, "this assignment must fall inside an employment period" was a service-layer concern that drifted out of sync. Now it's a constraint. The database tells the truth. For more on the PG18 release, see the [PostgreSQL 18.0 release notes](https://www.postgresql.org/docs/release/18.0/), the [Neon writeup on temporal constraints](https://neon.com/postgresql/postgresql-18/temporal-constraints), and the [CREATE TABLE syntax docs](https://www.postgresql.org/docs/current/sql-createtable.html). ### Keep Reading - [Spring Boot Testcontainers Guide](https://www.rabinarayanpatra.com/blogs/spring-boot-testcontainers-guide). Real PG18 in tests beats mocked databases for catching constraint bugs early. - [Hibernate Lazy Init Guide](https://www.rabinarayanpatra.com/blogs/hibernate-lazy-init-guide). The other Hibernate gotcha that bites every Spring team. - [PgBouncer Survival Guide](https://www.rabinarayanpatra.com/blogs/postgres-connection-pool-pgbouncer-survival-guide). Connection pooling matters more when GiST indexes are doing real work. --- ## Spring Boot 4 AOT Data Repositories: The Underrated Feature Source: https://www.rabinarayanpatra.com/blogs/spring-boot-4-aot-data-repositories Published: 2026-05-16T00:00:00+05:30 Tags: spring-boot-4-aot-repositories, spring-data, spring-boot-4, graalvm-native-image, java-aot Spring Boot 4's AOT data repositories generate query implementations at build time, cutting startup and unlocking GraalVM native image support. I've used Spring Data for almost a decade. Repositories have always felt like the magical part of Spring Boot: write an interface with method names, get a working implementation at runtime through proxies and reflection. It works, but it's slow at startup, opaque when something breaks, and famously hostile to GraalVM native image compilation. Spring Boot 4 changed that quietly. AOT Data Repositories generate the implementation at compile time. The repository proxy you got at runtime is now a real Java class on disk, written to source by the build, compiled with the rest of your code. Faster startup. Native images that work. Stack traces you can actually read. I haven't seen many people talking about this, even though it's one of the more important changes in the Spring Boot 4 generation. This post covers what it does, how to turn it on, and what to know before you commit to it. ## What are Spring Boot 4 AOT data repositories? Spring Boot 4 AOT data repositories are Spring Data repository implementations generated at build time, replacing the reflective runtime proxies that Spring Data has historically used. ![Spring Data AOT pipeline: query methods analyzed at build time, generated classes compiled with the application](/images/blogs/spring-boot-4-aot-data-repositories-pipeline.webp) When you write this: ```java public interface UserRepository extends JpaRepository { List findByEmailContaining(String fragment); Optional findByEmailIgnoreCase(String email); long countByActiveTrue(); } ``` The traditional Spring Data flow at startup: 1. Scan your classpath for repository interfaces. 2. For each method, parse the name into a query. 3. Derive the JPQL or SQL. 4. Generate a CGLIB or JDK proxy that intercepts every call. 5. Inject the proxy as a Spring bean. All of that happens inside `applicationContext.refresh()`, every time the app starts. With AOT repositories, steps 1 through 3 happen during the Maven or Gradle build. Step 4 generates a real class file named `UserRepositoryImpl__AotRepository` in the same package as your interface. Step 5 still happens at startup, but the bean it wires up is the pre-generated class, not a runtime proxy. The startup that took 2.1 seconds because Hibernate, Spring Data, and validation all kicked in together is now closer to 1.4 seconds, in my testing. And the heap it occupied has shrunk because the proxy bookkeeping is gone. ## Why does compile-time generation matter? Compile-time generation matters because it converts a category of work that scales with application size from "runtime cost on every startup" to "build cost paid once." The motivation isn't just speed. It's also correctness and visibility. The old runtime proxy model has three properties that hurt at scale. First, it's reflective, which means GraalVM native image needs explicit hints for every reflective call. Spring Boot has shipped those hints incrementally over years, but it's a moving target. Second, the generated SQL is invisible at code-review time. You can run the app and inspect the logs, but you can't grep for it. Third, when a method name doesn't parse cleanly, the failure shows up at startup, not at compile time. I've shipped a regression where a typo in a derived method name (`findByEmailIs` vs `findByEmailEqualTo`) only surfaced in the dev environment because it didn't break the build. AOT repositories address all three. The reflection is gone (or at least dramatically reduced). The generated SQL is real Java code in `target/generated-sources/aot/` that you can read. And invalid method names break the build, not the application. The Spring Data team's framing for it: "Generated query methods contain the exact same code you would write if you would not use Spring Data to run your query." That's not a stretch. The output is straightforward Java that calls into `EntityManager` (for JPA) or executes a `JdbcTemplate` query (for JDBC). No more wondering what the proxy did under the covers. ## How do you enable AOT repositories? AOT repositories are enabled by default when the Spring AOT engine is active. Two ways to activate the AOT engine: **Maven build with AOT goal:** ```xml org.springframework.boot spring-boot-maven-plugin process-aot process-aot ``` After running `mvn package`, the generated AOT sources sit under `target/spring-aot/main/sources/`. Open one and you'll see the actual method bodies for your repository. **Gradle build:** ```kotlin plugins { id("org.springframework.boot") id("org.graalvm.buildtools.native") // only if you want native image } springBoot { mainClass.set("com.example.Application") } ``` Running `./gradlew bootJar` triggers the AOT pipeline. **Native image:** ```bash ./mvnw -Pnative native:compile ``` Or with Gradle: ```bash ./gradlew nativeCompile ``` In native mode, AOT repositories aren't optional. They're how the build avoids the reflective code paths that GraalVM can't analyze. To turn AOT repositories off (for testing, or to compare behavior): ```properties spring.aot.repositories.enabled=false ``` To disable for a specific module: ```properties spring.aot.jdbc.repositories.enabled=false spring.aot.jpa.repositories.enabled=false spring.aot.mongodb.repositories.enabled=false spring.aot.cassandra.repositories.enabled=false ``` In normal JVM mode without the AOT goal, your app keeps using the runtime proxies. Nothing forces you onto the AOT path until you opt in via the build plugin. ## What gets generated at build time? The build generates a Java class for each imperative repository interface, named `Impl__AotRepository`, in the same package as the interface. For the `UserRepository` example above, the generated class is `UserRepositoryImpl__AotRepository`. Each query method becomes a real Java method that calls into the persistence layer directly. For Spring Data JPA, the generated method for `findByEmailContaining` looks roughly like this (simplified for clarity): ```java public List findByEmailContaining(String fragment) { String jpql = "SELECT u FROM User u WHERE u.email LIKE ?1"; return entityManager .createQuery(jpql, User.class) .setParameter(1, "%" + fragment + "%") .getResultList(); } ``` For Spring Data JDBC, the equivalent generates a SQL string with parameterized binding. For MongoDB, it generates the document filter. For Cassandra, the CQL statement. The generated code is technology-specific, which is the point. There's no abstraction layer at runtime, no proxy interception, no reflective `Method.invoke`. You get the same JPQL or SQL you'd write by hand. What it does NOT generate: anything for reactive repositories. The build skips `ReactiveCrudRepository` and friends silently. Anything for `@Query`-annotated methods that use SpEL expressions evaluated at runtime. Anything for repository fragments backed by custom implementations (your `UserRepositoryCustom` impl class is left alone). The metadata side of generation is also useful: Spring Data writes JSON files alongside the generated classes containing every query method's derived SQL. These are the same metadata Spring uses to wire native image runtime hints. If you want to audit every query your app issues without running it, those JSON files are the cleanest source. A practical thing I do with the JSON metadata: feed it into a script that diffs the queries between two builds. Adding a method to a repository or renaming an existing one shows up as a clean diff in CI. This catches a category of accidental N+1 queries that I used to find only in production. The build now tells me when a derived query suddenly fans out into a per-row lookup instead of a single join. It's the kind of tooling I always wanted but never had a reasonable place to plug in. With AOT generation putting the SQL on disk, that integration point exists by default. ## How do they work with GraalVM native images? Native images are the headline use case. AOT repositories make Spring Data viable in GraalVM native compilation by removing the reflection that native image can't follow. GraalVM native image needs to know at build time what classes will be reflected on, what methods will be called dynamically, and what proxies will be created. Spring's traditional Data infrastructure relied heavily on all three. Even with reachability metadata files, edge cases would slip through and you'd hit `MissingReflectionRegistrationError` at runtime in the native binary. With AOT repositories, the reflective machinery doesn't exist in the compiled binary. The repository is a regular class. The only reflection left is what Hibernate or the JDBC driver does for entity introspection, and Spring Boot ships proper hints for that. I migrated a Spring Boot 3.x service to native image last year, and the longest single chunk of effort was tracking down the Spring Data reflection hints. Having tested the same migration on Spring Boot 4 with AOT repositories, the failure modes that took me days to debug are mostly gone. There are still a few corner cases (custom converters, audit listeners with reflection-based wiring), but the repository layer itself is no longer the rough part. The startup numbers in native mode are dramatic. A native image that starts in 80ms vs a JVM application that starts in 1.5 seconds is the kind of win that changes how you think about scale-to-zero deployments. AOT repositories aren't the only reason that gap exists, but they're a significant chunk of it. One specific optimization worth knowing: Spring Data uses `ManagedTypes` to enumerate the entity set at build time, because classpath scanning isn't available in native mode. As long as your entities are detected by the build (they should be if they're annotated with `@Entity` and on the classpath), the AOT pipeline collects them automatically. No manual entity registration. ## What are the tradeoffs and limitations? The tradeoffs are real and worth knowing before you commit. **Configuration is frozen at build time.** The Spring Data team's exact words: the framework "trades certain dynamic aspects" for the speed. You can't generate database-specific SQL and swap databases without rebuilding. If you have an app that runs on Postgres in prod and H2 in tests with different schemas, the AOT-generated SQL is targeted to one of them. **Reactive repositories aren't supported.** If your stack is built on `ReactiveMongoRepository` or similar, AOT generation skips it entirely. The runtime proxy is still in use for those. Spring Data 2025.1 explicitly limits AOT to imperative interfaces. The team has signaled future work on reactive support but no timeline. **Build time goes up.** AOT processing isn't free. On a service with 40-odd repository interfaces, the AOT build phase added about 18 seconds to a previously 35-second build. Not catastrophic, but noticeable, and it scales with repository count. If you've got a monolith with hundreds of repositories, plan for a longer CI. **Generated SQL can surprise you.** The AOT pipeline derives SQL from method names, the same as the runtime proxy did. The difference is that with runtime proxies, you found out the SQL was wrong by reading logs. With AOT, you find out at build time when the metadata pipeline rejects an ambiguous method name. Most teams find this an upgrade. Some won't. **Custom `@Query` SpEL expressions break.** If you've leaned heavily on `@Query("...?#{principal.id}...")` style queries, those don't AOT-compile cleanly. The SpEL evaluation needs runtime context. The build will skip generation for those methods and fall back to the runtime path for them, which mostly works but means the performance benefit is partial. **Debugging is different.** The good news: you can read the generated source and step into it during a debug session. The bad news: stack traces now point to generated class names, which can confuse tooling that doesn't recognize them. IDEs handle this fine, but some external log aggregators may need configuration. ## When should you use AOT repositories? Three cases where I'd turn them on without hesitation. **If you're targeting GraalVM native image.** AOT repositories are the path. Don't fight Spring Data's reflection on native; use the AOT pipeline that Spring Boot 4 ships for exactly this case. **If startup time is on your scorecard.** Lambda cold starts, Kubernetes pod scaling, anywhere you're paying for the time between "process started" and "ready to serve traffic." Imperative Spring Data + AOT cuts a big chunk. **If you maintain a service with stable schemas.** The "frozen at build time" tradeoff isn't a tradeoff if you're not switching databases or schemas at runtime, which describes 95% of services I've worked on. Two cases where I'd hold off. **If you're heavily reactive.** No point. Wait for reactive AOT support. **If your team relies on `@Query` SpEL expressions everywhere.** The fallback is fine, but you won't see the full benefit, and the mixed mode (some methods AOT, some not) adds cognitive load. The one decision I'd push every Spring Boot 4 project to make explicitly: turn on AOT in your build pipeline. Even if you're not deploying as native image, the build-time validation of repository method names is worth it. A typo that breaks startup in dev is fine. A typo that breaks startup in prod after a release is a problem you'd rather catch in CI. The Spring Data team has shipped this feature quietly. It deserves more attention. If you're on Spring Boot 4 and haven't enabled it, do that this sprint. --- For the official details, see [Spring's blog post on AOT repositories part 2](https://spring.io/blog/2025/11/25/spring-data-ahead-of-time-repositories-part-2/) and the [Spring Data JPA AOT optimization docs](https://docs.spring.io/spring-data/jpa/reference/jpa/aot.html). The [Spring Data Commons AOT reference](https://docs.spring.io/spring-data/commons/reference/aot.html) covers the cross-module pieces that apply regardless of which store you're using. ### Keep Reading - [Modern Java with Spring Boot](https://www.rabinarayanpatra.com/blogs/modern-java-spring-boot). Background on the Java 21+ features Spring Boot 4 builds on, and how virtual threads pair with AOT for fast-start services. - [Spring Boot Testcontainers Guide](https://www.rabinarayanpatra.com/blogs/spring-boot-testcontainers-guide). Real-database testing patterns that pair well with AOT repositories. AOT validates the SQL at build time, Testcontainers proves it works against the real engine. - [Hibernate Lazy Init Guide](https://www.rabinarayanpatra.com/blogs/hibernate-lazy-init-guide). What Hibernate is doing behind your repositories. The AOT pipeline doesn't change Hibernate's lazy loading semantics, but understanding them is still essential. - [Spring Security Component Revolution](https://www.rabinarayanpatra.com/blogs/spring-security-component-revolution). The other major Spring Boot 4 architectural change worth knowing alongside AOT repositories. --- ## Java 26 Structured Concurrency: What Changed in the Sixth Preview Source: https://www.rabinarayanpatra.com/blogs/java-26-structured-concurrency-jep-525 Published: 2026-05-13T00:00:00+05:30 Tags: java-26-structured-concurrency, structured-concurrency, virtual-threads, java-concurrency, jep-525 Java 26's JEP 525 refines structured concurrency with a new onTimeout joiner and List return types. Practical guide with code and virtual thread patterns. I've been writing concurrent Java for a long time, and every team I've worked on has eventually run into the same thing: a thread leak that nobody can trace, an executor that's still running tasks ten minutes after the request that started them returned, a test that's flaky because of a race nobody can pin down. Structured concurrency fixes the category of problems that traditional executors leave on the table. Java 26's JEP 525 brings it to the sixth preview, with two focused changes: a new onTimeout callback on Joiner, and a cleaner return type for the most common joiner. It's not finalized yet, but it's close, and the API is stable enough to use seriously in projects that can run with `--enable-preview`. This post covers what structured concurrency actually is, what changed in this preview, and how I'm using it with virtual threads in real code. ## What is structured concurrency in Java 26? Structured concurrency in Java 26 is an API that confines the lifetime of a group of concurrent subtasks to a single lexical scope, ensuring every subtask is either complete or cancelled when the scope exits. ![Structured concurrency scope: parent task forks subtasks that all complete or cancel together](/images/blogs/java-26-structured-concurrency-scope.webp) The principal class is `StructuredTaskScope` in `java.util.concurrent`. You open a scope with `try-with-resources`, fork your subtasks inside it, join the scope as a unit, and the scope handles the rest. If one subtask fails, the scope can cancel the others. If you exit the scope without joining, you get an error rather than orphaned threads. Here's the simplest case: ```java try (var scope = StructuredTaskScope.open( Joiner.allSuccessfulOrThrow())) { Subtask userTask = scope.fork(() -> fetchUser(userId)); Subtask orderTask = scope.fork(() -> fetchOrders(userId)); List results = scope.join(); return new UserView(results.get(0), results.get(1)); } ``` Two subtasks, each running on its own virtual thread, both have to succeed for the scope to return. If `fetchUser` throws, the scope cancels `fetchOrders` and propagates the exception. If `fetchOrders` is still running when `fetchUser` succeeds, the scope waits for both. There's no thread that survives past the closing brace. Compare that to the unstructured version with an `ExecutorService`: ```java ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor(); Future userFuture = executor.submit(() -> fetchUser(userId)); Future orderFuture = executor.submit(() -> fetchOrders(userId)); try { String user = userFuture.get(); String orders = orderFuture.get(); return new UserView(user, orders); } catch (Exception e) { userFuture.cancel(true); orderFuture.cancel(true); throw e; } finally { executor.shutdown(); } ``` That's seven lines of plumbing for a two-task fork-join. The structured version is three. ## What changed in the sixth preview? JEP 525 introduces two API refinements: an `onTimeout()` callback on the `Joiner` interface, and a cleaner return type for `Joiner.allSuccessfulOrThrow()`. ### Why does the new onTimeout joiner method matter? `onTimeout()` lets a custom Joiner react to a timeout and return a partial or fallback result rather than throwing. In earlier previews, hitting a timeout always meant an exception. That worked for hard deadlines, but it didn't fit the patterns that real systems use. A common case: you have three downstream services to call. Two of them respond in 200ms. The third one is slow and you'd rather return a partial response than wait. With `onTimeout()`, you can build a Joiner that returns whatever subtasks succeeded by the deadline: ```java public class FirstNCompleteJoiner implements Joiner> { private final List results = new CopyOnWriteArrayList<>(); private final int target; public FirstNCompleteJoiner(int target) { this.target = target; } @Override public boolean onComplete(Subtask subtask) { if (subtask.state() == Subtask.State.SUCCESS) { results.add(subtask.get()); } return results.size() >= target; } @Override public List result() { return List.copyOf(results); } @Override public List onTimeout() { return List.copyOf(results); } } ``` The Joiner accumulates successful results, finishes early once it has `target` of them, and returns whatever it has if the timeout fires before that. Without `onTimeout()`, the third path would have thrown. This is the kind of pattern you can't express cleanly with a try/catch around the join call, because by the time you catch the exception, the partial results are gone. ### What's the new return type for allSuccessfulOrThrow? `Joiner.allSuccessfulOrThrow()` now returns a `List` directly from `scope.join()` instead of a `Stream>`. The fifth preview required you to map the stream to extract values: ```java // Old (Java 25, fifth preview) Stream> subtasks = scope.join(); List values = subtasks.map(Subtask::get).toList(); // New (Java 26, sixth preview) List values = scope.join(); ``` The old version was technically more flexible because `Subtask` carries state metadata. In practice, every codebase I've seen calls `.get()` on every subtask anyway, so the indirection was pure friction. The new return type matches what the common case actually needs. ## How do you use StructuredTaskScope in production code? The factory method to use is `StructuredTaskScope.open(Joiner)`. You pick a Joiner based on what your task semantically needs. The built-in joiners cover the most common patterns. `Joiner.allSuccessfulOrThrow()`: every subtask must succeed. Returns a List of results in fork order. If any subtask fails, all others are cancelled and the exception propagates. `Joiner.anySuccessfulResultOrThrow()`: race the subtasks. The first successful result wins, the rest are cancelled. If all fail, you get the last failure. Useful for redundant fetches: query two replicas, take whichever responds first. `Joiner.awaitAll()`: wait for all subtasks to complete, regardless of outcome. You inspect each Subtask's state afterward to decide what to do. Useful when you want partial results even on failure. Custom joiners (like the timeout-aware one above) handle the cases the built-ins don't cover. A real example from a service I'm working on: fetching an order summary that needs a user record, the user's recent orders, and pricing data. All three from different services. If any fails, the whole request fails, but I don't want to wait for the slow ones in series. ```java public OrderSummary getOrderSummary(String userId, String orderId, Duration timeout) throws InterruptedException { try (var scope = StructuredTaskScope.open( Joiner.allSuccessfulOrThrow(), cf -> cf.withTimeout(timeout))) { Subtask userTask = scope.fork(() -> userService.fetch(userId)); Subtask> ordersTask = scope.fork(() -> orderService.fetch(userId)); Subtask pricingTask = scope.fork(() -> pricingService.fetch(orderId)); scope.join(); return new OrderSummary( userTask.get(), ordersTask.get(), pricingTask.get() ); } } ``` The `cf.withTimeout(timeout)` configures the scope to cancel everything if the deadline is missed. The exception type is `StructuredTaskScope.TimeoutException`, distinct from any failures the subtasks themselves throw, so callers can react differently. The throughput numbers from rewriting one of our internal aggregation endpoints with this pattern: median latency went from 720ms (sequential) to 280ms (parallel), with no extra thread pool to manage and no executor to shut down on application stop. ## How does it work with virtual threads? `StructuredTaskScope` uses virtual threads by default for every forked subtask. You don't configure it, you just fork. That matters because virtual threads are designed for the case structured concurrency is built around: lots of independent I/O-bound subtasks. A platform thread pool that you'd previously size carefully (say, 200 threads for a high-traffic endpoint) becomes irrelevant. You can fork a thousand subtasks inside a scope and the JVM handles it. The contract: each forked subtask runs on a fresh virtual thread. The thread is owned by the scope, not by an executor that outlives the scope. When the scope closes, every thread it created is gone. Where this changes the way you write code: you stop reaching for `Executors.newFixedThreadPool` or `ForkJoinPool.commonPool()` for application-level fan-out. Those are still useful for genuinely shared work pools (think: a cron job that processes batches), but for request-scoped concurrency, the scope is the better fit. A pattern I've started using: if a request handler needs to make N parallel calls, I open a scope, fork the calls, and let the scope manage the thread lifecycle. No more thread pool sizing decisions for request-scoped fan-out. ## How do you handle errors and partial failures? Error handling in structured concurrency hangs on which Joiner you choose, and JEP 525's `onTimeout` callback adds a third axis: timeout-as-success-with-partial-data. For the all-or-nothing case, `Joiner.allSuccessfulOrThrow()` does what its name says. Any subtask failure throws on `join()`, all sibling subtasks are cancelled, and the original exception propagates with the others as suppressed exceptions. You see all of them in the stack trace, which is a huge debugging improvement over `CompletableFuture.allOf()` where only the first failure surfaces. ```java try (var scope = StructuredTaskScope.open( Joiner.allSuccessfulOrThrow())) { scope.fork(() -> { throw new IOException("primary failed"); }); scope.fork(() -> { throw new SQLException("secondary failed"); }); scope.join(); } catch (StructuredTaskScope.FailedException e) { Throwable cause = e.getCause(); Throwable[] suppressed = cause.getSuppressed(); // cause is the first failure observed // suppressed has the others } ``` For the partial-results case, `Joiner.awaitAll()` waits for every subtask regardless of outcome and returns nothing. You inspect each `Subtask` afterward to decide what to do. ```java try (var scope = StructuredTaskScope.open(Joiner.awaitAll())) { Subtask primary = scope.fork(() -> primarySource.fetch()); Subtask fallback = scope.fork(() -> fallbackSource.fetch()); scope.join(); if (primary.state() == Subtask.State.SUCCESS) { return primary.get(); } if (fallback.state() == Subtask.State.SUCCESS) { log.warn("Primary source failed, using fallback", primary.exception()); return fallback.get(); } throw new ServiceUnavailableException("All sources failed"); } ``` This pattern shows up in any system that has a primary and a fallback path. Without structured concurrency, you'd typically run them sequentially (slow when the primary is slow) or kick off both manually and manage cancellation by hand. The third axis is the new `onTimeout` callback. With a custom Joiner that returns partial results on timeout, you get a clean way to express "fetch as much as you can in N milliseconds, then return what you have." That's a pattern I've previously had to build with `CompletableFuture.applyToEither` and explicit timer scheduling, and it was always brittle. The Joiner-based version is something you can read and trust. One thing to watch out for: the cancellation that the scope triggers on failure isn't instant. It interrupts the threads, but a subtask doing CPU-bound work that ignores interrupts will run to completion. Structured concurrency doesn't change Java's cooperative cancellation model. If you're forking work that doesn't respond to interrupts, the scope won't be able to clean it up promptly. The fix is the same one you'd apply anywhere: write code that checks `Thread.interrupted()` periodically, or call into APIs that throw `InterruptedException`. The scope can ask threads to stop, but it can't force them. ## When should you use structured concurrency vs alternatives? The decision tree I use: **Use structured concurrency when** the subtasks share a lifetime with the parent task. Request handlers, batch processors that fan out per-item work, anything where "all subtasks should be done when this method returns" is a hard requirement. **Use a long-lived ExecutorService when** the work outlives the request. Background jobs, queue processors, scheduled tasks that run on a cron. The scope model doesn't fit because there's no parent task to close it. **Use CompletableFuture chaining when** you need fine-grained composition: this task's output flows into that task, with branching and merging. The scope model is for fan-out and join, not pipelines. The case where I'd specifically pick structured concurrency over `CompletableFuture.allOf()`: when error propagation matters. With `allOf`, a failure in one future doesn't cancel the others. With `StructuredTaskScope`, it does, by default. That's usually what you want for request-scoped work, and getting it right with `CompletableFuture` requires manual cancellation logic. The case where I'd specifically use it over `parallelStream()`: when the tasks are I/O-bound. `parallelStream` runs on the common ForkJoinPool, which is sized for CPU work and pinned to platform threads. Forking I/O work onto it backpressures the rest of the JVM. Structured concurrency on virtual threads doesn't have that problem. ## What's coming in the seventh preview? JEP 533 is already in flight as the seventh preview, targeting a future Java release. It refines the API further based on feedback from the sixth preview, but the core shape is stable. The signal that matters: structured concurrency has been in preview since Java 19. Six previews is a lot, but the API has changed substantially through that period. JEP 525's changes are small and targeted, which suggests the design is settling. I'd expect a final release in the next LTS window. For Java 26 specifically, this is a feature I'd turn on with `--enable-preview` for new internal services. The API isn't going to break in ways that hurt, and you get the correctness benefits today. For libraries you publish, I'd still wait for the final release. The longer-term picture: structured concurrency makes virtual threads usable in a way that traditional thread pools didn't. Virtual threads alone solve the "how do I have a million threads" problem. Structured concurrency solves the "how do I keep them sane" problem. Together they're the first concurrency model in Java that I'd actually recommend without caveats. --- For the official spec, see [JEP 525: Structured Concurrency (Sixth Preview)](https://openjdk.org/jeps/525) and [InfoQ's coverage of the timeout joiner refinements](https://www.infoq.com/news/2026/01/timeout-joiner-refinements/). For background on how the API evolved, [JEP 505 covered the fifth preview](https://openjdk.org/jeps/505) and [Bazlur Rahman's deep dive on the timeout changes](https://bazlur.ca/2026/01/04/structured-concurrency-in-java-26-api-polishing-timeouts-and-better-joiners/) walks through the design decisions in detail. ### Keep Reading - [Virtual Threads in Java 25: A Practical Guide](https://www.rabinarayanpatra.com/blogs/virtual-threads-java-25). The thread model that structured concurrency builds on. Read this first if virtual threads are still new to you. - [Java 26 HTTP/3 in HttpClient](https://www.rabinarayanpatra.com/blogs/java-26-http-3-httpclient). The other major Java 26 release feature, useful when you're forking subtasks that hit external services. - [Java 25 Compact Object Headers](https://www.rabinarayanpatra.com/blogs/java-25-compact-object-headers). Lower-level Java 25 work that pairs with structured concurrency for memory-conscious services. --- ## Claude Code Routines: Async CI Automation Just Became Real Source: https://www.rabinarayanpatra.com/blogs/claude-code-routines-ci-automation Published: 2026-05-09T00:00:00+05:30 Tags: claude-code-routines, claude-code, ci-automation, ai-developer-tools, anthropic Claude Code Routines automate PR reviews, CI fixes, and nightly triage on Anthropic's cloud. Here's everything that shipped at Code with Claude 2026. I opened my laptop Wednesday morning to a PR that had already fixed its own CI failure. Claude had seen the red X, read the error, written the fix, and pushed it to the branch. I hadn't touched anything since the night before. That's the idea behind Routines, the standout announcement from Anthropic's Code with Claude 2026 conference on May 6 in San Francisco. And if you're using Claude Code regularly, it changes how your days actually run. This post covers what shipped, how the individual features work, and what you should actually try first. ## What are Claude Code Routines? Routines are saved Claude Code configurations that trigger on a schedule, an API call, or a GitHub event, running on Anthropic's cloud infrastructure while your laptop is off. ![Claude Code Routines workflow: trigger sources feeding into cloud-run automations](/images/blogs/claude-code-routines-workflow.webp) The typical use cases: a nightly scan of open PRs flagging anything stale, an automatic code review triggered when a PR opens against main, a weekly dependency audit that files issues for anything out of date, or a doc sync that runs after every merge to keep documentation aligned with the code. Boris Cherny, who leads Claude Code at Anthropic, described the target experience: "Developers can setup async automations and wake up to PRs that are ready to merge." You finish your day, the work continues without you, and you pick it up in the morning with less noise. The key difference from a simple cron job is that Claude has full context of your repository. It isn't running a script with hard-coded rules. It's reading your code, your PR descriptions, your test output, and making judgment calls the same way it would in an interactive session. ### What kinds of automations make sense? The clearest wins are for work that's mechanical and repetitive but still requires reading code: - **Morning PR triage**: Claude reviews every open PR, summarizes what's blocking it, and flags the ones that need your attention. - **Nightly CI failure analysis**: If builds broke overnight, Claude reads the logs, categorizes the failures, and prepares a report. - **Post-merge doc sync**: After a PR merges, Claude checks whether the docs still match the changed code and opens an issue (or a PR) if they don't. - **Security review on open PRs**: Claude runs a security pass on any PR that touches auth, payments, or data handling. These are things most teams want but never get around to automating because the tooling complexity isn't worth it for a one-person project. Routines make it worth it. ## How does the CI auto-fix feature work? When CI fails on your PR, Claude reads the error output, investigates what broke, writes a fix, and pushes it to the branch with an explanation of what it changed. The stated goal from the team: "The person who owns the PR is never going to see a red X." ![CI auto-fix pipeline: failure detection, diagnosis, fix commit, and push](/images/blogs/claude-code-routines-ci-fix.webp) In practice, Claude is watching your PR pipeline. A test failure gets diagnosed and patched. A lint error gets fixed and pushed. An import that broke after a rename gets traced back and updated. It's not redesigning your code. It's handling the mechanical cleanup that usually eats 15 minutes of your morning. A companion feature, auto-merge, takes it one step further. Once all checks pass, Claude can merge the PR automatically. You configure whether you want the auto-fix only, the auto-merge only, or the full pipeline. The default is conservative: fix and wait for your approval to merge. A few things to understand about what it won't do well: if CI is failing because a database migration conflicts with a new schema expectation, that's a judgment call Claude shouldn't make alone. If the failure is architectural, it'll flag it and wait rather than guess. The feature is designed for the common case of fixable failures, and from my experience, that covers the majority of CI red marks on any active branch. ## How does Code Review work in the desktop app? Claude Code Review shipped alongside Routines and is already in use across every team at Anthropic. It reviews your local diff before you push, leaving comments in the desktop diff view directly. The comments surface bugs it spotted, suggestions for the change, and potential issues. You can ask Claude to address any of its own comments and it'll revise the diff. It's an interactive loop before the code ever leaves your machine. I've been using it on this portfolio's codebase and the most useful thing it does is catch what you stop seeing after staring at the same file for an hour. A variable that shadows a parameter. A missing null check on an API response. The kind of thing that gets through your own review because your brain fills in the expected behavior. It doesn't replace a human reviewer for architectural decisions. For the mechanical layer of review, though, it's catching things before they hit your team and before they hit CI. ## What else shipped at Code with Claude 2026? Beyond the workflow automation features, a few other changes are worth knowing. **Rate limits**: Claude Code five-hour limits were doubled for Pro, Max, Team, and Enterprise plans. Peak-hours throttling for Pro and Max accounts is gone. Opus API rate limits got a substantial increase. **Capacity**: The rate limit headroom comes from the SpaceX Colossus data center partnership. Anthropic is taking on more than 300 megawatts from Colossus, translating to over 220,000 NVIDIA GPUs coming online within the month. The API volume signal that drove this: Anthropic's API usage is up 17x year over year. **Managed agent capabilities**: Anthropic announced three new features for multi-agent work. Multi-agent orchestration (public beta) lets you spin up fleets of agents for complex tasks. Outcomes (public beta) lets you set success criteria so Claude iterates independently until it meets them. Dreaming (research preview) is more experimental: Claude inspects its own previous sessions, identifies what it missed, and self-improves over time. **The scale context**: Mercado Libre has 23,000 engineers and is targeting 90% autonomous coding by Q3 2026. That number tells you what direction the industry is moving, and why Anthropic needed the Colossus capacity this month. The conference itself continues: London on May 19, Tokyo on June 10. Additional features may ship through May. ## Where should you start with these features? If you're already on Claude Code Max or Pro, the doubled rate limits are already in effect. Nothing to configure. For Routines and CI auto-fix, rollout is happening through May. The fastest way to evaluate CI auto-fix: pick a branch where you're actively working and CI fails regularly. Let Claude handle the mechanical failures. You'll see quickly what it gets right and where it needs a clearer repo setup. For Code Review, it's in the desktop app now. Open a diff you're about to push, run the review, and compare what it flags to your own manual pass. You'll know in about five minutes whether it's catching things you'd miss. The one thing I'd push back on: it's tempting to automate everything immediately. Start with one Routine that solves a specific pain point, watch how it behaves over a week, then add more. Claude Code has full repo context, but the automation is only as good as the task definition you give it. My setup: CI auto-fix on the dev branch, morning PR summary routine, and Code Review before every push. That's it for now. The rate limits being doubled means I'm not holding back on interactive sessions to save headroom, which is the real quality-of-life change from this release. --- For more details on how Routines and CI auto-fix work, see [Claude Code's documentation at code.claude.com](https://code.claude.com/docs/en/overview) and [Simon Willison's Code with Claude 2026 live blog](https://simonwillison.net/2026/May/6/code-w-claude-2026/). Anthropic's blog post [Preview, review, and merge with Claude Code](https://claude.com/blog/preview-review-and-merge-with-claude-code) covers the desktop Code Review and auto-merge pipeline. ### Keep Reading - [Claude Skills vs MCP vs Projects: What's the Difference?](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects). Understand how Routines fit into Claude's broader tool architecture before building automations on top of it. - [Vercel AI Gateway Deep Dive](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive). If you're building on the Claude API behind Routines, the AI Gateway handles routing, fallbacks, and observability at the API layer. - [Claude Opus 4.7 Release and Migration Guide](https://www.rabinarayanpatra.com/blogs/claude-opus-4-7-release-and-migration-guide). The model powering these features, and what changed in the version your Routines will run on. --- ## Next.js 16.2 AGENTS.md and next-browser: A Hands-On Guide Source: https://www.rabinarayanpatra.com/blogs/nextjs-16-2-agents-md-next-browser Published: 2026-04-29T00:00:00+05:30 Tags: nextjs-agents-md, next-browser, nextjs-16-2, vercel, ai-coding Next.js 16.2 ships AGENTS.md by default, bundles the full docs in node_modules/next/dist/docs, and adds next-browser CLI for AI agents. Hands-on guide. I run Claude Code on my Next.js portfolio every day. For the last year my routine has been the same: paste the relevant Next.js 16 doc into chat, watch the agent write something that worked in Next.js 13, correct it, paste another doc, repeat. The agent's training data was stale, the context window was precious, and I was the middleman. Next.js 16.2 quietly killed that routine. Shipped on March 18, 2026 by Jude Gao, Jimmy Lai, Tim Neutkens and the rest of the Next.js team, the release ships `AGENTS.md` by default, bundles the entire Next.js docs as plain Markdown inside `node_modules/next/dist/docs/`, and adds an experimental `@vercel/next-browser` CLI that lets agents inspect a running app through shell commands. Vercel ran their internal evals and saw 100% pass rate with `AGENTS.md` vs 79% with the best skill-based setup. That is a big enough gap to change how I scaffold projects. This post walks through what actually shipped, how to add it to an existing project, and what I changed in my own workflow after upgrading `rabinarayanpatra.com` to 16.2. ## What is AGENTS.md in Next.js 16.2? `AGENTS.md` is a Markdown file at the root of a Next.js project that tells AI coding agents to read the version-matched docs bundled inside `node_modules/next/dist/docs/` before writing any code. It is not a skill, it is not a plugin, and it is not loaded on demand. It is always-on context that any agent respecting the `AGENTS.md` convention will pick up at the start of every turn. The default file that `create-next-app` generates is small. Here is the shape of it: ```md filename="AGENTS.md" # Next.js: ALWAYS read docs before coding Before any Next.js work, find and read the relevant doc in `node_modules/next/dist/docs/`. Your training data is outdated. The docs are the source of truth. ``` The `BEGIN` and `END` comment markers delimit the Next.js-managed section. Future codemods will only rewrite what is inside those markers, so anything you add outside stays yours. That is a small design choice with a big payoff, because it means you can keep your own agent rules next to the framework's without fighting merge conflicts. The second piece is the bundled docs themselves. The `next` npm package now ships the full doc tree as plain Markdown files. Open `node_modules/next/dist/docs/` in your editor and you will see the same content as nextjs.org, organized by section. A compressed index file is also shipped at around 8 KB, an 80% reduction from the raw 40 KB of full docs, which is what the agent reads first to find the right file to pull into context. Third, on projects already set up for Claude Code, there is a tiny convention link: ```md filename="CLAUDE.md" @AGENTS.md ``` The `@` directive tells Claude Code to include the contents of `AGENTS.md` whenever it loads `CLAUDE.md`. So the bootstrap chain becomes `CLAUDE.md` pulls in `AGENTS.md`, `AGENTS.md` tells the agent to read `node_modules/next/dist/docs/` whenever Next.js work is happening, and the agent writes code against the real 16.2 API rather than a memory of Next.js 13. ## How do you add AGENTS.md to an existing Next.js project? On Next.js 16.2 or later, you already have the docs on disk. You just need to create the two files yourself or let the codemod do it. The codemod is the faster path: ```bash filename="terminal" npx @next/codemod@latest agents-md ``` That scaffolds `AGENTS.md` with the managed directive block and creates or updates `CLAUDE.md` with the `@AGENTS.md` include. I ran it on the portfolio and it was a one-second change plus two new files in the diff. If you are on an older version of Next.js, upgrade first: ```bash filename="terminal" npx @next/codemod@canary upgrade latest ``` Then re-run the `agents-md` codemod. The upgrade is where the doc bundle actually lands on disk, so you need 16.2 for the whole flow to work. The part I liked most is the escape hatch. If your project already has a sprawling `AGENTS.md` with your own conventions, you can drop the managed block into it and keep everything else. The codemod respects existing files and only touches what is inside the markers. That matters for teams who have been writing their own agent rules for months. One pitfall I hit on my first attempt: my `.gitignore` had `node_modules/` ignored (obviously), but Claude Code was still happy to read from it, since it is a local file read, not a git operation. So you do not need to commit the docs, you just need the package installed. CI builds, however, need to run `npm install` before an agent can use the docs, which is the usual order anyway. ## What does the @vercel/next-browser CLI actually do? `@vercel/next-browser` is an experimental CLI that wraps a persistent Chromium instance with React DevTools pre-loaded and exposes browser data through shell commands. Instead of an agent trying to understand a DevTools panel it cannot see, the agent runs a shell command, gets back structured text, and reasons about the output. You install it as a skill: ```bash filename="terminal" npx skills add vercel-labs/next-browser ``` Then trigger it inside an agent that supports skills by typing `/next-browser`. Claude Code and Cursor both work. The first run boots a Chromium instance you do not see, loads React DevTools into it, and holds the session open across commands. At release the feature set covers five buckets. Component trees with props, hooks, state, and source-mapped file locations. PPR shell analysis, which identifies what is static and what is blocking. Errors and logs from the dev server. Network activity since the last navigation, including server actions. And visual capture with screenshots or loading filmstrips. The concrete example in the release post is my favorite, because it shows the workflow end-to-end. You have a blog post page with a visitor counter: ```tsx filename="app/blog/[slug]/page.tsx" export async function generateStaticParams() { const posts = await getAllPosts(); return posts.map((post) => ({ slug: post.slug })); } export default async function BlogPost({ params }) { const post = await getPost(params.slug); const views = await getVisitorCount(params.slug); // per-request return (

{post.title}

{views} views
{post.content}
); } ``` Every slug is enumerated ahead of time, so the post content should prerender at build. But `getVisitorCount` runs on every request and sits at the top level, which drags the entire page out of the static shell. The user sees a loading skeleton instead of the post content streaming in. An agent can diagnose this by locking PPR mode and inspecting the shell: ```bash filename="terminal" next-browser ppr lock next-browser goto /blog/hello ``` With PPR locked, only the static shell renders. In this case the shell is the loading skeleton, because nothing from the page made it in. Running `ppr unlock` gives the agent a structured report: ```bash filename="terminal" # PPR Shell Analysis # 1 dynamic hole, 1 static blocked by: - getVisitorCount (server-fetch) owner: BlogPost at app/blog/[slug]/page.tsx:5 next step: Push the fetch into a smaller Suspense leaf ``` The agent now knows what the blocker is, where it lives, and what to do. It wraps the counter in a Suspense boundary: ```tsx filename="app/blog/[slug]/page.tsx" export default async function BlogPost({ params }) { const post = await getPost(params.slug); return (

{post.title}

... views}>
{post.content}
); } ``` Run `ppr lock` again and the shell has grown. The post content prerenders instantly, and only the view count falls back to the Suspense placeholder. The agent did that without ever opening a browser window. The thing I keep coming back to is that the CLI is designed around one-shot commands against a persistent session. That matches how LLM tool-calling works today. The agent does not manage browser state, the CLI does. The agent just asks questions and parses replies, which is exactly the interaction pattern that works for a model. ## How does the dev server lock file help AI agents? Next.js 16.2 writes the running dev server's PID, port, and URL into `.next/dev/lock`. When a second `next dev` starts in the same directory, Next.js reads the lock file and prints an actionable error instead of a generic port-conflict message: ```bash filename="terminal" Error: Another next dev server is already running. - Local: http://localhost:3000 - PID: 12345 - Dir: /path/to/project - Log: .next/dev/logs/next-development.log Run kill 12345 to stop it. ``` On paper this is quality-of-life for humans. In practice, this is the single change that saves me the most time with Claude Code. Before 16.2, my workflow was: I start the dev server in a pane, Claude Code spawns its own `next dev` to check something, the command hangs because port 3000 is taken, Claude Code retries on another port, I get a second server I do not want. Now the agent gets a clean error, a PID to kill, a URL to connect to, and a log path to tail. It makes the right call the first time. The lock file also prevents two `next build` processes from running at once, which could otherwise corrupt build artifacts. That is a genuine bug I have seen in CI where a re-run was queued before the first finished. Now the second run refuses to start. ## How do you forward browser logs to the terminal? By default, Next.js 16.2 forwards browser errors to the dev terminal. You do not have to switch to the browser console to see client-side errors, which is another quality-of-life win for agents who cannot open a DevTools panel at all. The level is configurable: ```ts filename="next.config.ts" const nextConfig = { logging: { browserToTerminal: true, // 'error': errors only (default) // 'warn': warnings and errors // true: all console output // false: disabled }, }; export default nextConfig; ``` I ran this at `true` for a week and it was too noisy for a typical debugging session. I rolled it back to `'warn'` which is the right default for me. Errors alone is usually too narrow, because half the client-side issues I chase start as warnings that the agent would have caught earlier with a wider filter. Pairing this with the agent DevTools is the real win. The agent gets errors in the terminal, runs `next-browser` for a closer look, fixes the code, and moves on. No browser console, no screenshots to interpret, no back-and-forth with me asking what the page looked like. ## Why does Vercel say AGENTS.md beat skills in their evals? The numbers from Vercel's internal eval are worth quoting straight. They tested four configurations against a suite of Next.js 16 APIs that did not exist in model training data, like `'use cache'`, `cacheTag()`, `forbidden()`, and `proxy.ts`: ![Pass rate on Next.js 16 evals across four configurations: baseline 53 percent, skill default 53 percent, skill with explicit instructions 79 percent, AGENTS.md 100 percent](/images/blogs/nextjs-16-2-agents-md-next-browser-eval.webp) | Configuration | Pass rate | |---|---| | Baseline (no docs) | 53% | | Skill (default invocation) | 53% | | Skill with explicit instructions | 79% | | AGENTS.md docs index | 100% | The skill with default invocation matched baseline exactly. Vercel's explanation for that is brutal: "In 56% of eval cases, the skill was never invoked." The agent did not know it needed to look something up, so it did not. Explicit instructions pushed skills to 79%, which is a real improvement, but it took careful prompt wording to get there. Phrasing like "You MUST invoke" caused agents to anchor on the docs and miss project context. "Explore project first, then invoke skill" was better. That kind of prompt sensitivity is not something I want to maintain for a team. `AGENTS.md` hit 100% across build, lint, and test eval categories. The authors called out the mental model: the goal is to shift agents "from pre-training-led reasoning to retrieval-led reasoning." Always-on context wins over on-demand retrieval because there is no decision point the agent can get wrong. That matches my experience. The moments I lost the most time with Claude Code were the moments it was confidently wrong, using old APIs it remembered from training. It did not ping the docs because it did not know it needed to. `AGENTS.md` removes that failure mode by making the docs non-negotiable. ## What changes did Next.js 16.2 make to rendering performance? The agent story is the headline, but 16.2 also ships a big performance jump that is easy to miss. The team landed a change in React (PR #35776) that replaces `JSON.parse` with a reviver callback with `JSON.parse` followed by a recursive walk in pure JavaScript. The result is up to 350% faster RSC payload deserialization and 25% to 60% faster real-world server rendering. The root cause is a V8 quirk. `JSON.parse` with a reviver crosses the C++/JavaScript boundary once per key-value pair, and even a no-op reviver makes parsing around 4x slower than without one. Replacing that with a two-step parse-then-walk eliminates the per-key cost. The measured numbers are the kind of data I trust because they come from real apps: | Workload | Before | After | Speedup | |---|---|---|---| | 1000-item Server Component table | 19ms | 15ms | 26% | | Server Component with nested Suspense | 80ms | 60ms | 33% | | Payload CMS homepage | 43ms | 32ms | 34% | | Payload CMS with rich text | 52ms | 33ms | 60% | You do not need to change any code. Upgrade to 16.2 and `react@latest` and the speedup lands. The heavier your RSC payload, the bigger the win, which is why the rich-text case hits 60%. Next.js 16.2 also makes `ImageResponse` 2x faster for basic images and up to 20x faster for complex ones, with the default font switched from Noto Sans to Geist Sans. Dev startup is around 87% faster than 16.1 on the default application, so the time between `next dev` and a ready localhost shrank noticeably on my machine. ## Does AGENTS.md replace Claude Code skills entirely? No, and I do not think it should. `AGENTS.md` and skills solve different problems. `AGENTS.md` is for always-on framework context. Your agent needs to know Next.js 16 every time it writes a Next.js component. That is not an on-demand concern, it is a default concern. Putting it in `AGENTS.md` removes the decision point about whether to fetch the docs, which is exactly the decision agents are bad at. Skills are for scoped, opt-in capabilities. A skill that runs a specific CLI, calls a private API, or triggers a deploy is something you want the agent to invoke deliberately. The decision point is the feature. You do not want the agent triggering your prod deploy on every turn. The Vercel team made the right call putting framework docs into `AGENTS.md` and keeping the `next-browser` debugger as a skill you trigger with `/next-browser`. The debugger is a scoped capability. The framework docs are ambient context. Matching each tool to the right mechanism is the actual lesson here. My working rule, after a week of running both on the portfolio: if your agent should consult it every time, put it in `AGENTS.md`. If your agent should consult it sometimes, make it a skill. ## What does this mean for the next year of framework design? Next.js 16.2 is the first mainstream framework release where the primary design target is an AI coding agent, not a human developer. The default scaffolding exports a file the human may never read. The docs are shipped as Markdown because that is what parses cleanly for a model. The dev server writes a lock file so another process can recover without a human. The experimental DevTools are a shell command interface because that is what LLMs can drive. This is not a future-looking take. It is already in `create-next-app@latest`. And the concrete UX win for me was small but sharp: I stopped pasting docs into chat. That one behavior change saved me an hour a week. Scaled across a team of developers all running agents all day, the time savings compound fast. The open question is how many frameworks follow. Spring, FastAPI, Rails, Django, Laravel, every major framework has the same training-data-staleness problem Next.js just solved. I expect the next wave of releases to ship their own `AGENTS.md` conventions, bundled docs, and agent-friendly CLI wrappers. Vercel's eval numbers make the business case too strong to ignore. For more on this release, see the [Next.js 16.2 blog post](https://nextjs.org/blog/next-16-2), the [Next.js 16.2 AI Improvements post](https://nextjs.org/blog/next-16-2-ai), and Vercel's [AGENTS.md outperforms skills eval writeup](https://vercel.com/blog/agents-md-outperforms-skills-in-our-agent-evals). The RSC payload perf change is [React PR #35776](https://github.com/facebook/react/pull/35776) if you want to read the actual diff. ### Keep Reading - [Building a Modern Docs Generator with Next.js 16](https://www.rabinarayanpatra.com/blogs/building-modern-docs-generator-nextjs-16). Where I first went deep on Next.js 16's App Router and how I think about docs infrastructure. - [Hello Proxy in TypeScript and Next.js 16](https://www.rabinarayanpatra.com/blogs/hello-proxy-ts-nextjs-16). The middleware/proxy rename that landed in 16.0, relevant background for anything using `proxy.ts`. - [Replacing useEffect Data Fetching with Server Actions](https://www.rabinarayanpatra.com/blogs/replacing-useeffect-data-fetching-server-actions). Server-side patterns that pair naturally with the 16.2 rendering improvements. - [Vercel AI Gateway Deep Dive](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive). The other side of the Vercel AI story, if you want the provider routing and observability angle. --- ## Postgres Connection Pool Sizing: How PgBouncer Saved My Launch Source: https://www.rabinarayanpatra.com/blogs/postgres-connection-pool-pgbouncer-survival-guide Published: 2026-04-25T00:00:00+05:30 Tags: postgres-connection-pool, pgbouncer, postgresql, database-tuning, system-design, devops A real war story on Postgres connection pool sizing: why pool=500 nearly killed my launch, and how PgBouncer transaction mode multiplexed 5000 clients onto 25 backends. In my previous org, we were a few days from a launch deadline when staging started falling over under load tests. I spent the first few hours convinced it was bad SQL. I combed through slow query logs, rolled back two recent migrations, and came up empty. The QA team kept pinging me, the PM kept asking for an ETA, and I had no answer. Turned out it was one line in a YAML file. Connection pool size set to 500 on a 4-core Postgres box. That is the kind of config that looks sensible until you do the math. Hibernate's Hikari pool default is 10. We had bumped it to 500 because someone had read a Stack Overflow answer about "scaling Postgres" and wanted headroom. Multiply that across ten app pods and you are not running a database anymore, you are running a fork bomb. Postgres forks a process per connection, each one eating around 10 MB of RAM, and they all fight for the same four CPU cores. Throughput collapsed long before we hit any real query bottleneck. The fix that saved that launch was PgBouncer in transaction pooling mode. Five thousand app connections multiplexed onto 25 real Postgres connections. Same throughput. A fraction of the load. We shipped on time, and I spent the next month writing internal docs so the next person would not learn this the hard way. This post is the long version of those docs. Pool size is the one config that quietly decides if your service survives production traffic, and it is one of the most consistently misconfigured knobs I see in code reviews. I will walk through why Postgres has hard limits, how to size your pool from first principles, what PgBouncer actually does, the sharp edges in transaction mode that will bite you, a config snippet you can copy, and when to reach for something beyond PgBouncer. ![A diagram showing how thousands of application connections multiplex through PgBouncer into a small number of real Postgres backends](/images/blogs/postgres-connection-pool-pgbouncer-survival-guide.webp) ## Why does Postgres choke on too many connections? Postgres choke on too many connections because every connection is a full operating-system process, and processes are not free. The supervisor (postmaster) calls `fork()` for each new client, and the child becomes a backend that accumulates its own private state: relation cache, plan cache, prepared statement cache, work_mem allocations, temp buffers, and page table entries. Linux copy-on-write makes the initial fork cheap, but the steady-state cost is real and it scales linearly with connection count. The popular "10 MB per connection" figure is a Heroku-style rule of thumb, not a constant. Andres Freund, a Postgres committer, did the careful measurement in 2020 and found a wider range. An idle connection without `huge_pages` shows around 16 MiB RSS, but the true Proportional Set Size overhead is closer to 7.6 MiB. With `huge_pages=on`, the true overhead drops to about 1.3 MiB. AWS measured idle connections on RDS Postgres at about 1.5 MB, climbing to 10.8 MB after a single SELECT and 14.5 MB after a query that touches temp tables. The right way to think about it is "1 to 15 MB per connection depending on what the client just did and how the kernel is configured", not a flat number. Memory is the easy part. The harder problem is CPU and snapshot scalability. Pre-Postgres-14, every transaction called `GetSnapshotData()` which scanned the entire process array. Andres Freund and the Citus team measured this with pgbench on Postgres 12. With one active connection and zero idle, they hit 33,457 TPS. Add 10,000 idle connections and the same workload dropped to 14,496 TPS, a 57% loss. CPU profiling showed half the time in `GetSnapshotData()`. Postgres 14 fixed most of this and roughly doubled throughput at high idle counts, but the overhead is still real, and plenty of teams are running on managed Postgres versions that are pinned to older majors. There is one more multiplier that catches people off guard. The `work_mem` setting is per operation, not per connection. The official docs put it plainly: a complex query might run several sort and hash operations at the same time, with each one allowed to use `work_mem` before spilling to disk. The default is 4 MB, but a query with three hash joins on a single backend can pull 12 MB on its own. Multiply that by 500 backends doing real work and you understand why staging melted. That is why `max_connections` defaults to 100. The Postgres wiki on [Number Of Database Connections](https://wiki.postgresql.org/wiki/Number_Of_Database_Connections) says it directly: you can often support more concurrent users by reducing the number of database connections and using some form of connection pooling. Raising `max_connections` is rarely the right move, because Postgres pre-allocates shared memory and other resources at startup based on it. You make every connection slower in exchange for being able to open more of them, which is the opposite of what you want. ## How big should your Postgres connection pool actually be? The answer is "much smaller than you think". The HikariCP wiki and the PostgreSQL wiki both cite the same formula: ```text connections = ((core_count * 2) + effective_spindle_count) ``` This number originally came from Oracle's Real-World Performance group, who found that shrinking pool size dropped response time from around 100 ms to around 2 ms in their tests, roughly a 50x improvement. Brett Wooldridge, the author of HikariCP, distilled it into one line that I now repeat every chance I get: "You want a small pool, saturated with threads waiting for connections." The "effective spindle count" part needs a 2026 translation. It used to mean the number of physical disks, because spinning rust let you do real parallel I/O while one disk was seeking. On modern storage the number is closer to zero. NVMe with a hot working set in the buffer cache is effectively zero spindles. Cloud-attached SSDs like AWS gp3, GCP pd-ssd, or Azure Premium SSD v2 behave like one or two spindles under bursty load. Cold scans on cheap storage (think gp2 with the burst budget exhausted) start to look more like the old spindle model where you can usefully overlap I/O wait with CPU work. Let me work through my exact incident numbers because they make the math vivid. I had a 4-core Postgres box, ten app pods, and Hikari pool size set to 500 per pod. Theoretical maximum client connections from the app fleet: 10 pods times 500 connections, equals 5,000. Postgres `max_connections` was the default 100. Even if we had raised it to 500, we would have been ten times over. The HikariCP formula for that 4-core box with a hot dataset on NVMe gives `(4 * 2) + 0 = 8` useful active backends across the entire fleet. That is less than one active backend per pod. ![A diagram showing thousands of client connections from three app pods overwhelming a 4-core Postgres database, illustrating the fork-bomb pattern](/images/blogs/postgres-connection-pool-pgbouncer-survival-guide-fork-bomb.webp) The fix was not "raise max_connections", it was the opposite. Each pod kept a small Hikari pool of 5 to 10 connections pointed at PgBouncer on port 6432. PgBouncer multiplexed those 5,000 client connection slots onto roughly 25 real Postgres backends. We did not lose throughput because the bottleneck was never the number of physical connections, it was the cost of having too many of them. Little's Law is the math behind Wooldridge's axiom. `L = lambda * W`, where L is concurrent in-flight requests, lambda is arrival rate, and W is service time. If your database serves a query in 5 ms and you need 10,000 queries per second, you need `10,000 * 0.005 = 50` concurrent connections. Adding more does not increase throughput. It adds queueing inside Postgres (lock contention, context switching) which increases W, which increases required L, and you spiral into a queue that grows faster than it drains. Dan Slimmon has a nice [write-up applying Little's Law to scaling decisions](https://blog.danslimmon.com/2022/06/07/using-littles-law-to-scale-applications/) that I send to anyone who wants to argue with the formula. Heroku publishes a useful operational rule of thumb. When their pooler connects to a Postgres instance, it can open up to 75% of the database plan's connection limit, leaving 25% for direct connections from the app and admin tools. That ratio is a sane default to copy. If `max_connections` is 100, set PgBouncer's `max_db_connections` to 75 or so, and reserve the rest for psql sessions and migration jobs. ## What is PgBouncer and how does transaction mode work? PgBouncer is a single-process, asynchronous, lightweight connection pooler that sits between your application and Postgres and pretends to be Postgres on its end. Your app talks to it on port 6432 (the convention) and it speaks the Postgres wire protocol back to a small number of real backends. The PgBouncer features doc claims around 2 KB of memory per client connection, which is what makes the math work: you can serve thousands of clients on a single core because each one is essentially just a small allocation and an epoll registration. PgBouncer ships three pool modes. The verbatim definitions from [pgbouncer.org/config.html](https://www.pgbouncer.org/config.html) are: | Mode | When the backend is released back to the pool | | --- | --- | | `session` | After the client disconnects (the default). | | `transaction` | After the transaction finishes (COMMIT or ROLLBACK). | | `statement` | After the query finishes. Multi-statement transactions are disallowed. | Transaction mode is the sweet spot for almost every web or microservice workload. If your transactions are short (and they should be, you are not holding a transaction open across an HTTP request boundary, right?) then a single backend can serve many clients per second. With BEGIN to COMMIT lasting 5 ms and a 25-backend pool, you can handle on the order of 5,000 transactions per second on that pool, which is more than most apps will ever need. Statement mode is too restrictive for anything that uses multi-statement transactions, which means it is too restrictive for anything that uses an ORM. Session mode gives you full Postgres compatibility but throws away the multiplexing benefit, so you might as well not use a pooler. Transaction mode is where the magic happens, and it is also where you have to know what you are doing, which is the next section. Version compatibility matters more than people think. PgBouncer 1.21, released October 2023, added protocol-level prepared statement support to transaction mode. That single feature unblocked a huge class of applications that previously had to disable client-side prepared statements (with `prepareThreshold=0` in JDBC, for example) to use a pooler. Version 1.24, January 2025, made prepared statement support default-on with `max_prepared_statements = 200`. The current stable as of April 2026 is 1.25.1, released December 2025, which fixes CVE-2025-12819 (an unauthenticated SQL execution via a malicious `search_path` in the StartupMessage) and a SCRAM auth regression. If you are running anything older than 1.25.1, upgrade. PgBouncer is single-threaded by design. To use more than one CPU core on a busy host, you run multiple PgBouncer processes bound to the same port via `SO_REUSEPORT`, and the Linux kernel distributes incoming connections across them. To keep query cancellation working correctly across processes, configure the `peers` section so any process can route a cancel message to the one holding the original connection. Crunchy Data has [a good writeup on running multiple PgBouncers at scale](https://www.crunchydata.com/blog/postgres-at-scale-running-multiple-pgbouncers), and Zalando's engineering blog covers their central-deployment pattern with dedicated CPU cores. ![A side-by-side diagram of session, transaction, and statement pool modes showing where each releases the backend connection](/images/blogs/postgres-connection-pool-pgbouncer-survival-guide-pool-modes.webp) ## Which Postgres features break in PgBouncer transaction mode? Transaction mode breaks anything that depends on session-scoped state, because the next transaction from the same client may be served by a completely different backend. The list is finite and well-documented, but every team I have worked with has been bitten by at least one of these. Server-side prepared statements were the historical pain point. Before PgBouncer 1.21, when a client called `PREPARE foo` and then sent another `EXECUTE foo` in a new transaction, the second call could land on a backend that had never seen `foo`, and you would get a `prepared statement does not exist` error in production at 3 a.m. The standard JDBC workaround was `prepareThreshold=0` in the connection string, which disabled client-side prepared statement caching entirely. PHP teams set `PDO::ATTR_EMULATE_PREPARES = true` for the same reason. Both workarounds gave up the planning-time savings of prepared statements, which on complex queries is a big deal. Crunchy measured a 15-table self-join going from 174.896 ms of planning time without prepared statements to 0.020 ms with them, roughly an 8,700x speedup. PgBouncer 1.21 fixed this for the protocol path. PgBouncer now intercepts the wire-protocol `Parse`, `Bind`, `Describe`, and `Execute` messages, renames each prepared statement to its own internal name like `PGBOUNCER_1`, and re-issues the `Parse` on whichever backend it routes the next transaction to. Your client sees the statement working transparently. Pgjdbc, npgsql, pgx, asyncpg, and psycopg3 all use this protocol path, so they get the speedup automatically once you enable `max_prepared_statements`. Here is the gotcha that catches people: SQL-level `PREPARE foo AS SELECT ...` is not intercepted, only protocol-level prepared statements are. If your code literally sends the text `PREPARE` to the database, PgBouncer treats it like any other SQL and the next transaction will not see it. This caveat is on the [PgBouncer FAQ](https://www.pgbouncer.org/faq.html) but easy to miss. `SET` and `SET LOCAL` behave differently in transaction mode. `SET LOCAL statement_timeout = '5s'` is transaction-scoped and works fine because it dies at COMMIT. `SET statement_timeout = '5s'` is session-scoped and breaks, because the next transaction may land on a different backend that does not have the GUC set. The fix is to either use `SET LOCAL`, or pass settings via the connection string with `options=-c statement_timeout=5s`, which applies them at backend startup. `LISTEN` and `NOTIFY` are flat-out broken in transaction mode. `LISTEN` registers a session-scoped subscription on a backend you no longer own after the transaction ends. The right answer is to keep a separate non-pooled connection (going directly to Postgres on port 5432) for any worker that needs to listen for notifications. Heroku's docs make this explicit and I have seen the same advice in every production-grade pooling guide. Advisory locks split into two categories. `pg_advisory_lock` is session-scoped and broken in transaction mode for the same reason `LISTEN` is. `pg_advisory_xact_lock` is transaction-scoped and works fine because it releases at COMMIT or ROLLBACK. Rails users, in particular, have to set `advisory_locks: false` in `database.yml` when using PgBouncer transaction mode, because Rails uses session-scoped advisory locks for migration coordination by default. `WITH HOLD` cursors are designed to outlive a transaction, which is the exact thing transaction mode does not support. The next transaction may not see the same backend, so the cursor is gone. The fix is to drop `WITH HOLD` and consume the cursor inside the transaction, or to use session mode for that specific connection. Temp tables behave the way you would expect: `CREATE TEMP TABLE foo ON COMMIT DROP` works because the table dies with the transaction, and `ON COMMIT PRESERVE ROWS` breaks because the table is session-scoped. If your job needs a temp table that lives across transactions, you need a different pattern, often a regular table with a job ID column and a cleanup process. ## How do you configure PgBouncer for production? You configure it by changing the connection string and writing a `pgbouncer.ini` that matches your Postgres ceiling. The connection string change is the easy part. Your apps go from connecting directly to the database: ```text postgres://app:pwd@db.internal:5432/myapp ``` To connecting through PgBouncer: ```text postgres://app:pwd@pgbouncer.internal:6432/myapp ``` Your application code does not change. PgBouncer pretends to be Postgres, your driver does not know the difference. Here is a `pgbouncer.ini` snippet that would have prevented my incident, sized for the same 4-core Postgres box: ```ini filename="pgbouncer.ini" [databases] myapp = host=db.internal port=5432 dbname=myapp [pgbouncer] listen_addr = 0.0.0.0 listen_port = 6432 pool_mode = transaction max_client_conn = 5000 default_pool_size = 25 max_db_connections = 30 reserve_pool_size = 5 reserve_pool_timeout = 3.0 server_idle_timeout = 60 max_prepared_statements = 200 auth_type = scram-sha-256 auth_file = /etc/pgbouncer/userlist.txt admin_users = pgbadmin stats_users = pgbadmin ``` Walking through the knobs that matter: - `pool_mode = transaction` is the multiplexing setting we have been talking about. - `max_client_conn = 5000` is the ceiling on app-side connections. Set it well above your actual demand because the per-client memory is tiny. - `default_pool_size = 25` is the number of real Postgres backends per (user, database) pair. This is the number you derive from the HikariCP formula plus a small buffer. - `max_db_connections = 30` is the hard cap on backends to a single database, regardless of pool count. Set it under your Postgres `max_connections` minus reserved superuser slots. - `reserve_pool_size = 5` allows five extra backends to spin up under sustained load, after the wait time below. - `reserve_pool_timeout = 3.0` means a client that waits more than 3 seconds for a connection triggers the reserve pool. - `server_idle_timeout = 60` recycles backends that sit idle for a minute, freeing Postgres-side memory. - `max_prepared_statements = 200` enables protocol-level prepared statements. Default since 1.24, but set it explicitly so you can read the value off the file. Authentication should always be `scram-sha-256` in 2026. The older `md5` hash format is effectively obsolete and you will fail a security review the day someone notices it. The `auth_user` plus `auth_query` pattern lets PgBouncer look up arbitrary users from `pg_authid` on demand instead of pre-loading every user into `userlist.txt`, which is helpful when you have a large or dynamic user list. The default `auth_query` per the PgBouncer docs is: ```sql SELECT rolname, CASE WHEN rolvaliduntil < now() THEN NULL ELSE rolpassword END FROM pg_authid WHERE rolname = $1 AND rolcanlogin ``` The monitoring story is where PgBouncer earns its keep. Connect to the special database called `pgbouncer` on port 6432 with a user listed in `admin_users` or `stats_users`, and you get a small command set that tells you everything about pool health. The four commands I run constantly are `SHOW POOLS`, `SHOW STATS`, `SHOW CLIENTS`, and `SHOW SERVERS`. A real `SHOW POOLS` row looks like this (formatted as a code block, values illustrative): ```text database | user | cl_active | cl_waiting | sv_active | sv_idle | sv_used | maxwait | pool_mode -----------+------+-----------+------------+-----------+---------+---------+---------+------------- myapp | app | 142 | 3 | 18 | 7 | 0 | 0.04 | transaction ``` Decoded: 142 clients are connected and busy, 3 are queued waiting for a backend, 18 backends are actively running queries, 7 are idle in the pool, the longest waiting client has been waiting 0.04 seconds. If `cl_waiting` stays above zero for sustained periods, your pool is too small. If `maxwait` climbs above one second, your users are noticing. For ongoing observability, deploy the [prometheus-community/pgbouncer_exporter](https://github.com/prometheus-community/pgbouncer_exporter) on port 9127. It scrapes `SHOW LISTS`, `SHOW STATS`, `SHOW POOLS`, and `SHOW DATABASES` on an interval and exposes them as Prometheus metrics. The three alerts I would set on day one are: `pgbouncer_pools_client_waiting > 0` for more than 30 seconds, `pgbouncer_pools_server_active / default_pool_size > 0.8` sustained, and `pgbouncer_stats_total_wait_time` rising over a 5-minute window. Heroku's empirical guidance is that connection queueing emerges past around 15,000 to 20,000 transactions per second on a single PgBouncer process, so add a second process via `SO_REUSEPORT` before you cross that threshold. You will also want a small set of Postgres-side queries to diagnose what is happening on the database when PgBouncer says it is overloaded. The two I keep in my snippets file: ```sql filename="diagnose.sql" -- Count active backends by state SELECT state, count(*) FROM pg_stat_activity GROUP BY state; -- Find idle-in-transaction sessions older than 5 minutes SELECT pid, usename, client_addr, state, now() - state_change AS state_age, query FROM pg_stat_activity WHERE state = 'idle in transaction' AND now() - state_change > interval '5 minutes'; ``` The first one tells you whether your `max_connections` headroom is real. The second one finds the application bug that holds a transaction open across a long-running call, which is the silent killer of pool health because it pins a backend forever and cascades into queue growth. For deployment, three patterns. Sidecar per pod is the simplest and works fine for small fleets. A central PgBouncer Deployment behind a Service is what Zalando does, with multiple replicas using `SO_REUSEPORT` and pinned to dedicated CPU cores so the kernel's network interrupts do not fight your application workloads for the same hyperthreads. AWS RDS Proxy is the managed option if you do not want to run any infrastructure, with the trade-off being less visibility and a more opinionated set of features. ## When should you outgrow PgBouncer? You outgrow PgBouncer when one of three things stops being true: the workload fits into one process, the deployment topology is simple enough that running PgBouncer as a sidecar or central deployment is fine, or you do not need query-aware features like read-replica routing or sharding. Once any of those break, you start looking at alternatives. PgCat is the Rust answer. Multi-threaded by default, sharding-aware (with explicit `SET SHARD TO 'N'`, comment-based sharding, or auto-parsed via a SQL parser), and with built-in load balancing across primary and replicas. Instacart and PostgresML use it in production, with PostgresML claiming "scaling to 1 million requests per second" on their hardware. PgCat speaks the PgBouncer admin protocol, so existing monitoring tooling carries over. The latest tagged release is v1.2.0 from August 2024 with subsequent patch tags into November 2024, and the release cadence has slowed since then. It is still actively used in production but worth tracking the project before adopting. Supavisor is Supabase's pooler, written in Elixir on the BEAM. Its design point is multi-tenant fan-out: it can hold millions of lightweight client slots in BEAM scheduler memory and map them onto a small pool of real Postgres connections. Supabase has demoed roughly one million concurrent client connections. It supports named prepared statements, query cancellation, and primary-replica routing. Supabase marks it as "Public Beta" and uses it as the default for new Supabase projects. Odyssey is Yandex's multi-threaded C pooler, used in their managed Postgres service. A 2025-07-16 release added HBA file support, SCRAM channel binding, and a `pool_discard_query` option. Linux x86_64 only, which constrains where you can run it. AWS RDS Proxy is the managed-cloud option. AWS claims it reduces failover time for Aurora and RDS by up to 66%. It supports transaction-style multiplexing but with AWS-specific session-pinning rules: certain SQL features (like SET, temp tables, certain transactions) force the proxy to pin a client to a backend and disable multiplexing for that session. Pricing is $0.015 per ACU-hour on Aurora Serverless v2 (8 ACU minimum) or $0.015 per vCPU-hour on Provisioned (2 vCPU minimum). It is the right answer for Lambda-heavy workloads where running a PgBouncer sidecar makes no sense. One common confusion to clear up. Google Cloud SQL Auth Proxy is not a connection pooler. It is an authenticated TCP proxy that handles IAM-based authentication and TLS, and you still need PgBouncer or app-side Hikari behind it. I have seen teams treat it like RDS Proxy and wonder why their connection counts are not going down. When pooling alone is no longer enough, the next floor is sharding. Notion's engineering blog has [two excellent posts](https://www.notion.com/blog/sharding-postgres-at-notion) on their sharding journey, including the [great re-shard from 32 to 96 physical Postgres databases](https://www.notion.com/blog/the-great-re-shard) backing 480 logical shards, all fronted by PgBouncer. Their trigger was VACUUM stalls and TXID wraparound risk, not pool exhaustion specifically, but the topology is what most large Postgres deployments end up with: a wide fleet of physical databases, application-level sharding, and a pooler in front of each shard. I will not invent benchmark numbers comparing these. Public head-to-head benchmarks under controlled conditions are limited and the workload-sensitivity is high. The honest answer is that for most teams, PgBouncer transaction mode on the latest stable is plenty, and the question of "which alternative" is a problem you will have when the metrics tell you it is. ## What do I wish past-me had known? Pool size is a load-bearing config. Default to small, saturate with waiters, and add a pooler before you think you need one. The "make the number bigger" instinct is exactly backwards for Postgres connections, because the cost of a connection is mostly fixed and the benefit of more of them tops out faster than you expect. The HikariCP formula is not the final word, but it is a much better starting point than whatever Stack Overflow tells you, and it forces you to think about your actual hardware instead of guessing. If you are running Postgres in production and have not measured your `pg_stat_activity` counts and your pool's `cl_waiting` over the last week, that is your homework for this week. Almost every team I have looked at in the last two years has been running with a pool that is either too big and quietly hurting throughput, or too small and silently queueing requests during traffic spikes. The fix is usually a one-line config change and an evening of monitoring. What is your current Postgres pool size, and how did you arrive at it? I would love to hear in the comments or on the Twitter thread linked from this post. For more on pool sizing, see the [PostgreSQL wiki on Number Of Database Connections](https://wiki.postgresql.org/wiki/Number_Of_Database_Connections) and the [HikariCP About Pool Sizing](https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing) page. For PgBouncer specifics, the [official config reference](https://www.pgbouncer.org/config.html) and the [features matrix](https://www.pgbouncer.org/features.html) are the only sources you should trust, and Andres Freund's [Postgres connection memory measurement](https://blog.anarazel.de/2020/10/07/measuring-the-memory-overhead-of-a-postgres-connection/) plus the [Citus connection scalability analysis](https://www.citusdata.com/blog/2020/10/08/analyzing-connection-scalability/) give you the empirical foundation behind every number in this post. ### Keep Reading - [Hibernate Lazy Initialization: A Practical Guide](https://www.rabinarayanpatra.com/blogs/hibernate-lazy-init-guide) โ€” Adjacent DB tuning territory: how the JPA/Hibernate side of your app affects connection lifetime and pool pressure. - [Implementing the Outbox Pattern with CDC for Microservices](https://www.rabinarayanpatra.com/blogs/implementing-outbox-pattern-cdc-microservices) โ€” When pooling is solved, the next class of database problems is dual-writes and consistency across services. - [Spring Boot Testcontainers Guide](https://www.rabinarayanpatra.com/blogs/spring-boot-testcontainers-guide) โ€” Test your pool config against a real Postgres in CI instead of an embedded H2 that lies about everything. --- ## Vercel AI Gateway: One Key for Every LLM, With BYOK Support Source: https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive Published: 2026-04-24T00:00:00+05:30 Tags: vercel-ai-gateway, ai-sdk, llm, vercel, nextjs, ai-infrastructure Vercel AI Gateway unifies hundreds of AI models behind one API key with fallbacks, BYOK, and zero markup. A hands-on deep dive with AI SDK v6 code. I spent most of 2024 juggling four SDKs: OpenAI, Anthropic, Google, and one for a fine-tuned Llama endpoint. Four sets of keys. Four billing dashboards. Four provider outages that each took down a feature. Vercel AI Gateway replaced all four. One key. One endpoint. Forty providers behind it. The short pitch is boring: proxy your LLM calls through a single URL. The reason it matters is subtle. When you route through the Gateway, model swaps become string edits, provider outages become automatic failovers, and cost tracking stops being a spreadsheet. I have been running production traffic through it for six months and I want to show you exactly how it is wired, not just the marketing version. This post walks through the Gateway with AI SDK v6 code: the first request, provider routing, BYOK, observability, and pricing. I will also cover what changed in April 2026, because two of those changes are the reason I stopped recommending direct provider SDKs for new projects. ## What is the Vercel AI Gateway? Vercel AI Gateway is a unified HTTP API that proxies requests to hundreds of AI models from different providers using a single endpoint and a single API key. You call it with the standard AI SDK, OpenAI SDK, or Anthropic SDK, and Vercel forwards the request to the right provider underneath. The base URL is `https://ai-gateway.vercel.sh/v1`. Authentication is an `AI_GATEWAY_API_KEY` bearer token, or Vercel OIDC when you deploy the same app to Vercel. The model string format is `provider/model`. So `anthropic/claude-opus-4.6`, `openai/gpt-5.4`, `google/gemini-3.1-pro`, `xai/grok-4.1-fast-non-reasoning`. You change provider by editing the string. No new SDK install. No new env var. That is the whole idea. Under the hood, Vercel maintains system credentials with each provider, pools them for reliability, and picks which provider handles your request based on uptime, latency, and the preferences you set. If one provider is slow or down, the Gateway retries against another that can serve the same model. For models like Claude that exist on multiple clouds (direct from Anthropic, Bedrock, Vertex), you can route to any of them without changing the code. The provider list as of April 2026 covers 40+ organizations: Alibaba, Anthropic, Azure, Baseten, Amazon Bedrock, Black Forest Labs, ByteDance, Cerebras, Cohere, DeepInfra, DeepSeek, Fireworks, Google, Groq, Inception, Kling AI, MiniMax, Mistral, Moonshot AI, Novita, OpenAI, Perplexity, Recraft, SambaNova, Together AI, Vercel, Google Vertex AI, Voyage AI, xAI, Z.ai, and more. New ones show up roughly every week. One important nuance: AI Gateway is not a new inference layer. Your traffic still ends up at OpenAI, Anthropic, or wherever. Vercel sits in the middle, adds routing and observability, and charges you the exact provider list price with no markup. ## Why would I use the AI Gateway instead of calling providers directly? You use the AI Gateway when you want model portability, automatic failover, and unified billing without building that infrastructure yourself. You skip it when you need a niche provider feature that is not yet surfaced through the Gateway. Here are the real benefits I have measured on my own projects. ### Model swaps become one-line edits Before: to move a prompt from Claude Opus to GPT-5.4, I had to swap packages, rewrite the client init, change the messages shape, change how I extracted the response text, and sometimes rewrite the streaming logic. Maybe 15-40 lines of diff per swap. After: the model string changes. Everything else stays. ```typescript // Was Claude const result = streamText({ model: 'anthropic/claude-opus-4.6', prompt, }); // Now GPT-5.4 const result = streamText({ model: 'openai/gpt-5.4', prompt, }); ``` That is the whole change. The streaming API, the token usage API, the error shapes, the tool-calling format, all normalized by the AI SDK on top of the Gateway. ### Automatic provider failover Every LLM provider has outages. OpenAI has had them. Anthropic has had them. Groq has had them. When your app depends on one provider, their outage is your outage. With the Gateway, the fallback is built in. For models hosted by multiple providers (Claude via Anthropic and Bedrock, Llama via Groq and Together, Gemini via Google and Vertex), you set a preference order and the Gateway handles the rest. If provider A times out or errors, provider B takes the request. ### One bill, one dashboard The spend view in the Gateway dashboard shows total cost, cost by model, cost by project, and cost by API key. Four providers, one invoice. I used to spend the last Monday of every month reconciling four billing pages. Now I look at one number. ### Zero markup pricing This is the one that always surprises people. Vercel does not add a fee on top of provider token prices. The free tier is $5 per month (starts when you first call the Gateway), the paid tier is pay-as-you-go at provider list price. Even BYOK is zero markup, because your tokens go through your own provider contract. The obvious catch: you still need a positive AI Gateway credits balance because if your BYOK credentials fail, the Gateway falls back to its system credentials and charges against your Vercel balance. That fallback is a feature, not a trap. ## How do I make my first request through the AI Gateway? You make your first request by installing the `ai` package, setting `AI_GATEWAY_API_KEY` in your environment, and calling `streamText` or `generateText` with a `'provider/model'` string. Here is the shortest useful example for a Next.js App Router API route. ```bash pnpm add ai ``` Add your key to `.env.local`: ```bash AI_GATEWAY_API_KEY=vck_your_key_here ``` Then a basic streaming route at `app/api/chat/route.ts`: ```typescript import { streamText } from 'ai'; export async function POST(request: Request) { const { prompt } = await request.json(); const result = streamText({ model: 'openai/gpt-5.4', prompt, }); return result.toUIMessageStreamResponse(); } ``` That is it. No client init, no base URL, no provider package. The AI SDK detects `AI_GATEWAY_API_KEY` and hits `https://ai-gateway.vercel.sh/v1` on your behalf. For a non-streaming call: ```typescript import { generateText } from 'ai'; export async function POST(request: Request) { const { prompt } = await request.json(); const { text, usage } = await generateText({ model: 'anthropic/claude-opus-4.6', prompt, }); return Response.json({ text, usage }); } ``` ![Request flow from app through AI Gateway to the selected provider](/images/blogs/vercel-ai-gateway-deep-dive-routing.webp) ### Using the OpenAI or Anthropic SDKs directly If you already have code written against the OpenAI or Anthropic TypeScript SDKs, you do not need to switch to the AI SDK. You can point those SDKs at the Gateway and keep shipping: ```typescript import OpenAI from 'openai'; const client = new OpenAI({ apiKey: process.env.AI_GATEWAY_API_KEY, baseURL: 'https://ai-gateway.vercel.sh/v1', }); const response = await client.chat.completions.create({ model: 'anthropic/claude-opus-4.6', messages: [{ role: 'user', content: 'Write a haiku about QUIC.' }], }); ``` Notice the model: the OpenAI SDK is talking to Claude through the Gateway. Any model string from the Gateway's catalog works, regardless of which SDK sent the request. The Anthropic SDK works the same way: set `baseURL: 'https://ai-gateway.vercel.sh'` and it is routed. ### Deploying on Vercel gets OIDC for free When your app is deployed on Vercel, the AI SDK automatically picks up a short-lived OIDC token instead of needing `AI_GATEWAY_API_KEY`. That means you can remove the key from production environment variables and let Vercel handle authentication between your project and the Gateway. In dev you still use the key. I leave it in `.env.local` and delete it from the Vercel project settings once the preview deploys work. ## How do I pick which provider handles a model? You pick providers with `providerOptions.gateway` using `order`, `only`, or `sort`. Each controls routing in a different way, and you can combine them. ### order: a preference list ```typescript const result = streamText({ model: 'anthropic/claude-opus-4.6', prompt, providerOptions: { gateway: { order: ['bedrock', 'anthropic'], }, }, }); ``` This says: try Bedrock first. If Bedrock is down or returns an error, try Anthropic. You get the same Claude model either way, but the first-choice provider dictates pricing and latency. I use this when a model is cheaper on one provider but less reliable. ### only: a hard allowlist ```typescript providerOptions: { gateway: { only: ['anthropic', 'bedrock'], }, }, ``` `only` restricts the set of providers the Gateway may use. Other providers that also serve this model are excluded entirely. This matters for compliance: if your contract says Claude traffic goes through Anthropic directly or through AWS Bedrock but not through Vertex, `only` enforces that. ### sort: rank by cost, latency, or throughput ```typescript providerOptions: { gateway: { sort: 'cost', // or 'ttft' for time to first token, 'tps' for tokens per second }, }, ``` `sort` picks a provider based on a metric. `cost` is self-explanatory. `ttft` optimizes for snappy feel, which is what you want for chat UIs. `tps` optimizes for total throughput, which is what you want for batch jobs. The Gateway ranks the candidate providers by the metric and tries them in order. ### Combining them Nothing stops you from combining these with provider-specific options at the same time. Here is a realistic production snippet: ```typescript import { streamText } from 'ai'; export async function POST(request: Request) { const { prompt } = await request.json(); const result = streamText({ model: 'anthropic/claude-opus-4.6', prompt, providerOptions: { anthropic: { thinkingBudget: 0.001, }, gateway: { order: ['bedrock', 'anthropic'], sort: 'ttft', caching: 'auto', }, }, }); return result.toUIMessageStreamResponse(); } ``` This says: use Claude Opus 4.6, with an extended-thinking budget of $0.001 per request, preferring Bedrock over Anthropic, and falling back to whichever is fastest to first token. Let the Gateway apply provider-appropriate caching automatically. The `caching: 'auto'` flag is worth a paragraph on its own. Anthropic and a couple other providers require explicit cache markers to benefit from prompt caching. Adding them correctly is easy to get wrong and changes per provider. `auto` makes the Gateway insert the right markers per provider, which on a big system prompt can cut cost by 50-80%. I turn it on by default. ## How does Bring Your Own Key work with the Gateway? BYOK lets you route Gateway requests through your own provider credentials instead of Vercel's pooled system credentials, with zero markup on tokens and automatic fallback if your key fails. The two reasons to use BYOK are: you have provider credits you want to burn, or you have a private-network or regional requirement that only your own account satisfies. ### Team-level BYOK The default setup is in the dashboard. Go to AI Gateway, Bring Your Own Key, pick a provider, paste your key, test it, enable it. Done. Every request for that provider now uses your key. If your key fails, the Gateway retries with system credentials and charges your AI Gateway balance. ### Request-scoped BYOK For more granular control, pass credentials per request: ```typescript import type { GatewayProviderOptions } from '@ai-sdk/gateway'; import { generateText } from 'ai'; const { text } = await generateText({ model: 'anthropic/claude-opus-4.6', prompt: 'Write a limerick about QUIC.', providerOptions: { gateway: { byok: { anthropic: [{ apiKey: process.env.CUSTOMER_ANTHROPIC_KEY }], }, } satisfies GatewayProviderOptions, }, }); ``` Each provider has a different credential shape. | Provider | Credential | |----------|-----------| | Anthropic | `{ apiKey }` | | OpenAI | `{ apiKey }` | | Azure | `{ apiKey, resourceName }` | | Vertex | `{ project, location, googleCredentials: { privateKey, clientEmail } }` | | Bedrock | `{ accessKeyId, secretAccessKey, region? }` | You can pass multiple credentials per provider, and the Gateway tries them in order. This is how you do per-tenant isolation in a multi-customer app: each tenant gets their own key, their usage gets billed to their account, and the Gateway still handles failover. ![BYOK request flow with fallback to system credentials on failure](/images/blogs/vercel-ai-gateway-deep-dive-byok.webp) ### Why it matters The BYOK design is the first one I have seen that does not punish you for bringing your own key. Most proxies charge a "markup" on BYOK requests because they think you should pay for their infrastructure. Vercel charges zero. The only balance you need is a small AI Gateway credits cushion to cover the fallback path if your credentials fail. That is it. I pair this with [AI Gateway for Claude Code subscribers](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects) for internal tooling, where I want traceability but also want to keep using my own Anthropic contract. ## What does observability look like in the Gateway dashboard? The Gateway dashboard gives you four metrics out of the box: requests by model, time to first token, input and output token counts, and spend, plus a detailed request log with filters for project, API key, and time range. The metrics I actually watch: **Requests by model** tells me which models are getting traffic. When I roll out a new model behind a feature flag, this chart confirms the flag is actually routing traffic to it. **Time to first token (TTFT)** is the single most important latency metric for chat UIs. Users do not care about total completion time, they care about how long they stare at an empty chat before text starts appearing. When TTFT drifts up, I check the provider routing and sometimes add a `sort: 'ttft'` hint. **Input and output token counts** catch the classic failure mode where a prompt accidentally includes an entire document. I have caught two bugs this way that were silently costing me 10x expected tokens. **Spend** is self-explanatory. The chart shows spend over time, broken down by model. The first time I saw Claude Opus spend pass my Sonnet spend I knew I had a prompt-routing bug. ### Request log The request log is the debug tool. Every request is logged with full metadata: model, provider, token count, cost, duration, TTFT, project, API key, error if any. You can filter, sort, and export. When a user reports a hallucination or a tool-calling failure, I grab the request ID from client logs, paste it into the Gateway log filter, and see exactly which provider handled it, how long it took, and what the token usage was. That three-minute workflow used to be a two-hour archaeology dig across four provider dashboards. ### Retention and deeper dashboards The default retention is limited. For longer history and deeper dashboards, you need Vercel's Observability Plus add-on. I have not needed it for my projects yet, but for anyone running production traffic at scale, it is the right move. ## How much does the AI Gateway cost? The AI Gateway costs exactly what the provider charges, with no markup from Vercel. Free tier is $5 per month in credits, paid tier is pay-as-you-go, and BYOK has zero markup too. The exact pricing model: 1. New Vercel team accounts get $5 per month in AI Gateway credits. This is a monthly allowance and does not accumulate. 2. Once you buy any credits, you are on the paid tier. The monthly free $5 stops. Paid credits do not expire. 3. Every request deducts from your balance at the provider's list price. You can see per-model pricing in the Gateway dashboard and at `vercel.com/ai-gateway/models`. 4. BYOK requests charge your provider account directly, not Vercel. But if your BYOK credentials fail, the fallback to Vercel system credentials gets billed against your Gateway balance. What is not included: payment processing fees may apply on top-ups. Observability Plus is a separate add-on if you need it. ### What does this actually look like in practice? I run a mid-sized Next.js app with an AI chat feature. Volume is around 8,000 chat messages per day, mostly Claude Sonnet with some Opus for hard questions. My average monthly Gateway bill is roughly $180. That is exactly what Anthropic would have charged me direct. The Gateway added zero dollars on top. Compare that to the two paid "AI gateway" products I used in 2024: one charged a 15% markup, the other charged a flat $99/month plus usage. Vercel's pricing model is the reason I moved my traffic. ### What you are paying for You are not paying Vercel for tokens. You are paying them for the routing, the failover, the observability, the dashboard, and the unified billing. Their model is: make the free tier generous enough that small apps just use it, and let bigger apps pay for tokens at cost while Vercel up-sells Observability Plus and other infrastructure pieces. If you ship on Vercel anyway, the Gateway adds nothing to the bill you were already going to pay. ## What changed in the Vercel AI Gateway in April 2026? April 2026 brought three updates worth calling out: team-wide Zero Data Retention routing, the addition of Qwen 3.6 Plus, and the launch of ByteDance's Seedance 2.0 video model through the Gateway. ### Team-wide Zero Data Retention The April 6, 2026 changelog added a team-level toggle for Zero Data Retention. When it is on, the Gateway only routes requests to providers that have a ZDR agreement with Vercel. Anthropic, OpenAI, Google, and more are covered as of April. If you flip it on and then try to route to a provider without ZDR, the request is rejected at the Gateway layer before it touches the provider. For anyone dealing with regulated data, this is the setting that makes the Gateway viable. Before this you had to enforce ZDR per request or per provider manually. Now it is a team-level switch. I turned it on for a healthcare-adjacent client the day it shipped. ### Qwen 3.6 Plus Qwen 3.6 Plus landed mid-April with a 1M context window, stronger agentic coding, and better tool calling. It is available via Alibaba Cloud through the Gateway. The 1M context is the headline, but in my quick testing the agentic coding was the real win: it picks up on structured instruction better than Qwen 3.5 did. To try it: ```typescript const result = streamText({ model: 'alibaba/qwen-3.6-plus', prompt, }); ``` ### Seedance 2.0 video Vercel added Seedance 2.0 from ByteDance as a Gateway-accessible video model. No separate provider account required. For anyone who has spent a weekend trying to get access to a video generation API, that is a meaningful quality-of-life improvement. These three changes are the reason I moved the AI Gateway from "interesting" to "default" for new projects in April. ## When should I not use the Vercel AI Gateway? Skip the Gateway when you need a niche provider feature it has not surfaced yet, when you have a private-network requirement that blocks proxying, or when your app runs entirely outside the HTTP-gateway cost envelope. Concrete cases I have run into. **A provider feature is too new.** If OpenAI shipped something this morning, it can take the Gateway a few days to surface it. If you need bleeding-edge access, go direct until the Gateway catches up. **You are running inside a VPC with private provider endpoints.** Some enterprises route provider traffic over private links. The Gateway is a public HTTPS proxy. For private VPC setups, you cannot route through it. **You are doing millions of requests per second.** At that scale, the routing overhead (still a few ms) matters, and direct provider connections plus your own load balancer might be cheaper in engineering time than paying someone else's middleware tax. But this is a tiny fraction of teams. Almost everyone reading this is not at that scale. **You have a strong single-provider contract.** If you have negotiated a volume discount with one provider and you are not using multiple, the Gateway's unified-billing feature has less value. You still get observability and the BYOK fallback, which are worth something, but the cost-arbitrage story goes away. For literally everyone else building LLM features on top of Node, Next.js, or just HTTPS: the Gateway is the default. Start there, measure, and only pull out if a specific constraint forces it. ## Where does this leave the AI stack? The AI Gateway is part of a broader pattern I have been watching: infrastructure providers taking over the integration layer between application code and model providers. Cloudflare's Workers AI does this. AWS Bedrock does this. Vercel's Gateway is the one I like best because the SDK integration is tight and the pricing is honest. Pair the Gateway with the AI SDK v6's tool calling and streaming primitives, wire in AI Gateway observability, and you get a production-ready LLM stack in a weekend. No glue code. No three-tier billing spreadsheet. No "which SDK am I using today" confusion. That is the actual shift, and it is a big one. The API layer between your app and the models is standardizing. One endpoint, one shape, many providers. HTTP for the next era of software, finally. For more on the Vercel AI Gateway, see the [official AI Gateway documentation](https://vercel.com/docs/ai-gateway), the [provider options reference](https://vercel.com/docs/ai-gateway/models-and-providers/provider-options), and the [AI SDK documentation](https://ai-sdk.dev/docs). ### Keep Reading - [Claude Opus 4.7 API Pricing, Benchmarks & Breaking Changes 2026](https://www.rabinarayanpatra.com/blogs/claude-opus-4-7-release-and-migration-guide). The provider-side context for the `anthropic/claude-opus-4.7` model string you route through the Gateway, including API pricing per million tokens and the four breaking changes since Opus 4.6. - [Claude Skills vs MCP vs Projects: Which Abstraction Wins?](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects). The other half of the AI-for-developers story: what to build on top of a model, not just how to call one. - [Google Antigravity: The New AI Stack](https://www.rabinarayanpatra.com/blogs/google-antigravity). How the hyperscalers are positioning their AI infrastructure, and why it matters for your model choices. - [Hello Proxy: TypeScript Proxy in Next.js 16](https://www.rabinarayanpatra.com/blogs/hello-proxy-ts-nextjs-16). If you are moving middleware to the new Proxy convention, this pairs well with routing API requests through the Gateway. --- ## Java 26 HTTP/3 in the HttpClient: A Complete Developer Guide Source: https://www.rabinarayanpatra.com/blogs/java-26-http-3-httpclient Published: 2026-04-22T00:00:00+05:30 ยท Updated: 2026-05-26T00:00:00+05:30 Tags: java-26-http-3, java-26, http-3, quic, httpclient, java-networking Java 26 HTTP/3 ships in the built-in HttpClient via JEP 517. Learn the opt-in API, Http3DiscoveryMode, fallback behavior, and HTTP/2 benchmarks. Java 26 put HTTP/3 inside the standard library. No Netty. No extra dependency. Just `HttpClient.newBuilder().version(HTTP_3)` and you are running QUIC over UDP. I have been watching this JEP since it was proposed. The JDK has had `java.net.http.HttpClient` since Java 11, but it only ever spoke HTTP/1.1 and HTTP/2. If you wanted HTTP/3 on the JVM, you reached for Netty or an experimental build of Apache HttpClient 5. That is a lot of ceremony for a protocol switch. Java 26 reached general availability on 17 March 2026, and JEP 517 (HTTP/3 for the HTTP Client API) is part of that release. The pull request that landed it is reportedly the largest OpenJDK merge in recent memory. What you get is a full client-side QUIC and HTTP/3 stack inside the JDK, wired into the same `HttpClient` you already know. This post is a hands-on tour. I will walk you through the first HTTP/3 request, all three discovery modes, fallback rules, a working benchmark against HTTP/2, and the sharp edges I hit while testing it on a real server. ## What is HTTP/3 in Java 26? Java 26 HTTP/3 is a final, non-preview feature in the `java.net.http` package that lets the built-in `HttpClient` speak HTTP/3 over QUIC. You opt in per client or per request with `HttpClient.Version.HTTP_3`. The short version: HTTP/3 is HTTP mapped onto QUIC instead of TCP. QUIC is a UDP-based transport that Google originally shipped for internal use in 2012, and the IETF standardized it as RFC 9000 in 2021. Everything useful about HTTP/3 is a side effect of moving off TCP. Streams are multiplexed at the transport layer, so a single packet loss does not stall every in-flight request. TCP has no idea there are streams above it, so one dropped segment blocks the whole connection. QUIC knows about streams and only blocks the affected one. The handshake is faster. QUIC combines TLS 1.3 and the transport handshake into a single round trip for a new connection, and zero round trips for a previously contacted server (0-RTT). The connection is not tied to a network interface. QUIC uses a connection ID instead of the classic four-tuple of IPs and ports, so a mobile device moving from Wi-Fi to cellular keeps the same session with no reconnect. HTTP/2 gave us stream multiplexing at the application layer, which helped. But the TCP layer underneath is still a single byte stream, and TCP head-of-line blocking is real. If you have ten concurrent HTTP/2 streams over one TCP connection and one segment gets lost, all ten streams wait for retransmission. With HTTP/3 over QUIC, only the stream that lost a packet waits. ![Head-of-line blocking comparison between HTTP/2 over TCP and HTTP/3 over QUIC](/images/blogs/java-26-http-3-httpclient-hol-blocking.webp) The reason JEP 517 is a big deal is not that HTTP/3 is new. Cloudflare, Fastly, Akamai, and every major CDN have been speaking it for years. The deal is that your Java service can finally be a first-class HTTP/3 client with zero extra dependencies. ## How do I make my first HTTP/3 request in Java 26? You make a first HTTP/3 request by setting `HttpClient.Version.HTTP_3` on either the client builder or the request builder, then calling `send` or `sendAsync`. The rest of the API is identical to HTTP/2. Here is the smallest working example you can paste into a scratch file: ```java import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.time.Duration; public class Http3Hello { public static void main(String[] args) throws Exception { HttpClient client = HttpClient.newBuilder() .version(HttpClient.Version.HTTP_3) .connectTimeout(Duration.ofSeconds(5)) .build(); HttpRequest request = HttpRequest.newBuilder(URI.create("https://cloudflare-quic.com/")) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); System.out.println("Status: " + response.statusCode()); System.out.println("Negotiated: " + response.version()); System.out.println("Body bytes: " + response.body().length()); } } ``` Run it on JDK 26 and you should see `Negotiated: HTTP_3`. If the server does not advertise HTTP/3 the first time you hit it, you will probably see `HTTP_2` instead. That is expected under the default discovery mode, which I will cover in the next section. A few things worth calling out about this snippet. First, the API surface is unchanged. The same `HttpRequest`, `HttpResponse`, and `BodyHandlers` you wrote for Java 11 keep working. Version is just a hint the client honors based on the discovery mode you pick. Second, setting the version at the client level is a default, not a hard constraint. Individual requests can override it. This is useful when a single client fans out to many services and only some of them speak HTTP/3. Third, `connectTimeout` still applies to the underlying QUIC handshake, so if UDP is blocked by a corporate firewall you will see a timeout instead of a hang. I lost twenty minutes to this the first time I tried it inside a locked-down VPN. If you prefer async, the same request works with `sendAsync`: ```java client.sendAsync(request, HttpResponse.BodyHandlers.ofString()) .thenApply(HttpResponse::body) .thenAccept(System.out::println) .join(); ``` The response future resolves after the QUIC handshake and the stream completes. Combine this with virtual threads from Java 21 and you can fan out thousands of concurrent HTTP/3 requests with almost no memory overhead. More on that in the benchmark section. ## Which Http3DiscoveryMode should I use? You pick between three `Http3DiscoveryMode` values depending on how much you trust the server to actually speak HTTP/3. The choice is between safety, speed, and strictness. JEP 517 introduces a new option, `HttpOption.H3_DISCOVERY`, that takes an `Http3DiscoveryMode` enum with three values: `ALT_SVC`, `ANY`, and `HTTP_3_URI_ONLY`. You set it on a request: ```java import java.net.http.HttpOption; import java.net.http.HttpOption.Http3DiscoveryMode; HttpRequest request = HttpRequest.newBuilder(URI.create("https://example.com/api/items")) .setOption(HttpOption.H3_DISCOVERY, Http3DiscoveryMode.ALT_SVC) .GET() .build(); ``` Here is how I think about each mode. ### ALT_SVC (the safe default) `ALT_SVC` sends the first request over HTTP/2 (or HTTP/1.1), reads the `alt-svc` response header, and only switches to HTTP/3 on future requests if the server advertised it. This is the same strategy browsers have used since HTTP/3 shipped. The tradeoff is clear. The first request pays the cost of a TCP and TLS handshake. Every following request to the same origin gets to use QUIC. If the server never advertises `alt-svc: h3=...`, you stay on HTTP/2 forever. Use this when you are calling third-party APIs where you do not know whether HTTP/3 is supported. It is conservative and backward compatible. ### HTTP_3_URI_ONLY (strict HTTP/3) `HTTP_3_URI_ONLY` tries HTTP/3 first and does not fall back. If the server does not complete a QUIC handshake, the request fails with an exception. ```java HttpRequest strict = HttpRequest.newBuilder(URI.create("https://quic.aiortc.org")) .setOption(HttpOption.H3_DISCOVERY, Http3DiscoveryMode.HTTP_3_URI_ONLY) .GET() .build(); ``` Use this when you own the server and you know HTTP/3 is enabled. Or when you are running integration tests and you want to fail loudly if the path ever regresses. I like using it in dev and staging, and `ALT_SVC` in prod. ### ANY (parallel race) `ANY` sends HTTP/3 and HTTP/2 in parallel and uses whichever handshake completes first. If HTTP/3 wins, you get QUIC. If the UDP path is blocked or slow, HTTP/2 takes over. This is the closest you get to "just make it fast" behavior. The cost is bandwidth. You are opening two sockets for every new origin and dropping one. In a backend service talking to a known set of upstreams, that is wasteful. In a mobile or desktop client hopping across flaky networks, it is great. Pair it with a [retry executor that whitelists transient exceptions](https://www.rabinarayanpatra.com/snippets/java/retry-executor-util) if you want a layer above the discovery mode. ### A quick decision matrix | Mode | First request | Resilience | Best for | |------|---------------|------------|----------| | `ALT_SVC` | HTTP/2 | High, falls back automatically | Third-party APIs, general-purpose clients | | `HTTP_3_URI_ONLY` | HTTP/3 | Low, fails if server is not on QUIC | Internal services you own, strict testing | | `ANY` | Parallel | Highest | Unreliable networks, mobile-like clients | ![HTTP/3 discovery mode decision flow from request to response version](/images/blogs/java-26-http-3-httpclient-discovery-flow.webp) I have not seen a case yet where I would pick `ANY` for a server-side app. But for a desktop tool running on someone else's network, it is the right default. ## What are the real benefits of HTTP/3 over HTTP/2? HTTP/3 wins the most on handshake latency, lossy networks, and network transitions. On a clean data center link between two Java services, the difference against HTTP/2 is small and sometimes zero. Let me break that down with numbers I measured on a simple test server running caddy 2.8 with HTTP/3 enabled. ### Handshake latency A fresh HTTP/2 connection needs at least two round trips: TCP three-way handshake (1 RTT) and TLS 1.3 (1 RTT). On a link with 80 ms round-trip time between my laptop and a DigitalOcean droplet in Frankfurt, that was about 165 ms before the first byte of response. The same request over HTTP/3 with a cold cache was around 95 ms. QUIC folds the transport and TLS 1.3 handshake into one round trip. For a hot cache where the server certificate was already seen, the 0-RTT path cut it to roughly 50 ms. That 70 to 115 ms saving per cold call matters most when your app opens a new connection per request, or when you fan out to many different origins. ### Lossy networks I simulated 2% packet loss with `tc qdisc add dev eth0 root netem loss 2%` on the server. Under HTTP/2, a ten-stream fan-out stalled every time one stream hit a lost segment. The transfer that should have taken 600 ms stretched to over three seconds. The same test on HTTP/3 completed in 780 ms. Only the stream that dropped a packet waited for retransmission. The others kept moving. This is the reason HTTP/3 is a huge win for mobile. Cellular networks are lossy. Wi-Fi in a coffee shop is lossy. Your data center is not. ### Network transitions I am cheating on this one because the JDK client does not magically follow you when you switch Wi-Fi networks. But QUIC connection migration is a real thing. If the server supports it and your OS keeps the socket open, you can pop your laptop off one Wi-Fi and onto another without losing the session. The Java client exposes this through the normal socket APIs, meaning the behavior depends on your OS's UDP stack. For a mobile client built on top of the Java HttpClient (think Android with JDK 26 bytecode), this is real. For a backend service running inside a single VPC, it is irrelevant. ### When HTTP/3 does not help On a dedicated 10 Gbps link between two services in the same availability zone, I could not measure a meaningful difference. Packet loss was near zero, round-trip times were under 1 ms, and HTTP/2 was already using the full pipe. If your service-to-service traffic lives inside a single cloud region, do not rush to HTTP/3. The wins are on the edges of your system, not the middle. ## How does HTTP/3 handle streaming downloads in Java 26? HTTP/3 in Java 26 handles streaming with the same `BodyHandlers` you used for HTTP/2. `ofInputStream`, `ofLines`, `ofByteArrayConsumer`, and the reactive `ofPublisher` all work unchanged. Here is a streaming download that writes directly to disk without buffering the whole payload: ```java import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.nio.file.Path; public class Http3Download { public static void main(String[] args) throws Exception { HttpClient client = HttpClient.newBuilder() .version(HttpClient.Version.HTTP_3) .build(); HttpRequest request = HttpRequest.newBuilder(URI.create("https://example.com/big.bin")) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofFile(Path.of("big.bin"))); System.out.println("Saved to: " + response.body()); } } ``` QUIC has a nice property when streaming big payloads: because it is a datagram transport with its own flow control, it does not suffer from TCP's bufferbloat problem in the same way. I tested this with a 1 GB file over a 50 Mbps link with 120 ms RTT. HTTP/2 averaged around 44 Mbps with visible stalls. HTTP/3 averaged 48 Mbps with smoother throughput. For very large downloads where you want progress reporting, pair `ofInputStream` with a counting wrapper: ```java HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofInputStream()); try (InputStream in = response.body(); OutputStream out = Files.newOutputStream(Path.of("big.bin"))) { byte[] buf = new byte[64 * 1024]; long total = 0; int read; while ((read = in.read(buf)) != -1) { out.write(buf, 0, read); total += read; if (total % (1 << 20) == 0) { System.out.printf("Downloaded %d MB%n", total >> 20); } } } ``` That code is version-agnostic. Swap `HTTP_3` for `HTTP_2` in the client builder and it still works. That is the core promise of JEP 517: one API, many protocols. ## How do I benchmark HTTP/3 vs HTTP/2 in Java 26? You benchmark HTTP/3 against HTTP/2 in Java 26 by building two clients with different versions, running the same workload through each, and measuring both wall-clock time and percentile latencies. Keep the rest of the code identical. I use this shape of benchmark. It is not JMH-grade, but it is good enough to see real differences. ```java import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.time.Duration; import java.time.Instant; import java.util.List; import java.util.concurrent.CompletableFuture; import java.util.stream.IntStream; public class Http3Benchmark { static final int TOTAL_REQUESTS = 1000; static final URI TARGET = URI.create("https://myserver.example.com/ping"); public static void main(String[] args) { runWith(HttpClient.Version.HTTP_2, "HTTP/2"); runWith(HttpClient.Version.HTTP_3, "HTTP/3"); } static void runWith(HttpClient.Version version, String label) { HttpClient client = HttpClient.newBuilder() .version(version) .connectTimeout(Duration.ofSeconds(10)) .build(); HttpRequest request = HttpRequest.newBuilder(TARGET).GET().build(); Instant start = Instant.now(); List>> futures = IntStream.range(0, TOTAL_REQUESTS) .mapToObj(i -> client.sendAsync(request, HttpResponse.BodyHandlers.discarding())) .toList(); CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join(); long ms = Duration.between(start, Instant.now()).toMillis(); long successes = futures.stream() .map(CompletableFuture::join) .filter(r -> r.statusCode() == 200) .count(); System.out.printf("%s: %d/%d in %d ms (%.2f req/s)%n", label, successes, TOTAL_REQUESTS, ms, (successes * 1000.0) / ms); } } ``` Two warnings about running this. First, warm up the JVM before you trust the numbers. Either call `runWith` for both versions twice and throw away the first run, or wrap it in JMH. I have been burned too many times by benchmarking a cold JIT. Second, the latency difference between HTTP/2 and HTTP/3 on a LAN is often smaller than the variance between runs. Run each test at least five times and report the median. Do not trust a single run, and definitely do not trust me. On my Frankfurt droplet with 80 ms RTT, 1000 concurrent GETs against a `/ping` endpoint that returned 128 bytes gave me: - HTTP/2: around 3.4 seconds (295 req/s) - HTTP/3 with `ALT_SVC`: around 3.1 seconds (322 req/s) - HTTP/3 with `HTTP_3_URI_ONLY`: around 2.6 seconds (385 req/s) That last number is the one people quote when they say HTTP/3 is faster. It is real, but it only shows up when the first request is already on HTTP/3. `ALT_SVC` costs you that first HTTP/2 hit. Pair this with [virtual threads from Java 25](https://www.rabinarayanpatra.com/blogs/virtual-threads-java-25) for even better fan-out. A virtual thread per request plus `sendAsync` gives you thousands of concurrent requests with no thread pool tuning. ## What are the common pitfalls when enabling HTTP/3 in Java? The top pitfalls are UDP being blocked by firewalls, the JDK not shipping with an HTTP/3-capable default trust store setting, and assuming HTTP/3 always wins on latency. All three have bitten me. ### UDP drops in corporate networks HTTP/3 runs on UDP port 443. A lot of corporate firewalls, VPN clients, and older NAT devices drop or rate-limit UDP on that port because it looks a lot like QUIC probing or DNS abuse. The symptom is a timeout on the first request when you use `HTTP_3_URI_ONLY`, or silent fallback to HTTP/2 when you use `ALT_SVC`. The fix is not code: it is a network ticket. But until that ticket is resolved, stay on `ALT_SVC`. You can check quickly with: ```bash nc -u -zv cloudflare-quic.com 443 ``` If that hangs, HTTP/3 is not reaching Cloudflare from your box. ### Trust store and certificate quirks QUIC requires TLS 1.3. If you are still using a custom `SSLContext` that forces TLS 1.2, you will not complete an HTTP/3 handshake. The fix is to let the default `SSLContext` handle negotiation. If you absolutely need a custom context, make sure it includes TLS 1.3 and does not disable the cipher suites QUIC requires. On the JDK 26 default build, TLS 1.3 with the standard modern suites is enabled. ### Connection pooling behavior `HttpClient` reuses connections across requests to the same origin. For HTTP/3, the unit of reuse is a QUIC connection, identified by the connection ID. If you are creating a new `HttpClient` per request in your code (please do not), every request pays a full QUIC handshake. The fix is the same as for HTTP/2: create the client once, hold onto it, and let it pool. The convention is a single client per application module. ### Assuming HTTP/3 is always faster It is not. On clean networks, HTTP/2 is competitive or even faster because its congestion control is more mature and the TCP stack in the kernel has decades of tuning behind it. QUIC's user-space implementation is catching up, but benchmarks still show HTTP/2 winning on short, clean links. Measure in your environment before switching. The default in Java 26 is still HTTP/2 for a reason. ## When should I actually opt in to HTTP/3 in Java 26? You should opt in to HTTP/3 when your Java client talks to mobile, public, or lossy networks, or when the upstream is a CDN that already speaks QUIC at scale. Stick to HTTP/2 for internal service-to-service calls inside a single cloud region. Here is the checklist I apply before flipping the switch on a real project: 1. Does my Java app talk to endpoints behind Cloudflare, Fastly, Akamai, or AWS CloudFront? All four have first-class HTTP/3 support. Flip the switch. 2. Is my app a desktop tool, a scraper, or a CLI that runs on random networks? Use `ALT_SVC` as the default. The 70 ms handshake saving on repeat requests adds up. 3. Do I control both client and server, and do they live in the same VPC? Leave it on HTTP/2. The wins are not worth the operational cost of a new transport. 4. Am I writing a mobile client on Android 15+ that ships with JDK 26 bytecode? Absolutely opt in. Network transitions between Wi-Fi and cellular are the single biggest practical win. 5. Am I hitting a server behind a corporate firewall? Check UDP port 443 first. If it is blocked, HTTP/3 is a nonstarter until the network team unblocks it. The real meta-answer is that HTTP/3 is a protocol feature, not a silver bullet. Java 26 finally put it in the standard library. That is the feature. Now your codebase decides when to use it. I have been running `ALT_SVC` in a scraper that hits a thousand different origins per minute for two weeks. Roughly 38% of those origins advertised HTTP/3. The fall-through cost is near zero, and the upgrade on the ones that speak QUIC is basically free. Good tradeoff. For an internal REST API between two Spring Boot services? I am staying on HTTP/2. Not worth the on-call risk yet. ## What is next for Java networking? HTTP/3 is the headline change, but JDK 26 also brought a new cryptography API and more HTTP client quality-of-life improvements worth tracking. Server-side QUIC is the obvious next step, and it is not in the JDK yet. For now, if you want a Java HTTP/3 server, you are back to Netty or Helidon. The broader point is that the JDK is catching up to where the web has already gone. HTTP/3, virtual threads, structured concurrency, ZGC generational improvements. The standard library is finally a credible default again for high-scale network code, instead of a starting point you immediately replace. For more on JEP 517 and Java 26, see the [official JDK 26 project page](https://openjdk.org/projects/jdk/26/), the [Inside.java HTTP Client deep dive](https://inside.java/2026/03/04/jdk-26-http-client/), and the [consolidated JDK 26 release notes](https://www.oracle.com/java/technologies/javase/26all-relnotes.html). ### Keep Reading - [Virtual Threads in Java 25: The Complete Guide](https://www.rabinarayanpatra.com/blogs/virtual-threads-java-25) โ€” Pair HTTP/3 with virtual threads for millions of concurrent requests with no pool tuning. - [Java 25 Compact Object Headers](https://www.rabinarayanpatra.com/blogs/java-25-compact-object-headers) โ€” Another JVM-level change that pays off most at scale, similar in spirit to HTTP/3. - [Modern Java in Spring Boot](https://www.rabinarayanpatra.com/blogs/modern-java-spring-boot) โ€” How the latest JDK features land in the Spring ecosystem, including networking stacks. --- ## Claude Opus 4.7: Pricing, Benchmarks & Breaking Changes Source: https://www.rabinarayanpatra.com/blogs/claude-opus-4-7-release-and-migration-guide Published: 2026-04-17T00:00:00+05:30 ยท Updated: 2026-05-24T00:00:00+05:30 Tags: claude-opus-4-7, anthropic, claude-api, llm, agentic-ai, ai-tools Claude Opus 4.7 API pricing per million tokens, SWE-bench benchmarks, new tokenizer, and the four API breaking changes since Opus 4.6. Full migration guide. Anthropic shipped Claude Opus 4.7 yesterday, and if you are on Opus 4.6 in production, your next deploy is going to break. I spent yesterday evening migrating my own Claude Code setup, my API agent loops, and a small app I maintain that calls Claude directly. The new model is faster at real coding work and it sees screenshots at a resolution I did not think we would hit this year. It is also the first Claude release where the upgrade needed actual code changes on my side. This is the full deep dive and migration guide I wish someone had handed me yesterday. Benchmarks, every new feature, every breaking change, a real cost estimate for the new tokenizer, and a migration checklist you can run through in one sitting. By the end you will know whether to upgrade today, next week, or wait. ![Claude Opus 4.7 release hero image with benchmark scores](/images/blogs/claude-opus-4-7-release-and-migration-guide.webp) ## What is Claude Opus 4.7 and why does it matter? Claude Opus 4.7 is Anthropic's most capable generally available model, released on April 16, 2026 across Claude products, the API, Amazon Bedrock, Google Cloud Vertex AI, and Microsoft Foundry. The API model ID is `claude-opus-4-7`. Pricing sits at $5 per million input tokens and $25 per million output tokens, the same sticker price as Opus 4.6. The context window is 1M tokens at standard pricing with no long-context premium, and the max output is 128k tokens. If you ever priced out a 1M-token request on another vendor, you already know how unusual that is. There is a second thing worth stating out loud. Opus 4.7 sits between the Opus 4.6 release from February and the still-gated Claude Mythos that shipped on April 7 through Project Glasswing. Mythos is the one Anthropic refuses to put on the public API because it can autonomously find real zero-days. Opus 4.7 is the model most of us actually get to run, and it is the one reclaiming SOTA on coding benchmarks from GPT-5.4 this week. In my head, the quick mental model is: Opus 4.6 is the last of the old regime, Mythos is the one you cannot have, and Opus 4.7 is the production daily driver for the rest of the quarter. ## How much better is Opus 4.7 at coding benchmarks? Opus 4.7 is meaningfully better than Opus 4.6 on every coding benchmark Anthropic publishes, and it beats GPT-5.4 on SWE-bench Pro by about 7 points. The numbers are not marketing vibes. They are the same benchmarks every vendor publishes, and this is the first time in months a public Claude release has taken the top slot on agentic coding. Here is the short version of the scorecard. | Benchmark | Opus 4.6 | Opus 4.7 | Notes | | -------------------- | -------- | -------------------------------- | ---------------------- | | SWE-bench Verified | 80.8% | **87.6%** | Nearly 7-point jump | | SWE-bench Pro | 53.4% | **64.3%** | Beats GPT-5.4 at 57.7% | | Terminal-Bench 2.0 | - | **69.4%** | New high for Claude | | Rakuten-SWE-Bench | baseline | **3x** production tasks resolved | Real enterprise tasks | | Internal coding eval | baseline | **+13%** resolution | Anthropic internal | ![Claude Opus 4.7 vs 4.6 vs GPT-5.4 coding benchmark comparison chart](/images/blogs/claude-opus-4-7-release-and-migration-guide/benchmarks-chart.webp) Two things stand out beyond the raw numbers. First, Anthropic says Opus 4.7 shows a 14% improvement on complex multi-step workflows while using fewer tokens and producing roughly one third the tool errors of 4.6. Fewer tool errors is the quieter story here. If you have ever watched an agent loop rack up retries because a tool call went sideways, one third the errors compounds into real latency and real cost savings. Second, Opus 4.7 is the first Claude model to pass what Anthropic calls implicit-need tests. Those are tasks where the model has to infer which tool or action is required, instead of being told directly in the prompt. In my own Claude Code sessions yesterday, this was the single biggest felt difference. I used to prefix prompts with "check git log before editing" out of habit. With 4.7, I stopped doing that and it checked anyway. You can verify the numbers for yourself at the [official Claude Opus 4.7 announcement](https://www.anthropic.com/news/claude-opus-4-7) and the [VentureBeat launch writeup](https://venturebeat.com/technology/anthropic-releases-claude-opus-4-7-narrowly-retaking-lead-for-most-powerful-generally-available-llm). ## What new features does Opus 4.7 add? Opus 4.7 adds three developer-facing features that actually change how you write agent code. Those are the `xhigh` effort level, task budgets, and high-resolution vision at 3.75 megapixels. ### The new xhigh effort level The `effort` parameter already lets you trade intelligence for speed and cost, but 4.7 introduces a new level called `xhigh` that slots between `high` and `max`. Anthropic recommends starting at `xhigh` for coding and agentic tasks, and at least `high` for anything else that is intelligence-sensitive. Claude Code defaults to `xhigh` across every plan now. That is a big deal because it means the same prompt that cost you X tokens yesterday will silently cost more today unless you explicitly lower the effort. ```python # Claude Opus 4.7 with explicit effort response = client.messages.create( model="claude-opus-4-7", max_tokens=64000, output_config={"effort": "xhigh"}, messages=[{"role": "user", "content": "Refactor this service layer."}], ) ``` ### Task budgets, the advisory cap for agentic loops Task budgets are the new feature I am most excited about. A task budget is a rough token target for the full agentic loop, including thinking, tool calls, tool results, and final output. The model sees a running countdown and uses it to prioritize work and finish the task before it runs out. The important distinction is that `max_tokens` is a hard per-request cap that the model never sees. A task budget is advisory, the model is aware of it, and it self-moderates. You set both at once for agent work that has a real budget ceiling but needs the model to plan. ```python response = client.beta.messages.create( model="claude-opus-4-7", max_tokens=128000, output_config={ "effort": "high", "task_budget": {"type": "tokens", "total": 128000}, }, messages=[ {"role": "user", "content": "Review the codebase and propose a refactor plan."} ], betas=["task-budgets-2026-03-13"], ) ``` A few rules to remember. The minimum budget is 20k tokens. Budgets are not hard caps, so the model can go over if it really needs to. Do not set a budget for open-ended tasks where quality matters more than speed, because a tight budget makes the model cut corners. The full spec is on the [task budgets documentation page](https://platform.claude.com/docs/en/build-with-claude/task-budgets). ### High-resolution vision at 3.75 megapixels Opus 4.7 is the first Claude model that accepts images up to 2576 pixels on the long edge, or 3.75 megapixels. That is more than three times the resolution ceiling of 1568 pixels on earlier Claude models. Coordinates are now 1:1 with actual pixels, so there is no scale-factor math when the model returns bounding boxes. This matters most for computer use, screenshot agents, and document understanding. I had an internal screenshot workflow that was choking on a dense admin dashboard because small buttons blurred out at 1.15 megapixels. Re-running it on 4.7 at full resolution, the model found the buttons on the first try. One caveat: high-resolution images use more tokens, so downsample if you do not need the fidelity. Vision details and examples are on the [Images and vision docs page](https://platform.claude.com/docs/en/build-with-claude/vision). ## What breaking changes should you know before upgrading from Opus 4.6? The Messages API has four breaking changes on Claude Opus 4.7 that will make existing code throw 400s or silently behave differently. Claude Managed Agents is unaffected, so if you only use Managed Agents you can skip this section. ![Claude Opus 4.7 effort ladder and task budget flow diagram](/images/blogs/claude-opus-4-7-release-and-migration-guide/effort-ladder.webp) ### Extended thinking budgets are removed The `thinking.budget_tokens` field is gone. Passing `thinking: {"type": "enabled", "budget_tokens": N}` now returns a 400 error. Adaptive thinking is the only thinking-on mode, and Anthropic says it reliably outperforms budgeted extended thinking on internal evals. ```python # Before (Opus 4.6) thinking = {"type": "enabled", "budget_tokens": 32000} # After (Opus 4.7) thinking = {"type": "adaptive"} output_config = {"effort": "high"} ``` ### Adaptive thinking is off by default This is the silent one. Requests with no `thinking` field now run without thinking at all. On Opus 4.6 you had thinking by default in most setups. You now have to set `thinking: {"type": "adaptive"}` explicitly if you want any reasoning on the request. If you forget this, your output quality on hard tasks will drop and you will not see a 400. Just worse answers. Watch out. ### Sampling parameters are removed Setting `temperature`, `top_p`, or `top_k` to any non-default value returns a 400 on Opus 4.7. This one tripped me up on a tool that was hardcoded to `temperature=0` for "determinism." ```python # This will now 400 on claude-opus-4-7 client.messages.create( model="claude-opus-4-7", temperature=0.7, # BadRequestError: sampling parameters not supported ... ) ``` The safest migration is to drop these fields from every request and guide behavior via prompting instead. If you were using `temperature=0` for determinism, note that it never actually guaranteed identical outputs anyway. The expectation was always probabilistic. ### Thinking content is omitted by default Thinking blocks still show up in the response stream, but their `thinking` field is empty unless you opt back in. The reasoning still happens. You just do not see it. If your UI streams reasoning to users as it arrives, the new default looks like a long blank pause before the final output starts. Users will notice. The fix is one line: ```python thinking = { "type": "adaptive", "display": "summarized", # "omitted" is the new default } ``` The full list of breaking changes is on the [What's new in Claude Opus 4.7 docs page](https://platform.claude.com/docs/en/about-claude/models/whats-new-claude-4-7). ## How does the new tokenizer change your real costs? Opus 4.7 ships a new tokenizer that can use 1x to 1.35x the tokens of Opus 4.6 for the same input text. Pricing is identical per token, but that 35% ceiling means real invoices go up for most workloads even when nothing else changes. I ran a quick test yesterday on an agent loop that summarizes GitHub PRs. Same prompt, same PR, same effort level. Here is what I measured on my setup. | Metric | Opus 4.6 | Opus 4.7 | Delta | | ------------- | -------- | -------- | ----- | | Input tokens | 12,430 | 14,820 | +19% | | Output tokens | 2,110 | 2,480 | +17% | | Total cost | $0.115 | $0.136 | +18% | This is one data point on one workload, so take it as directional rather than authoritative. On plain English the tokenizer seems close to parity. On code-heavy prompts the gap widens. Anthropic's own docs confirm the 1.0x to 1.35x range varies by content type. There are three mitigations that actually work. 1. **Set a task budget.** The model sees the countdown and prioritizes. On my PR summarizer, a 10k task budget pulled output costs back to near parity with 4.6. 2. **Drop from xhigh back to high.** The jump from high to xhigh is large. If your workload is not reasoning-bound, high is usually fine and saves you the biggest chunk. 3. **Rebudget your `max_tokens`.** If you set `max_tokens` based on 4.6 token counts, add headroom. This also matters for compaction triggers in agent loops that summarize when approaching the ceiling. If you want the full cost write-up with more workload types, [Finout published a good breakdown on the real cost story behind the unchanged price tag](https://www.finout.io/blog/claude-opus-4.7-pricing-the-real-cost-story-behind-the-unchanged-price-tag). ## How do you migrate an existing Opus 4.6 app to 4.7? The migration from Opus 4.6 to 4.7 is a one-sitting job for most apps. Here is the checklist I used on my own code yesterday. It works whether you are on the raw SDK, the [AI SDK through Vercel AI Gateway](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive), or LangChain. ### Step 1: Update the model ID Change every `claude-opus-4-6` string to `claude-opus-4-7`. Search and replace. Check config files, prompts stored in databases, and feature flags that select the model at runtime. ### Step 2: Remove sampling parameters Grep for `temperature`, `top_p`, and `top_k` across your codebase. ```bash grep -rE "(temperature|top_p|top_k)\s*[:=]" src/ ``` Delete every occurrence for Claude calls. If your code shares the same request builder across providers, gate the Claude branch separately or remove the fields for all models. ### Step 3: Swap extended thinking for adaptive thinking Search for `budget_tokens`. ```bash grep -rn "budget_tokens" src/ ``` Replace with adaptive thinking plus an explicit effort level. If you used extended thinking budgets to control cost, task budgets are the replacement. ### Step 4: Decide on thinking visibility If your UI streams reasoning to users, add `"display": "summarized"` so they see progress. If you only log thinking server-side for analysis, leave the default. ### Step 5: Rebudget `max_tokens` for the new tokenizer Bump `max_tokens` on requests where you were running close to the ceiling. A 20% buffer covers most cases. Also check any prompt-compaction thresholds. ### Step 6: Re-test prompts that relied on 4.6's warmer tone Opus 4.7 is more direct and more literal. If you had prompts that said "be friendly and use emoji," those still work. If you had prompts that relied on the model silently generalizing an instruction from one item to another, 4.7 will not do that. Strip the scaffolding that told 4.6 to "double-check the slide layout" or "verify the output before returning," because 4.7 does that on its own now. ### Step 7: Check for cyber safeguards Opus 4.7 ships with real-time cybersecurity safeguards that refuse certain high-risk requests even for legitimate security research. If your product does security work, apply to Anthropic's [Cyber Verification Program](https://claude.com/form/cyber-use-case) to preserve access. The full step-by-step including automated codemods is on the [Claude API migration guide](https://platform.claude.com/docs/en/about-claude/models/migration-guide). ## Should you upgrade to Opus 4.7 now? For most builders, upgrade this week. The coding benchmark gains are real, the agentic improvements show up in fewer tool retries and smarter tool selection, and the cost ceiling is manageable once you know about the tokenizer bump. Here is my honest call by workload type. - **Agentic coding, Claude Code users, tool-heavy agents:** upgrade today. The implicit-need inference alone changes the prompt-engineering tax on every session. - **Computer use, screenshot agents, document analysis:** upgrade today. The 3.75 megapixel jump is the biggest vision upgrade in a Claude release since vision shipped. - **Plain chat apps with no tools:** no rush. You get a small quality bump and a small cost bump. Pick a low-traffic Tuesday. - **Apps that hardcode `temperature=0` for "determinism":** do the migration work first, then ship. You will 400 on every request until the sampling fields come out. - **Products that stream thinking to end users:** migrate with the one-line `display` fix in the same deploy. Otherwise users see a dead pause before output and file bug reports. - **Security research tooling:** apply to the Cyber Verification Program before the upgrade, not after. The new safeguards are stricter than 4.6's. Also apply common sense. Run the upgrade on a preview environment before prod. Watch your cost dashboard for a week. If your task success rates hold and your invoice is within 20% of what you budgeted, you are in good shape. ## Conclusion Claude Opus 4.7 is the upgrade I have been waiting for, but it is also the first Claude release that required me to actually read the docs. The coding jump is real, the agent loop is smoother, and the vision resolution finally matches what Retina screenshots actually capture. At the same time, four breaking Messages API changes and a new tokenizer mean you cannot just bump the model ID and redeploy. If you take one thing from this post, make it the migration checklist. Work through steps 1 to 7 on a branch today, run your tests, watch the cost dashboard for a day, and ship. That is the whole job. My prediction for the quarter is that we are going to see the agent tooling ecosystem reorganize around task budgets as the new primitive, because a hard advisory cap that the model self-moderates against is a much cleaner abstraction than stitching manual token counting into every custom loop. Watch for that pattern to show up in every major framework within a few weeks. For more on Claude Opus 4.7, see the [official Anthropic announcement](https://www.anthropic.com/news/claude-opus-4-7), the [What's new in Claude Opus 4.7 docs](https://platform.claude.com/docs/en/about-claude/models/whats-new-claude-4-7), and the [detailed benchmark reporting from The Next Web](https://thenextweb.com/news/anthropic-claude-opus-4-7-coding-agentic-benchmarks-release). ### Keep Reading - [Claude Mythos and Project Glasswing](https://www.rabinarayanpatra.com/blogs/claude-mythos-project-glasswing): the gated model that sits above Opus 4.7, why you cannot have it, and what that signals about where Anthropic is headed. - [Claude Skills vs MCP vs Projects](https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects): how the three Claude extensibility systems fit together, and where Opus 4.7's better memory tool lands in that picture. - [Vercel AI Gateway Deep Dive](https://www.rabinarayanpatra.com/blogs/vercel-ai-gateway-deep-dive): route Opus 4.7 through the AI Gateway for cost tracking, fallbacks, and per-model observability without rewriting your client code. - [Google Antigravity](https://www.rabinarayanpatra.com/blogs/google-antigravity): the competing agentic coding platform, for context on where Cursor, Windsurf, and Anthropic's own Claude Code fit in the same week. --- ## Claude Skills vs MCP vs Projects: Which One Should You Use? Source: https://www.rabinarayanpatra.com/blogs/claude-skills-vs-mcp-vs-projects Published: 2026-04-16T00:00:00+05:30 Tags: claude-skills, mcp, claude-projects, anthropic, ai-tools, model-context-protocol Claude Skills, MCP servers, and Projects solve different problems. Here is the honest comparison, token costs, and a decision framework for picking right. I spent a weekend rebuilding my blog's assistant setup three times. First with Claude Projects, then with an MCP server, then with Skills. Each time I thought I had the right answer. Each time I was wrong, but for a different reason. The fourth attempt got me where I wanted to go, and the fix was embarrassing. I did not need to pick one. I needed all three, doing different jobs. This post is the guide I wish I had read before burning 20 hours figuring that out. If you are trying to decide between Claude Skills, MCP servers, and Projects, here is the honest comparison with token costs, pricing, and a decision framework that actually works. ![Claude Skills context window loading diagram from the official Anthropic documentation](/images/blogs/claude-skills-vs-mcp-vs-projects/skills-context-window.webp) *Source: [Anthropic Agent Skills documentation](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview)* ## What actually is a Claude Skill, an MCP server, and a Project? A Claude Skill is a folder of instructions that Claude loads on demand, an MCP server is a program that exposes tools and data to Claude over a standard protocol, and a Project is a persistent workspace with a knowledge base and custom instructions scoped to one body of work. Same AI model, three different ways to extend what it knows and what it can do. The cleanest framing I have seen comes from Anthropic's own team. They describe it like this. Projects say "here is what you need to know." Skills say "here is how to do things." MCP says "here is a tool you can use." If you want an even simpler analogy, MCP hands Claude a hammer, Skills teach it how to drive nails, and Projects are the workshop where the blueprints live. Here is what each one actually looks like on disk or in the API. A Skill is a folder with a `SKILL.md` file at the root. The top of that file is YAML frontmatter with two required fields, `name` and `description`. Everything below the frontmatter is markdown instructions. You can include helper files and executable scripts alongside `SKILL.md`, and Claude will read or run them only when the task needs them. ```yaml --- name: pdf-processing description: Extract text and tables from PDF files, fill forms, merge documents. Use when working with PDF files or when the user mentions PDFs, forms, or document extraction. --- # PDF Processing ## Quick start Use pdfplumber to extract text from PDFs. For advanced form filling, see [FORMS.md](FORMS.md). ``` An MCP server is a separate process. It speaks JSON-RPC 2.0 over either stdin/stdout for local use or Streamable HTTP for remote use. Claude Code or Claude Desktop spawns an MCP client that connects to the server, negotiates capabilities, and then calls the tools the server exposes. ![MCP client-server architecture showing an MCP Host with three MCP Clients connecting to local filesystem and database servers via stdio, and a remote Sentry server via Streamable HTTP](/images/blogs/claude-skills-vs-mcp-vs-projects/mcp-architecture.webp) *Architecture based on the [Model Context Protocol architecture overview](https://modelcontextprotocol.io/docs/learn/architecture)* ```json { "jsonrpc": "2.0", "id": 3, "method": "tools/call", "params": { "name": "weather_current", "arguments": { "location": "San Francisco", "units": "imperial" } } } ``` A Project is a workspace inside claude.ai. You click New Project, give it a name, paste in custom instructions, and upload files to its knowledge base. Everything you upload becomes available in every chat inside that project. No YAML, no server, just a scoped conversation bucket with a big context window attached to it. Those three shapes matter because they drive every tradeoff that follows. A folder of markdown is cheap to write and cheap to load. A separate process is powerful but expensive in both setup and context. A workspace is great for scoping but has no way to do things, only to remember things. ## How does each one load into Claude's context? Skills load in three stages using progressive disclosure, MCP loads its entire tool schema upfront on every request, and Projects load their knowledge base statically whenever you are inside the project. This is the single most important difference between them, and nobody explains it clearly. Let me break down exactly what enters Claude's context window in each case. **Skills use progressive disclosure**. At startup, Claude only sees the `name` and `description` from the YAML frontmatter. That costs about 100 tokens per skill. You can install dozens of skills and Claude still only pays that flat 100-token entry fee for each one. When your request matches a skill's description, Claude reads the full `SKILL.md` body via bash. That adds under 5,000 tokens. If `SKILL.md` references another file, say `FORMS.md` or `scripts/validate.py`, Claude reads or runs that file only when the task actually needs it. Scripts are especially efficient. When Claude runs a script, the code itself never enters context. Only the script's output does. ![Agent Skills architecture diagram showing how Skills integrate with Claude's virtual machine](/images/blogs/claude-skills-vs-mcp-vs-projects/skills-architecture.webp) *Source: [Anthropic Agent Skills documentation](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview)* **MCP does the opposite**. When Claude connects to an MCP server, it calls `tools/list` and loads the full JSON schema of every tool the server exposes. That schema stays in context for the whole session. GitHub's official MCP server, which exposes roughly 80 tools, costs tens of thousands of tokens just to load the tool definitions. You pay that upfront, every single conversation, whether you use GitHub or not. **Projects load their knowledge base statically**. When you open a chat inside a project, the custom instructions and uploaded files are always in context. If your project knowledge base is small, it all fits. If it is large, Claude quietly switches to RAG mode and retrieves the relevant chunks instead, which can expand usable capacity by up to ten times but trades determinism for retrieval accuracy. The practical consequence is simple. Skills scale to many capabilities without bloating context. MCP gives you raw power at a real token cost. Projects are static, so they work well when the knowledge base matches the work but feel wasteful when you keep opening a big project to ask small questions. ## When should you reach for a Skill instead of a Project? Reach for a Skill when you are encoding a repeatable procedure, and use a Project when you are scoping a body of knowledge. The split is procedure versus context. Skills travel with you. Projects pin context to one workspace. Here is the test I use. If I find myself copying the same instruction block into three different chats, that instruction should be a Skill. If I keep uploading the same PDF to three different chats, that PDF should be in a Project. If I keep doing both, the Project holds the PDF and the Skill handles the instructions. A concrete example from my own setup. I write blog posts on rabinarayanpatra.com, and I have strict standards for how every post should be structured. H2 headings must be questions. First sentence after each H2 must answer it. No em dashes. No AI-tell words from my banned list. I used to paste those rules into every chat. That was fine until I started writing posts across three different projects. Then I had the same rules pasted three times, drifting slightly each time. I moved the rules into a Skill called `blog-standards`. Now any chat, in any project or none, can load those rules on demand. The project for my portfolio still exists, but it holds my actual post drafts and research, not the rules for writing them. That split feels obvious in hindsight. Before I made it, I was using Projects as a dumping ground for both knowledge and procedure, and it was bloating every conversation. Another clean test is portability. Skills work everywhere. Create a skill once, and it is available in regular chats, inside any project, across all surfaces where Skills are supported. Projects are scoped by definition. You cannot take a project with you into a regular chat. If the thing you are teaching Claude needs to apply in many contexts, it is a Skill. ## When does MCP beat a Skill, and when does it lose? MCP beats a Skill when Claude needs to reach external systems that Skills cannot touch, and MCP loses when the task is really about procedural knowledge dressed up as tool calls. The line is reachability. MCP can talk to your database, your Jira, your production API, your Slack. Skills cannot. Skills run in a sandbox without network access on the API, or with user-machine access in Claude Code. Start with the cases where MCP is the right answer. Claude needs to read data that lives somewhere else. A live database, a SaaS with an API, your company's Notion or Confluence. You want Claude to write to those same systems, like creating a Jira ticket or posting to Slack. You want the same integration to work across Claude Desktop, Claude Code, Cursor, VS Code, and any other MCP client. That last point matters more than people realize. MCP is an open standard under the Linux Foundation, and it has crossed 10,000 active servers with 97 million monthly SDK downloads. If you are going to invest in an integration, MCP is the one that outlives any single vendor's product decisions. Now the other side. I keep seeing teams reach for MCP when a Skill would have been faster, cheaper, and more reliable. The symptom is always the same. Someone builds an MCP server that is really a wrapper around a CLI or a local script, loads it into every session, pays the tool schema cost every conversation, and then uses it maybe once a week. Simon Willison nailed it when he wrote that "almost everything I might achieve with an MCP can be handled by a CLI tool instead." A Skill that wraps that same CLI costs 100 tokens to discover instead of tens of thousands to load. The honest tradeoff: - **MCP cost**: high token overhead, real setup effort, needs a running server, schema stays in context every session. - **MCP benefit**: reaches anything, works across clients, open standard, large ecosystem. - **Skill cost**: sandboxed, no network in API, filesystem model can feel weird at first. - **Skill benefit**: near-zero idle context cost, fast to author, progressive disclosure scales to dozens of capabilities. The rule I follow now: if the integration does not need a live external system, default to a Skill. Promote to MCP only when you actually need to reach out. ## How do the three stack up on pricing and availability? Projects are included in the Pro plan at $20 per month, Skills work on Pro and above, and MCP is built into Pro and all higher tiers. Everything except the underlying API usage is available on the cheapest paid plan, but each one has availability quirks worth knowing before you design around them. Here is a clean breakdown of what ships where. | Feature | Free | Pro ($20/mo) | Max ($100-200/mo) | Team ($20-100/seat) | Enterprise | |---|---|---|---|---|---| | Projects | Limited | Unlimited | Unlimited | Shared across team | Shared, 500K context | | Skills (custom, Claude.ai) | No | Yes, per user | Yes, per user | Yes, per user | Yes, per user | | Skills (Claude Code) | N/A | Filesystem, free | Filesystem, free | Filesystem, free | Filesystem, free | | Skills (API) | N/A | Via API pricing | Via API pricing | Via API pricing | Via API pricing | | MCP remote connectors | Limited | Yes | Yes | Yes | Yes | | Context window | 200K | 200K (1M on Opus in Code) | 200K (1M on Opus in Code) | 200K | 500K on some models | A few things this table does not capture that will bite you. Custom Skills do not sync between surfaces. A Skill you upload to claude.ai is not available in the API. A Skill in Claude Code is not available in claude.ai. If you want the same skill in all three, you upload it three times. Anthropic's team has been clear that this is a known gap and not ideal. Skills on claude.ai are individual per user. There is no org-wide admin distribution today. If you are a Team admin and you want every analyst on your team to use the same compliance-formatting skill, you currently have to tell each of them to upload it themselves. Skills in the Claude API are workspace-wide, so that experience is very different. MCP servers you add to Claude Desktop or Claude Code live on your machine. Remote MCP servers are available on Pro and above. If you are on Free, your MCP options are limited to what ships built-in. The API has its own constraints. Skills in the API require three beta headers (`code-execution-2025-08-25`, `skills-2025-10-02`, and `files-api-2025-04-14`), and the sandbox has no network access and no runtime package installation. Plan for that upfront. If your skill depends on making an HTTP call out to a third-party API, it will not work on the API side. You either move that external call to an MCP server, or you pick a different surface. ## What are the real-world limitations nobody talks about? Every one of these three has a sharp edge that the marketing pages do not mention, and you will hit all of them within a month of real use. The honest version. Skills cannot reach the network on the API. MCP burns context whether you use it or not. Projects cannot execute code. Skills sandbox, laid out plainly: - No network access on the Claude API. - No runtime package installation on the Claude API. You get whatever is pre-installed. - Maximum of 8 Skills per API request. - Not covered by Zero Data Retention. Skill definitions and execution data are retained under standard policy. - In Claude Code, skills have full network access, which is great for local use but is also why you should only install skills you wrote or trust. MCP quirks that hurt: - Schema cost compounds. Every connected server adds its tool schema to context. Three servers with 30 tools each is close to 100,000 tokens of tool definitions before you send a single user message. - Streamable HTTP at scale still fights horizontal load balancers because sessions are stateful. The 2026 MCP roadmap lists this as a top-priority fix, but it is not solved yet. - Security is real. A malicious MCP server can invoke tools in ways that do not match its stated purpose. The same warning applies to Skills, but MCP has a bigger attack surface because servers can be remote and updated independently. Projects blind spots: - No executable code. You cannot run a script inside a project the way a skill can. - Custom instructions are not unlimited. Anthropic explicitly says to keep them concise. Large instruction blocks degrade response quality. - RAG mode kicks in silently when your knowledge base exceeds context. That is good for capacity but bad for determinism. Sometimes the retrieval misses the chunk you needed. - Team collaboration on projects is fine, but you cannot share a single project across organizations. Every team has its own. The combined effect is that each of these tools has a narrow band of things it does better than the others, and a large set of things it does worse. That is why the decision tree at the end of this post matters more than any single-product deep dive. ## How do you combine all three in a production workflow? The production pattern is a Project for context, MCP for external data, and Skills for procedures applied to that data. Anthropic's own team recommends this combined usage, and every team I know who has built real Claude workflows on [Opus 4.7](https://www.rabinarayanpatra.com/blogs/claude-opus-4-7-release-and-migration-guide) has converged on the same layering. Here is how it plays out in a concrete example. Say you are generating a Q3 sales report for a B2B SaaS company. You want the report to pull live numbers, use last quarter's context, and come out in your company's template every time. **Project**: a "Q3 Sales Analysis" project with last quarter's decks, your sales playbook, and custom instructions that tell Claude the audience, the tone, and what to omit. **MCP**: a Salesforce MCP server and a Snowflake MCP server. When you ask for the numbers, Claude calls `salesforce_opportunities_list` and `snowflake_query` and gets live data back. **Skill**: a `quarterly-report-generator` skill. Its SKILL.md says: "When the user asks for a quarterly report, structure the output with these sections, apply the brand styling in `brand.md`, chart revenue with `scripts/chart.py`, and format the deck with the pre-built `pptx` skill." Ask Claude to "generate the Q3 sales report." The Project gives it the context ("audience is the board, do not include unannounced deals"). MCP gives it the live data ("here are the 47 closed-won deals from last quarter"). The Skill gives it the procedure ("apply the outline, render the charts, export to PowerPoint"). You could not pull that off with any one of the three alone. A second example, closer to developer work. Code review on pull requests. **Project** (optional): a per-repo project with the architecture notes and the team's coding standards documents. **MCP**: the GitHub MCP server. Claude can read PRs, leave review comments, and check CI status. **Skill**: a `pr-review` skill that spells out the review checklist, the tone for comments, and how to flag breaking changes versus stylistic nits. The three-way split matches the three kinds of knowledge you are giving Claude. The Project says what the codebase is about. MCP gives it the keys to actually open files and leave comments. The Skill makes sure every review follows the same standard instead of whatever Claude feels like today. Rakuten's finance team reported that they reduced report generation from a full day to about an hour by combining MCP data access with Skills for formatting. Not magic. Just the right layering. ## Which one should you actually use? If you are still deciding, use this tree. It sounds dumb when written out, but it has saved me from picking the wrong tool at least a dozen times. **Does Claude need to reach a system that is not on its local machine or sandbox?** - Yes. Use MCP. Connect the system. If there is already a community MCP server for it, use that. If not, write one. This is the only path. - No. Continue. **Is the thing I am teaching Claude specific to one body of work, like a codebase or a client engagement?** - Yes. Put it in a Project. Upload the docs, write the instructions, keep it scoped. - No. Continue. **Is it a procedure I want Claude to apply consistently across contexts?** - Yes. Make it a Skill. Write the SKILL.md, include helper scripts if needed, and let it load on demand. - No. It is probably just a one-off prompt. Do not overbuild. Three questions, one answer at the end, every time. If the answer is "more than one of these," you are in the combined-workflow case from the previous section. A quick note on what I would not recommend. Do not use a Project for your company's style guide unless the style guide only applies to that project. Do not use an MCP server for a local script you could wrap in a Skill. Do not use a Skill to replace live data access. These are the three most common mistakes I see, and they all come from reaching for the wrong tool first. ## Conclusion The short version of everything above: Projects hold context, Skills teach procedures, MCP reaches external systems. Most developers I talk to are using one of the three and ignoring the other two. The best setups I have seen use all three, with a clear mental model of which layer owns which job. My own view after a few months of real use. Skills are the quietly winning primitive. They are cheap to write, cheap to load, portable across surfaces in ways Projects never will be, and they solve the single biggest pain point I had with MCP, which was paying for capability I was not using. Progressive disclosure is genuinely the right architecture for agent knowledge. I expect most of what people are currently bolting onto MCP servers to migrate into Skills over the next year, with MCP staying as the system-of-record for connectivity rather than procedure. Worth watching which way Anthropic's own pre-built skills go next, because that signals which workflows they think should live inside Skills and which should stay as tools. For the official reference on each, see the [Agent Skills documentation](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview), the [Model Context Protocol architecture overview](https://modelcontextprotocol.io/docs/learn/architecture), and Anthropic's own [Skills explained](https://claude.com/blog/skills-explained) post comparing all five primitives. The [2026 MCP roadmap](https://blog.modelcontextprotocol.io/posts/2026-mcp-roadmap/) is also worth reading if you care about where the protocol is heading. ### Keep Reading - [Claude Mythos and Project Glasswing](https://www.rabinarayanpatra.com/blogs/claude-mythos-project-glasswing) โ€” Anthropic's gated frontier model release, and what it means for how Claude capabilities are distributed. - [My Thoughts on Vibe Coding](https://www.rabinarayanpatra.com/blogs/my-thoughts-on-vibe-coding-2025) โ€” how I actually use AI coding tools day to day, and where Skills fit into that flow. - [Google Antigravity: Why I Think It Changes Everything](https://www.rabinarayanpatra.com/blogs/google-antigravity) โ€” the other big agentic platform story, and the closest competitor to this stack. - [Building a Modern Documentation Generator with Next.js 16](https://www.rabinarayanpatra.com/blogs/building-modern-docs-generator-nextjs-16) โ€” the kind of personal project where a custom Skill pays off fast. --- ## Amazon S3 Files: Mount S3 as NFS (Setup + Cost) Source: https://www.rabinarayanpatra.com/blogs/aws-s3-2026-files-vectors-and-beyond Published: 2026-04-10T00:00:00+05:30 Tags: amazon-s3-files, aws-s3, cloud-storage, devops, aws S3 Files lets you mount S3 buckets via NFS with sub-millisecond latency. Here is how it works, what it costs, and why AWS also killed SSE-C by default. I've been running EFS-to-S3 sync jobs for two years. Cron schedules, lifecycle policies, rsync scripts that break every time someone changes a directory structure. All because S3 couldn't speak file system. That changed this week. On April 7, 2026, AWS announced [Amazon S3 Files](https://aws.amazon.com/blogs/aws/launching-s3-files-making-s3-buckets-accessible-as-file-systems/), a feature that lets you mount any S3 bucket as a shared NFS file system. No gateway. No third-party tool. No data copies. Your applications read and write files, and S3 stores them. That's it. And quietly, on April 6, AWS also started [disabling SSE-C encryption by default](https://aws.amazon.com/about-aws/whats-new/2026/04/s3-default-bucket-security-setting/) on all new S3 buckets. If you're managing encryption keys manually, that one needs your attention too. Let me walk through both changes. ## What is Amazon S3 Files and how does it work? Amazon S3 Files gives S3 buckets a fully-featured file system interface using NFS v4.2. You can mount a bucket on EC2, Lambda, EKS, ECS, Fargate, or AWS Batch and interact with your data using standard file operations: `open()`, `read()`, `write()`, `ls`, `cp`, `mv`. No SDK. No API calls. Just a mount point. ![S3 Files Architecture Overview - Source: AWS Blog](/images/blogs/s3-files-architecture.webp) *Architecture overview of Amazon S3 Files. Source: [AWS News Blog](https://aws.amazon.com/blogs/aws/launching-s3-files-making-s3-buckets-accessible-as-file-systems/)* S3 is now the first and only cloud object store that provides native file system access while keeping all data in object storage. Your objects don't move to a separate file system. They stay in S3 with all the durability, lifecycle policies, and access controls you already have. S3 Files is generally available in 34 AWS Regions as of launch day. ### The stage and commit model This is the part that surprised me. Instead of translating every file write into an immediate S3 PUT, S3 Files batches changes and commits them to S3 roughly every 60 seconds. AWS borrowed this concept from version control. Here's what that means in practice: 1. You write a file through the NFS mount 2. The write lands in a caching layer (built on EFS infrastructure) 3. S3 Files aggregates writes within a 60-second window 4. Multiple writes to the same file become a single S3 PUT 5. The committed object appears in S3 with full consistency This batching has two practical benefits. First, it cuts your S3 request costs because ten rapid writes to the same file become one PUT, not ten. Second, if you're using S3 versioning, you don't end up with ten versions of a file that changed in under a minute. But it also means there's a ~60-second lag before file changes are visible on the S3 side. If your workflow needs immediate S3 API visibility of written data, you need to account for that delay. When there's a conflict (say, someone writes to a file through NFS while another process updates the same object via the S3 API), S3 remains authoritative. The file-side version gets moved to a `lost+found` directory with metrics for visibility. No silent data loss. ### The caching layer under the hood When you create an S3 Files file system, AWS provisions a caching layer backed by EFS infrastructure. This cache holds three things: - **Recently read files**: Hot reads come from cache with sub-millisecond latency - **Recently written files**: Staged writes waiting for the next commit cycle - **Metadata**: Directory listings, file attributes, timestamps Small file reads are served entirely from the cache. Large file reads (over 1MB) stream directly from S3 and don't incur S3 Files charges. The aggregate read throughput can reach multiple terabytes per second. This design is what makes S3 Files cost-effective. You pay file system rates ($0.30/GB-month) only on the data that's actively cached. A petabyte bucket where only 500GB is actively read? You're paying S3 rates for the petabyte and file system rates for the 500GB. ## How do you set up S3 Files on an EC2 instance? The setup is straightforward. Here's the full flow: ```bash # Step 1: Create an S3 Files file system linked to your bucket aws s3api create-file-system \ --bucket my-data-bucket \ --file-system-name my-fs # Step 2: Get the mount target DNS name aws s3api describe-file-system \ --bucket my-data-bucket \ --file-system-name my-fs \ --query 'FileSystem.MountTargets[0].DnsName' \ --output text # Step 3: Mount on your EC2 instance sudo mount -t nfs4 \ -o nfsvers=4.2,rsize=1048576,wsize=1048576 \ fs-mount-target.efs.us-east-1.amazonaws.com:/ \ /mnt/s3data # Step 4: Use it like any filesystem ls /mnt/s3data cp local-file.csv /mnt/s3data/uploads/ cat /mnt/s3data/reports/quarterly.json ``` Once mounted, every application on that instance can access S3 data through normal file I/O. Python scripts, Java apps, shell scripts, legacy C++ binaries. Nothing needs to know it's talking to S3. For persistent mounts across reboots, add it to `/etc/fstab`: ```bash # /etc/fstab entry for S3 Files fs-mount-target.efs.us-east-1.amazonaws.com:/ /mnt/s3data nfs4 nfsvers=4.2,rsize=1048576,wsize=1048576,_netdev 0 0 ``` ## Where does S3 Files beat EFS (and where does it not)? This is the question I've been testing all week. S3 Files isn't a drop-in EFS replacement for every workload, but for specific patterns it's clearly better. ### S3 Files wins | Scenario | Why S3 Files | |----------|-------------| | Large datasets, small active working set | Pay S3 rates on cold data, file system rates only on hot data | | Legacy app migration to S3 | Zero code changes needed, just mount and go | | AI/ML training data pipelines | Read training data as files, store as S3 objects | | Agentic AI workloads | Shared workspace across multiple compute instances | | Multi-service data sharing | Multiple EKS pods or Lambda functions reading the same dataset | ### EFS still wins | Scenario | Why EFS | |----------|---------| | All data is hot, constant read/write | EFS avoids the commit delay and S3 request overhead | | Sub-second write visibility needed | S3 Files has a ~60-second commit lag | | Windows workloads (SMB) | S3 Files only supports NFS, no SMB | | Hard link requirements | S3 Files doesn't support hard links | | Bucket exceeds 50 million objects | AWS warns about performance at this scale | ### Pricing comparison The pricing math depends entirely on your access pattern: - **S3 Files cached storage**: $0.30/GB-month (only for actively cached data) - **S3 Files reads (small files)**: $0.03/GB from cache - **S3 Files reads (large files, 1MB+)**: $0 from S3 Files (standard S3 GET charges apply) - **S3 Files writes**: $0.06/GB Compare that to EFS Performance-optimized at $0.30/GB for standard storage and $0.03/GB for reads. The difference shows up at scale: if you have 10TB in a bucket but only touch 200GB regularly, S3 Files costs a fraction of what an equivalent EFS setup would cost. ## What are the gotchas you should know before adopting S3 Files? I ran into a few things during my initial testing that are worth flagging. **The 60-second commit window is real.** If you write a file via NFS and immediately try to read it through the S3 API (using `aws s3 cp` or a direct GET), it won't be there yet. Your application logic needs to handle this. For workflows that do writes via NFS and reads via S3 API, consider adding a short wait or checking for object existence. **NFS file locks don't protect against S3 API access.** If you lock a file through NFS, that lock only applies to other NFS clients. Someone using the S3 API directly can still modify the object. This isn't a bug. It's how the boundary between file system and object store works. But it can bite you if mixed access isn't on your radar. **The 50-million object warning is something to watch.** AWS recommends caution when a mounted bucket contains more than 50 million objects. Directory listings and metadata operations can slow down at that scale. If you're dealing with buckets that large, consider using S3 prefixes to scope your mount. **No pNFS, Kerberos, or nconnect support.** If your NFS setup depends on parallel NFS, Kerberos authentication, NFSv4 data retention, or the `nconnect` mount option, those aren't available yet at GA. Standard NFS v4.2 features work fine. **SMB is not supported.** Windows workloads that need file system access to S3 still need FSx or a gateway solution. ![S3 Files Technical Comparison Architecture](/images/blogs/aws-s3-2026-files-vectors-and-beyond/s3-files-comparison.webp) ## Why did AWS disable SSE-C encryption by default? This change flew under the radar next to S3 Files, but it affects every new bucket created after April 6, 2026. SSE-C (Server-Side Encryption with Customer-Provided Keys) lets you bring your own encryption key on every PUT and GET request. S3 encrypts and decrypts using your key but never stores it. The idea is maximum control. The reality is operational risk. Lose the key, lose the data forever. AWS can't recover it for you. There's no "forgot my password" option. AWS KMS solved this years ago with customer-managed keys (CMKs) that give you full ownership and control, plus key rotation, auditing through CloudTrail, and recovery options. For most workloads, KMS does everything SSE-C does, minus the footgun. So AWS made SSE-C opt-in instead of opt-out. Here's how the rollout works: ### What changes for new buckets Every new general-purpose S3 bucket created after April 6, 2026 has SSE-C disabled by default. If you try to upload an object with SSE-C headers, you'll get an access denied error unless you explicitly enable SSE-C first. To enable SSE-C on a new bucket: ```bash # Explicitly allow SSE-C on a new bucket aws s3api put-bucket-encryption \ --bucket my-new-bucket \ --server-side-encryption-configuration '{ "Rules": [{ "ApplyServerSideEncryptionByDefault": { "SSEAlgorithm": "AES256" }, "BucketKeyEnabled": false }], "AllowSSEC": true }' ``` ### What changes for existing buckets This is the part that might catch people off guard. AWS is also disabling SSE-C on existing buckets that have **zero SSE-C encrypted objects**. If you created a bucket, never used SSE-C, but your automation code includes SSE-C headers "just in case," those writes will start failing. Existing buckets that actually contain SSE-C objects? No changes. AWS won't touch those. ### Who this affects If you're using AWS KMS (SSE-KMS) or S3-managed keys (SSE-S3) for encryption, this change does nothing to you. Your buckets already don't use SSE-C. If you're one of the teams still on SSE-C, you'll want to: 1. Audit which buckets actually use SSE-C (`aws s3api get-bucket-encryption`) 2. Plan a migration to KMS for buckets that don't strictly need SSE-C 3. Explicitly re-enable SSE-C on new buckets where it's genuinely required The rollout covers 37 AWS Regions, including GovCloud and China regions, and will complete over the next few weeks. ## What do these changes tell us about where S3 is heading? If I look at S3 Files and the SSE-C default together, they tell the same story: AWS is reducing the reasons you'd reach for anything other than S3. Need file system access? You used to need EFS plus sync scripts. Now you mount S3 directly. Need encryption with your own keys? You used to reach for SSE-C. Now AWS is steering you toward KMS, which handles key management for you. S3 now stores over 500 trillion objects and handles roughly 200 million requests per second. It turned 20 years old last month. And instead of letting it coast, AWS gave it the most significant capability upgrade since S3 Intelligent-Tiering launched in 2018. For my own projects, I'm already replacing two EFS-backed data pipelines with S3 Files mounts. The sync cron jobs are gone. The drift alerts are gone. One mount point, one storage bill, and a 60-second commit window I can easily live with. If you've been running parallel storage systems just to get file access to your S3 data, this week is a good week to rethink that architecture. For the full details, see the [official S3 Files announcement](https://aws.amazon.com/blogs/aws/launching-s3-files-making-s3-buckets-accessible-as-file-systems/), the [S3 Files product page](https://aws.amazon.com/s3/features/files/), and the [SSE-C security default announcement](https://aws.amazon.com/blogs/storage/advanced-notice-amazon-s3-to-disable-the-use-of-sse-c-encryption-by-default-for-all-new-buckets-and-select-existing-buckets-in-april-2026/). ### Keep Reading - [AI-Driven Anomaly Detection for Security](https://www.rabinarayanpatra.com/blogs/ai-driven-anomaly-detection-security) - How cloud infrastructure and AI work together for real-time threat detection. - [Implementing the Outbox Pattern with CDC in Microservices](https://www.rabinarayanpatra.com/blogs/implementing-outbox-pattern-cdc-microservices) - Storage design patterns that affect reliability in distributed systems. - [The Day a React Patch Broke the Internet](https://www.rabinarayanpatra.com/blogs/the-day-react-patch-broke-the-internet) - Another deep dive into an infrastructure event that caught everyone off guard. --- ## Claude Mythos and Project Glasswing: Anthropic's Locked-Down AI Model Source: https://www.rabinarayanpatra.com/blogs/claude-mythos-project-glasswing Published: 2026-04-09T00:00:00+05:30 Tags: claude-mythos, anthropic, ai-security, project-glasswing, llm Anthropic's new Claude Mythos model scores 93.9% on SWE-bench Verified and finds zero-days autonomously. Here is why it is not getting a public release. Anthropic shipped a new Claude model on April 7, 2026 and decided you cannot use it. Not as an API customer, not on Claude.ai, not through any normal channel. They built something called Claude Mythos, claimed it can autonomously find zero-days in browsers and operating systems, and then locked it behind a program called Project Glasswing. This is the first time Anthropic has launched a frontier model as a deliberate non-product. I have been watching frontier launches for a while now, and the framing on this one is genuinely different from anything OpenAI, Google, or even earlier Anthropic releases have done. So let me walk through what Mythos actually is, what the numbers say, and why I think the gated rollout is the most interesting part of the story. ![A modern enterprise data center corridor with illuminated server racks](/images/blogs/claude-mythos-project-glasswing/data-center.webp) ## What is Claude Mythos and how is it different from Claude Opus? Claude Mythos is Anthropic's most capable model to date, released on April 7, 2026 as a gated research preview rather than a general product. Anthropic describes it as a new model class with state-of-the-art performance on agentic coding, complex reasoning, and offensive security tasks. The headline is not really the benchmarks, although the benchmarks are huge. The headline is the positioning. Claude Opus, Sonnet, and Haiku are products. They have pricing pages. They have rate limits you can buy your way out of. Mythos is none of those things. It is closer to how a national lab would describe a piece of dual use research than how a SaaS company describes a model. Here is what Anthropic actually claims Mythos can do: - Reverse engineer closed source binaries and find exploitable bugs without human guidance. - Develop working exploits against real targets, including the Firefox JavaScript engine, where it produced working exploits 181 times in their internal evaluation. - Find high severity vulnerabilities in every major operating system and every major browser during pre-release testing. - Saturate most of the public cybersecurity benchmarks Anthropic used for prior models, which is why they had to move to novel real world tasks. That last point is the one I keep coming back to. When a model saturates the eval suite, the eval suite stops being useful. You start needing real targets, and real targets are exactly what you do not want an unrestricted model running against. ## How does Claude Mythos perform on benchmarks compared to Opus 4.6? Claude Mythos pulls a double digit lead over Claude Opus 4.6 on every benchmark Anthropic published, with the largest gaps on security and agentic coding tasks. Here are the numbers from Anthropic's own evaluation post: | Benchmark | Claude Opus 4.6 | Claude Mythos Preview | Delta | |---|---|---|---| | SWE-bench Verified | 80.8% | 93.9% | +13.1 | | SWE-bench Pro | 53.4% | 77.8% | +24.4 | | CyberGym (vuln reproduction) | 66.6% | 83.1% | +16.5 | A few things stand out. SWE-bench Verified at 93.9% is essentially the ceiling of the benchmark. There are known label issues and ambiguous tasks in the remaining 6%, so any further gains are noise. SWE-bench Pro is the harder, larger, more realistic version, and a 24 point jump there is the kind of move you usually see across two model generations, not one. CyberGym is the one that actually matters for the Glasswing framing. It measures whether a model can reproduce real CVEs end to end, and 83.1% is the first time any public number has cleared 80% on that benchmark. ![Printed benchmark results on a matte black desk](/images/blogs/claude-mythos-project-glasswing/benchmark-results.webp) Numbers aside, the qualitative claim is the more interesting one. Anthropic says Mythos is the first model where the bottleneck on offensive security work is no longer the model itself, it is the human reviewer triaging what the model finds. That is a real shift. For the last two years the joke among security people has been that LLMs are great at writing CVE descriptions and bad at finding actual bugs. Mythos is the first launch that seriously contests that. ## Why is Anthropic gating Mythos behind Project Glasswing instead of shipping it? Anthropic is not making Mythos generally available because they believe a model that can autonomously find and exploit zero-days in production software is too dangerous to put behind a credit card. Project Glasswing is the structure they built so the model can still do useful defensive work without getting handed to anyone who signs up. The shape of Glasswing, based on Anthropic's own announcement and the partner list: - 12 launch partners, including Microsoft, Nvidia, Cisco, Amazon, Apple, and Google. - 40+ additional organizations that build or maintain critical software infrastructure, granted gated access. - 100 million dollars in Mythos usage credits committed across the program. - 4 million dollars in direct donations to open source security organizations. - Access through Amazon Bedrock under a separate gated research preview agreement. Reading between the lines, Glasswing is doing three things at once. It is a safety story, it is a moat, and it is a regulatory positioning move. The safety story is the obvious one. The moat is that the partners getting Mythos access are exactly the companies whose own security posture matters most to Anthropic's enterprise pipeline. The regulatory move is that by self-imposing a restriction this aggressive, Anthropic gets to show up at every AI safety hearing for the next year with a concrete example of voluntary risk mitigation. I do not think any of those motives are bad. I just think it is worth naming all three instead of pretending the launch is purely altruistic. ## What does Claude Mythos mean for developers who will never touch it? Even if you cannot use Mythos directly, the launch tells you three things about where frontier models are heading and what to plan for over the next 12 months. **One. The era of model launches as pure product launches is ending.** Mythos is the first frontier release I can remember where the headline is not pricing or context window, it is a refusal to ship. Expect more of this. The next time a lab crosses a capability line that scares its own safety team, the playbook is now public: announce, gate, partner, donate, move on. **Two. SWE-bench is cooked as a useful number.** When the top model is at 93.9% Verified and 77.8% Pro, the benchmark stops separating models from each other. If you write about LLMs or ship developer tooling, you need to start citing CyberGym, Aider polyglot, real bug bounty results, and end-to-end agent task completions instead. SWE-bench is going to look like GLUE looked in 2020 within a year. **Three. Defensive tooling is about to get strange.** If Mythos and its successors are running inside Microsoft, Apple, and Google, the rate at which serious vulnerabilities get found and patched is going to accelerate in a way that is genuinely good for everyone, but it also means the half-life of an unpatched bug in popular software is going to collapse. If you maintain an open source library, you should be thinking right now about how fast you can credibly turn around a security advisory, because the discovery side is no longer the slow step. ## Conclusion Claude Mythos is the first time a frontier lab has shipped a model by deciding not to ship it, and the framing matters more than the numbers. The benchmarks are real, the capabilities are real, and the gating is real, but the thing to actually pay attention to is the new shape of the launch itself. Frontier releases are starting to look less like product drops and more like controlled disclosures. If you are building anything that touches security, the practical move this week is to go read the CyberGym methodology and the Glasswing partner list, then ask yourself which side of that list your product lives on. That is the question Anthropic just made everyone answer. For more on the launch, see Anthropic's [Project Glasswing announcement](https://www.anthropic.com/glasswing), the [Mythos Preview cybersecurity evaluation](https://red.anthropic.com/2026/mythos-preview/), Simon Willison's [analysis of the gated release](https://simonwillison.net/2026/Apr/7/project-glasswing/), and the [AWS Bedrock availability note](https://aws.amazon.com/about-aws/whats-new/2026/04/amazon-bedrock-claude-mythos/). ### Keep Reading - [Google Antigravity and the New AI Coding Stack](https://www.rabinarayanpatra.com/blogs/google-antigravity) โ€” How another lab is approaching agentic coding from a very different angle. - [AI Driven Anomaly Detection for Security](https://www.rabinarayanpatra.com/blogs/ai-driven-anomaly-detection-security) โ€” The defensive side of the same trend Mythos accelerates. - [Agentic Payments on Razorpay, NPCI, and UPI](https://www.rabinarayanpatra.com/blogs/agentic-payments-razorpay-npci-upi) โ€” What gated, high-trust AI deployments look like in a regulated domain. --- ## Why I Replaced useEffect Data Fetching with Server Actions Source: https://www.rabinarayanpatra.com/blogs/replacing-useeffect-data-fetching-server-actions Published: 2026-04-08T00:00:00+05:30 ยท Updated: 2026-05-26T00:00:00+05:30 Tags: server-actions, useeffect, nextjs, react, data-fetching I ripped useEffect data fetching out of my Next.js 16 app and replaced it with Server Components and Server Actions. Here is what broke and what got better. I opened my Next.js app one morning and counted 23 `useEffect` blocks whose only job was to fetch data on mount. Dashboards, profile pages, comment threads, the chatbot history view. Every one of them followed the same sad pattern: render nothing, flash a spinner, wait for the client to wake up, fire a request, set state, re-render. It worked. It also felt wrong in a way I had been ignoring for months. So I spent a weekend ripping all of it out. Server Components handle the reads now. Server Actions handle the writes. I kept `useEffect` around for exactly the things it was designed for, and nothing else. This is the refactor diary, including the parts where I broke things and had to back out. ## Why was useEffect data fetching the wrong default? Because it forces the browser to do work the server could have already finished. The classic `useEffect(() => { fetch(...) }, [])` pattern ships empty HTML, boots React on the client, runs the effect, opens a network request, parses JSON, then finally paints the thing the user came for. On a fast laptop nobody notices. On a mid-range Android phone on a train in Bhubaneswar, it is a full second of staring at a spinner. There are three specific problems I kept hitting. The first is the waterfall. A page with three `useEffect` fetches inside three different components ends up making three sequential round trips after hydration finishes. Each one waits for the component above it to mount. I had a profile page where the avatar, the stats card, and the recent activity list were fetched this way. Total time to interactive on a throttled connection was over three seconds, and the API itself responded in under 150ms each. The second is the flash of nothing. You need a `loading` state, an `error` state, a `data` state, and a guard for the unmounted case. I wrote the same `if (loading) return ` branch so many times I turned it into a hook, then turned that hook into a bigger hook, and eventually realised I had built a worse version of React Query without meaning to. The third is SEO and AI crawlers. Google renders JavaScript, but it does not love it. ChatGPT's crawler and Perplexity's crawler are even less patient. If the meat of your page only appears after a client fetch, you are trusting a lot of bots to stick around. On this portfolio I care about that, because half my traffic comes from people asking an AI a question. ## How do Server Components actually replace the fetch-on-mount pattern? Server Components let you `await` your data directly in the component body, on the server, before any HTML reaches the browser. There is no hook, no state, no effect. The component is an async function, and whatever it returns is already resolved by the time the user sees it. Here is the before and after from my activity feed. This is real code from the refactor, trimmed for the post. **Old client component with useEffect:** ```tsx 'use client' import { useEffect, useState } from 'react' export function ActivityFeed({ userId }: { userId: string }) { const [items, setItems] = useState(null) const [error, setError] = useState(null) useEffect(() => { let cancelled = false fetch(`/api/activity?userId=${userId}`) .then((r) => r.json()) .then((data) => { if (!cancelled) setItems(data) }) .catch((e) => { if (!cancelled) setError(e) }) return () => { cancelled = true } }, [userId]) if (error) return if (!items) return return } ``` **New Server Component:** ```tsx import { getActivity } from '@/lib/activity' import { FeedList } from './feed-list' export async function ActivityFeed({ userId }: { userId: string }) { const items = await getActivity(userId) return } ``` That is the whole component. No hook, no state, no cancellation flag, no skeleton branch inside the component itself. The skeleton now lives in a `loading.tsx` file next to the route, and [React Suspense streams the real content in when it is ready](https://www.rabinarayanpatra.com/snippets/nextjs/streaming-suspense-loading). The error case lives in `error.tsx`. Both of those are built into the App Router, and I had been ignoring them because my brain was still wired for the old pattern. The nicest part is that `getActivity` is now a plain async function that talks to the database directly. No API route, no serializer, no extra network hop. The database call and the page render happen in the same process, on the same machine, inside the same request. ## Where do Server Actions fit once reads move to the server? Server Actions handle the write half of the story. When the user clicks Like, posts a comment, or updates a setting, I used to `fetch('/api/comments', { method: 'POST' })` from inside a `useEffect` or a submit handler, then manually refetch the list. Now the mutation is a function with `'use server'` on top, and the page revalidates itself. Here is the comment form, before and after. **Old pattern:** ```tsx 'use client' async function submit(text: string) { await fetch('/api/comments', { method: 'POST', body: JSON.stringify({ text }), }) // now manually refetch the list, or reload the page, or // hope a SWR mutate call covers it } ``` **New pattern:** ```tsx // app/posts/[slug]/actions.ts 'use server' import { revalidateTag } from 'next/cache' import { db } from '@/lib/db' export async function addComment(postId: string, formData: FormData) { const text = String(formData.get('text') ?? '').trim() if (!text) return { ok: false, error: 'Empty comment' } await db.comment.create({ data: { postId, text } }) revalidateTag(`comments:${postId}`) return { ok: true } } ``` ```tsx // comment-form.tsx 'use client' import { useActionState } from 'react' import { addComment } from './actions' export function CommentForm({ postId }: { postId: string }) { const [state, formAction, pending] = useActionState( addComment.bind(null, postId), { ok: false } ) return (