Upgrading to Java 25 & Spring Boot 4: Concurrency, Virtual Threads, and Latency Tuning
- Published on
- • 13 mins read•––– views
For over two decades, high-throughput Java backend engineering was shaped by a fundamental constraint: Java platform threads are 1:1 mappings to Operating System (OS) kernel threads.
Because each platform thread consumes roughly 1 MB of stack memory and incurs expensive OS context switching, JVM applications could rarely scale past a few thousand concurrent threads without exhausting memory or choking CPU schedulers. To survive at multi-million RPM scale, the industry pivoted toward asynchronous reactive frameworks like WebFlux, RxJava, and Netty. But reactive programming came at a steep cognitive price: fragmented stack traces, unreadable backpressure pipelines, and debugging sessions that drained developer morale.
Enter Project Loom and Virtual Threads, first previewed in Java 19/21 and reaching mature production stability in Java 25 LTS alongside Spring Boot 4.0.
Over the past three quarters, we upgraded 16 core microservices across our checkout domain at Trendyol to Java 25 and Spring Boot 4. Our services absorb 4M RPM steady-state and peak to 7M RPM, with over 90% of our latency budget spent in I/O operations (HTTP REST calls, Couchbase document lookups, PostgreSQL queries, and Kafka event publishing).
The promise sounded almost too good to be true: flip spring.threads.virtual.enabled=true, ditch reactive callback hell, write simple blocking imperative code, and achieve massive concurrent throughput.
Here is the unvarnished engineering reality of what happened when we deployed virtual threads to production: how we discovered and eliminated catastrophic carrier thread pinning, why our database connection pools almost melted, how Generational ZGC stabilized latency, and how we maintained Grade A SonarQube code quality.
1. Virtual Threads Architecture: M:N Scheduling
To understand why things broke before they got better, we must understand how Java 25 executes virtual threads under the hood.
Unlike a platform thread, a virtual thread (java.lang.VirtualThread) is a lightweight user-space construct managed by the JVM rather than the OS kernel.
[Virtual Threads (User Space)]
VT-1 VT-2 VT-3 ... VT-50,000
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────────────────────────────────────┐
│ ForkJoinPool Carrier Worker Threads (Kernel) │
│ [Carrier 1] [Carrier 2] ... [Carrier 16]│ (Matches CPU Core Count)
└──────┬─────────────┬────────────────────┬────┘
│ │ │
▼ ▼ ▼
[ OS Core 0 ] [ OS Core 1 ] [ OS Core 15 ]
- When a virtual thread executes on-CPU computation, the JVM mounts it onto a carrier thread (a standard platform thread from an internal
ForkJoinPool). - When the virtual thread encounters a non-CPU blocking operation (such as waiting on an HTTP response, reading a Couchbase document via socket I/O, or sleeping), the JVM unmounts the virtual thread from its carrier thread and parks its call stack on the heap.
- The carrier thread is immediately free to run other runnable virtual threads.
- When the socket I/O completes, the OS epoll/kqueue notifies the JVM, which remounts the virtual thread on any available carrier thread and resumes execution.
At steady state, our applications can maintain 80,000 concurrent virtual threads mounted across just 16 carrier threads (matching our Kubernetes node CPU core allocation), slashing thread memory overhead from 80 GB down to under 250 MB.
2. The Silent Killer: Carrier Thread Pinning
During our initial synthetic load test at 250,000 requests per minute in staging, the application ran flawlessly. But when we ramped the load test toward our campaign target of 4 Million RPM, something terrifying happened:
Our p99 latency suddenly spiked from 35ms to over 8,000ms, CPU usage dropped to near zero, and the application became completely unresponsive. Prometheus reported that all incoming HTTP connections were timing out at the reverse proxy.
We were experiencing Carrier Thread Pinning.
What Causes Pinning?
A virtual thread is pinned to its carrier thread when it executes a blocking operation while:
- Inside a
synchronizedblock or method. - Executing a native method (JNI) or Foreign Function & Memory (FFM) call.
When a virtual thread is pinned, it cannot be unmounted. If it blocks on network I/O while pinned, its underlying carrier thread remains hostage.
In a cluster where the carrier pool only has 16 carrier threads:
Carrier 1: [ Pinned in synchronized block waiting for Couchbase socket ]
Carrier 2: [ Pinned in synchronized block waiting for HTTP REST call ]
...
Carrier 16: [ Pinned in synchronized block waiting for PostgreSQL lock ]
-------------------------------------------------------------------------
RESULT: Carrier pool 100% starved! 79,984 other virtual threads FROZEN!
Even though the CPU had 95% idle capacity, not a single virtual thread could be scheduled.
Hunting Down Pinning with JVM Tracing & JFR
To identify where pinning was occurring, we enabled the JDK pinned thread diagnostic flag on our staging deployments:
# JVM Diagnostic Flags
-Djdk.tracePinnedThreads=full
We also configured Java Flight Recorder (JFR) to capture jdk.VirtualThreadPinned events:
<!-- JFR recording profile snippet -->
<event name="jdk.VirtualThreadPinned">
<setting name="enabled">true</setting>
<setting name="stackTrace">true</setting>
<setting name="threshold">10ms</setting>
</event>
The logs immediately pointed us to two legacy areas of our codebase:
Culprit 1: Legacy In-Memory Cache Mutexes
A shared internal token cache used a synchronized method around an HTTP token-refresh call:
// Anti-Pattern under Virtual Threads: Pinning the Carrier
public class LegacyOAuthTokenManager {
private String cachedToken;
private Instant expiryTime;
public synchronized String getValidToken() {
if (Instant.now().isAfter(expiryTime)) {
// BLOCKING HTTP CALL INSIDE SYNCHRONIZED METHOD!
// Pins the carrier thread for the entire network duration (~45ms)
this.cachedToken = authClient.requestNewTokenBlocking();
this.expiryTime = Instant.now().plusSeconds(300);
}
return this.cachedToken;
}
}
The Fix: Refactoring to ReentrantLock and StampedLock
Unlike synchronized, java.util.concurrent.locks.ReentrantLock integrates seamlessly with Project Loom's unmounting mechanics. When a virtual thread blocks waiting for a ReentrantLock, it unmounts cleanly from the carrier thread:
// Fixed: Virtual-Thread Friendly with ReentrantLock
public class ResilientOAuthTokenManager {
private final ReentrantLock lock = new ReentrantLock();
private volatile String cachedToken;
private volatile Instant expiryTime = Instant.MIN;
public String getValidToken() {
// Fast path: lock-free volatile read
if (Instant.now().isBefore(expiryTime) && cachedToken != null) {
return cachedToken;
}
lock.lock();
try {
// Double-checked locking
if (Instant.now().isBefore(expiryTime) && cachedToken != null) {
return cachedToken;
}
// Blocking I/O unmounts virtual thread cleanly
this.cachedToken = authClient.requestNewTokenBlocking();
this.expiryTime = Instant.now().plusSeconds(300);
return cachedToken;
} finally {
lock.unlock();
}
}
}
We systematically audited all 16 microservices, replacing legacy synchronized methods enclosing I/O calls with ReentrantLock, StampedLock, or non-blocking primitives. Carrier thread pinning dropped from 42,000 events/minute to absolute zero.
3. Database Connection Pool Sizing: HikariCP with Virtual Threads
The second major operational landmine when adopting virtual threads is database connection pools.
In the old platform-thread world, your thread pool was naturally bound by server.tomcat.threads.max=200. Since you could only have 200 concurrent requests executing, a HikariCP pool of maximum-pool-size: 50 was reasonably well-balanced.
With virtual threads enabled, your application can effortlessly accept and begin processing 10,000 concurrent requests.
If each of those 10,000 virtual threads immediately invokes couponRepository.findById(id):
- 10,000 virtual threads concurrently attempt to borrow a connection from HikariCP.
- HikariCP's internal lock experiences extreme contention.
- Connection acquisition timeouts (
Connection is not available, request timed out after 30000ms) explode.
The Naive Trap: Don't Increase the Pool to 1,000
Junior and intermediate engineers often suggest: "If we have 10,000 threads, let's just scale HikariCP maximumPoolSize from 50 to 1,000!"
Do not do this.
A PostgreSQL or Oracle instance cannot handle 1,000 active concurrent queries without massive context switching, disk seek contention, and buffer pool eviction. PostgreSQL spawns a dedicated backend process for every single connection. Having 1,000 concurrent processes executing queries will quickly cause CPU load on the database server to spike past 100, collapsing your database.
The Solution: Little's Law & Concurrency Limiting via Semaphores
We kept our HikariCP pool lean and placed a virtual-thread-aware bulkhead semaphore in front of database operations:
# application.yml
spring:
threads:
virtual:
enabled: true
datasource:
hikari:
maximum-pool-size: 40
minimum-idle: 40
connection-timeout: 5000
idle-timeout: 600000
max-lifetime: 1800000
@Component
public class DatabaseAccessBulkhead {
// Bound DB access to match physical connection pool capacity
private final Semaphore dbSemaphore = new Semaphore(40, true);
public <T> T executeWithPermit(Supplier<T> dbOperation) {
try {
if (!dbSemaphore.tryAcquire(2, TimeUnit.SECONDS)) {
throw new DatabaseSheddingException("Database pool congested. Request shed.");
}
try {
return dbOperation.get();
} finally {
dbSemaphore.release();
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "Operation interrupted");
}
}
}
By bounding concurrent database borrowers to 40 permits, virtual threads queue in memory without creating connection pool stampedes or exhausting database CPU.
4. Replacing ThreadLocal with Java 25 Scoped Values
For decades, ThreadLocal was the default mechanism to propagate contextual data (such as security principals, tracing IDs, and transaction contexts) down the call stack.
In a platform thread pool, you allocate a fixed number of ThreadLocal maps. But when you spawn 100,000 virtual threads, each holding multiple ThreadLocal variables:
- Memory Bloat: 100,000 copies of request metadata accumulate on the heap.
- Mutability & Leak Risks: If a
ThreadLocalis not cleaned up viaremove()in afinallyblock, references linger and corrupt subsequent workloads.
In Java 25, we replaced ThreadLocal with Scoped Values (java.lang.ScopedValue).
Scoped Values are immutable, bound to a specific lexical execution scope, and designed explicitly for lightweight virtual threads:
public class SecurityContextHolder {
public static final ScopedValue<UserContext> CURRENT_USER = ScopedValue.newInstance();
public static <T> T runAs(UserContext user, Callable<T> task) throws Exception {
// Bound exclusively for the duration of the callable
return ScopedValue.callWhere(CURRENT_USER, user, task);
}
}
// In service layer:
public CouponEvaluationResult evaluateCoupon(String couponCode) {
// Zero-overhead lookup; immutable and garbage-collected automatically with scope exit
UserContext user = SecurityContextHolder.CURRENT_USER.get();
return couponEngine.apply(couponCode, user.getCustomerId());
}
Scoped Values cut our per-request context memory overhead by 72% and completely eliminated memory leak warnings.
5. Garbage Collection in Java 25: Generational ZGC
Virtual thread workloads produce a distinct memory allocation profile: millions of short-lived virtual thread objects, frame stacks, and continuation buffers allocated and discarded in milliseconds.
Under the default G1GC collector, our minor collection frequency jumped significantly during 6M RPM load bursts, causing latency jitter.
We switched to Generational ZGC (available as the production standard in Java 25). Generational ZGC separates objects into young and old generations while performing all phases of garbage collection concurrently with application threads.
# Production JVM Flags for Java 25 Checkout Services
-XX:+UseZGC
-XX:+ZGenerational
-XX:MaxRAMPercentage=75.0
-XX:+AlwaysPreTouch
-XX:+UseNUMA
The Latency Stability Transformation
Here is the latency comparison between G1GC and Generational ZGC under a sustained 5M RPM synthetic checkout load test:
Garbage Collector Latency Profiles (5M RPM Sustained)
Latency Percentile │ Java 17 + G1GC │ Java 25 + Generational ZGC │ Improvement
───────────────────┼────────────────┼────────────────────────────┼────────────
p50 (Median) │ 18.2 ms │ 11.4 ms │ -37.4%
p90 │ 42.1 ms │ 24.6 ms │ -41.5%
p99 │ 84.5 ms │ 38.2 ms │ -54.8%
p99.9 │ 285.0 ms │ 52.8 ms │ -81.5%
Max GC Pause Time │ 48.0 ms │ 0.65 ms │ -98.6%
p99.9 Latency Under High Allocation Spikes
300ms ──┐ (Java 17 G1GC: Pause Spikes up to 285ms)
│ ▄
200ms │ █ ▄ ▄
│ █ █ ▄ █
100ms │ █ █ █ █ █
50ms ──┼─────────────────────────────────────── (Java 25 Generational ZGC: Held at ~50ms)
0ms └───────────────────────────────────────
By keeping maximum GC pause times below 1 millisecond, Generational ZGC gave us deterministic sub-60ms p99.9 latencies even during aggressive promotion allocation bursts.
6. Code Quality & SonarQube Grade A Compliance
Upgrading 16 production services across a major framework version requires rigorous quality gates. In Spring Boot 4 and Java 25, several legacy conventions are deprecated:
- Legacy
javax.*packages are gone; 100% Jakarta EE namespace migration. - Modern record patterns and pattern matching for
switchreplace verbose instanceof casts. - Structured concurrency (
StructuredTaskScope) replaces raw uncoordinatedCompletableFuturechains.
We enforced our internal SonarQube Quality Gate rules across all repositories:
- 0 Blocker or Critical Bugs.
- 0 Security Hotspots.
- >85% Unit and Integration Test Coverage (validated with Testcontainers and WireMock).
- A-Grade Maintainability, Reliability, and Security ratings.
// Example: Modern Java 25 Pattern Matching & Structured Concurrency
public CouponValidationResult validateBasket(Basket basket) {
try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
Supplier<UserQuota> quotaSubtask = scope.fork(() -> quotaClient.fetchQuota(basket.userId()));
Supplier<List<Rule>> rulesSubtask = scope.fork(() -> ruleEngine.fetchApplicableRules(basket.items()));
scope.join(); // Join subtasks in parallel
scope.throwIfFailed(); // Propagate first exception if any
UserQuota quota = quotaSubtask.get();
List<Rule> rules = rulesSubtask.get();
return switch (quota.status()) {
case ACTIVE when quota.remainingLimit() > 0 -> new CouponValidationResult.Approved(rules);
case EXHAUSTED -> new CouponValidationResult.Rejected("Campaign quota depleted");
case SUSPENDED -> new CouponValidationResult.Suspended("Account verification required");
};
} catch (ExecutionException | InterruptedException e) {
log.error("Failed parallel basket validation", e);
return new CouponValidationResult.Error("Service temporarily degraded");
}
}
Summary Metrics: Java 17 Platform vs Java 25 Virtual Threads
Here is the consolidated production benchmark across our 16 checkout microservices:
Benchmark Metric │ Java 17 + Spring Boot 3 │ Java 25 + Spring Boot 4 │ Delta
──────────────────────────────────┼─────────────────────────┼─────────────────────────┼───────────
Max Concurrent Requests / Pod │ 650 │ 12,000 │ +17.4x
Thread Memory Overhead / Pod │ 650 MB │ 38 MB │ -94.1%
p99 Read Latency │ 78 ms │ 39 ms │ -50.0%
p99.9 Read Latency │ 240 ms │ 54 ms │ -77.5%
Max GC Pause Time │ 42 ms │ 0.65 ms │ -98.4%
Carrier Thread Pinning Incidents │ N/A │ 0 (Grade A Monitored) │ Clean
Kubernetes Pod Count for Peak │ 180 Pods │ 95 Pods │ -47.2%
The Staff Engineer Checklist for Virtual Threads
If your team is planning to adopt Java 25 and Spring Boot 4 for mission-critical services, follow these engineering guidelines:
- Verify Your Dependencies for
synchronizedBlocks: Run-Djdk.tracePinnedThreads=fullin staging under stress. Any third-party library that usessynchronizedaround socket I/O will pin your carrier threads and freeze your JVM. - Never Scale HikariCP Linearly with Virtual Threads: Virtual threads decouple concurrent requests from OS threads, but your database is still bounded by CPU and disk IOPS. Guard connection pools with Semaphores.
- Adopt Scoped Values Over
ThreadLocal: Eliminate memory leaks and cut heap allocation overhead by binding request context to lexical scopes. - Pair Virtual Threads with Generational ZGC: The massive volume of short-lived virtual thread objects demands a concurrent collector that eliminates stop-the-world pauses.
- Embrace Structured Concurrency: Replace fragile
CompletableFuturechains withStructuredTaskScopeto maintain clear parent-child thread lifecycles and clean cancellation semantics.