🔴 The Error You're Seeing
Confirm this matches your console output. If it does, you're in the right place.
java.util.concurrent.ExecutionException: java.sql.SQLException: no such row
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:124)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:193)
at java.base/java.util.concurrent.AbstractExecutorService.doInvokeAny(AbstractExecutorService.java:207)
at java.base/java.util.concurrent.AbstractExecutorService.invokeAny(AbstractExecutorService.java:236)
at com.devinhyderabad.failover.HedgedReader.fastestReplica(HedgedReader.java:52)
Caused by: java.sql.SQLException: no such row
at com.devinhyderabad.replica.ReplicaClient.lookup(ReplicaClient.java:88)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:545)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)⚡ Quick Fix Works 80% of the time
Record each task’s failure inside its own Callable before rethrowing — invokeAny discards everything except one representative cause when all replicas fail.
List<Throwable> failures = Collections.synchronizedList(new ArrayList<>());
try {
return executor.invokeAny(replicas.stream()
.map(r -> (Callable<Row>) () -> {
try {
return r.lookup(key);
} catch (Exception e) {
failures.add(e); // preserve what invokeAny drops
throw e;
}
}).toList());
} catch (ExecutionException e) {
failures.add(e.getCause()); // the one cause invokeAny kept
throw new AllReplicasFailed(key, failures); // checked-safe domain wrapper
}🧠 Why this Happens
Tap to expand the deep technical explanation
doInvokeAny races the submitted tasks through a completion queue and returns the first successful result immediately, cancelling stragglers. Each failing future deposits its exception into an internal ee holder that keeps being overwritten by later failures; only when the last task fails does the loop rethrow an ExecutionException wrapping whichever cause happened to be stored last — verified in jdk8u and jdk21u sources, where the assignment ee = ex is unconditional, the inline comment reads "throw the last exception we got", and the nearby if (ee == null) merely fabricates a fallback when nothing failed. Earlier causes never reach your log unless you capture them inside the tasks themselves.
The HITEC City Parking Spot Analogy:
Asking three friends for concert tickets and taking whoever texts back first. If all three come back empty-handed, they hand you one combined shrug — you never learn who was sold out versus who never left the house unless each friend kept their own note.
🔁 How to Reproduce Confirm this is your error
Submit two Callables to invokeAny where both throw immediately (one IOException, one SQLException): the EE wraps whichever task failed LAST — doInvokeAny overwrites its holder unconditionally, so earlier causes vanish. Rerun and the wrapped type can flip; only one failure ever survives. (Lab capture: OpenJDK Temurin 25.0.2 — SQLException won that run.)
🛠️ Solutions (5 Ways to Fix)
Capture per-task failures before they are discarded
👉 Use this when/if diagnosis of which replica failed matters operationally.
Because invokeAny keeps only one cause, wrap every callable body so failures append to a shared list on their way out. After the aggregate exception you hold the complete failure set — essential for telling a shared-database outage apart from one bad replica.
Callable<Row> instrumented(Replica r) {
return () -> {
try {
return r.lookup(key);
} catch (Exception e) {
failures.add(e);
throw e;
}
};
}Switch to invokeAll when every outcome matters
👉 Use this when/if partial success plus a full error report beats fastest-wins.
invokeAll waits for all futures and never discards causes: iterate the returned list, collect successes, and inspect each get() individually. Slightly slower than racing, but observability becomes structural instead of bolted on.
List<Future<Row>> all = executor.invokeAll(callables);
for (Future<Row> f : all) {
try {
return f.get(); // first success wins here too
} catch (ExecutionException e) {
log.warn("replica failed", e.getCause());
}
}
throw new AllReplicasFailed(failures);Use the timed overload and handle its distinct TimeoutException
👉 Use this when/if hedging must respect a hard latency budget.
invokeAny(tasks, timeout, unit) throws a bare TimeoutException when nothing succeeds within budget — a different class from the all-failed ExecutionException, so handlers can distinguish too-slow from all-broken without unwrapping. Outstanding tasks are cancelled automatically on either exit path.
try {
return executor.invokeAny(callables, 250, TimeUnit.MILLISECONDS);
} catch (TimeoutException e) {
metrics.increment("hedge.slow"); // nobody finished in time
return cachedValue();
}Hedge with independent, idempotent backends
👉 Use this when/if identical causes reveal a shared dependency rather than flaky replicas.
If every wrapped cause is connection-refused, the problem is the shared database, not replication lag. Hedging pays off only when tasks fail independently; verify idempotency of lookups and diversity of backing stores, or you pay double load for correlated failure.
if (failures.stream().map(Throwable::getClass).distinct().count() == 1
&& failures.size() == replicas.size()) {
page.onCall("shared-dependency down: " + failures.get(0));
}DEV ONLY: resubmit the whole invokeAny in a tight retry loop
👉 Use this only to measure how fast retries melt a struggling cluster — never ship it.
Blind resubmission multiplies in-flight load precisely when the backend is least able to bear it, and because earlier causes were already dropped, each round hides more evidence. Retry decisions belong after cause analysis with backoff.
// ANTI-PATTERN — do not ship
while (!done) {
try { row = executor.invokeAny(callables); done = true; }
catch (ExecutionException e) { /* instant resubmit */ }
}📋 Version Notes
Same last-cause-only semantics from doInvokeAny; frame names identical.
Unchanged; timed overload still throws bare TimeoutException on budget expiry.
Hedging via virtual threads makes per-replica cost negligible, raising the value of per-task failure capture.
🛡️ How to Prevent This Next Time
Wrap hedged callables with failure-recording middleware once, centrally; require idempotency checks for any task eligible for racing; monitor distinct-cause counts so correlated failures trigger dependency alerts instead of replica retries.