Correct Java Micro-Benchmarks with JMH

A reliable Java micro-benchmark measures a very small piece of code while accounting for JVM behavior such as warm-up, JIT compilation, garbage collection, and compiler optimizations. By the end, you will know why JMH is the standard tool for this job, how to choose benchmark modes, and why hand-written stopwatch loops usually produce misleading results.

A micro-benchmark measures the cost of a small operation: for example, parsing one value, looking up one map entry, or calculating a hash.

Java makes this harder than it first appears because code does not always run the same way throughout a program's lifetime:

• Java source is compiled to bytecode first.
• The JVM initially interprets or lightly compiles code.
• The Just-In-Time (JIT) compiler detects frequently executed code and optimizes it while the program runs.
• The optimizer may inline methods, remove unused calculations, eliminate allocations, or specialize code based on observed values.
• Garbage collection, CPU scheduling, cache state, and other running processes can add noise.

A loop timed with System.nanoTime() often measures a mixture of application work and these effects. It can also accidentally let the JIT remove the work being measured.

JMH (Java Microbenchmark Harness) is the standard Java tool for micro-benchmarks. It is designed by OpenJDK contributors and handles important details such as:

• Warm-up iterations before measurement
• Multiple measurement iterations
• Separate JVM forks
• Result consumption to reduce dead-code elimination
• Statistical reporting
• Benchmark modes such as throughput and average time

A benchmark is useful only when it models a meaningful operation and prevents the JVM from optimizing away that operation.

Think of the JVM as a kitchen that learns its busiest orders.

The first few times you order a dish, the cook follows the recipe slowly. After seeing the same dish many times, the cook prepares ingredients in advance and removes unnecessary steps. That is similar to JIT compilation and optimization.

If you start a stopwatch on the first order, you measure training and setup rather than normal service. If you ask the cook to prepare food but throw it away without checking it, the cook may decide not to make it at all. That is similar to dead-code elimination.

JMH lets the kitchen practice first, checks that the dish is actually produced, and then records several normal-service measurements.

JMH benchmarks are methods annotated with @Benchmark.

import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;

@State(Scope.Thread)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class StringLengthBenchmark {

    private String text;

    @org.openjdk.jmh.annotations.Setup
    public void setUp() {
        text = "microbenchmark";
    }

    @Benchmark
    public int stringLength() {
        return text.length();
    }
}

What each important part does:

• @Benchmark marks the method JMH should measure.
• @State(Scope.Thread) gives each benchmark thread its own instance and fields.
• @Setup prepares data outside the measured method.
• Returning int makes the result observable, helping prevent the calculation from being discarded.
• Mode.AverageTime reports average time per operation.
• @OutputTimeUnit(TimeUnit.NANOSECONDS) requests a readable unit for this tiny operation.

For a benchmark that creates an object, use JMH's Blackhole if returning the value is not appropriate:

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.infra.Blackhole;

public class AllocationBenchmark {

    @Benchmark
    public void createMessage(Blackhole blackhole) {
        String message = new String("hello");
        blackhole.consume(message);
    }
}

Blackhole.consume tells JMH that the value matters without forcing you to redesign the benchmark just to return it.

Consider this benchmark:

import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Setup;

@State(Scope.Thread)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class ArraySumBenchmark {
    private int[] numbers;

    @Setup
    public void setUp() {
        numbers = new int[] {3, 7, 11, 13};
    }

    @Benchmark
    public int sumNumbers() {
        int sum = 0;
        for (int number : numbers) {
            sum += number;
        }
        return sum;
    }
}

When JMH runs it:

1. JMH creates benchmark state. With Scope.Thread, each worker thread gets its own ArraySumBenchmark object.
2. JMH calls setUp() before the benchmark trial, so array creation is not included in the timed sumNumbers() operation.
3. JMH performs warm-up iterations. During this stage, the JVM can profile and optimize sumNumbers().
4. JMH performs measurement iterations. Each iteration invokes sumNumbers() many times, making timer resolution less significant.
5. The returned sum is used by the harness, so the JVM cannot simply treat the entire loop as irrelevant.
6. JMH reports measurements across iterations, including an average and error information rather than one lucky or unlucky stopwatch reading.

Do not interpret a result such as 5 ns/op as a permanent universal fact. It describes that benchmark configuration on that machine, JVM, and environment.

Micro-benchmarks are useful when you have a focused performance question and a realistic workload.

• Data processing: Compare two ways to parse timestamps or convert records in a high-volume import job.
• API services: Measure JSON serialization, validation, token parsing, or a frequently called cache-key function.
• Collections: Compare HashMap, EnumMap, or an array lookup when the key type and access pattern are known.
• Database-adjacent code: Measure object mapping or SQL-string construction separately from actual network and database latency.
• Libraries: Verify whether a change improves a hot method without relying on a full application's noisy performance test.
• Regression prevention: Keep representative JMH benchmarks to detect substantial slowdowns after dependency or implementation changes.

A micro-benchmark does not replace load testing. A web service's real latency also includes networking, database calls, lock contention, request queues, and deployment configuration.

In production codebases, developers usually benchmark a candidate decision, not random snippets.

Benchmark a realistic input

Use @Param to test input sizes or strategies without editing source code each time:

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.Param;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;

@State(Scope.Thread)
public class SearchBenchmark {
    @Param({"10", "1000", "100000"})
    public int size;

    private int[] values;

    @org.openjdk.jmh.annotations.Setup
    public void setUp() {
        values = new int[size];
        for (int i = 0; i < size; i++) {
            values[i] = i;
        }
    }

    @Benchmark
    public int findLastValue() {
        for (int i = 0; i < values.length; i++) {
            if (values[i] == values.length - 1) {
                return i;
            }
        }
        return -1;
    }
}

Keep setup out of the operation when appropriate

If production code receives an already-created String, create that string in @Setup; do not include string construction unless construction is part of the question.

Include setup when it is part of real work

If a real request must create a parser and parse a document, benchmark the whole request-level operation too. Separating costs is useful, but excluding required costs gives an unrealistic result.

Use forks for JVM isolation

JMH forks separate JVM processes by default in typical configurations. Forks help prevent one benchmark's compilation profile and state from affecting another benchmark. Do not disable forks merely to make a benchmark run faster.

Treat results as evidence, then validate

Use JMH to select a promising implementation. Then test the full application with representative traffic and data, because a faster micro-operation may not improve end-to-end performance.

Timing a single call

long start = System.nanoTime();
int result = "hello".length();
long elapsed = System.nanoTime() - start;

This is not a useful micro-benchmark. The operation is far smaller than normal timing noise, and the first call may not represent optimized execution.

Avoid it: use JMH, which repeats operations and records multiple measurement samples.

Forgetting warm-up

for (int i = 0; i < 1_000_000; i++) {
    work();
}

Timing this entire loop combines early, unoptimized execution with later optimized execution.

Avoid it: let JMH warm up first. Do not guess a fixed warm-up count in a home-made harness.

Letting work disappear

@Benchmark
public void brokenBenchmark() {
    "hello".toUpperCase();
}

Because the result is unused, the JVM may be able to eliminate or drastically simplify the operation.

Avoid it: return the result or consume it with Blackhole.

@Benchmark
public String validBenchmark() {
    return text.toUpperCase();
}

Benchmarking a compile-time constant

@Benchmark
public int brokenConstantBenchmark() {
    return 2 + 2;
}

The compiler can replace this with 4; it does not measure arithmetic work.

Avoid it: use state prepared at runtime and choose inputs that reflect actual use.

Including unrelated work

@Benchmark
public String parseAndCreateInput() {
    String json = loadJsonFromDisk();
    return parse(json);
}

Disk I/O dominates parsing, so this cannot answer a parsing-only question.

Avoid it: prepare input in @Setup when measuring parsing alone. Create another benchmark if you want to measure the full workflow.

Assuming nanoseconds are perfectly accurate

System.nanoTime() has nanosecond-oriented units, but that does not mean every reading has nanosecond precision or that a 1 ns difference is meaningful.

Avoid it: compare distributions and error ranges, not just the smallest displayed number.

Running on a busy or changing machine

Downloads, IDE indexing, power-saving behavior, containers with changing CPU limits, and other processes add noise.

Avoid it: run repeated measurements in a reasonably stable environment and investigate only meaningful, repeatable differences.

Time per iteration or iterations per time?

These are two views of the same basic rate:

• Average time answers: “How long does one operation take?”
• Throughput answers: “How many operations fit in one second?”

For a stable single-threaded operation, they are approximately reciprocal. For example, 2,000,000 ops/s is roughly 500 ns/op. They can tell a different practical story when concurrency, batching, pauses, or latency variation matter.

Use the measure that matches the decision:

• Choose average time for a latency-sensitive operation on a request path.
• Choose throughput for a worker, encoder, queue consumer, or bulk-processing task.
• Use sample time when tail latency is important.

JMH controls how work is repeated during an iteration, so you should not manually calculate a per-iteration value from one timer reading.

• Use JMH for Java micro-benchmarks; avoid hand-written stopwatch loops for small operations.
• Put the measured operation in a method annotated with @Benchmark.
• Use @State(Scope.Thread) for mutable per-thread benchmark data.
• Use @Setup to prepare inputs outside the measured operation.
• Return a result or use Blackhole.consume(value) so work is not optimized away.
• Use @Param to test realistic input sizes and options.
• Use Mode.AverageTime for time per operation.
• Use Mode.Throughput for operations per unit of time.
• Use Mode.SampleTime when latency distribution matters.
• Keep JVM forks enabled unless you have a specific, documented reason not to.
• Benchmark the exact question: include required production work, exclude unrelated work.
• Compare repeatable differences, not tiny changes within noise.

@Benchmark
public int operation() {
    return input.hashCode();
}

@Benchmark
public void operationWithNoReturn(org.openjdk.jmh.infra.Blackhole blackhole) {
    blackhole.consume(createValue());
}

What is JMH in Java?

JMH is the Java Microbenchmark Harness. It is a framework for running reliable micro-benchmarks while accounting for JVM warm-up, optimization, and measurement issues.

Why is System.nanoTime() not enough for a Java micro-benchmark?

System.nanoTime() is appropriate for elapsed-time measurement, but a manual benchmark usually fails to handle JIT warm-up, dead-code elimination, multiple samples, forks, and other JVM effects. It is more suitable for coarse timing of larger operations.

Should I measure nanoseconds per operation or operations per second?

Use nanoseconds per operation (AverageTime) when you care about latency of one call. Use operations per second (Throughput) when you care about processing capacity. They are often approximately reciprocal for stable work.

Why does JMH need warm-up iterations?

The JVM optimizes frequently executed code during runtime. Warm-up lets compilation and optimization settle before JMH records the main measurements.

Why should a JMH benchmark return a value?

If a calculation has no observable result, the JVM may remove it as unnecessary. Returning the value, or consuming it through Blackhole, helps ensure the intended work remains observable.

Can I benchmark code that allocates objects?

Yes. Use realistic inputs and consume or return the result. Be aware that escape analysis may remove allocations that cannot be observed; this can be correct if it matches how the code is used in the real application.

Can I run JMH benchmarks from my IDE?

You can, but command-line runs in a stable environment are usually more reproducible. Ensure the benchmark is launched through JMH rather than by directly calling a benchmark method.

Does a faster JMH result guarantee a faster application?

No. It proves only that the measured operation was faster under the benchmark conditions. Confirm important decisions with integration, load, or production-like performance tests.

• JIT compilation — Explains why Java code changes performance characteristics after it has run repeatedly.
• Dead-code elimination — Explains why unused benchmark results can invalidate a measurement.
• Garbage collection — Allocation and collection behavior can affect benchmark results and application latency.
• Big O notation — Helps predict how algorithms scale, while micro-benchmarks measure actual constants and runtime behavior.
• Java profiling — Profilers identify where an application spends time; micro-benchmarks compare focused implementation choices.
• Load testing — Measures end-to-end system behavior under realistic concurrency and traffic.
• Escape analysis — A JVM optimization that can remove or transform allocations, especially relevant in allocation benchmarks.
• Concurrency in Java — Required when benchmark results depend on contention, shared state, or multi-threaded throughput.