Skip to content

Just-in-Time Compilation

Runtime tool — instantiates Demand-Triggered Deferred Evaluation

Defers translating code into fast native form until running demand — and repeated use — proves a path hot enough to repay the compilation.

A just-in-time (JIT) compiler leaves code in a cheap-to-start form — bytecode, or an interpreter's own representation — and compiles a routine to optimized native machine code only when execution actually reaches it and reaches it often enough to matter. What makes it THIS mechanism rather than a generic "compile lazily" is the second trigger: demand alone is not enough, so the JIT waits for heat. It counts invocations and loop iterations and spends the expensive optimization budget only on the small fraction of code that runs hot, keeps the compiled output in a cache, and discards it when the runtime assumptions it bet on stop holding. Ahead-of-time compilation pays for every path up front; the JIT pays per path, when and only when a path proves it will repay the cost.

Example

A server runs a nightly analytics job that scans tens of millions of records. At startup none of the job's code is compiled — the inner scoring loop runs interpreted, perhaps an order of magnitude slower than native. As the loop repeats, an invocation counter climbs; once it crosses the compile threshold the runtime marks the method hot and hands it to the optimizing compiler in the background while interpretation continues. A few hundred milliseconds later the optimized native version is installed in the code cache and later iterations jump to it. The compiler even speculated that one polymorphic call site only ever sees a single concrete type and inlined it — until, minutes in, a record of a different type arrives, the speculation fails, and the runtime deoptimizes: it throws away that compiled version, drops back to the interpreter, and (if the path stays hot) recompiles without the bad assumption.

The payoff is the shape of the whole run: the first seconds are slow (warmup), a handful of hot methods get compiled, and the cold setup-and-teardown code that runs once is never compiled at all — its native version would cost more to produce than the microseconds it would save.

How it works

  • Count, don't guess. Instrument invocations and loop back-edges; a method becomes a candidate only when its counter crosses a threshold, so demand is measured as frequency, not first touch.
  • Compile in tiers. Cheap, fast-to-produce code first; the expensive fully-optimizing pass reserved for the hottest methods, so warmup cost scales with how much a path is actually used.[1]
  • Cache the native code keyed to the method and its speculative assumptions, and reuse it on every later call.
  • Deoptimize on violation. When a speculative assumption is contradicted at runtime, discard the compiled version and fall back to the interpreter rather than run wrong-fast code.

Tuning parameters

  • Compile threshold — how many invocations/iterations count as "hot." Lower compiles sooner (less time interpreted, but more compile time and cache spent on paths that may not repay it); higher delays warmup but wastes less on lukewarm code.
  • Optimization tier depth — how aggressively to optimize a hot method. Deeper optimization runs faster but costs more compile time and deoptimizes more often when speculations break.
  • Code-cache budget — how much memory compiled code may occupy. Too small and hot methods get evicted and recompiled (thrash); too large and memory sits under rarely-run natives.
  • Speculation aggressiveness — how boldly to bet on observed types and branches. Bolder bets run faster when they hold and cost a full deopt-and-recompile when they don't.
  • Background vs. blocking compile — whether execution waits for the compiler or keeps interpreting while it works. Background hides warmup latency at the cost of running slow a little longer.

When it helps, and when it misleads

Its strength is that it spends optimization effort where a whole-program compiler cannot see to spend it: on the paths runtime data proves are hot, with type and branch information no ahead-of-time compiler has. Long-running workloads with a small hot core — servers, data jobs, language runtimes — get most of native's speed while paying to compile only a sliver of the code.

Its costs are warmup and unpredictability. Short-lived programs may exit before the JIT ever pays back its compilation cost — the whole run is warmup — which is exactly why ahead-of-time compilation still wins for CLI tools and cold-start-sensitive functions. The heat counters can also be fooled: a benchmark that exercises one path leads the JIT to optimize for a shape the real workload never takes, and the tidy "it gets fast eventually" story hides tail-latency spikes at each compile and deopt. The discipline that keeps it honest is to measure steady-state performance separately from warmup, and to treat a path's compiled speed as contingent on its speculations holding rather than as a fixed property.[2]

How it implements the components

Just-in-Time Compilation fills the heat-triggered-production side of the archetype — the parts a runtime compiler operates, not the whole contract:

  • activation_rule — the hotness threshold: compile a method only once its invocation/loop counter crosses the bar.
  • memoized_result_store — the code cache holding compiled native routines, keyed to the method and reused on later calls.
  • invalidation_rule — deoptimization: discard and recompile a native version when the assumptions it was compiled under are violated.
  • latency_and_waste_budget — the warmup-vs-payoff trade the thresholds encode: don't spend compile time on code too cold to repay it.

It does not model the deferred value itself or its force semantics — that is Thunk or Lazy Promise — nor keep the general-purpose result cache keyed by inputs, which is Memoization.

References

[1] Tiered compilation — real runtimes such as the HotSpot JVM compile in stages (interpreter → a quick baseline compiler → a fully optimizing compiler), promoting a method to the next tier only as it gets hotter. It is the standard way to keep warmup cost proportional to how much a path is used. ↩

[2] Deoptimization — a speculative JIT's ability to abandon optimized native code and resume in the interpreter when a runtime assumption is violated. It is what makes aggressive speculation safe, and why compiled speed is contingent, not guaranteed. ↩