Inference economics have become the main constraint on LLM team design. Serving bills now rival or exceed training spend for many production teams, and hiring priorities have followed the money. The engineer who can shave latency and cost per request is more valuable than the one who can only fine-tune. This piece lays out what that shift means for engineers choosing a specialisation and for managers writing job descriptions in 2026.

The New Cost Curve Tension

Unit economics now drive hiring decisions in LLM teams. The mechanism is easiest to see in long-context chat: a request with a 100k-token prompt forces the server to hold a proportionally larger KV cache in GPU memory, and memory is what caps how many requests fit on a card at once. Halve the free memory and you roughly halve concurrency, which doubles the GPU-hours needed to serve the same traffic. Finance teams have started asking engineering for per-feature spend because the bill scales with prompt length and concurrency rather than with anything visible to the user. Roles then split by who owns that number: training infrastructure, serving optimisation, and product-side cost accountability.

Inference bills rival training budgets at many companies, though the ratio varies by workload. A chat product with high concurrency and long context can burn more on GPUs per month than the initial fine-tuning run. Managers who once hired generalist ML engineers now look for people who can reason about batching, memory bandwidth, and token budgets.

This site has argued that KV cache reuse shifts serving costs toward memory bandwidth, and that framing shows up in hiring. Candidates who understand why memory bandwidth, not raw FLOPs, limits throughput are easier to place.

Why Inference Economics Changed the Job

Serving costs often exceed training costs for production LLM features. Training is a one-time capital expense; inference is a recurring operational one. A team that trains once and serves millions of requests pays the inference bill every month, and that recurring cost lands on the engineering budget. The job description now includes words like quantisation, caching, and batching because those are the levers that move the monthly bill.

Latency and throughput have become product features. Users notice when a response takes three seconds instead of one, and product managers notice when the GPU bill doubles after a launch. Engineers who can hold both constraints, a latency budget and a cost ceiling, are scarce. Hiring loops test for that dual awareness. A candidate who talks only about model accuracy without mentioning serving cost reads as junior for these roles.

Quantisation and caching enter job descriptions directly. A role that once asked for PyTorch experience now asks for experience with 8-bit or 4-bit inference, paged attention, and prefix caching. These are not research topics; they are operational skills. The shift is visible in compensation bands too, where serving specialists command premiums over generalist ML engineers at many companies.

What Optimisation Engineers Actually Get

Optimisation engineers get deep ownership of the serving stack. They choose the inference framework, set the batching policy, and decide when to quantise. That ownership is broader than it sounds. Move a model from FP16 to 8-bit weights and the memory footprint roughly halves, which can let the same card hold twice the batch or serve a longer context; the trade is a small accuracy drop the team has to measure against its eval set. A single batching decision can change both latency and cost by a large margin, and the engineer sees the result on a dashboard within hours.

Direct impact on gross margins is the second reward. A serving optimisation that cuts cost per request by a meaningful fraction shows up in the company's margin, not just in a model card. That visibility is rare for infrastructure work. It also means the engineer is in the room when product and finance discuss pricing, which some engineers love and others find distracting.

Scarce skills command salary premiums. As of late 2025, serving optimisation experience is still less common than general ML experience, and companies pay for it. The premium is not uniform; it depends on the candidate's ability to show measured cost reductions. A portfolio with a before-and-after cost chart is worth more than a list of frameworks.

The Trade-offs Behind the Specialisation

The scope is narrower than a generalist ML role. An optimisation engineer may spend months on the serving path and never touch training data or model architecture, and the research problems that attract many people in LLMs live in training and post-training rather than in the serving loop. Some teams mitigate this by rotating engineers through both, but rotation is rare when serving cost is the constraint, because the person who owns the p95 latency graph is hard to spare. A candidate who wants to lead model research should think carefully before committing to serving.

On-call pressure for cost regressions is the other cost. A bad deploy that doubles token usage or breaks caching can page the engineer at night. This site has covered how training budgets stall while maintainers stay on call, and the same dynamic applies to serving. Career paths also tie to infrastructure bets: if the company switches serving frameworks or cloud providers, the specialist's leverage can shift quickly.

Hiring Signals That Predict Cost Sense

Candidates who cite concrete latency budgets stand out. A strong answer names a target, such as a p95 latency under a second, and explains what they traded to hit it. Weak answers stay abstract. The interviewer is listening for the word budget and for a number attached to it. That single habit separates engineers who have owned a serving path from those who have only read about one.

Experience with batching and KV-cache tuning is a strong signal, provided the candidate can describe the mechanics. Larger batch sizes raise GPU utilisation and cut cost per token, but they also queue requests, so p95 latency climbs while throughput looks better on the dashboard. Paged attention reduces memory fragmentation at the cost of an extra lookup on every attention step. A candidate who can explain why a batch of 64 is cheaper per token but worse for a chat UI has likely debugged it in production. Familiarity with GPU memory constraints matters too: engineers who know why a model fits on one card but not another can reason about cost before writing code.

A portfolio that shows cost-per-request reasoning is the best predictor. A side project with a cost dashboard, even a simple one, demonstrates that the engineer thinks in unit economics. Managers should ask for a specific example of a cost regression and how it was found. Candidates who can walk through the detection and the fix are the ones worth hiring.

Practical Moves for Engineers and Hiring Managers

Engineers should build a cost dashboard side project. Instrument a small open-source model, log tokens per request, and compute cost per thousand requests. The exercise teaches where the money goes. It also produces a concrete artifact to discuss in interviews, which is more persuasive than a certificate.

Learn one serving framework deeply rather than skimming five. Pick vLLM, TensorRT-LLM, or another widely used option and understand its batching and caching internals. Depth in one framework transfers better than shallow familiarity with many. The goal is to explain trade-offs, not to memorise APIs.

Managers should interview for trade-off articulation. Ask candidates to describe a time they chose latency over throughput, or cost over quality, and what happened next. Pair an infrastructure interviewer with a product interviewer so both perspectives are tested. A candidate who can satisfy both is the one who will own the cost curve.

Both engineers and managers should track inference spend per feature. A simple monthly review of cost by endpoint surfaces regressions before they compound. This site's piece on build minutes as a real line item makes a similar point about hidden operational costs.

When the Serving Bet Loses

The case for a dedicated optimisation hire rests on serving cost being the binding constraint. Plenty of teams are not in that position. A startup that rents inference from a hosted API pays per token and has little control over batching or memory layout; its cost problem is a procurement and prompt-design problem, not a CUDA problem. A product whose differentiation is model quality, where a few points of accuracy decide whether customers stay, may rationally spend more per request rather than less. In both cases a serving specialist has less leverage than the job description implies, and the hire can stall once the obvious wins are gone. Managers should check which constraint actually binds before writing the requisition, and engineers should ask the same question before taking the role.