What determines the trade-off between local and hosted language-model inference?

Memory, serving architecture, workload and the measurement of deployment resources

This retrospective review considers source versions available by 2 April 2026. The cutoff defines the evidence included; preparation and source checking took place on 21 October 2026.

Keywords: local inference; hosted inference; memory; quantisation; serving systems; energy

1. The deployment question

Running an AI model locally changes where computation occurs. It does not, by itself, establish which model is available, whether the model meets the task's quality requirements or how efficiently it will run. A hosted service similarly describes an operating arrangement rather than a single capability level. Comparing local with hosted inference requires two separate comparisons: the quality of the candidate configurations and the resources needed to deliver that quality under the intended workload.

This review examines the technical evidence relevant to that comparison. It focuses on memory, context processing, generation, concurrency and energy accounting. Marketing is the application setting because its workloads range from occasional interactive drafting to large batches of classification or extraction. These requests place different demands on a serving system. The same hardware and model can appear adequate under one workload and inadequate under another without any change in the underlying weights.

The question is consequently more precise than whether a laptop can run a language model. A model may load successfully but respond too slowly, support too little context or consume enough memory to interfere with other work. Conversely, a modest local model may meet a narrow task's requirements with an acceptable response time. The research does not supply a universal break-even point. It supplies mechanisms and measurement distinctions needed to estimate one for a defined deployment.

2. Review method and comparison boundary

Primary-source retrieval on 21 October 2026 covered transformer inference, low-bit quantisation, memory management, attention kernels, speculative decoding and energy measurement. Fourteen references were selected. The accessible PagedAttention and July 2024 AWQ manuscripts were inspected for their architectural and workload discussions. Other research claims are limited to author abstracts and publication records. This is a narrative technical review, with no reproduced hardware benchmark or current supplier price comparison.

The comparison boundary is inference, while training and adaptation are discussed where needed to prevent accounting errors. No particular computer, accelerator or hosted endpoint was tested. Reported research speedups remain attached to the models, baselines and workloads in the underlying studies. Derived storage arithmetic is explicitly labelled as arithmetic and excludes runtime overhead. The review does not convert a theoretical reduction in weight bytes into an unmeasured latency, energy or financial saving.

3. What occupies memory?

3.1 Persistent weights

Transformer models use learned parameters in attention and other layers [1]. At inference, the weights must be available to the runtime, whether in accelerator memory, system memory or a combination of storage levels. For a dense model, a first approximation to raw weight storage is the number of parameters multiplied by the bits allocated to each parameter, divided by eight. The approximation describes the weight values only. Additional metadata, scales, temporary buffers and application resources are outside it.

For an illustrative dense model with seven billion parameters, that arithmetic yields 14 decimal gigabytes at sixteen bits per parameter, seven gigabytes at eight bits and 3.5 gigabytes at four bits. These are derived quantities, not observed footprints of a released checkpoint. A runtime can require more memory because of quantisation metadata, uncompressed components, caching and allocation behaviour. Confusing raw storage with the memory needed to serve a request can make a configuration appear feasible before it is actually tested.

3.2 Request-dependent state

Autoregressive generation conditions each new token on the prompt and preceding output. Serving implementations can retain attention keys and values to avoid recomputing them. This key-value cache is associated with active sequences rather than simply with the model artifact. Longer contexts and more simultaneous sequences can therefore increase working memory even when the weight file remains unchanged. A benchmark that reports only checkpoint size omits a central part of the serving workload [5].

The exact cache requirement depends on architecture as well as sequence length and precision. It should not be inferred from a single universal number per parameter. The operational question is how many active requests, at which lengths, fit under the memory limit while preserving acceptable latency. The answer also depends on whether prompts share prefixes, whether outputs branch and how the serving engine allocates memory. These are configuration properties that a statement such as runs on one GPU leaves unspecified.

4. Lower precision and usable performance

4.1 Eight-bit and lower-bit representations

LLM.int8 develops a mixed-precision procedure that handles outlier features while using eight-bit multiplication for most values. The authors report large inference-memory reductions while retaining evaluated performance [2]. The method shows why precision reduction is more involved than rounding every value identically. The numerical distribution of activations and weights affects error. Its historical results support a specific representation and implementation rather than a guarantee for every model converted to eight bits.

GPTQ uses approximate second-order information for post-training weight quantisation and reports strong quality retention at low bit widths in its evaluated models [3]. A compressed model can fit within a different memory envelope from its higher-precision counterpart. Whether that produces a useful deployment depends on the available kernels and the task's sensitivity to the resulting approximation. Memory feasibility and output equivalence are separate evaluation outcomes.

The July 2024 AWQ manuscript uses activation statistics to identify salient weight channels and scaling to reduce quantisation error [4]. Its accompanying implementation addresses execution as well as representation. A smaller weight file and a faster service are different observations. Realised speed depends on whether the runtime and hardware use the compressed representation efficiently. No numerical speed forecast for an untested local computer or hosted service is inferred here.

4.2 Quantisation and adaptation are different stages

QLoRA is a fine-tuning method that combines a quantised frozen base with trainable low-rank adapters [11]. Its memory result concerns the adaptation procedure. A comparison of local inference cannot treat that training-memory figure as the memory needed for every final request. The deployed artifact, context length and serving configuration still determine inference demand. Using the same word memory for training and inference hides which tensors must remain available at each stage.

Quality testing must use the final representation. If a model is evaluated at high precision and then quantised for local execution, the test no longer describes exactly the deployed artifact. The same applies when an adapter or different decoding rule is introduced. A defensible comparison holds the task inputs and scoring fixed while evaluating both representations. The purpose is to measure whether the intended resource saving preserves the required output properties, including rare but consequential errors.

5. Prompt processing and token generation

5.1 Two phases of one request

Processing a prompt and generating a continuation place different patterns of work on hardware. The prompt can be processed across its token positions using parallel operations permitted by the architecture. Continuation generation proceeds through successive conditional steps. A long input with a short output therefore differs from a short input with a long output, even when the total token count is similar. The elapsed time cannot be estimated reliably from a single aggregate tokens-per-second figure without knowing which phase that figure describes.

Time to first token captures part of the interactive experience. Time per generated token describes the continuation once it begins. End-to-end completion time includes both, together with queuing, retrieval, transport and application processing. A hosted endpoint may add network delay while using a more efficient serving engine. A local process may avoid that network delay while generating tokens more slowly. Neither component alone determines which arrangement finishes a task sooner.

5.2 Attention and memory movement

FlashAttention reduces memory reads and writes through an IO-aware exact-attention algorithm. Dao and colleagues show that reorganising computation around the memory hierarchy can improve performance without replacing exact attention with an approximate alternative [6]. This provides a useful distinction between algorithmic arithmetic and implementation efficiency. The same mathematical operation can have different practical costs depending on how intermediate values move through memory.

Attention optimisation does not remove every serving bottleneck. A workload can be constrained by weight movement, cache capacity or scheduling rather than by the same operation that dominated a training benchmark. The review therefore keeps phase, batch size and architecture visible when interpreting a speedup. A percentage improvement in one kernel is not an equal percentage improvement in complete application time, because other components remain in the denominator.

6. Concurrency, batching and memory management

6.1 Why simultaneous use changes the comparison

Batching can improve hardware utilisation by processing work from several requests together. It also introduces a scheduling problem: requests arrive at different times, have different prompt lengths and finish after different numbers of generated tokens. A system optimised for high aggregate throughput may permit more queuing than an interactive user accepts. The workload definition must therefore state whether the target is a single response, a stream of concurrent requests or a deadline for completing a large batch.

The PagedAttention system divides key-value caches into fixed-size blocks and maps logical blocks to physical memory. Blocks are allocated as needed, allowing sequences to grow without reserving their maximum size in advance. The full-text description also supports block-level sharing [5]. This is a concrete example of serving software changing the number of requests that can be handled within a memory budget. It improves memory management rather than increasing the model's factual knowledge.

The same paper reports throughput gains relative to its evaluated baselines. Its architectural account is more transferable than any one speedup ratio: allocation and sharing affect how much usable work fits in memory [5]. An organisation assessing a local engine should measure the actual workload rather than assume that the research ratio applies unchanged. A hosted provider may use related techniques internally, but an external user cannot infer its precise serving configuration from the endpoint alone.

6.2 Offloading and workload tolerance

FlexGen investigates high-throughput, latency-insensitive generation with limited GPU memory, using GPU, CPU and disk resources together [14]. The design exploits a different operating objective from a fast single-user conversation. Offloading can make a large model executable on a memory-constrained accelerator while introducing data movement. A claim that a model runs on limited hardware must therefore state whether the result concerns batch throughput or interactive response time.

For marketing, that distinction separates overnight categorisation of a document collection from an assistant waiting inside a live briefing session. Both can be valid uses of local inference. Their acceptance criteria differ. The research does not imply that an architecture suitable for one is appropriate for the other, even when the model and hardware are unchanged. The user-visible objective should determine which performance measure is treated as primary.

7. Speculative decoding and mixed-model systems

Leviathan and colleagues propose speculative decoding, in which a cheaper approximation proposes tokens that a target model verifies. Their method accelerates sampling while preserving the target output distribution under the algorithm's conditions [7]. It provides a different route to efficiency from reducing the target model's precision or replacing it with a weaker model. The relevant evidence concerns a joint system with a draft model, verification procedure and target model.

The benefit depends on the relationship between draft proposals and target acceptance, together with the execution costs of both models. A draft model that is cheap but rarely accepted can add work. A close approximation that is itself expensive can leave little saving. These are consequences of the algorithm's structure, not numerical predictions for a particular pair. Evaluating the complete path is necessary before treating the draft model's speed as a system-level improvement.

This complicates a simple local-versus-cloud taxonomy. A local application could use a draft model and a remotely executed target; a hosted service could use several internal models. The user-visible endpoint does not necessarily reveal the computational arrangement. The comparison should therefore be based on observed task quality, latency, costs and data flows rather than an assumed one-request-one-model implementation.

8. Context capacity and context use

A large supported context window establishes that an interface can accept a long input. It does not establish that the model uses every part of that input equally well. Lost in the Middle reports sensitivity to the position of relevant information in long contexts on the tasks evaluated, with information in the middle often used less effectively [12]. The result concerns specific historical models and tasks, but it identifies a measurement problem that raw context capacity cannot answer.

For a marketing document collection, the input may contain duplicated sections, conflicting dates or a small amount of crucial information among irrelevant text. Expanding the context can increase resource demand while leaving the selection and interpretation problem unresolved. Retrieval is one possible application design, but its effect depends on whether the right material is retrieved and whether the answer remains supported. The present review makes no quantitative claim that one context strategy is universally superior.

A local configuration with a shorter context and a carefully scoped task may therefore be preferable on one evaluation, while a hosted long-context system may perform better on another. That comparison must be made empirically. The technical ability to load an entire report and the semantic ability to synthesise it correctly are different properties. Both belong in the acceptance criteria when document reasoning is the task.

9. Energy, carbon and the accounting denominator

9.1 Inference is a recurring activity

Luccioni, Jernite and Strubell compare the energy and carbon associated with inference across task-specific and general-purpose systems. Their study reports substantial differences between categories on the evaluated tasks [8]. The result demonstrates that parameter count alone does not capture the resources used by a complete inference workload. It does not supply a universal energy value for a prompt, because model, task, output and hardware conditions differ.

Henderson and colleagues propose systematic reporting of energy and carbon footprints and provide a measurement framework [9]. The distinction between energy and carbon is important. The same electricity use can imply different emissions under different accounting assumptions or electricity supplies. A local-versus-hosted comparison that observes one and guesses the other should state that boundary. The review does not estimate either quantity for an untested deployment.

9.2 Training, development and use

Strubell and colleagues examine the financial and environmental costs associated with training and developing NLP models [10]. Those costs are distinct from the repeated cost of serving requests. A comparison that assigns the entire original training run to one local application while excluding it from a hosted option is asymmetric. An inference-only comparison should say so, while a lifecycle comparison needs a defensible allocation rule for shared development and infrastructure.

Adaptation adds another stage. A locally specialised model may require data preparation, training runs and evaluation before serving begins. A hosted general model may avoid that local training expense but incur continuing inference charges. The balance depends on use volume and how long the adapted artifact remains suitable. The literature establishes no fixed number of requests at which adaptation becomes preferable for every organisation.

The denominator can be requests, tokens, completed tasks or accepted outputs. These choices produce different interpretations. A system generating longer answers may use more energy per request without using more per token. A system producing fewer valid outputs may look efficient per generated response while consuming more resources per usable result. Reporting the denominator alongside the measurement is necessary to make a resource comparison intelligible.

10. Constructing an interpretable deployment comparison

10.1 Workload and quality constraints

MLPerf Inference develops benchmarking methods intended to make diverse hardware and software systems comparable under defined rules [13]. Its relevance here is methodological. A deployment comparison needs a representative workload, a quality constraint and a reproducible measurement process. A single demonstration prompt, especially one selected because it ran well, cannot estimate performance over an organisation's actual request distribution.

The workload should retain the properties that drive resource use: input length, output length, concurrent demand and the mix of easy and difficult items. Removing long documents from a test can hide a memory bottleneck. Testing only long documents can exaggerate that bottleneck relative to ordinary use. Stratified reporting exposes those differences without requiring an arbitrary composite score. It also makes clear when a configuration is suitable for only part of the workload.

Quality must be checked under the measured configuration. Changing precision, context truncation or generation limits to improve speed can change the result's correctness. A latency claim obtained by cutting outputs short is not equivalent to completing the original task more quickly. The comparison should report which settings changed and whether the same acceptance requirements remained in force. Otherwise the resource and quality results refer to different tasks.

10.2 End-to-end boundaries

For local inference, model loading can dominate a short session but be amortised across a long-running process. For hosted inference, network transit and queuing can dominate a small request while becoming less important for a long generation. Reporting warm and cold behaviour separately can reveal that difference. These are accounting choices about elapsed time; neither should be omitted merely because it favours one operating arrangement.

The application also performs work outside model generation. Retrieval, file parsing, validation, storage and user-interface updates can contribute to completion time. An engine benchmark isolates some of those costs to study a technical mechanism. A deployment assessment needs to restore the components that the user actually waits for. Maintaining both measurements permits technical diagnosis without presenting a narrow kernel result as the whole experience.

11. Operating control and capability remain separate

Local operation can permit version retention, offline use and direct control of the runtime. Each depends on the actual implementation: a local interface can still call remote services for retrieval, telemetry or selected tasks. A hosted endpoint can supply managed scaling and other service functions, but those functions vary by service and contract. The current review does not infer a provider's guarantees from the research systems discussed.

Operating control also imposes work. The model artifact, runtime dependencies and evaluation set must remain compatible as the application changes. A tested checkpoint may be retained successfully while a changed parser or prompt alters output. This is why a deployment configuration contains more than weights. The capability result should be tied to the complete system used during evaluation, while the operational record identifies which components the organisation can actually retain or replace.

The choice can be hybrid. A narrowly scoped local task can be evaluated independently from a broader hosted synthesis task. The scientific question is then whether the partition preserves task quality and how it changes total resources and data movement. A hybrid arrangement should not be assumed to inherit the best properties of both sides automatically. Routing, transfer and fallback introduce additional steps that need their own measurements.

11.1 Capacity and demand over time

A fixed local installation provides a finite amount of capacity whether or not that capacity is being used. A hosted service can expose a different charging and scaling arrangement, although its actual limits and commitments depend on the service. The comparison therefore changes with the arrival pattern of work. A regular high-volume batch may use retained hardware differently from a team whose demand is concentrated into a few unpredictable periods. No price or utilisation assumption is inserted here because none was measured for an organisation.

Average utilisation can conceal operational peaks. A system with sufficient total capacity over a day can still produce unacceptable queues during a short busy period. Adding capacity may improve that peak while increasing idle resources during the rest of the day. A deployment study should report the request arrival pattern and the service objective together. The objective might concern a completion deadline for a batch or an upper bound on a specified portion of interactive response times. Those are different constraints and can favour different configurations.

11.2 Failure, retries and service continuity

Resource accounting also needs the unsuccessful path. A request that exceeds available memory, times out or produces invalid structure may be repeated. If the retry uses a shorter context, a different model or a remote fallback, the final result comes from a changed configuration. The elapsed time and resource use of the failed attempt remain part of the workload. Reporting only the successful final call can make a brittle system appear more efficient than the complete user experience.

Availability can be defined at several levels. A process may remain running while returning unusable outputs; a hosted endpoint may respond while a required retrieval integration is unavailable. For a marketing application, completion of the intended task is the relevant service outcome. A continuity test can therefore include model startup, dependency access, input processing and recovery from a failed request. The inference literature isolates mechanisms to study them, while an operational assessment must reconnect those mechanisms to the task.

11.3 A fair counterfactual

Comparing an optimised local runtime with an unoptimised hosted integration would mix deployment location with implementation quality. The reverse comparison would have the same defect. A useful analysis can either hold the model and task path as constant as possible to study location, or compare complete realistic options and describe all their differences. The first supports a narrower explanation; the second supports a practical choice. Neither permits attributing every observed difference to hosting location alone.

The counterfactual also determines how shared resources are assigned. If a computer is already used for other work, allocating its full purchase cost to one AI task is a different question from measuring the task's additional operating demand. If new equipment is required, excluding acquisition and maintenance would be equally incomplete. These are accounting boundaries that should be chosen and stated consistently. The paper offers no universal allocation rule; it makes the dependence of the comparison on that rule explicit.

12. Limitations and conclusion

This review does not test hardware or compare current commercial prices. Several sources concern historical models or research engines, and most were consulted at abstract level. The full-text architectural examples support explanation of mechanisms rather than a claim to reproduced performance. Derived weight-storage values exclude overhead and should not be used as equipment specifications. No pooled speedup, carbon estimate or universal local-hosted break-even point is calculated.

The technical literature shows that inference performance depends on numerical representation, memory movement, cache allocation, scheduling and workload. Model size and physical hosting location are incomplete proxies for those properties. A useful comparison measures the exact candidate configurations against the same task requirements, includes the whole path to an accepted result and states its resource-accounting boundary. Local and hosted inference can each be appropriate under different observed conditions; the evidence must identify those conditions rather than assume them from the deployment label.

Source access and appraisal register

The accessible manuscripts [4,5] were inspected for hardware, memory-management and serving-design sections. Sources [1,2,3,6,7,8,9,10,11,12,13,14] were consulted through author abstracts and publication records. The storage figure uses explicitly stated arithmetic rather than an empirical dataset. No hardware test, independent replication, current pricing exercise or complete lifecycle assessment was performed.

References