Developer Tools / AI

Does RTX 5090 Need CUDA 13.5 for vLLM? 2026 Environment Troubleshooting

Do not install CUDA 13.5 just because you have an RTX 5090: as of September 30, 2026, NVIDIA’s toolkit archive lists CUDA 13.4.1, while vLLM’s published build options include CUDA 12.8, 12.9, and 13.0. Choose a matching driver, PyTorch CUDA backend, and vLLM build instead; this is the safer route unless documentation for the exact versions you use says otherwise.

This guide is for developers installing or upgrading vLLM on an RTX 5090, engineers testing large or FP4 models, and teams diagnosing conflicts between containers, PyTorch, CUDA, and vLLM.

Last updated September 30, 2026. Version and hardware claims were checked against the NVIDIA CUDA Toolkit Archive, vLLM installation documentation, and the RTX 5090 specifications.

Check the version stack before changing CUDA

The question “Which CUDA version should I use to install vLLM on an RTX 5090?” has no safe answer based on the GPU name alone. Start with the artifacts supported by the vLLM version you intend to run. The official CUDA installation instructions and vLLM release history list build options that vary by release. The available information does not establish CUDA 13.5 as a requirement for RTX 5090.

That distinction matters because “CUDA version” can refer to different parts of the stack. The host may have a CUDA toolkit installed, while a PyTorch wheel includes its own CUDA runtime components, and a container may package yet another user-space runtime. The driver is a separate component. These layers have to work together; installing a newer toolkit on the host does not automatically replace the CUDA backend used by your PyTorch installation.

Use this comparison to choose what to check first:

Option or layer When it is the right decision What to verify
vLLM-supported wheel or container You want the least ambiguous setup and a matching published build exists The exact vLLM release, CUDA build label, PyTorch backend, and driver requirements
Source build No suitable prebuilt artifact matches your required stack, or you need a specific build The release’s build instructions, compiler dependencies, CUDA toolkit, and PyTorch version
Host CUDA toolkit change A documented build or operation explicitly requires a different toolkit Whether that requirement applies to your host build, container, or PyTorch backend
CUDA 13.5 assumption You have heard it is mandatory but have not found version-specific support evidence Do not treat the claim as a requirement; check the archive and the relevant vLLM release

For a reproducible baseline, record the driver version, whether vLLM runs in a container or directly on the host, the PyTorch version and CUDA backend, the vLLM version and installation source, and the GPU reported by the runtime. Save the output with the exact startup command. If the install fails, this record helps distinguish a driver issue from a wheel or build mismatch.

NVIDIA’s CUDA compatibility guidance explains why driver and toolkit relationships matter, but it does not replace the compatibility requirements for a particular vLLM release. Check the release-specific installation instructions before changing multiple layers. Changing the driver, host toolkit, PyTorch wheel, and vLLM build at once can make an error disappear without showing which mismatch caused it.

Separate installation failures from runtime failures

When an RTX 5090 installation stops, an extension fails to import, or startup reports a CUDA error, first identify the stage where the failure occurs. These symptoms do not all point to a missing host toolkit.

  • Installation stops before vLLM is available: Check whether the requested package or container tag exists for the chosen release and CUDA build. Then check the package manager output for dependency resolution or compilation errors.
  • Python imports vLLM but a compiled extension fails: Compare the PyTorch CUDA backend with the extension and vLLM build. A package compiled against a different backend can fail even when the host reports a capable GPU.
  • vLLM imports but startup fails while initializing the GPU: Check driver visibility from inside the process or container, device permissions, and the exact CUDA error. A container that cannot access the host GPU is not fixed by installing a toolkit inside the container.
  • The server starts but a model fails to load: Stop treating this as an installation issue. Check the model’s architecture, quantization metadata, supported kernels, and available memory.

How can you tell whether a build failure is a PyTorch or CUDA mismatch? Compare the installed PyTorch version and its reported CUDA backend with the vLLM artifact or source build requirements. If the installed vLLM package expects a different CUDA build than the PyTorch components available in that environment, align those packages before changing the host toolkit. If the driver cannot initialize the device, investigate driver visibility and compatibility instead. Keep the full error log: a compiler error, an import-time symbol error, and a driver initialization error call for different fixes.

Do not carry one installation command across releases without checking its assumptions. A command can be valid for one wheel or container and wrong for another. Use the current vLLM GPU installation guide to select the artifact for the release you actually plan to run; then save the package source and version alongside your environment record.

Diagnostic note: A CUDA toolkit installed on the host is not proof that vLLM is using that toolkit. Record the container boundary and PyTorch backend before adding or removing system packages.

Check the model format before diagnosing FP4

A model labelled “4-bit” is not necessarily compatible with every 4-bit kernel or execution path. Before changing vLLM settings, inspect the model card and its weight configuration for the precise format, quantization method, and intended backend. In particular, distinguish an FP4 floating-point format such as NVFP4 from INT4 integer quantization. Their bit widths may sound similar, but their representations, metadata, kernels, and hardware support are not interchangeable.

The vLLM documentation on choosing a quantization scheme describes NVFP4, while the ModelOpt checkpoint support documentation identifies supported checkpoint paths. Use these as format-specific checks, not as a blanket guarantee that every model described as FP4 will load in every vLLM release.

What should you check when an FP4 model fails to load on an RTX 5090? First verify the exact quantization format declared by the checkpoint and whether that format is supported by the vLLM release and execution backend in your environment. Then classify the failure: a missing or inconsistent configuration suggests a checkpoint or loader problem; an unsupported operator points to a backend or kernel limitation; output anomalies after a successful load require a separate quality check. Do not respond to all three by changing the CUDA toolkit.

Successful loading also does not prove that a quantized model fits comfortably or produces acceptable results. The nominal bit width does not include every scale, metadata item, runtime workspace, or cache allocation. Quantization can reduce the weight footprint, but it does not remove the need to measure actual memory use and test output quality for the workload.

Test Chunked Prefill as a scheduling variable

Chunked Prefill changes how prompt processing is scheduled; it is not a substitute for a memory budget. If a long prompt produces an error, compare behavior with the feature disabled or adjusted using the setting documented for your vLLM release. Keep the model, prompt, concurrency, and other launch options unchanged during that comparison.

What should you check first when Chunked Prefill causes memory errors or unexpected output? Record the prompt length, number of simultaneous requests, maximum batched-token setting, and available GPU memory. Then run a controlled comparison with the release-appropriate Chunked Prefill setting changed and nothing else. If the error disappears only on the long-prompt path, investigate scheduling and batch limits; if memory is exhausted in both tests, revisit the model and KV cache budget.

Do not copy a parameter name or default from a different release. vLLM options can change, and the exact behavior depends on the version in use. Check the documentation that corresponds to your installed release, then record the selected setting with the reproduction command. A server that handles a short prompt is not yet validated for the intended context length or concurrent load.

Recalculate the memory budget before claiming a model fits

The RTX 5090’s official specifications list 32 GB of GDDR7 memory. That capacity is a hard constraint, not a promise that the entire amount is available for model weights. The process also needs space for quantization scales and metadata, temporary runtime workspaces, and the KV cache used during inference.

Use a component-by-component estimate rather than a parameter-count shortcut:

  • Weights: Estimate from the checkpoint’s actual quantized representation, not just the model’s advertised bit width. Include scales and other quantization data where the format requires them.
  • Runtime workspace: Reserve room for kernels and other allocations. The actual requirement depends on the model, backend, and request pattern; measure it rather than assuming it is zero.
  • KV cache: Its use depends on architecture, context length, and concurrent requests. A model that loads at startup can still run out of memory when prompts grow or requests overlap.
  • Headroom: Keep enough free memory for workload variation. A configuration that only starts under an idle, short-prompt test may not be stable in normal use.

Llama 4 illustrates why active parameter count is not a safe shortcut for memory planning. Meta describes Llama 4 Scout as 109B total parameters with 17B active parameters, and Maverick as 400B total parameters with 17B active parameters in its official Llama 4 overview. The active count describes the MoE model’s computation per token; it does not mean all inactive experts can be omitted from the checkpoint. Unless the serving configuration explicitly offloads or distributes weights, estimate for the complete weight set that must be available to the runtime.

For a rough lower-bound illustration only, multiplying Scout’s 109B total parameters by four bits gives about 54.5 GB of raw weight data before scales, metadata, runtime allocations, or KV cache. This arithmetic assumes every parameter is represented at exactly four bits and uses decimal gigabytes; it is not a measured checkpoint size or an inference result. It already exceeds the RTX 5090’s 32 GB capacity, so a claim that Scout simply “fits because only 17B parameters are active” is not a valid single-GPU memory argument.

This does not prove that every offloaded, sharded, or specially compressed configuration is impossible. It does mean that a successful model load, a short test, and stable target-context inference are separate acceptance points. Do not describe a 109B model as a confirmed 120B model, and do not promise that any 120B-class workload will run stably on one RTX 5090 based only on nominal FP4 bit width.

Keep a reproducible troubleshooting record

Use a short checklist before changing the environment:

  • [ ] Record driver, operating system, container or host setup, PyTorch version and CUDA backend, vLLM version, and installation source.
  • [ ] Save the complete error output, not only the final line.
  • [ ] Record the model identifier, architecture, quantization format, and relevant checkpoint configuration.
  • [ ] Save all launch settings, including the version-specific Chunked Prefill and batch-related settings.
  • [ ] Note GPU memory status before startup, after model loading, and during the failing request.
  • [ ] Change one variable at a time and keep the previous command for comparison.
  • [ ] Validate in stages: package installation, Python import, model load, short request, then the intended prompt length and concurrency.

This progression prevents a common false pass: installation succeeds, the model loads, and a short prompt returns text, so the team assumes deployment is ready. If the target workload uses longer context or overlapping requests, test those conditions explicitly. A community issue can suggest a failure mode, but an unreplicated report is only a case lead; it does not establish official support or a universal RTX 5090 limitation.

Split development work from CUDA inference when useful

For troubleshooting and cost planning, compare the current all-in-one setup with a split workflow. A single local workstation keeps editing, orchestration, and GPU inference together, but it also ties those tasks to one machine’s availability, makes a broken GPU environment disruptive to unrelated development, and can consume the local system’s resources during long inference tests. A cloud Mac does not run CUDA inference and cannot replace the RTX 5090 for this workload. It can be considered separately for Mac-side development tasks, while CUDA inference remains on the local GPU.

If that separation fits the team’s workflow, review the KuzCloud service overview and KuzCloud help information before deciding whether to use a cloud Mac for non-CUDA development. If the team needs direct access to a CUDA GPU, has a sustained heavy inference workload, or depends on physical interfaces, keep that work on appropriate local GPU hardware instead. For this troubleshooting problem, the practical decision is to match the vLLM stack and measure the RTX 5090 workload—not to rent a Mac expecting it to perform CUDA inference.

Run Your Mac Workloads on Dedicated Hardware

Use KuzCloud for remote macOS development, builds, and testing on a dedicated Mac mini M4.

Connect by SSH or remote desktop with full root access and no virtualization overhead.