NVIDIA published a tutorial on October 30, 2026 showing how to download, compile and run Nemotron 3.5 Lightning on the 64 GB DGX Spark. The workflow uses TensorRT Edge-LLM 0.11.1 and exposes an OpenAI-compatible API for local application development.
NVIDIA reports generation speeds of roughly 73 tokens per second without DSpark speculative decoding and 105–127 tokens per second with it. Those figures need context: the tests used one concurrent request, a 128-token output limit and a specific software stack.
For embedded and physical-AI developers, the useful part is the complete path from model download to a local endpoint. The benchmark results then help identify what to measure before moving an application onto deployment hardware.
What You Need Before Installing
TensorRT Edge-LLM is NVIDIA’s lightweight C++ inference runtime for embedded workloads. Its trtedge command-line tool handles model downloads and builds optimized TensorRT engines.
The tutorial’s setup requirements are specific:
| Component | Requirement or tested configuration |
|---|---|
| Hardware | DGX Spark; the guide and measurements use the 64 GB configuration |
| NVIDIA software stack | A supported DGX Spark software stack |
| CUDA | Version 13 |
| TensorRT | 10.16.1.11 or newer, remaining within the 10.x series |
| Python | 3.10–3.12 |
| TensorRT Edge-LLM | Version 0.11.1 in the examples |
| Model | nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 |
| Download storage | Approximately 21 GB for the model, plus space for compiled engines |
The TensorRT upper bound matters. The guide specifies a compatible 10.x release, so installing version 11 would fall outside its documented requirement.
NVIDIA tested the examples with TensorRT 10.16.1.11. Its reported benchmark environment used DGX OS 7.6.0, based on Ubuntu 24.04.4 LTS. That OS version describes the measured setup; it should not be confused with a separately stated minimum requirement.
Install and Check the Runtime
Open a terminal on the DGX Spark and create a Python virtual environment:
python3 -m venv --system-site-packages .venv-edgellm
source .venv-edgellm/bin/activate
python -m pip install --upgrade pip
python -m pip install "tensorrt-edgellm[server]==0.11.1"
The --system-site-packages option lets the environment access system Python packages, including TensorRT bindings, while keeping newly installed packages in the virtual environment. The [server] extra installs the package’s server dependencies.
Next, check the TensorRT version and load the runtime. This reformats NVIDIA’s verification command into a more readable block:
python - <<'PY'
import tensorrt as trt
version = tuple(map(int, trt.__version__.split(".")))
assert (10, 16, 1, 11) <= version < (11,), (
f"TensorRT >=10.16.1.11 and <11 required; found {trt.__version__}"
)
from tensorrt_edgellm import runtime
runtime.load()
PY
The version check rejects an unsupported TensorRT release; the final call checks whether the Edge-LLM runtime loads. It does not benchmark the model.
If compatible TensorRT bindings are unavailable, NVIDIA directs developers to install a compatible TensorRT 10.x version inside the activated environment and repeat the check. I’d resolve that dependency before downloading the model: a successful package installation alone does not confirm that the inference runtime can load.
Download Nemotron and Start a Chat
With the environment active, launch the text-only model:
trtedge chat nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
The first launch downloads approximately 21 GB of model files and builds the TensorRT engines. NVIDIA does not quantify the download or compilation time.
The default cache is:
~/.cache/tensorrt-edgellm
Allow disk space for both the downloaded model and generated engines. Later launches reuse matching cached engines, so the initial startup and a cached startup are different operations.
Once the chat prompt appears, enter a short request, such as a temperature conversion. Type /exit to leave the session.
This is a useful functional check before adding an application client. It separates model-loading problems from HTTP request formatting or application integration issues.
Expose an OpenAI-Compatible Local Endpoint
Start the server from the activated environment:
trtedge serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
--host 127.0.0.1 \
--port 8000
The server builds or reuses an engine for that configuration. Wait for startup, then open a second terminal on the same DGX Spark and check readiness:
curl -fsS http://127.0.0.1:8000/health/ready
A response whose status field is ready confirms that the service is ready to accept requests.
Send a streaming chat-completions request:
curl -fsS -N http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
"messages": [
{
"role": "user",
"content": "Convert 25 degrees Celsius to Fahrenheit."
}
],
"stream": true
}'
The JSON selects the model and supplies a user message. Setting stream to true requests incremental output, while curl’s -N option disables its output buffering.
NVIDIA documents these endpoints in its local serving walkthrough. The practical addresses are:
- API base URL:
http://127.0.0.1:8000/v1 - Chat completions:
http://127.0.0.1:8000/v1/chat/completions - Readiness:
http://127.0.0.1:8000/health/ready
Binding to 127.0.0.1 keeps the listener on the Spark’s loopback interface. A client on another computer cannot reach that address as though it were the Spark’s network address.
I’d keep this binding during initial development. Making the endpoint reachable over a network is a separate deployment decision involving access control and network configuration. OpenAI compatibility describes the API interface; the inference runs locally on the Spark.
What NVIDIA’s DSpark Results Actually Measure
DSpark uses speculative decoding: a draft model proposes several tokens, and the main model verifies them in one pass. The tutorial’s measured configuration uses three draft tokens.
NVIDIA’s results, reproduced in TokenPost’s report, show a clear difference between generation speed and time to first token:
| Workload | Mean generation, DSpark off | Mean generation, DSpark on | Median first-token latency, off/on |
|---|---|---|---|
| SpeedBench 1K | 73.3 tokens/s | 108.5 tokens/s | 201.0 / 203.0 ms |
| SpeedBench 2K | 72.8 tokens/s | 105.3 tokens/s | 329.5 / 331.6 ms |
| HumanEval prompts | 73.4 tokens/s | 126.9 tokens/s | 98.1 / 98.7 ms |
These are NVIDIA measurements, not independent results. The published conditions were:
Frequently Asked Questions
3 questions
1Is Nemotron 3.5 Lightning Multimodal?
The model used in this tutorial is text-only. This recipe does not demonstrate image, video or sensor-input processing.
