A torch.AcceleratorError: CUDA error: device-side assert triggered message means a check failed while a CUDA operation was running on the GPU. It is a symptom, not a diagnosis: the message alone does not identify which input or operation caused the failure.
GPU operations can be asynchronous. As a result, the line where Python reports the error may be later than the operation that caused it. Start by making the failure easier to locate, then check the inputs to the suspected operation.
Common causes to check
- Invalid class or category index: A target passed to a classification loss may be outside the range expected by the model’s output classes.
- Invalid embedding index: An index supplied to an embedding layer may be negative or greater than or equal to the number of embedding rows.
- Incorrect tensor shapes or types: The operation may receive tensors with shapes or data types that do not match its requirements.
- Bad batch data: One example in a batch may contain a malformed label, out-of-range index, or unexpected value.
- An earlier GPU operation failed: Because work is asynchronous, the error can surface at a later CUDA call.
These are investigation leads, not proof that any one of them caused a particular failure. Check the documentation for the PyTorch operation and version you are using before relying on its exact input requirements.
Minimal example: validate classification targets
The following example illustrates a common check for class-index targets. It is a diagnostic example, not a report of a tested failure. For a model with num_classes output classes, class indices are generally expected to be in the range from zero through num_classes - 1. Confirm the requirements for your loss function and PyTorch version.
import torch
import torch.nn.functional as F
num_classes = 4
logits = torch.randn(3, num_classes)
targets = torch.tensor([0, 2, 4])
print("logits shape:", logits.shape)
print("targets shape:", targets.shape)
print("target range:", targets.min().item(), targets.max().item())
assert targets.min().item() >= 0
assert targets.max().item() < num_classes
loss = F.cross_entropy(logits, targets)Here, the last target is outside the illustrated range: with four classes, valid class indices are 0, 1, 2, and 3. The assertions are intended to stop execution with a clearer Python error before the loss call. Do not fix an out-of-range target by blindly changing its value; determine whether the labels, class count, or label mapping is wrong.
Step-by-step diagnostic path
1. Record the context
Note the PyTorch version, CUDA-related environment details, GPU device, operation that fails, and the shapes and data types of the relevant tensors. Check which versions and devices are supported by the PyTorch installation you actually use. Do not assume that a fix for one release applies unchanged to another.
2. Restart after a device-side assertion
After a device-side assertion, stop the current run and start a fresh process before testing again. A failed GPU operation can leave subsequent checks in the same run difficult to interpret. Restarting does not fix the underlying data or code; it gives you a clean attempt.
3. Make CUDA work easier to locate
For a debugging run, set CUDA_LAUNCH_BLOCKING=1 in the environment before starting Python. This requests synchronous CUDA launches, which can help associate an error with the operation that triggered it. It may reduce performance, so use it as a diagnostic aid rather than a production setting.
# Linux or macOS shell
CUDA_LAUNCH_BLOCKING=1 python train.py# Windows Command Prompt
set CUDA_LAUNCH_BLOCKING=1
python train.pyEnvironment setup differs by shell. If the variable is not available to the Python process, the run will not use that diagnostic setting. Consult the documentation for your environment and PyTorch installation for current guidance.
4. Reduce the failing case
Try a single batch, then a single example if possible. Temporarily disable unrelated parts of the model or data pipeline. Print the inputs just before the suspected operation, including shapes, data types, minimum and maximum values, and any relevant index ranges.
print("input shape:", inputs.shape)
print("input dtype:", inputs.dtype)
print("target shape:", targets.shape)
print("target dtype:", targets.dtype)
print("target min/max:", targets.min().item(), targets.max().item())Only use minimum and maximum checks when the tensor is non-empty and the values can be inspected safely. For floating-point tensors, also check for unexpected non-finite values when they could affect the operation:
print("all finite:", torch.isfinite(inputs).all().item())5. Check the operation’s input contract
Look up the exact operation in the official PyTorch documentation for your installed version. Confirm its expected tensor shapes, data types, and valid value ranges. For example, investigate class targets separately from embedding indices: both involve indices, but their valid ranges depend on different model dimensions.
6. Compare CPU and GPU carefully
If practical, run a small input through the same operation on CPU. A CPU run may provide a more direct error message, but a successful CPU run does not prove that the GPU path is correct. Make sure the CPU and GPU runs use the same data, model settings, and relevant library versions.
Choose a fix based on the cause
- Targets are out of range: Correct the dataset labels, class count, or label-to-index mapping. Add validation before training so invalid values are caught early.
- Embedding indices are out of range: Check the vocabulary or lookup table size and the values generated by tokenisation or preprocessing. Ensure the model’s embedding dimension matches the intended index space.
- Shapes or data types do not match: Follow the documented input requirements for the operation. Make the conversion or reshaping explicit, and verify it does not change the meaning of your data.
- Only a particular batch fails: Inspect that batch and trace its records through preprocessing. Keep the problematic example so you can use it as a small regression check after correcting the data.
Avoid suppressing the error, skipping arbitrary batches, or clamping indices unless that behaviour is deliberately part of your application’s data policy. Those changes can hide the actual problem or silently alter labels.
Verify the fix
- Restart Python after the failed GPU run.
- Reproduce the relevant path using the smallest input that previously failed.
- Check tensor shapes, types, and valid ranges immediately before the operation.
- Run with synchronous CUDA launches while debugging, if appropriate for your environment.
- Run the corrected example and then test several normal batches, not only the smallest case.
- Remove temporary debugging settings or prints when they are no longer needed, and keep useful input validation in place.
A fix is more convincing when the suspected bad input has been identified and corrected, and the same operation works on both the reduced case and representative data. If the error remains, revisit the operation immediately before the failure and confirm that you are debugging a fresh process.
Common follow-up questions
Does the reported Python line always identify the faulty operation?
No. GPU work may be asynchronous, so an error can be reported at a later call. Synchronous launch debugging can help narrow the location, but you still need to inspect the operation’s inputs.
Should I reinstall CUDA or PyTorch first?
Not as the first step. Check input ranges, shapes, data types, and the operation’s requirements first. If a small, valid example still fails, then investigate whether your installed PyTorch build, CUDA environment, and hardware combination are supported. Use current official documentation for the versions involved rather than applying an unverified version-specific fix.
References
The Python documentation is useful for Python language and runtime questions. For PyTorch operation requirements and CUDA-specific behaviour, also consult the official PyTorch documentation that matches your installed version.
Dont SPAM