> For the complete documentation index, see [llms.txt](https://docs.cloudeka.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cloudeka.ai/reference/how-to-use-gpu-h100-with-rdma-in-kubernetes/troubleshooting.md).

# Troubleshooting

This section covers issues related to Pod configuration only. If the issue comes from the node or infrastructure, contact support. The following issues are ones you can handle yourself.

## Pod stuck in Pending state

If the Pod status shows `Pending`, like the example below:

```bash
NAME               READY   STATUS     RESTARTS   AGE
h100-rdma-test    0/1     Pending    0          5m
```

To diagnose the issue, run the following command.

```bash
kubectl describe pod h100-rdma-test
```

Check the `Events` section for error messages. Here are the common causes and the actions you can take.

### Insufficient RDMA for Multi-Node Workload

```bash
Event: 0/2 nodes are available: 2 Insufficient rdma/ib_shared_device
```

{% hint style="info" %}
RDMA is required for multi-node workloads with 16+ GPUs.
{% endhint %}

If the `Events` section shows the message above, the Pod is requesting RDMA but no RDMA resource is available. For a single-node setup (≤8 GPUs), update the RDMA requirement in your Pod manifest. For a multi-node setup (16+ GPUs), contact Cloudeka Support.

## Pod crashing or exiting with error

If the Pod status shows `CrashLoopBackOff`, like the example below:

```bash
NAME               READY   STATUS             RESTARTS   AGE
h100-rdma-test    0/1     CrashLoopBackOff   5          2m
```

To diagnose the issue, first check the Pod logs with the following command.

```bash
kubectl logs h100-rdma-test
```

If you need more detailed logs, run the following command.

```bash
kubectl describe pod h100-rdma-test
```

The logs contain the error details. Here are the common causes and the actions you can take.

### Image Not Found or Pull Failed

If the logs show `ImagePullBackOff` or registry errors, try the following solution:

#### 1. Verify CUDA image compatible

Make sure the CUDA image you use is compatible.

```bash
image: nvidia/cuda:12.4.1-base-ubuntu22.04
```

{% hint style="info" %}
If the image pull fails, you may have a network issue, and contact Cloudeka Support team or open [ticket ](/service-portal-ai/ticket/create-ticket.md)using Service Portal Cloudeka.
{% endhint %}

### Insufficient memory for container

If the logs show `OOMKilled` or a memory error, increase the memory request by adding this configuration.

```bash
resources:
  requests:
    memory: "16Gi"
    nvidia.com/gpu: 1
  limits:
    memory: "32Gi"
    nvidia.com/gpu: 1
```

### Permission error with user 1000

If the logs show `permission denied`, verify that the image supports a non-root user by adding this configuration.

```bash
securityContext:
  runAsUser: 1000
  runAsNonRoot: true
```

## GPU not detected in the container

If the command below returns the message `NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver`, the GPU is not detected inside the container.

```bash
$ kubectl exec -it pod/h100-rdma-test -- nvidia-smi
```

To diagnose the issue, first check whether the Pod was allocated a GPU.

```bash
kubectl describe pod h100-rdma-test | grep -A 5 "nvidia.com/gpu"
```

Then check the Pod logs with the following command.

```bash
kubectl logs h100-rdma-test
```

The logs contain the error details. Here are the common causes and the actions you can take.

### Verify that the manifest includes an `nvidia.com/gpu` request

```yaml
resources:
  requests:
    nvidia.com/gpu: 1  # HARUS ada ini
  limits:
    nvidia.com/gpu: 1
```

### Verify that `runtimeClassName` is set correctly

```yaml
runtimeClassName: nvidia
```

{% hint style="info" %}
If you have tried all the solutions above and the issue still occurs, contact the Cloudeka team.
{% endhint %}