Troubleshooting

This section covers issues related to Pod configuration only. If the issue comes from the node or infrastructure, contact support. The following issues are ones you can handle yourself.

Pod stuck in Pending state

If the Pod status shows Pending, like the example below:

NAME               READY   STATUS     RESTARTS   AGE
h100-rdma-test    0/1     Pending    0          5m

To diagnose the issue, run the following command.

kubectl describe pod h100-rdma-test

Check the Events section for error messages. Here are the common causes and the actions you can take.

Insufficient RDMA for Multi-Node Workload

Event: 0/2 nodes are available: 2 Insufficient rdma/ib_shared_device

RDMA is required for multi-node workloads with 16+ GPUs.

If the Events section shows the message above, the Pod is requesting RDMA but no RDMA resource is available. For a single-node setup (≤8 GPUs), update the RDMA requirement in your Pod manifest. For a multi-node setup (16+ GPUs), contact Cloudeka Support.

Pod crashing or exiting with error

If the Pod status shows CrashLoopBackOff, like the example below:

NAME               READY   STATUS             RESTARTS   AGE
h100-rdma-test    0/1     CrashLoopBackOff   5          2m

To diagnose the issue, first check the Pod logs with the following command.

kubectl logs h100-rdma-test

If you need more detailed logs, run the following command.

kubectl describe pod h100-rdma-test

The logs contain the error details. Here are the common causes and the actions you can take.

Image Not Found or Pull Failed

If the logs show ImagePullBackOff or registry errors, try the following solution:

1. Verify CUDA image compatible

Make sure the CUDA image you use is compatible.

image: nvidia/cuda:12.4.1-base-ubuntu22.04

If the image pull fails, you may have a network issue, and contact Cloudeka Support team or open ticket using Service Portal Cloudeka.

Insufficient memory for container

If the logs show OOMKilled or a memory error, increase the memory request by adding this configuration.

resources:
  requests:
    memory: "16Gi"
    nvidia.com/gpu: 1
  limits:
    memory: "32Gi"
    nvidia.com/gpu: 1

Permission error with user 1000

If the logs show permission denied, verify that the image supports a non-root user by adding this configuration.

securityContext:
  runAsUser: 1000
  runAsNonRoot: true

GPU not detected in the container

If the command below returns the message NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, the GPU is not detected inside the container.

$ kubectl exec -it pod/h100-rdma-test -- nvidia-smi

To diagnose the issue, first check whether the Pod was allocated a GPU.

kubectl describe pod h100-rdma-test | grep -A 5 "nvidia.com/gpu"

Then check the Pod logs with the following command.

kubectl logs h100-rdma-test

The logs contain the error details. Here are the common causes and the actions you can take.

Verify that the manifest includes an nvidia.com/gpu request

resources:
  requests:
    nvidia.com/gpu: 1  # HARUS ada ini
  limits:
    nvidia.com/gpu: 1

Verify that runtimeClassName is set correctly

runtimeClassName: nvidia

If you have tried all the solutions above and the issue still occurs, contact the Cloudeka team.