Deployment LLM with Deka GPU + NIM
Pre-requisite
Contact [email protected] to get licensed for NVIDIA AI Enterprise.
You need NGC API Keys, click this link.
Contact [email protected] to get root privileged access.
Download Helm Chart
After you have fulfilled the prerequisites above, the next step is to download Helm Chart by executing the following syntax.
helm fetch https://helm.ngc.nvidia.com/nim/charts/nim-llm-1.3.0.tgz --username=\$oauthtoken --password=<your token> After successfully downloading Helm Chart, the next step is to extract the file by executing the following syntax.
tar -xvzf nim-llm-1.3.0.tgz Enter the extracted folder by running the syntax below.
cd nim-llmChange the security context in the values.yaml file
Determine the base model used, in this guide using meta llama 8b.
If you want use other model, you can refer to NVDIA web page. The following are the steps for getting the URL for the model that will be used.
On the NVIDIA website, in the Search API Catalog column, enter the name of the model that will be used.
The Search API Catalog column displays several models, and select the model to use.
On the selected model page there is an Experience tab.
On the Experience tab on the right there are several options, namely Python, LangChain, Node, Shell, and Docker. Select Docker.
In the Docker section under "Pull and run the NVIDIA NIM with the command below. This will download the optimized model for your infrastructure." in the bottom row there is the model URL. Copy the URL.
Edit the gpu based on the needs.
Put your api key in line "ngcAPI Key".
Edit the storage based on the needs.
Save the file after updating, press CTRL + O to save, and CTRL + X to exit.
Add Runtime NVIDIA
Open the deployment.yaml file in the files folder by executing the following syntax.
nano files/deployment.yamlAfter the deployment.yaml file opens, look for the specsection and add runtimeClassName: nvidia according to the image below.
Save the file after updating, press CTRL + O to save, and CTRL + X to exit.
Install Helm Chart
Install Helm Chart by executing the following syntax.
helm install nim . -f values.yaml 
Wait until the installation process is complete and run the following syntax to see the list of available pods.
kubectl get pods
Testing
When finished, run the following syntax to display a list of nim services in the namespace.
kubectl get svc -n nim
In the image above, you can see nim-llm using the IP cluster 10.250.225.0 and the status is active. Run the following syntax to log into a container running inside a pod in Kubernetes.
kubectl exec -it multitoll -- bash
Last updated