> For the complete documentation index, see [llms.txt](https://docs.cloudeka.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cloudeka.ai/reference/deployment-llama-3.1-70b-with-vllm-on-kubernetes/create-deployment.md).

# Create Deployment

Create a deployment to run the LLaMA container. If you are using a Linux operating system, then run the following syntax to create the pvc.yaml file.

```bash
nano deployment.yaml
```

If you are using a Windows operating system, open a text editor such as Notepad or Notepad++.

<figure><img src="/files/BsjbwkBGhwTv6O7RQ0QD" alt="" width="375"><figcaption><p>Text Editor</p></figcaption></figure>

Enter the following syntax.

{% hint style="success" %}
Running the model in 2 GPUs.
{% endhint %}

{% code lineNumbers="true" %}

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-31-70b-instruct
  namespace: vllm
  labels:
    app: llama-31-70b-instruct
spec:
  replicas: 1
  selector:
    matchLabels:
      app: llama-31-70b-instruct
  template:
    metadata:
      labels:
        app: llama-31-70b-instruct
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: nvidia.com/gpu.product
                operator: In
                values:
                - NVIDIA-H100-80GB-HBM3
                # - NVIDIA-L40S
      volumes:
      - name: cache-volume
        persistentVolumeClaim:
          claimName: llama-31-70b-instruct
      - name: shm
        emptyDir:
          medium: Memory
          sizeLimit: "2Gi"
      containers:
      - name: llama-31-70b-instruct
        image : dekaregistry.cloudeka.id/cloudeka-system/vllm-openai:v0.11.2
        args: [
          "meta-llama/Llama-3.1-70B-Instruct",
          "--trust-remote-code",
          "--gpu-memory-utilization", "0.9",
          "--tensor-parallel-size", "2",
          "--max-model-len", "16000",
        ]
        env:
        - name: HF_HOME
          value: /.cache/huggingface
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token-secret
              key: token
        ports:
        - containerPort: 8000
        resources:
          limits:
            cpu: "16"
            memory: 64Gi
            nvidia.com/gpu: "2" 
          requests:
            cpu: "8"
            memory: 32Gi
            nvidia.com/gpu: "2"
        volumeMounts:
          - name: cache-volume
            mountPath: /.cache
          - name: shm
            mountPath: /dev/shm
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 30
          failureThreshold: 40
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 30
          failureThreshold: 40
        securityContext:
          runAsUser: 1000
          runAsNonRoot: true
          allowPrivilegeEscalation: false
      runtimeClassName: nvidia
```

{% endcode %}

{% hint style="warning" %}
There are several lines of syntax above that you have to change.

* You need to adjust the version tag used according to the LLaMA model page. You can adjust on line 58.

```yaml
image : dekaregistry.cloudeka.id/cloudeka-system/vllm-openai:v0.11.2
```

* The model can be run on 2 GPUs, but you need to reduce max-model-len to 16000. The consequence is that the model can only process 16000 input tokens. To do this, change the args section on line 39 in deployment.yaml as follows.

```yaml
args: [
          "meta-llama/Llama-3.1-70B-Instruct",
          "--trust-remote-code",
          "--gpu-memory-utilization", "0.9",
          "--tensor-parallel-size", "2",
          "--max-model-len", "16000",
        ]
```

* Detailed Parameters for vllm serve. The vllm serve command is used to start the vLLM server. Here are the detailed parameters used in the command:
  1. `<model>`: The name of the model to serve. In this case, it is meta-llama/Llama-3.1-70B-Instruct.
  2. `--trust-remote-code` : Allows executing custom python from model repository.
  3. `-gpu-memory-utilization <float>`: The GPU memory utilization factor. This controls how much of the available GPU memory is used by the model. A value of 0.9 means 90% of the GPU memory will be used.
  4. `--tensor-parallel-size <int>` : The number of GPUs to use for tensor parallelism. This helps in distributing the model across multiple GPUs. In this case, it is set to 2
  5. **`--max-model-len <int>`**: The maximum length of the input tokens the model can process. Reducing this value can help in running the model with fewer GPUs.
     {% endhint %}

If you are using a **Linux** operating system, run the following syntax but If you are using a **Windows** operating system, after save the file as secret.yaml, in CMD navigate to the folder that contains the secret.yaml file and run the following syntax.

```bash
kubectl apply -f deployment.yaml
```

<figure><img src="/files/7eODj4OtnT4n1lb007Mg" alt=""><figcaption><p>Success</p></figcaption></figure>

{% hint style="warning" %}
To delete the pvc.yaml configuration that has been applied, run the following syntax.

```bash
kubectl delete -f deployment.yaml -n [namespace]
```

**Replace \[namespace] with the namespace you created in the sub-chapter** [**Create Namespace**](/reference/deployment-llama-3.1-70b-with-vllm-on-kubernetes/create-namespace.md)**.**
{% endhint %}