> For the complete documentation index, see [llms.txt](https://docs.cloudeka.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cloudeka.ai/deka-gpu/deka-gpu-autoscaling/keda-autoscalling/example-autoscaling-vllm-with-keda-based-on-gpu-kv-cache-usage/keda-configuration-example.md).

# KEDA Configuration Example

Below is a sample ScaledObject definition for KEDA to scale a vLLM deployment based on GPU KV cache metrics collected from Prometheus:

```yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: prometheus-scaledobject
  namespace: vllm
spec:
  maxReplicaCount: 2
  minReplicaCount: 1
  cooldownPeriod: 120        # Cooldown period in seconds before scale-in
  pollingInterval: 30        # Polling interval for metrics (in seconds)
  scaleTargetRef:
    name: llama-32-3b-instruct
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://l2-jatiluhur.metric.cloudeka.ai
      threshold: '0.5'   # This is equivalent to 50%
      query: vllm:gpu_cache_usage_perc{job="goto-vllm-exporter", model_name="meta-llama/Llama-3.2-3B-Instruct"}
```

Explanation:

* `minReplicaCount`: Minimum number of vLLM pods (set to 1).
* `maxReplicaCount`: Maximum number of vLLM pods (set to 2).
* `cooldownPeriod`: Wait time before scaling down.
* `pollingInterval`: Interval to check the metric value.
* `threshold`: The GPU KV cache usage threshold for scaling. In this example:
* 0.003 represents 0.3% (Prometheus metric returns values between 0 to 1).
* Adjust this value based on your scaling policy, e.g., 0.5 for 50%.

Prometheus Query Example:

```powerquery
vllm:gpu_cache_usage_perc{job="goto-vllm-exporter", model_name="meta-llama/Llama-3.2-3B-Instruct"}
```

Notes: metrics to scale need to be provided by the customer and inform the metrics endpoint (e.g. `http://<SERVICE_IP>:<SERVICE_PORT>/metrics`) to the Lintasarta operation team.

{% hint style="success" %}
**Recommendations**

* Start with a conservative threshold (e.g., 50% or 0.5 in Prometheus) and monitor the behavior.
* Adjust maxReplicaCount according to your GPU and cluster capacity.
* Keep an eye on GPU KV cache metrics to avoid cache evictions and performance degradation.
* Combine this autoscaler with proper request queueing and batching logic in vLLM to maximize GPU efficiency.
  {% endhint %}

These metrics are exposed via the /metrics endpoint on the vLLM OpenAI compatible API server.