# Log Aggregator

**Log Aggregator** functions as the processing and consolidation layer: it merges logs from multiple sources into a central location, automatically classifies them, detects patterns and anomalies, and enables cross-log search and analytics. Through its aggregation and advanced processing capabilities, it helps users gain insights from large volumes of log data

## Aggregate Kubernetes Audit & Container Logs to Other Platforms

Log Aggregator functions to aggregate Kubernetes audit and container logs into other platforms. This process involves deploying Logstash within the cluster to receive both types of logs, and configuring Logstash pipelines to parse, enrich, and export the logs to the existing logging platform. This allows all logs to be consolidated and analyzed centrally, improving system monitoring and troubleshooting efficiency.  This section describes two methods for configuring the Log Aggregator with Logstash:

###  1. Stdout Output *(For debugging purposes)*

Create a YAML file named logstash-stdout.

```bash
nano logstash-stdout.yaml
```

Copy and paste the following YAML contents.

```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-stdout
spec:
  version: 8.17.2
  count: 1
  podTemplate:
    spec:
      containers:
        - name: logstash
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 1000
            runAsNonRoot: true
          resources:
            limits:
              cpu: 500m
              memory: 1024Mi
            requests:
              cpu: 200m
              memory: 256Mi
      initContainers:
        - name: logstash-internal-init-config
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 1000
            runAsNonRoot: true
          resources:
            limits:
              cpu: 100m
              memory: 128Mi
            requests:
              cpu: 50m
              memory: 64Mi
  volumeClaimTemplates:
    - metadata:
        name: logstash-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
  pipelines:
    - pipeline.id: audit
      config.string: |
        input {
          elasticsearch {
            hosts => "https://dekalog.cloudeka.ai"
            index => "logs-<project_name>*"
            api_key => "api_key"
            query => '{
              "query": {
                "bool": {
                  "must": [
                    { "range": { "@timestamp": { "gte": "now-1m" } } }
                  ]
                }
              }
            }'
            ssl_enabled => true
            ssl_verification_mode => "full"
            schedule => "* * * * *"
            docinfo => true
          }
        }
        output {
          stdout { codec => rubydebug }
        }
```

### 2. Elasticsearch Output (For production use)

Create a YAML file named logstash-stdout.

```bash
nano logstash-elasticsearch.yaml
```

Copy and paste the following YAML contents.

```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-elasticsearch
spec:
  version: 8.17.2
  count: 1
  podTemplate:
    spec:
      containers:
        - name: logstash
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 1000
            runAsNonRoot: true
          resources:
            limits:
              cpu: 500m
              memory: 1024Mi
            requests:
              cpu: 200m
              memory: 256Mi
      initContainers:
        - name: logstash-internal-init-config
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 1000
            runAsNonRoot: true
          resources:
            limits:
              cpu: 100m
              memory: 128Mi
            requests:
              cpu: 50m
              memory: 64Mi
  volumeClaimTemplates:
    - metadata:
        name: logstash-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
  pipelines:
    - pipeline.id: audit
      config.string: |
        input {
          elasticsearch {
            hosts => "https://dekalog.cloudeka.ai"
            index => "logs-<project_name>*"
            api_key => "<api_key>"
            query => '{
              "query": {
                "bool": {
                  "must": [
                    { "range": { "@timestamp": { "gte": "now-1m" } } }
                  ]
                }
              }
            }'
            ssl_enabled => true
            ssl_verification_mode => "full"
            schedule => "* * * * *"
            docinfo => true
          }
        }
        
        output {
          elasticsearch {
            hosts => ["https://<es_url>:<es_port>"]
            api_key => "<api_key>"
            data_stream => "true"
            data_stream_type => "logs"
            data_stream_dataset => "kubernetes"
            data_stream_namespace => "default"
            ssl_enabled => true
            ssl_verification_mode => "full"
          }
        }
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.cloudeka.ai/guidance-for-individual/deka-gpu/deka-gpu-log-collection/log-aggregator.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.