Log Aggregator

Log Aggregator functions as the processing and consolidation layer: it merges logs from multiple sources into a central location, automatically classifies them, detects patterns and anomalies, and enables cross-log search and analytics. Through its aggregation and advanced processing capabilities, it helps users gain insights from large volumes of log data

Aggregate Kubernetes Audit & Container Logs to Other Platforms

Log Aggregator functions to aggregate Kubernetes audit and container logs into other platforms. This process involves deploying Logstash within the cluster to receive both types of logs, and configuring Logstash pipelines to parse, enrich, and export the logs to the existing logging platform. This allows all logs to be consolidated and analyzed centrally, improving system monitoring and troubleshooting efficiency. This section describes two methods for configuring the Log Aggregator with Logstash:

1. Stdout Output (For debugging purposes)

Create a YAML file named logstash-stdout.

nano logstash-stdout.yaml

Copy and paste the following YAML contents.

apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-stdout
spec:
  version: 8.17.2
  count: 1
  podTemplate:
    spec:
      containers:
        - name: logstash
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 1000
            runAsNonRoot: true
          resources:
            limits:
              cpu: 500m
              memory: 1024Mi
            requests:
              cpu: 200m
              memory: 256Mi
      initContainers:
        - name: logstash-internal-init-config
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 1000
            runAsNonRoot: true
          resources:
            limits:
              cpu: 100m
              memory: 128Mi
            requests:
              cpu: 50m
              memory: 64Mi
  volumeClaimTemplates:
    - metadata:
        name: logstash-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
  pipelines:
    - pipeline.id: audit
      config.string: |
        input {
          elasticsearch {
            hosts => "https://dekalog.cloudeka.ai"
            index => "logs-<project_name>*"
            api_key => "api_key"
            query => '{
              "query": {
                "bool": {
                  "must": [
                    { "range": { "@timestamp": { "gte": "now-1m" } } }
                  ]
                }
              }
            }'
            ssl_enabled => true
            ssl_verification_mode => "full"
            schedule => "* * * * *"
            docinfo => true
          }
        }
        output {
          stdout { codec => rubydebug }
        }

2. Elasticsearch Output (For production use)

Create a YAML file named logstash-stdout.

nano logstash-elasticsearch.yaml

Copy and paste the following YAML contents.

apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-elasticsearch
spec:
  version: 8.17.2
  count: 1
  podTemplate:
    spec:
      containers:
        - name: logstash
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 1000
            runAsNonRoot: true
          resources:
            limits:
              cpu: 500m
              memory: 1024Mi
            requests:
              cpu: 200m
              memory: 256Mi
      initContainers:
        - name: logstash-internal-init-config
          securityContext:
            allowPrivilegeEscalation: false
            runAsUser: 1000
            runAsNonRoot: true
          resources:
            limits:
              cpu: 100m
              memory: 128Mi
            requests:
              cpu: 50m
              memory: 64Mi
  volumeClaimTemplates:
    - metadata:
        name: logstash-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
  pipelines:
    - pipeline.id: audit
      config.string: |
        input {
          elasticsearch {
            hosts => "https://dekalog.cloudeka.ai"
            index => "logs-<project_name>*"
            api_key => "<api_key>"
            query => '{
              "query": {
                "bool": {
                  "must": [
                    { "range": { "@timestamp": { "gte": "now-1m" } } }
                  ]
                }
              }
            }'
            ssl_enabled => true
            ssl_verification_mode => "full"
            schedule => "* * * * *"
            docinfo => true
          }
        }
        
        output {
          elasticsearch {
            hosts => ["https://<es_url>:<es_port>"]
            api_key => "<api_key>"
            data_stream => "true"
            data_stream_type => "logs"
            data_stream_dataset => "kubernetes"
            data_stream_namespace => "default"
            ssl_enabled => true
            ssl_verification_mode => "full"
          }
        }