1. Guides & Tutorials
  2. Kubernetes
  3. How to Deploy the Elastic Stack on Kubernetes

How to Deploy the Elastic Stack on Kubernetes

Updated by Linode Written by Tyler Langlois

Contribute on GitHub

Report an Issue | View File | Edit File

What is the Elastic Stack?

The Elastic Stack is a collection of open source projects from Elastic that help collect and visualize a wide variety of data sources. Elasticsearch can store and aggregate data such as log files, container metrics, and more. The products in the stack include: Elasticsearch, Logstash, Kibana, and now Beats.

In this guide:

At the end of this guide, you will have a deployment installed and configured that you can further use for application logs or monitoring Kubernetes itself.

Caution

This guide’s example instructions will create the following billable resources on your Linode account: four (4) Linodes and three (3) Block Storage volumes. If you do not want to keep using the example cluster that you create, be sure to delete the cluster Linodes and volumes when you have finished the guide.

If you remove the resources afterward, you will only be billed for the hour(s) that the resources were present on your account. Consult the Billing and Payments guide for detailed information about how hourly billing works and for a table of plan pricing.

Before You Begin

Note
This guide uses Kubernetes services which are private by default. Local listeners are opened which allow you to access the services on your local browser, however, web servers and NodeBalancers are out scope for this guide. Due to this, you should complete the steps of this guide from your local computer or from a computer that will give you access to the web browser. If you wish to be able to access these services from a public domain, please see our guide on Getting Started with NodeBalancers.

  1. Install the Kubernetes CLI (kubectl) on your computer, if it is not already.

  2. Follow the How to Deploy Kubernetes on Linode with the k8s-alpha CLI guide to set up a Kubernetes cluster.

    Caution

    The k8s-alpha CLI is deprecated. On March 31st, 2020, it will be removed from the linode-cli. After March 31, 2020, you will no longer be able to create or manage clusters using the k8s-alpha CLI plugin.

    However, you will still be able to create and manage these clusters using Terraform. The Terraform module used is a public project officially supported by Linode, and is currently used to power the k8s-alpha CLI.

    Other alternatives for creating and managing clusters include:

    This guide will use a three node + master node cluster. You can use the following Linode k8s-alpha CLI command to create your cluster:

    linode-cli k8s-alpha create example-cluster --node-type g6-standard-2 --nodes 3 --master-type g6-standard-2 --region us-east --ssh-public-key ~/.ssh/id_rsa.pub

    • You should use this guide instead of manual installation via a method such as kubeadmin, as the k8s-alpha tool will setup support for persistent volume claims.

    • Node sizes are important when configuring Elasticsearch, and this guide assumes 4GB Linode instances.

    • This guide also assumes that your cluster has role-based access control (RBAC) enabled. This feature became available in Kubernetes 1.6. It is enabled on clusters created via the k8s-alpha Linode CLI.

  3. You should also make sure that your Kubernetes CLI is using the right cluster context. Run the get-contexts subcommand to check:

    kubectl config get-contexts

  4. Set up Helm in your Kubernetes cluster by following the How to Install Apps on Kubernetes with Helm guide and stop following the steps in this guide upon reaching the Use Helm Charts to Install Apps section.

Configure Helm

After following the prerequisites for this guide, you should have a Kubernetes cluster with Helm installed and configured.

  1. Add the elastic chart repository to your local installation of Helm:

    helm repo add elastic https://helm.elastic.co

  2. Fetch the updated list of charts from all configured chart repositories:

    helm repo update

  3. Search for the official elasticsearch chart to confirm Helm has been configured correctly. Note that this chart released by Elastic differs from the chart bundled with the default installation of Helm.

    helm search hub elasticsearch

    This command should return all the charts available for elasticsearch in the hub. The one we want is listed below. Note that your exact version numbers may be different.

      
NAME                                              CHART VERSION   APP VERSION   DESCRIPTION
https://hub.helm.sh/charts/elastic/elasticsearch  7.5.2        	  7.5.2      	Official Elastic helm chart for Elasticsearch

    Your Helm environment is now prepared to install official Elasticsearch charts into your Kubernetes cluster.

Install Charts

Install Elasticsearch

Before installing the chart, ensure that resources are set appropriately. By default, the elasticsearch chart allocates 1G of memory to the JVM heap and sets Kubernetes resource requests and limits to 2G. Using a Linode 4GB instance is compatible with these defaults, but if you are using a different instance type, you will need to provide different values to the chart at install time in order to ensure that running Pods are within the resource constraints of the node sizes you have chosen.

  1. Install the elasticsearch chart:

    helm install elasticsearch elastic/elasticsearch

  2. You should see an output like this:

      
NAME: elasticsearch
LAST DEPLOYED: Mon Feb 10 09:58:12 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Watch all cluster members come up.
  $ kubectl get pods --namespace=default -l app=elasticsearch-master -w
2. Test cluster health using Helm test.
  $ helm test elasticsearch

  3. A three-node Elasticsearch cluster is now configured and available locally to the Kubernetes cluster. To confirm this, first port-forward a local port to the Elasticsearch service. You should leave this command running in a terminal window or tab in in the background for the remainder of this tutorial.

    kubectl port-forward svc/elasticsearch-master 9200:9200

  4. In another terminal window, send a request to this port:

    curl http://localhost:9200/

  5. You should see a response similar to the following:

      
{
    "name" : "elasticsearch-master-0",
    "cluster_name" : "elasticsearch",
    "cluster_uuid" : "o66WYOm5To2znbZ0kOkDUw",
    "version" : {
    "number" : "7.5.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "8bec50e1e0ad29dad5653712cf3bb580cd1afcdf",
    "build_date" : "2019-09-06T14:40:30.409026Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
    },
    "tagline" : "You Know, for Search"
}

Note that your specific version numbers and dates may be different in this json response. Elasticsearch is operational, but not receiving or serving any data.

Install Filebeat

In order to start processing data, deploy the filebeat chart to your Kubernetes cluster. This will collect all Pod logs and store them in Elasticsearch, after which they can be searched and used in visualizations within Kibana.

  1. Deploy the filebeat chart. No custom values.yaml file should be necessary:

    helm install filebeat elastic/filebeat

  2. You should see a response similar to the following:

      
NAME: filebeat
LAST DEPLOYED: Mon Feb 10 11:17:18 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Watch all containers come up.
  $ kubectl get pods --namespace=default -l app=filebeat-filebeat -w

  3. Confirm that Filebeat has started to index documents into Elasticsearch by sending a request to the locally-forwarded Elasticsearch service port:

    curl http://localhost:9200/_cat/indices

    At least one filebeat index should be present, and output should be similar to the following:

      
green open filebeat-7.3.12-2019.09.30-000001 peGIaeQRQq-bfeSG3s0RWA 1 1 9886 0 5.7mb 2.8mb

Install Kibana

Kibana will provide a frontend to Elasticsearch and the data collected by Filebeat.

  1. Deploy the kibana chart:

    helm install kibana elastic/kibana

  2. You should see a response similar to the following:

      
NAME: kibana
LAST DEPLOYED: Mon Feb 10 11:20:02 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

  3. Port-forward the kibana-kibana service in order to access Kibana locally. Leave this command running in the background as well for the remainder of this tutorial.

    kubectl port-forward svc/kibana-kibana 5601:5601

Configure Kibana

Before visualizing Pod logs, Kibana must be configured with an index pattern for Filebeat’s indices.

  1. With the previous port-forward command running in another terminal window, open your browser and navigate to http://localhost:5601

  2. A page similar to the following should render in your browser.

    Initial Kibana Page

  3. To begin configuring index patterns, scroll down until the Index Patterns button appears, and click it.

    Kibana Home Page Index Patterns

  4. The Index Patterns page should be displayed. Click the Create index pattern button to begin.

    Kibana Index Patterns Page

  5. From this page, enter “filebeat-*” into the Index pattern text box, then click the Next step button.

    Kibana Create Index Pattern

  6. In the following page, select @timestamp from the Time Filter field name dropdown menu, then click the Create index pattern button.

    Kibana Create Index Pattern

  7. A page with the index pattern details will then be shown. Click the Discover compass icon from the sidebar to view incoming logs.

    Kibana Select Discover

  8. The Discover page provides a realtime view of logs as they are ingested by Elasticsearch from your Kubernetes cluster. The histogram provides a view of log volume over time, which by default, spans the last 15 minutes. The sidebar on the left side of the user interface displays various fields parsed from json fields sent by Filebeat to Elasticsearch.

  9. Use the Filters box to search only for logs arriving from Kibana Pods by filtering for kubernetes.container.name : "kibana". Click the Update button to apply the search filter.

    Note
    When searching in the filters box, field names and values are auto-populated.

    Kibana Filter

  10. In order to expand a log event, click the arrow next to an event in the user interface.

    Kibana Open Log Event

  11. Scroll down to view the entire log document in Kibana. Observe the fields provided by Filebeat, including the message field, which contains standard out and standard error messages from the container, as well as the Kubernetes node and Pod name in fields prefixed with kubernetes.

    Kibana Log Document

  12. Look closely at the message field in the log representation and note that the text field is formatted as json. While the terms in this field can be searched with free text search terms in Kibana, parsing this field will generally yield better results. The following section explains how to configure Filebeat and Kibana to achieve this.

Update Stack Configuration

At this point, the Elastic stack is functional and provides an interface to visualize and create dashboards for your logs from Kubernetes. This section will explain how to further configure the various components of the stack for greater visibility into your Kubernetes environment.

  1. Create a values file for Filebeat. This configuration will add the ability to provide autodiscover hints. Instead of changing the Filebeat configuration each time parsing differences are encountered, autodiscover hints permit fragments of Filebeat configuration to be defined at the Pod level dynamically so that applications can instruct Filebeat as to how their logs should be parsed.

    filebeat-values.yml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10

    ---

filebeatConfig:
filebeat.yml: |
filebeat.autodiscover:
providers:
  - type: kubernetes
hints.enabled: true
output.elasticsearch:
hosts: '\${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'

  2. Upgrade the filebeat deployment to use this new configuration file:

    helm upgrade --values filebeat-values.yml filebeat elastic/filebeat

  3. Once this command completes, Filebeat’s DaemonSet will have successfully updated all running Pods.

  4. Next, create a Kibana values file to append annotations to the Kibana Deployment that will indicate that Filebeat should parse certain fields as json values. This configuration file will instruct Filebeat to parse the message field as json and store the parsed object underneath the kibana field.

    kibana-values.yml 
    1
2
3
4
5

    ---

podAnnotations:
co.elastic.logs/processors.decode_json_fields.fields: message
co.elastic.logs/processors.decode_json_fields.target: kibana

  5. Upgrade the Kibana Helm release in your Kubernetes cluster, passing this file as an argument for the Chart values.

    helm upgrade --values kibana-values.yml kibana elastic/kibana

  6. Note, triggering a rolling Pod update of Kibana will cause the previous port-forward to lose track of running Pods. Terminate the previous Kibana port-forward command in the background terminal with Ctrl-C and start the command again:

    kubectl port-forward svc/kibana-kibana 5601:5601

  7. Open a browser window to http://localhost:5601 and navigate to the same Index Patterns page again:

    Kibana Home Page Index Patterns

  8. From the Index Patterns page, select the filebeat-* index pattern.

    Kibana Filebeat Index Pattern

  9. From the index pattern page for filebeat-*, select the Refresh field list button.

    Kibana Refresh Fields

  10. Confirm this action by selecting the Refresh button in the pop-up dialog.

    Kibana Refresh Fields Confirm

  11. Navigate to the “Discover” page.

    Kibana Select Discover

  12. Filter for kibana containers again, scroll down, and expand a log document. Note that various fields have been parsed into the kibana field, such as kibana.req.method, indicating which HTTP verb was issued for a request for Kibana.

    Kibana Parsed Fields

Metricbeat

In addition to collecting logs with Filebeat, Metricbeat can collect Pod and node metrics in order to visualize information such as resource utilization.

Install Metricbeat

  1. Deploy the metricbeat chart.

    helm install metricbeat elastic/metricbeat

  2. You should see output similar to the following:

      
NAME: metricbeat
LAST DEPLOYED: Mon Feb 10 11:32:12 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Watch all containers come up.
  $ kubectl get pods --namespace=default -l app=metricbeat-metricbeat -w

  3. Confirm that Metricbeat has started to index documents into Elasticsearch by sending a request to the locally-forwarded Elasticsearch service port:

    curl http://localhost:9200/_cat/indices

  4. At least one metricbeat index should be present, similar to the following:

      
green open metricbeat-7.3.2-2019.09.30-000001 N75uVk_hTpmVbDKZE0oeIw 1 1   455  0   1.1mb 567.9kb

Load Dashboards

Metricbeat can install default Dashboards into Kibana to provide out-of-the-box visualizations for data collected by Kubernetes.

Before following these steps, ensure that the port-forward command to expose Kibana over port 5601 locally is still running.

Run the following commands on your local machine. This will communicate with Kibana over 127.0.0.1:5601 to import default Dashboards that will be populated by data from Metricbeat.

Note

Your commands should use the same version of Metricbeat deployed to your Kubernetes cluster. You can find this version by issuing the following command:

helm get values --all metricbeat | grep imageTag

For Linux

  1. Get the Metricbeat package.

    wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.5.2-linux-x86_64.tar.gz

  2. Unzip the package.

    tar xvzf metricbeat-7.5.2-linux-x86_64.tar.gz

  3. Navigate to the directory.

    cd metricbeat-7.5.2-linux-x86_64

  4. Setup the dashboards.

    ./metricbeat setup --dashboards

For MacOS

  1. Get the Metricbeat package.

    wget https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.5.2-darwin-x86_64.tar.gz

  2. Unzip the package.

    tar xvzf metricbeat-7.5.2-darwin-x86_64.tar.gz

  3. Navigate to the directory.

    cd metricbeat-7.5.2-darwin-x86_64

  4. Setup the dashboards.

    ./metricbeat setup --dashboards

Explore Dashboards

  1. Open a browser window to http://localhost:5601 and click the Dashboards icon on the left sidebar.

    Kibana Dashboards Link

  2. In the search box, enter “kubernetes” and press Enter. Select the [Metricbeat Kubernetes] Overview ECS dashboard.

    Kibana Dashboards

  3. The following dashboard displays several types of metrics about your Kubernetes cluster.

    Kibana Kubernetes Dashboards

  4. You can explore the various visualizations on this page in order to view metrics about Pods, nodes, and the overall health of the Kubernetes cluster.

Next Steps

From this point onward, any additional workloads started in Kubernetes will be processed by Filebeat and Metricbeat in order to collect logs and metrics for later introspection within Kibana. As Kubernetes nodes are added or removed, the Filebeat and Metricbeat DaemonSets will automatically scale out Pods to monitor nodes as they join the Kubernetes cluster.

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

Join our Community

Find answers, ask questions, and help others.

This guide is published under a CC BY-ND 4.0 license.