How To Backup Elasticsearch On Kubernetes With Google Cloud Storage and Kibana

5 min readJun 21, 2020

Twitter | LinkedIn | YouTube | Instagram

We all make mistakes and we make them quite often. Deleting a pod, a persistence volume or even a whole namespace is an easy thing to do in Kubernetes and if you delete the right one you can say goodbye to your Elasticsearch cluster.

Today we will be going through how to backup our Elasticsearch cluster (Running on Kubernetes) data into Google Cloud Storage and protect it from unexpected data losses.

What will we be seeing?

Create Google Cloud Storage Bucket
Create Service Account
Create a Kubernetes Secret to Hold Your Key
Configure Elasticsearch To Connect To GCS
Take a snapshot through Elasticsearch
Take a snapshot through Kibana

Create Google Cloud Storage Bucket

Let’s get started by creating a Google Cloud Storage Bucket, you can do it by clicking here.

For this example, we will name our bucket elasticsearch-backup .

Create Service Account

Now that we have our GCS bucket, we need to set up a service account that will be used by Elasticsearch to Backup.

You can create it by clicking here.

Give it a name and grant it permissions as storage.objectAdmin

Download the JSON API key and save it i your computer as gcs_backup_key.json

Create a Kubernetes Secret To Hold Your Key

Now that you have your JSON key file in hands, let’s create a Kubernetes Secret to store that file and allow Elasticsearch to access it.

On the directory where you saved your JSON file, run:

kubectl create secret generic gcs-backup-key --from-file=gcs_backup_key.json=gcs_backup_key.json

Configure Elasticsearch To Connect To GCS

In order to do it, we will need to:

Install the Google Cloud Storage plug in
Add the JSON key to the configurations of Elasticsearch

Both actions must be done before the cluster actually starts and in order to accomplish it, we will use Init Containers.

Init containers allow us to run commands before the actual entrypoint is run and the Elasticsearch cluster has started. If you followed my Deploy the Elastic Stack with the Elastic Cloud On Kubernetes (ECK) story, you already have an init container configuring the vm.max_map_count that you can see as an example:

initContainers:
  - name: sysctl
  securityContext:
    privileged: true
  command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']

First, let’s use the secret we created before as a volume in our manifest. You can do it by adding the following to the volume sections:

volumes:
- name: gcs-backup-key
  secret:
    secretName: gcs-backup-key

We will need to add two more initContainers to this spec, the first one to install the plugin:

- name: install-plugins
  command:
  - sh
  - -c
  - |
  bin/elasticsearch-plugin install --batch repository-gcs

We will also need another init container to add the JSON key where we will mount our file specified in the previous step and add it to the Elasticsearch Keystore:

- name: add-gcs-key
  command:
  - sh
  - -c
  - |
  echo y | bin/elasticsearch-keystore add-file
  gcs.client.default.credentials_file ./key/gcs_backup_key.json
  volumeMounts:
  - name: gcs-backup-key
    mountPath: "/usr/share/elasticsearch/key"
    readOnly: true

We will have something like:

Now you can start your pod and we should be able to take our first Snapshot!

First we will learn how to do it using Elasticsearch directly and then we will be going through on how to do it through Kibana.

Take a snapshot through Elasticsearch

Before we create our first backup we need to set up a repository, that’s where your backups live, to do so you can send a request as:

PUT _snapshot/my_backup
{
  "type": "gcs",
  "settings":
  {
    "bucket": "elasticsearch-backup",
    "base_path": "backups"
  }
}

This will create a repository named my_backup in the bucket we created in the beginning of this tutorial and in the path “backups” inside this bucket. The base_path is optional.

We are now ready to take our first snapshot by sending the following request:

PUT /_snapshot/my_backup/snapshot_1

Depending on the size of your database, your backup might take a long time to save. To check its progress you can send a request as:

GET /_snapshot/my_backup/snapshot_1

Which will retrieve one of the information below:

Now let’s see how we can achieve the same thing in Kibana.

Take a snapshot through Kibana

In Kibana it is easier to set up a repository and take a snapshot. Let’s see how we can do it.

First of all open Kibana and go to Management > Snapshot and Restore.

On the Repositories tab, click Register a repository.
Provide a name for your repository and select type Google Cloud Storage.
Provide the following settings:
- Client: default
- Bucket: Your GCS bucket name (elasticsearch-backup in our case)
- Add any other settings that you wish to configure.
- Click Register.
- Click Verify to confirm that your settings are correct and the deployment can connect to your repository.

Your snapshot repository is now set up using GCS! Now let’s automate our snapshots!

Open the Policies view.
Click Create a policy.

3. As you walk through the wizard, enter the following values:

4. Review your input, and then click Create policy.

Congratulations, your new policy is created and you are now taking daily snapshots of your Elasticsearch cluster! 🎉

Contribute

Writing takes time and effort. I love writing and sharing knowledge, but I also have bills to pay. If you like my work, please, consider donating through Buy Me a Coffee: https://www.buymeacoffee.com/RaphaelDeLio

Or by sending me BitCoin: 1HjG7pmghg3Z8RATH4aiUWr156BGafJ6Zw

Follow Me on Social Media

Stay connected and dive deeper into the world of Elasticsearch with me! Follow my journey across all major social platforms for exclusive content, tips, and discussions.