Snapshot using Azure Repository Plugin in Elasticsearch(ECK) !

Elasticsearch

Elasticsearch comes with lot of cool features and index snapshot is one of it. A snapshot is a backup taken from a running Elasticsearch cluster. You can take snapshots of individual indices or of the entire cluster. Snapshots allows you to ensure proper backups are in place in case of any disaster.

These Snapshots can be placed in either local or remote repositories. Remote repositories can reside on cloud providers storage service. Elasticsearch can use them via plugins. In this post we talk about using Azure Repository plugin for keeping snapshots in Azure Cloud.

I have deployed a Elasticsearch cluster using ECK.
ECK is K8s operator based deployment mechanism provided by Elasticsearch.

Before we proceed further, following items need to be in place.

  1. The Image which I am using for Elasticsearch deployment is custom image build on top of base Elasticsearch and Azure plugin deployed. (https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-custom-images.html)
apiVersion: v1
kind: Secret
metadata:
namespace: dev
name: snapshot-secret
type: Opaque
data:
azure.client.default.account: <base64 account name(use only account name not the URL)>
azure.client.default.key: <base64 account key>

To generate base64 you can use command line

Encode: echo -n 'account-name' | base64
Decode: echo -n 'encrypted-account-name' | base64 -d

Now when you are ready with above all details spin-up a cluster for Elasticsearch. Make sure in spec you pass the secret information which you have created above.

spec:
secureSettings:
- secretName: snapshot-secret

Yaml for Elasticsearch :

You are done ! Once the cluster comes up. Connect it via Kibana or you can use postman to create repository. [Before you can take a snapshot, you have to “register” a snapshot repository. A snapshot repository is really just a storage location]

PUT _snapshot/my_backup
{
"type": "azure",
"settings":
{
"container": "my_azure_container",
"base_path" : "/"
}
}

Lets take a quick Snapshot of sample index.

PUT /_snapshot/my_backup/snapshot_1?wait_for_completion=true
{
"indices": "kibana_sample_data_logs",
"ignore_unavailable": true,
"include_global_state": false,
"metadata": {
"taken_by": "arun",
"taken_because": "backup before upgrading"
}
}

When you run it. Output —

{
"snapshot" : {
"snapshot" : "snapshot_1",
"uuid" : "Odde3dBTIfrgth5g335A",
"version_id" : 7070099,
"version" : "7.7.0",
"indices" : [
"kibana_sample_data_logs"
],
"include_global_state" : false,
"metadata" : {
"taken_by" : "arun",
"taken_because" : "backup before upgrading"
},
"state" : "SUCCESS",
"start_time" : "2020-06-22T20:50:40.010Z",
"start_time_in_millis" : 1592859040010,
"end_time" : "2020-06-22T20:50:42.231Z",
"end_time_in_millis" : 1592859042231,
"duration_in_millis" : 2221,
"failures" : [ ],
"shards" : {
"total" : 1,
"failed" : 0,
"successful" : 1
}
}
}

That’s It !

Snapshot

Issues Which I faced:

"Caused by: java.security.InvalidKeyException: Storage Key is not a valid base64 encoded string.",  "at com.microsoft.azure.storage.StorageCredentials.tryParseCredentials(StorageCredentials.java:68) ~[?:?]",  "at com.microsoft.azure.storage.CloudStorageAccount.tryConfigureServiceAccount(CloudStorageAccount.java:664) ~[?:?]",  "at com.microsoft.azure.storage.CloudStorageAccount.parse(CloudStorageAccount.java:285) ~[?:?]",  "at org.elasticsearch.repositories.azure.AzureStorageService.createClient(AzureStorageService.java:140) ~[?:?]",  "at org.elasticsearch.repositories.azure.AzureStorageService.buildClient(AzureStorageService.java:117) ~[?:?]",  "at org.elasticsearch.repositories.azure.AzureStorageService.client(AzureStorageService.java:110) ~[?:?]",  "... 11 more"] }

Sol:- Values for Account name and Key details were not proper !

Container Naming is imp: https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-azure-validation.html

If in case you have to use some other file share location for restore like a file system, then path.repo comes to rescue.

The path.repo setting supports Microsoft Windows UNC paths as long as at least server name and share are specified as a prefix and back slashes are properly escaped:

nodeSets:
- config:
node.data: true
xpack.monitoring.collection.enabled: true
path.repo: ["\\\\WINDOWSERVER\\file_share"]

In case of Unix share

nodeSets:
- config:
node.data: true
xpack.monitoring.collection.enabled: true
path.repo: ["/mount/restore", "/mount/restore_backups"]

register repository and now it can be used :

PUT /_snapshot/fs_location
{
"type": "fs",
"settings": {
"location": "\\\\WINDOWSERVER\\file_share",
"compress": true
}
}

Snapshot helpful APIs:

# see all snapshot repositories
GET _snapshot/_all
# To see all snapshots in a repository
GET _snapshot/fs_location/_all
#restore a snapshot
POST _snapshot/fs_location/1/_restore
{
"indices": "customndex*,systemindex*",
"ignore_unavailable": true,
"include_global_state": false,
"include_aliases": false,
"partial": false,
"index_settings": {
"index.blocks.read_only": false
},
"ignore_index_settings": [
"index.refresh_interval"
]
}

Ref:

https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-azure-usage.html
https://www.elastic.co/guide/en/elasticsearch/plugins/7.8/repository-azure-client-settings.html
https://github.com/elastic/elasticsearch/issues/20633

In quest of understanding How Systems Work !

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store