Typically, Persistent Volume Claims on Kubernetes are treated as a singular entity completely decoupled from your workload. The actual physical location doesn't really matter. But what if you wanted an atomic operation where all Persistent Volume Claims that make up an application in a microservice architecture need to be atomically protected to ensure referential integrity? Would you stop the application, sequence the operation or take a shotgun approach and hope for the best?

In this blog post, we'll use the HPE CSI Driver for Kubernetes to create Volume Groups that allow users to group Persistent Volume Claims together and use those Volume Groups to perform CSI Volume Snapshots through Snapshot Groups.

In other storage infrastructure management systems, the term "Volume Groups" is usually referred to as "Consistency Groups". It is the industry standard to create volume snapshots with referential integrity. This capability was introduced in the HPE CSI Driver for Kubernetes v1.4.0 and more information about the release may be found on Around The Storage Block.

TL;DR

A variant of the demonstrative steps below has been captured in a screencast that is available on YouTube. If you prefer watching and listening instead of reading, please go ahead and watch the screencast.

Just don’t forget to come back to read the "Learn more" section at the end of this article for important information.

Watch on YouTube!

Prerequisites

The examples we're going to walk through require that the HPE CSI Driver for Kubernetes v1.4.0 or later has been installed along with the CSI external snapshotter. Examples also assume a VolumeSnapshotClass named "hpe-snapshot" exists on the cluster.

• Install the HPE CSI Driver using the Helm Chart
• Enable CSI snapshots
• Add a HPE storage backend
• Create a VolumeSnapshotClass and of course a StorageClass

No particular parameters are needed in either the VolumeSnapshotClass or StorageClass but the backend Secret is assumed to be named "hpe-backend" and reside in the "hpe-storage" Namespace.

In the examples below, we're using HPE Nimble Storage. Any Container Storage Provider (CSP) will work that supports VolumeGroups and SnapshotGroups.

Pick an application

In order to illustrate the fact that multiple snapshots are being created, either pick an application that requires multiple volumes or deploy a microservice stack comprised of multiple stateful applications. In this example we'll use WordPress from the bitnami/wordpress Helm Chart.

We're using the following "values" file for the deployment:

mariadb:
  architecture: replication

Add the Bitnami repo:

helm repo add bitnami https://charts.bitnami.com/bitnami

Install the WordPress chart:

helm install my-wordpress bitnami/wordpress -f wp-values.yaml

Once deployed, there should be three PersistentVolumeClaims on the cluster.

kubectl get pvc -o name
persistentvolumeclaim/data-my-wordpress-mariadb-primary-0
persistentvolumeclaim/data-my-wordpress-mariadb-secondary-0
persistentvolumeclaim/my-wordpress

Adding content to the WordPress site being deployed is completely optional as restore procedures are not covered in this tutorial. However, if you're interested in performing a restore from the VolumeSnapshots we're creating, please see the CSI snapshots tutorial in this HPE DEV blog post.

Volume Groups

The CustomResourceDefinition (CRD) that users interact with is called a VolumeGroup. In order to facilitate the creation of VolumeGroups a Kubernetes cluster administrator needs to create another CRD called a VolumeGroupClass.

The Volume Group Provisioner (depicted below) is a Kubernetes CSI sidecar container that performs a number of duties to facilitate volume grouping.

The VolumeGroupContent CRD is managed solely by the Volume Group Provisioner.

Let's start by creating a VolumeGroupClass. The remaining YAML in this tutorial is assumed to be pasted into a terminal verbatim, like this:

kubectl create -f- <hit ENTER>
< paste the YAML content >
^D <hit CTRL+D on a new line>

Let's begin:

---
apiVersion: storage.hpe.com/v1
kind: VolumeGroupClass
metadata:
  name: my-volume-group-class
provisioner: csi.hpe.com
deletionPolicy: Delete
parameters:
  description: "HPE CSI Driver for Kubernetes Volume Group"
  csi.hpe.com/volume-group-provisioner-secret-name: hpe-backend
  csi.hpe.com/volume-group-provisioner-secret-namespace: hpe-storage

At this stage nothing has been created on the backend storage system. The next step is to create a blank VolumeGroup.

---
apiVersion: storage.hpe.com/v1
kind: VolumeGroup
metadata:
  name: my-volume-group
spec:
  volumeGroupClassName: my-volume-group-class

Now, logging in to the backend HPE Nimble Storage array, we'll see a new Volume Collection has been created.

Nimble OS $ volcoll --list
--------------------+---------------+-------------------------------------------
Volume Collection    Application     Owned By
Name                 Synchronization
--------------------+---------------+-------------------------------------------
volumegroup-e96aa858-93e9-424c-a593-6a6f216368c0 none            nva-grp

Inspecting further:

Nimble OS $ volcoll --info volumegroup-e96aa858-93e9-424c-a593-6a6f216368c0
Name: volumegroup-e96aa858-93e9-424c-a593-6a6f216368c0
Description: HPE CSI Driver for Kubernetes Volume Group
Owned by: nva-grp
Application synchronization: none
Application server: N/A
Application ID: N/A
Cluster name: N/A
Service name: N/A
VMware vCenter hostname: N/A
VMware vCenter username: N/A
VMware vCenter password: N/A
Backup agent hostname: N/A
Backup agent username: N/A
Backup agent password: N/A
Associated volumes: none
Associated pinned volumes: none
Snapshot collection count: 0
Created: Jan 13 2021 16:40:48
Last configuration change: Jan 13 2021 16:40:48
Replication Type: Periodic Snapshot
Synchronous Replication State: N/A
Synchronous Replication Last In Sync: N/A
Synchronous Replication Resync %: N/A

Adding members to the VolumeGroup is done by adding annotations to PersistentVolumeClaims. This can be done in a number of ways. Either the claims may be created with the annotation, using the kubectl patch command and the kubectl annotate command. I prefer using the kubectl annotate command. Let's do it.

kubectl annotate pvc/data-my-wordpress-mariadb-primary-0 \
  pvc/data-my-wordpress-mariadb-secondary-0 \
  pvc/my-wordpress \
  csi.hpe.com/volume-group=my-volume-group

Annotations doesn't need to be created all at once. They can be added (or removed) individually.

At this stage you can inspect the VolumeGroup on the Kubernetes cluster or the Volume Collection on the array to confirm that PersistentVolumeClaims have been added to the VolumeGroup.

kubectl get -o yaml volumegroup/my-volume-group \
  -o 'jsonpath={.spec.persistentVolumeClaimNames[*]}'

Next, let's create some atomic VolumeSnapshots.

Snapshot Groups

The SnapshotGroup CRD is primarily what users interact with. The Snapshot Group Snapshotter depicted below carries out all the backend work and populates Kubernetes with the necessary CRDs.

In a similar fashion to VolumeGroupClasses a SnapshotGroupClass needs to be created by an administrator.

---
apiVersion: storage.hpe.com/v1
kind: SnapshotGroupClass
metadata:
  name: my-snapshot-group-class
snapshotter: csi.hpe.com
deletionPolicy: Delete
parameters:
  csi.hpe.com/snapshot-group-snapshotter-secret-name: hpe-backend
  csi.hpe.com/snapshot-group-snapshotter-secret-namespace: hpe-storage

Now, users may create SnapshotGroups. Upon creation, the SnapshotGroups will automatically be populated by VolumeSnapshots by the Snapshot Group Snapshotter. Let's create a SnapshotGroup and see.

---
apiVersion: storage.hpe.com/v1
kind: SnapshotGroup
metadata:
  name: my-snapshot-group-1
spec:
  source:
    kind: VolumeGroup
    apiGroup: storage.hpe.com
    name: my-volume-group
  snapshotGroupClassName: my-snapshot-group-class
  volumeSnapshotClassName: hpe-snapshot

While unrelated to this tutorial, the "hpe-snapshot" VolumeSnapshotClass should already exist on the cluster as described in the prerequisites above. The .spec.source indicates our previously created VolumeGroup to snapshot. Now, let's check for VolumeSnapshots.

kubectl get volumesnapshots -o name
volumesnapshot.snapshot.storage.k8s.io/my-snapshot-group-1-data-my-wordpress-mariadb-primary-0
volumesnapshot.snapshot.storage.k8s.io/my-snapshot-group-1-data-my-wordpress-mariadb-secondary-0
volumesnapshot.snapshot.storage.k8s.io/my-snapshot-group-1-my-wordpress

Presto! Further, logging in to the backend array, we can now see a new Snapshot Collection with populated entries.

Nimble OS $ snapcoll --list
--------------------+---------------------------------------+-------+-----------
Volume Collection    Snapshot Collection                     Num     Replication
Name                 Name                                    Snaps   Status
--------------------+---------------------------------------+-------+-----------
volumegroup-e96aa858-93e9-424c-a593-6a6f216368c0 snapshot-fde72bb7-6633-4e6f-841c-3efaa0444710      3 N/A

A few points to note.

• SnapshotGroups may be deleted from the cluster and will in turn delete the VolumeSnapshots and backend Snapshot Collection
• VolumeSnapshots may be used to perform restores (clone) from as any other VolumeSnapshot

Learn more

This is just the tip of the iceberg when it comes to data management innovation from HPE. Stay tuned to the HPE Developer Community to learn more about upcoming features and capabilities.

• Explore the HPE CSI Driver for Kubernetes on GitHub
• Check out the release announcement on Around The Storage Block
• Install the latest HPE CSI Driver using the Helm Chart from ArtifactHub.io
• Visit the HPE Storage Container Orchestration Documentation (SCOD) portal for everything pertaining to the HPE CSI Driver
• Sign up for a HPE DEV Hack Shack Workshop to learn about CSI on the HPE Ezmeral Container Platform

HPE is eager to learn what our customers and partners are doing with Kubernetes and data management. Join the HPE DEV Slack community to share your thoughts and engage with the team. Sign up at slack.hpedev.io and sign in at hpedev.slack.com, we hang out in #nimblestorage, #3par-primera and #kubernetes.