Using the VMware Event Broker Appliance to Automatically Expand a Pure Storage FlashArray Datastore

Jun 24, 2020 VMware Event Broker Appliance PowerCLI vSphere FlashArray

Share on:

Something we get asked quite often it seems here at Pure Storage, is can we automatically expand a datastore when it gets full? The answer is yes! With all of the integrations with automation platformsm, Pure Storage Arrays have many options. This blog will cover how to handle this with the VMware Event Broker Appliance (VEBA)!

Ever since I first heard about VEBA it really interested me. William Lam and Michael Gasch presented on it at VMworld 2019. Many things have gotten in the way, but I finally had time to sit down and look at how to deploy and use VEBA.

Introduction

Before I cover why we may get asked how to automatically expand a volume, I want to cover one thing I love about Pure Storage FlashArray. The array itself has no per-volume performance limits. This means that if a customer wants to deploy a single large datastore or many small datastores it is really up to them. In some cases, customers have datastores carved up for certain roles’s such as Snapshots or Replication. In this case, if a datastore gets full they just want to be able to easily expand it. Currently when customers ask if this can be done, we point them over to Cody Hostermans blog. This configuration still works, but its back from 2016 and its SNMP (yuck) sorry Cody.

Deployment of VEBA

Since VEBA is a pre-built OVA appliance, deployment is extremely simple. It is still a VMware Fling but has quite the community behind it. When you deploy the OVA to your vCenter Server you have the choice on whether you wish to use OpenFaaS or AWS Event Bridge as the event broker. For my example I am using OpenFaas.

You can learn more about the deployment with OpenFaas or AWS EventBridge within their documentation pages.

Implementing the VEBA Function

Luckily there is a great community for VEBA, and it was useful for me getting up and running. There are prebuilt functions and a great #vmware-event-broker-appliance channel on vmwarecode.slack.com. For this particular use case an example already existed for datastore usage that I was able to build upon.

A few things I learned along the way…

Create your own image in DockerHub or in Harbor First

Docker must be installed previous to implementing functions.

To begin, head over to hub.docker.com and create a repository for your function. Once you have it created you should have a place to store your image such as username/repository, in my case it is dstamen/vebapowercli-increaseds-pure

Since this particular image requires PowerCLI and Powershell it will use the vmware/powerclicore:latest base image.

Using docker pull it down to your local machine.

1docker pull vmware/powerclicore:latest

Once it has been pulled down use docker images to see the image information.

1docker images|grep vmware/powerclicore

We will now need to tag and push the image to our own repository to use as a base image. For the tag identifier use the Image ID from docker images.

1docker tag a0fceeaed43e dstamen/test
2docker push dstamen/test

Now that our image is ready, we can proceed to configuring the function for VEBA.

Configure the VEBA Function

Next, we will need to update the function files for your environment. Grab a copy of the VEBA function from GitHub.

There are 3 main files we will need to look at.

stack.yml
vc-ds-config.json
/handler/script.ps1

stack.yml

In my experience, the stack.yml is similar to a kubernetes manifest where it specifies the details for the function to run. You will need to update this file to include the following parameters:

Gateway: This will be the FQDN or IP of your VEBA server.
Function Name: This will be the name of the function defined in OpenFaas.
Image: This will be the username/repository you pushed your image to in the previous step.

vc-ds-config.json

This file is the secrets file that stores your configuration such as passwords and data secrely. As I am also including a Slack notification there is a few extra fields.

SLACK_URL: This is the webhook URL you will setup as part of a slack integration.
SLACK_CHANNEL: This will be the channel you wish for the slack messages to be posted to.
SUBJECT: This will be the title of the slack notifications.
VC: This is the FQDN of the vCenter Server where the function is to be ran against.
VC_USERNAME: This is the username of a user that has rights to expand the datastore in the vCenter Server.
VC_PASSWORD: This is the password for the above user.
VC_ALARM_NAME: This is the alarm name for this function, do not change unless you have created a custom alarm.
DATASTORE_NAMES: This is an array of datastore names for the function to trigger against. If the datastore is not in this list, it will not be automatically expanded.
PURE_USERNAME: This is a user that has access to expand the volume on the FlashArray.
PURE_PASSWORD: This is the password for the above user.

/handler/script.ps1

This is the core script that is executed for the function. As this runs as a container, whatever is done must be run in Powershell Core. The script entails parsing the event when it comes in to make sure the datastore exists in the secrets file, and then based on threshold will expand the disk by that capacity. For my lab if the threshold reached warning I expanded by 10GB. If the threshold reached error I expanded by 20GB, this can be easily customized in the script for what works for your environment.

The script will only need to be modified to include the connection endpoints of your FlashArray. What the script will do is connect to those flasharrays and find the FlashArray the datastore volume resides on, expand it and then expand the datastore. Once complete it sends a notification to the slack channel.

Currently the Pure Storage PowershellSDK does not support PowershellCore. An update is coming, and when it is available I will update this post, with a much shorter script. For the time being I have added functions to the script that will allow this script to run in Powershell Core with no issues. Please see the GitHub repository for the file with the included functions.

Once the files are configured we can now push the function to the OpenFaas Gateway.

Deploying the function to OpenFaas Gateway

Pre-requisite: You will need to install faas-cli

Initially, this was one of the components I was having the most issues with, until someone mentioned a great shortcut!

1VEBA_GATEWAY=https://veba.lab.local
2export OPENFAAS_URL=${VEBA_GATEWAY}
3
4faas-cli login --username admin --password-stdin --tls-no-verify

Create function secret (only required once)

1faas-cli secret create vc-slack-config --from-file=vc-slack-config.json --tls-no-verify

The Hard Way

1faas-cli build -f stack.yml
2faas-cli push -f stack.yml
3faas-cli deploy -f stack.yml --tls-no-verify

The Easy Way

1faas-cli up --tls-no-verify

Validating the Deployment

After the deploy is complete, you can see the status. faas-cli up will build the image based on the files provided, pushing the script into a docker image. It will then push the image to your dockerhub repository. It will then push the function to your OpenFaas server where it will pull down your docker image and run the function.

Troubleshooting the Deployment

If at any time you experience issues with your function, there are two ways to see what is going on.

Option 1

You can use faas-cli to review the logs, however this will only look at the past 30 seconds and will end when the request finishes.

1faas-cli logs veba-powercli-increaseds-pure --tls-no-verify

Option 2

The preferred option is to tail the logs on the VEBA appliance itself. Since the function runs as a k8s container, there is a couple steps to view the logs.

1
2ssh [email protected] #ssh to veba server
3kubectl get pods -n openfaas-fn ##get k8s pods in the openfaas-fn namespace
4kubectl logs veba-powercli-increaseds-pure-59db6cd7f9-g4p4r -n openfaas-fn #review the pod logs. this will be regenerated on each deployment of your function

Conclusion

One thing I love about the Pure Storage FlashArray is the suite of integrations we have with different automation providers. This was just one great example how this could be accomplished with Powershell, PowerCLI and VEBA.

What would you like to see next? Leave a comment below!