This site has been formed to contain my ramblings about things on my mind related to technology I work with.
I would call myself a techie guy who has lucked out and had a great career doing what i love, working with new emerging, innovative solutions. As an engineering guy I hope to provide good insight into the technologies and challenges faced working with new tech and a personal view into what they are and the problems they solve.
I was in a meeting with a good customer of Pure Storage when I noticed that their operational panel on the floor was showing a PRTG dashboard. The PRTG Network Monitor tool from Paessler is a very nice agentless monitoring solution that is able to utilise a wide range of methods to monitor any accessible system or application on your network. It is a Windows based solution but supports all required methods to gain remote insights into your services such as SNMP, SSH, Scripts, REST, WMI, packaet sniffing, etc.
I am familiar with PRTG as I use it at home to monitor my home network and various components. This is possible due to the fact that you can run 100 sensors on a free license. The wonderful world of home automation and online media have made my network as critical to my household as it is for corporatations own services. Plus it is also fun to work with solutions like this, not to mention that it provides some interesting insights into your home network when combined with a Ubiquiti setup.
Anyway back to the task in hand, I thought that looking at how we could monitor Pure FlashArray's with PRTG could both help me to familiarise myself with the Purity API. It turns out we do have a couple of options at the Pure OpenSource community Pure/Code within the Python and PowerShell script packs written by some clever dudes in Pure (nod to Barkz). The only consideration for these was that they are focused on the array level performance metrics, where I wanted to see what else we can keep an eye on. Also, as a new Pure employee, as stated prior, it also gives me the chance to work with the Restful API of Purity.
I can say that writing this sensor was a bit of fun and the Restful API of Purity has to be one of the most logical I have worked with. Very simple in structure, versioned to allow for supportability across a range of Purity versions, and a rich capability around those supporting actions such as filtering. It took no time at all to get comfortable with the structure, limits were only my own and those discovered within PRTG.
So What Does It Do?
I wanted to provide a good overview of the current operational health of the arrays monitored while ensuring that the reporting was not overly complicated. As a first release my focus is on array level conditions covering:
Array capacity status
Array performance metrics
General hardware status of Controllers, Shelves and Chassis
General drive health
Array object sensor summary page
I have plans to add further granularity to cover 'volume performance', 'host performance' and 'protection compliance' in upcoming releases, but with the current method there is a limit of 50 channels to one sensor so still determining the best way to meet the number of instances required to monitor at that level and how to cater for lifecycle of objects that are subjected to CRUD operations (well create, update, delete anyway.
With the current script, the sensor reads the metrics from the array via a PowerShell script that directly queries each arrays Restful API. Although there is only a single script, it supports the multiple sensors through the 'Scope' parameter provided within the arguments section of each sensor (more details in installation instructions).
Each sensor can contain one or more channels, a channel can be thought of as a single value aligned to the sensor such as 'capacity consumed' within the 'capacity' sensor. Then as appropriate limits are assigned to the channel that define the thresholds that a channels value goes either above (maxlimit) or under (minlimit) to raise a warning or error condition for the sensor. Some channels have explicit limits set such as those related to capacity and health but others, such as the performance metrics, are left to self determine when an exceptional condition occurs based on operational patterns already recorded. For example latency may stay steady under a millisecond for normal operations but if suddenly it increases to 20 milliseconds for multiple recordings, PRTG will determine this to be an exception and raise a warning condition.
Array capacity sensor
Because each channel needs to provide a value that supports numeric comparisons to determine if limits are met, textual values returned from the array had to be converted to a representative numeric value. The textual values such as for Hardware components status to be reported as 'healthy', is enabled through the use of lookup tables included in the pack. Currently there are two included which provide lookup values for the channels in the 'Drive Health' and 'Hardware Health' sensor status values
Array Hardware Sensor
Where the Array performance and capacity sensors provide a collection of metrics related to the sensors scope of the whole array, the health sensor for hardware provides summary graphs for each major component (Chassis, Controller, Shelf) and for the Drive Health sensor, a status for each drive.
Hardware Health Sensor
For the Hardware Health sensor, summary values were required as reporting each individual hardware component that can be monitored exceeds the 50 channel to a sensor limit (not enforced but supportability and reliability can be compromised). I open to looking at alternative methods that can allow for all components to be individually reported in the future.
Drive Health Sensor
Installation
Requirements
First, get the scripts and associated lookup files from GitHub at Pure Storage Sensor Module. This will provide a Zip file of the required files but you can also download it directly from the GHitHub site at 'https://github.com/davlloyd/purestorage-prtg'
This set contains 4 files including:
Get-PureFA-Sensor.ps1- The main PowerShell script
prtg.standardlookups.purestorage.drivestatus.ovl - Drive status lookup file
prtg.standardlookups.purestorage.hardwarestatus.ovl - Hardware status lookup file
readme.md - Markdown file with Installation instructions
The Powershell script and lookup files need to be copied onto the PRTG server and then associated sensors created accordingly.
Ensure the PRTG server (or PRTG probe if running sensor from a probe)is running PowerShell 4.0
Script Installation
Copy the script Get-PureFA-Sensor.ps1to the directory 'C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML'
Copy the two lookup files prtg.standardlookups.purestorage.drivestatus.ovl and prtg.standardlookups.purestorage.hardwarestatus.ovl to the directory 'C:\Program Files (x86)\PRTG Network Monitor\lookups\custom'
Restart the Windows service 'PRTG Core Server Service'. This step is required to have PRTG read in the new custom lookup files
Sensor Creation
Create a new device in PRTG with the address (IP or FQDN of the FlashArray you want to monitor
Now, select <Add Sensor>. On the search field, type "Script Advanced" and then select the sensor type <EXE/Script Advanced Sensor> from the result list.
On the Add Sensor to Device screen, enter the following:
The sensor's name and tags (optional): There are currently four sensors to create. I have been naming them as follows:
Capacity
Performance
Drive Health
Hardware Health
Under Exe/Script, use the Drop-down to select the script 'Get-PureFA-Sensor.ps1' from the list.
Set the parameters for Array access and sensor scope as follows
[ArrayAddress] (use the PRTG variable '%host' to inherit from device entry
Either of the following combinations for security authentication:
[username] and [password] - This is ok for testing purposes but as password is stored in clear text not recommended for production. This will generate or read the APIKey for the account
[apikey] - preferred access method. The APIKey is generated from within the Purity Console for the preferred account. You do not need to enter account details if specifying an APIKey
[Scope] to set what is monitored for this sensor. Scope option values are:
capacity
performance
hardware
drive
By default the sensors will be run every 60 seconds, adjust accordingly. You can get a view of time for execution by running the scripts in debug mode but ensure that it is disabled when running the script as a sensor.
Is it a bird, plane or the Photon Platform? In the early days of Photon Platforms announcements, in my own mind, I had it pegged as a purpose built platform alternative to 'vSphere Integrated Containers (VIC)'. Keeping in mind that VICs sole purpose in life is to enable Containers in vSphere with a cool concept of a 1:1 alignment of VM to Container. This meant a container can have all the flexibility inherent in a container with all the control and security of a VM. Alright come on down Photon Platform just bigger, better more focused right? Well not really....
So one thing that strikes you with the Photon Platform is the flexibility of choices it provides. From a scheduling cluster manager's perspective you can utilise Swarm, Kubernetes or Mesos out of the box. On the other side, this is ESXi after all (marketing slides aside that keep alluding to the slimmed down hypervisor) so it will happily run Virtual Machines and Containers side by side (well containers within VMs anyway).
Being able to support both is important as not all workloads are created equal in this new age. Let's be honest, not everyone working in this new 'Cloud Native' world has grown a beard, rides a bicycle and wears skinny jeans, there is a variety of personalities that need to be catered for. What is common is we don't all need the assurance and crutches that come from the more traditional platforms. If I fall over it is ok, I will just get myself back up. In the traditional world I would have crutches, cushions, people watching me, ready to catch me if a stumble, etc. This is because in this traditional world I am seen as needed to be treated like a VIP (i.e. Very Important Process) and require all the care of advanced services to keep me going (and the costs associated with those advanced services). Sometimes a more traditional approach may be preferable to containerising (is that a word?) the workload so Virtual Machines live on even if they are seen as cattle (I'll let Duncan Epping define that for you).
This mixed capability is emphasised by the fact that all the management VMs are hosted out of the hypervisors that are flagged as Management hosts (you designate if a host is 'Management' so participates in the control plane or 'Cloud' which only provide a hosting platform capacity, or both). Anyway myself for one, I'm a big fan as by including the ability to host containers and VMs as this provides that flexible, cost and scale focused platform I want without having to look at alternatives such as Openstack. Add to this the fact that it also includes support for the 3 main container and workload cluster scheduling services out of the box and your covering a lot of bases for possible consumers of Photon Platform.
My personal choice is to go forward with Mesos at this stage as it gives me the open choices I want when paired with Marathon. I have played with all three though on Photon as all it takes is that you import the 3 images and enabled the 3 cluster types. I am not here to tell you how to do this or how to create a cluster as there are plenty of blogs that do that, I just want to give my view and some lessons I learnt on the way that may help someone out there.
It is important to note that when creating clusters, they align to there traditional model and not some Hybrid like in VIC. What I mean by that is that if you create a Docker Swarm cluster, the slave nodes will be running the containers in a shared multi-node format. The nodes are utilising PhotonOS as there Operating System but we don't have the 1:1 Container to VM alignment of VIC. This is important to consider as your sizing of those nodes should reflect your needs. There are default 'Flavors' (remember Flavors define the resource configuration profiles) aligned to the different clusters but they can be overwritten at the API, CLI level when creating a cluster.
This system is built as a multi-tenant service from the ground up so when you create workloads they need to be placed within a Project assigned to a Tenant. As you would expect, a Tenant gets an allotment of Resources (vCPUS, memory, Disk Capacity, VMs, etc) which are then subdivided into Resource Tickets (think Gold , Silver Bronze classes or whatever). A Project is then assigned a Resource Ticket which it then further carves out a resource reservation /limit out of. All the way down this logical structure you can add Access Controls via the integration with VMware Lightwave.
So importantly as a tenant, I can create my own cluster service to host workloads or just create VMs directly based on virtual appliances or virtual disks imported into the system. I can then control my VM workloads via the CLI and API into Photon Platforms Controller VM as you would of normally via vCenter, or use the cluster manager of choice to control the workloads within them.
As a final statement on the installation of the clusters what I can say is this. I have built all 3 flavours using simple sandbox methods as well as going through the full manual processes (more then once as I like punishment) and there is a highly compelling value proposition here. To be able to create these services as you require them on a production class platform so simply is huge. There will be challenges with the way it is done now such as how do they stay close to release parity (hence the compelling part of it being Open Source), but this is a new era platform that brings together the traditional on-premise controls to the new cloud native era's service requirements in the one stack. This is great stuff for a version 0.8 release!
Some Notes On Cluster Enablement:
Be Wary Of Resource Stinginess
My nature is to give less then more when allocating resources. This sent my on a dance to the requirements of the various clusters as I did not have enough resources aligned to my Project. Just be aware of the sizing needs (they vary of course depending on your own configuration requirements) and if you are resource constrained, use custom 'Flavors'. I prefer to bang my head on the wall so got very use to the 'Photon Cluster Create' operation being closely followed by the 'Photon Cluster Delete' operation to delete the failure and started again (side note, it didn't like me trying to recreate deleted clusters with the same name, didn't ping this down but got into the habit of using unique names each attempt) :)
Cluster Creation IP Address Requirements
Within the instructions it is emphasised that DHCP needs to be disabled / not present in the management network which is where the clusters are installed. Ironically, if DHCP is disabled the cluster deployment fails with a 'Unable to Obtain an IP Address' error. To progress forward I re-enabled DHCP in the management network (the one that the Photon Controller is installed into). The more accurate requirement description is to ensure the static IP addresses you provide (i.e. for the 'zookeeper' servers in Mesos and 'etcd' servers in Kubernetes and Swarm) are not also served within the active DHCP scopes.
Those Damn Certificates
Maybe you lucky that your company provides an open network and all traffic is created equal. That is not the case in mine, we trust no-one and inject ourselves into every certificate coming into the network. Any problems with this is mitigated by adding our own certificates into the trusted root but this is difficult for those environments that are created dynamically as these clusters are.
When you create a cluster PhotonOS based VMs are created corresponding to your configurations requirements (i.e. how many slaves, how many provide the interconnect service) and are then configured by templates contained on the Controller. Where the certificates being trusted becomes critical is the fact that the services run as containers within the VMs and as such the images need to be downloaded as part of the initial execution. The end result is the cluster creation operations fail with 'Time Exceeded' errors.
Anyway its an easy fix! I edited the template files contained in the controller directory '/usr/lib/esxcloud/deployer/scripts/clusters' with descriptive names such as 'kubernetes-master-user-data-template' to inject our certificates into the VMs. This at least made that problem go away and you of course could do this to apply any other customisations that may be required. It would be nice to see some option in the future to inject certificates into the process. I also provided this same feedback for 'vSphere Integrated Containers' as had the same problem there!
What am I saying, you are probably asking? I have had great fun over the last week getting down and dirty with one of the headline EMC {code} projects, RackHD. This is a very cool solution for taking care of your low level activities for bare metal infrastructure. Think configuration management for all those physical pieces in a rack in the data center. This is a way to manage the firmware through to the personas of these devices, very good stuff and it is Open Source!
I guess first up I should give some context around what I am referring to with EMC {code}. At EMC we can be perceived to be a player in the more traditional areas of IT Infrastructure, EMC {code} is one example (of many) where this is a long way from the current day EMC that I work for. EMC Code is the landing place for developer enablement and open source projects for EMC.
This is your one stop shop to find any open source projects supported by EMC, community projects by EMC staff, partners and customers. It also includes training content and projects aligned to helping enable the developer community on wide spectrum of those new Mode 2, Platform 3 Cloud Native, SDDC technologies (you get the idea). I strongly encourage anyone to have a look and join in. There are some very smart people in this community and very accessible via tools such as Git and Slack.
Anyway back to RackHD, what is it really. So, we know that there is a number of configuration management solutions out there today such as Puppet, Ansible, Salt, Chef, etc but these tend to have a common trait, they look after the nodes/hosts/clients through remote agentless access (SSH, WBEM, etc) or via agents that are installed in the target device. What this of course requires is that the device is ready to accept remote requests or have agents installed. in other words they take control and configuration management of the platform once it is operational. A few such as Puppet with Razor have the ability to control the physical world but not as an all inclusive service with mulitple action workflow smarts.
Looking at RackHD you have a solution that provides:
Bare metal configuration management across the physical infrastructure stack. So not just with the compute but all the bits that go in a rack (hence the name) including:
The compute
The Network
The Storage
The enclosures that may contain the nodes
The Racks themselves and PDUs (remember, just plug the thing in)
A strong but intuitive Restful API
Aligns to the 'Infrastructure as Code' model. Allows node definition, associated workflows and SKUs to be fed in as JSON files via the API (or UI if thats your preference)
Fully self contained service providing all the mechanisms required to control a physical environment such as DHCP, PXE,TFTP, HTTP, etc
A scale-out architecture that can grow to your environment needs
Full support of dynamic discovery and physical control through interactions with hardware via physical interaction standards like IPMI, SNMP, BMC, DMI
Ongoing low level configuration tracking and management through Pollers
Provides that one stop shop for physical telemetry data and alerts
This stuff is cool and to watch something be discovered and then have a profile assigned and provisioning actions kicked off is very cool. It does not matter if is using Zerotouch for Network switches or building out a Docker Cluster via Kickstart scripts, Ansible modules and Docker-Machine (did I mention that there is a Docker-Machine driver), it is great to watch.
The process of discovery and workflow execution
The best way to learn about tools like this is to start playing with it and luckily the EMC {code} have mode that very easy for all of us with a fully functional Vagrant Demo setup that leverages VirtualBox off your laptop. I highly recommend anyone to give this a go as I definitely have had some fun with it. The guys have also written a docker-machine driver that can also be tested with RackHD within Vagrant, get it from GitHub now in under an hour you would have your first workload up and going!
If you want to see this in action, Kendrick Coleman did a great demo video on YouTube.
