Hey Windows, Where Are All My iSCSI (boot) Paths?

Oh, the Pain! iSCSI Boot with Windows Server

OK, tongue in cheek as it is not all bad. I must say I am a big fan of 'Boot From SAN' as it aligns to the desired end state of decoupling operational state from the physical world. An important attribute of a software defined, automated world. An over simplification I know, as there are a lot more to it  then just the boot process externalisation. Anyway back to the problem in hand.

I recently had a reach out to see if I could help with a customer issue that had been raised. The customer was running Windows Server in an iSCSI boot configuration, but they were not seeing all paths once the server booted into Windows. A simple enough problem statement:

"When we iSCSI boot, we can only see 2 of an expected 4 paths between the host and the array"

Simple enough problem so as a first step, I looked to recreate the configuration within the local lab. This would allow me  to determine if the path condition was as expected or something that may be aligned to an issue within the environment. Thankfully the test lab environment was very similar to the clients which comprised the following:

• Compute:
• Cisco UCS Blade Server (B200M4)
• Adapter: UCS VIC 1340
• 4 vNICS configured, 2 for general traffic and 2 for iSCSI. 
• UCS Firmware 3.1(3c)
• Windows Server 2016
• Storage:
• Pure M20 FlashArray
• Purity 4.10.6
• Dual 10Gb interfaces in each controller assigned to iSCSI traffic (ETH0, ETH1). In total 4 iSCSI interfaces (CT0.ETH0, CT0.ETH1, CT1.ETH0, CT1.ETH1) are configured 
• Network:
• Dual Cisco Nexus 5672UP Switches
• Each controller has an iSCSI interface connected to each of the switches

To be able to leverage all four iSCSI interfaces we need to ensure we have visibility between the two iSCSI NICs configured in the UCS Service Profile and the four array iSCSI NICs. Within the lab setup this was done by configuring two static iSCSI target interfaces per iSCSI vNIC (two static targets is the maximum per vNIC). The target objective of this is to ensure that there is no single point of failure at the Network Switch or Storage Array Controller level. The diagram below shows how the connectivity is laid out.


.

Connection topology

As it demonstrates, from the iSCSI hardware initiator configuration, 4 paths are expected to be accessible. We could actually see this to be behaving as expected when powering on the Service Profile where we could see from within the Array, that all four paths were active on the initial boot.

A barebones setup though with minimal tweaking, once Windows is loaded, will only have two active paths, even with the Windows MPIO component (i.e. Windows builtin multi-pathing software) installed. Actually the number of active paths changed as follows at various stages of the boot and build process:

• Initial boot with iSCSI boot configuration loaded had 4 paths active
• Booted into install disk ready to commence the installation of Windows had 1 path active
• On completion of the installation of Windows 2016, 2 paths are shown as active
• Adding the Cisco Network drivers still only showed 2 paths active

I ran through a number of tests with varying configurations at both the UCS Service Profile and within Windows with varying results. In the end though, I was able to build and configure Windows so that all four paths were active when accessible.

To get to this point there are some considerations to keep in mind when setting your configuration parameters such as:

• Windows only supports a single iSCSI Qualified Name (IQN)
• Once Windows loads, only one configured target interface for each of the UCS Service Profiles iSCSI vNICs boot parameters will be active. 
• Within Windows, there is no ability to manipulate the physical iSCSI setup state and as such, you can not activate the additional paths with multi-pathing software (i.e. Windows Component MPIO) alone. 
• The physical iSCSI boot settings are reported within the software iSCSI initiator such as devices assigned and sessions established (the active paths) but only as Read Only and as such no settings can be changed or sessions added
• iSCSI Hardware and Software Initiator configurations can be applied to the same Network Interface

In the end a configuration that was able to ensure four active paths were enabled from the default Windows installation with the following configuration changes:

• Ensuring a single IQN was used within the UCS Service Profile
• Configuring the TCP/IP details for each of the iSCSI Interfaces within Windows as per the configuration settings within the vNICS iSCSI Boot Parameters. This step is required to allow us to add sessions within the iSCSI Software Initiator
• Installing and configuring the Windows Multi-path component MPIO
• Configuring the Windows software iSCSI Initiator to activate all  paths by adding new sessions for the missing paths

There are more specifics around some of these step below (specifics around path policy, session authentication, etc for iSCSI are not covered as there is plenty of material available around this and is not part of the problem statement).

UCS IQN Configuration

UCS allows IQNs to be defined at 3 different points within the profile as shown in the diagram below. The ability to configure multiple IQNs is achievable as you can set IQNs at the actual vNIC level within the vNIC settings themselves or within the associated iSCSI boot parameters assigned to each of the vNICs.

UCS Service Profile IQN Configuration Points

If you do not set IQNs then an order of IQN inheritance is applied in the order as shown below.

UCS Service Profile IQN Inheritance Order

As such you can set a single IQN to be applied to all iSCSI vNICS by setting the single IQN at the iSCSI vNIC global settings section. This point of configuration within the Service Profile is shown below.

UCS Service Profile Global IQN Configuration Point

Windows Network Interface Address Details

Even though Windows is able to recognise the address details applied within the UCS Service Profile's vNICS iSCSI Boot Parameters (i.e. running 'ipconfig' within a command prompt will list the address details for each NIC), the software initiator can only apply any settings for network adaptors when the address details are set within Windows. It will actually report the settings as Read Only if you try to manipulate any settings if the address details are not present locally but configured for iSCSI Boot.

By default the Interfaces associated within Windows will have the initial DHCP based configuration set as per any new interface. You can change this to reflect the settings as configured within the iSCSI Boot Parameters.

Install and Configure MPIO

Just a quick note to ensure that two required steps are completed for MPIO once installed to ensure that storage device IDs are added into MPIO and that support for iSCSI is enabled. Once this is done a reboot will be required.

Configure iSCSI Initiator

Within the iSCSI Administration tool you are able to note that settings such as the destination target, sessions and accessible devices are reported as expected. What is unable to be done is the addition of additional sessions (paths) until the address details are set in the corresponding network interfaces within Windows. Once this is done though you can then proceed to add the missing sessions do the following steps.

                                       
Within the iSCSI Admin Tool validate that the array is listed within the 'Target Portals' in the Discovery Tab. If not add the details of each of the arrays iSCSI interfaces (assuming iSNS services are not present)

Once done switch focus to the 'Targets' tab and select the 'Connect' button

Ensure 'Multipath' selected and click Advanced

You can now create additional sessions to cover the missing paths. In the case of Pure FlashArray the missing paths can be identified at the array by looking at the connection map for the target host.

for each Session you specify the local software initiator, the interface IP and the target array IP and select OK to have it created.

Sessions can then be validated within the iSCSI Admin Tools via the 'Targets' tab <Properties> button.

A PRTG Sensor for the Pure Storage FlashArray

 

Why A PRTG Sensor?

I was in a meeting with a good customer of Pure Storage when I noticed that their operational panel on the floor was showing a PRTG dashboard. The PRTG Network Monitor tool from Paessler is a very nice agentless monitoring solution that is able to utilise a wide range of methods to monitor any accessible system or application on your network. It is a Windows based solution but supports all required methods to gain remote insights into your services such as SNMP, SSH, Scripts, REST, WMI, packaet sniffing, etc.

I am familiar with PRTG as I use it at home to monitor my home network and various components. This is possible due to the fact that you can run 100 sensors on a free license. The wonderful world of home automation and online media have made my network as critical to my household as it is for corporatations own services. Plus it is also fun to work with solutions like this, not to mention that it provides some interesting insights into your home network when combined with a Ubiquiti setup. 

Anyway back to the task in hand, I thought that looking at how we could monitor Pure FlashArray's with PRTG could both help me to familiarise myself with the Purity API. It turns out we do have a couple of options at the Pure OpenSource community Pure/Code within the  Python and PowerShell script packs written by some clever dudes in Pure (nod to Barkz). The only consideration for these was that they are focused on the array level performance metrics, where I wanted to see what else we can keep an eye on. Also, as a new Pure employee, as stated prior, it also gives me the chance to work with the Restful API of Purity.

I can say that writing this sensor was a bit of fun and the Restful API of Purity has to be one of the most logical I have worked with. Very simple in structure, versioned to allow for supportability across a range of Purity versions, and a rich capability around those supporting actions such as filtering. It took no time at all to get comfortable with the structure, limits were only my own and those discovered within PRTG.

So What Does It Do?

I wanted to provide a good overview of the current operational health of the arrays monitored while ensuring that the reporting was not overly complicated. As a first release my focus is on array level conditions covering:

• Array capacity status
• Array performance metrics
• General hardware status of Controllers, Shelves and Chassis
• General drive health

Array object sensor summary page

I have plans to add further granularity to cover 'volume performance', 'host performance' and 'protection compliance' in upcoming releases, but with the current method there is a limit of 50 channels to one sensor so still determining the best way to meet the number of instances required to monitor at that level and how to cater for lifecycle of objects that are subjected to CRUD operations (well create, update, delete anyway.

With the current script, the sensor reads the metrics from the array via a PowerShell script that directly queries each arrays Restful API. Although there is only a single script, it supports the multiple sensors through the 'Scope' parameter provided within the arguments section of each sensor (more details in installation instructions).

Each sensor can contain one or more channels, a channel can be thought of as a single value aligned to the sensor such as 'capacity consumed' within the 'capacity' sensor. Then as appropriate limits are assigned to the channel that define the thresholds that a channels value goes either above (maxlimit) or under (minlimit) to raise a warning or error condition for the sensor. Some channels have explicit limits set such as those related to capacity and health but others, such as the performance metrics, are left to self determine when an exceptional condition occurs based on operational patterns already recorded. For example latency may stay steady under a millisecond for normal operations but if suddenly it increases to 20 milliseconds for multiple recordings, PRTG will determine this to be an exception and raise a warning condition.

Array capacity sensor

Because each channel needs to provide a value that supports numeric comparisons to determine if limits are met, textual values returned from the array had to be converted to a representative numeric value. The textual values such as for Hardware components status to be reported as 'healthy', is enabled through the use of lookup tables included in the pack. Currently there are two included which provide lookup values for the channels in the 'Drive Health' and 'Hardware Health' sensor status values

Array Hardware Sensor

Where the Array performance and capacity sensors provide a collection of metrics related to the sensors scope of the whole array, the health sensor for hardware provides summary graphs for each major component (Chassis, Controller, Shelf) and for the Drive Health sensor, a status for each drive.

Hardware Health Sensor

For the Hardware Health sensor, summary values were required as reporting each individual hardware component that can be monitored exceeds the 50 channel to a sensor limit (not enforced but supportability and reliability can be compromised). I open to looking at alternative methods that can allow for all components to be individually reported in the future.

Drive Health Sensor

Installation

Requirements

• First, get the scripts and associated lookup files from GitHub at Pure Storage Sensor Module. This will provide a Zip file of the required files but you can also download it directly from the GHitHub site  at 'https://github.com/davlloyd/purestorage-prtg' 

• This set contains 4 files including:

• Get-PureFA-Sensor.ps1 - The main PowerShell script
• prtg.standardlookups.purestorage.drivestatus.ovl - Drive status lookup file
• prtg.standardlookups.purestorage.hardwarestatus.ovl - Hardware status lookup file
• readme.md - Markdown file with Installation instructions 

• The Powershell script and lookup files need to be copied onto the PRTG server and then associated sensors created accordingly.

• Ensure the PRTG server (or PRTG probe if running sensor from a probe)is running PowerShell 4.0

Script Installation

1. Copy the script Get-PureFA-Sensor.ps1 to the directory 'C:\Program Files (x86)\PRTG Network Monitor\Custom Sensors\EXEXML' 
2. Copy the two lookup files prtg.standardlookups.purestorage.drivestatus.ovl and prtg.standardlookups.purestorage.hardwarestatus.ovl to the directory 'C:\Program Files (x86)\PRTG Network Monitor\lookups\custom'
3. Restart the Windows service 'PRTG Core Server Service'. This step is required to have PRTG read in the new custom lookup files

Sensor Creation

1. Create a new device in PRTG with the address (IP or FQDN of the FlashArray you want to monitor
2. Now, select <Add Sensor>. On the search field, type "Script Advanced" and then select the sensor type <EXE/Script Advanced Sensor> from the result list. 
3. On the Add Sensor to Device screen, enter the following: 
1. The sensor's name and tags (optional): There are currently four sensors to create. I have been naming them as follows:
1. Capacity
2. Performance
3. Drive Health
4. Hardware Health
2. Under Exe/Script, use the Drop-down to select the script 'Get-PureFA-Sensor.ps1' from the list.
3. Set the parameters for Array access and sensor scope as follows
1. [ArrayAddress] (use the PRTG variable '%host' to inherit from device entry
2. Either of the following combinations for security authentication:
1. [username] and [password]  - This is ok for testing purposes but as password is stored in clear text not recommended for production. This will generate or read the APIKey for the account
2. [apikey] - preferred access method. The APIKey is generated from within the Purity Console for the preferred account. You do not need to enter account details if specifying an APIKey
3. [Scope] to set what is monitored for this sensor. Scope option values are:
• capacity
• performance
• hardware
• drive
4. By default the sensors will be run every 60 seconds, adjust accordingly. You can get a view of time for execution by running the scripts in debug mode but ensure that it is disabled when running the script as a sensor.

-arrayaddress '%host' -scope 'hardware' -apikey '3bdf3b60-f0c0-fa8a-83c1-b794ba8f562c'

Sensor Setup

You are now ready to go. For a look at how to set a sensor up watch the video on this blog.

Sensor Screen

Performance Chart

VMware Photon Platform, Notes from the Field - Entry Two

Time to Build The Cluster Services

Is it a bird,  plane or the Photon Platform? In the early days of Photon Platforms announcements, in my own mind, I had it pegged as a purpose built platform alternative to 'vSphere Integrated Containers (VIC)'. Keeping in mind that VICs sole purpose in life is to enable Containers in vSphere with a cool concept of a 1:1 alignment of VM to Container. This meant a container can have all the flexibility inherent in a container with all the control and security of a VM. Alright come on down Photon Platform just bigger, better more focused right? Well not really....

So one thing that strikes you with the Photon Platform is the flexibility of choices it provides. From a scheduling cluster manager's perspective you can utilise Swarm, Kubernetes or Mesos out of the box. On the other side, this is ESXi after all (marketing slides aside that keep alluding to the slimmed down hypervisor) so it will happily run Virtual Machines  and Containers side by side (well containers within VMs anyway).

Being able to support both is important as not all workloads are created equal in this new age. Let's be honest, not everyone working in this new 'Cloud Native' world has grown a beard, rides a bicycle and wears skinny jeans, there is a variety of personalities that need to be catered for. What is common is we don't all need the assurance and crutches that come from the more traditional platforms. If I fall over it is ok, I will just get myself back up. In the traditional world I would have crutches, cushions, people watching me, ready to catch me if a stumble, etc. This is because in this traditional world I am seen as needed to be treated like a VIP (i.e. Very Important Process) and require all the care of advanced services to keep me going (and the costs associated with those advanced services). Sometimes a more traditional approach may be preferable to containerising (is that a word?) the workload so Virtual Machines live on even if they are seen as cattle (I'll let Duncan Epping define that for you).

This mixed capability is emphasised by the fact that all the management VMs are hosted out of the hypervisors that are flagged as Management hosts (you designate if a host is 'Management' so participates in the control plane or 'Cloud' which only provide a hosting platform capacity, or both). Anyway myself for one, I'm a big fan as by including the ability to host containers and VMs as this provides that flexible, cost and scale focused platform I want without having to look at alternatives such as Openstack. Add to this the fact that it also includes support for the 3 main container and workload cluster scheduling services out of the box and your covering a lot of bases for possible consumers of Photon Platform.

My personal choice is to go forward with Mesos at this stage as it gives me the open choices I want when paired with Marathon. I have played with all three though on Photon as all it takes is that you import the 3 images and enabled the 3 cluster types. I am not here to tell you how to do this or how to create a cluster as there are plenty of blogs that do that, I just want to give my view and some lessons I learnt on the way that may help someone out there. 

It is important to note that when creating clusters, they align to there traditional model and not some Hybrid like in VIC. What I mean by that is that if you create a Docker Swarm cluster, the slave nodes will be running the containers in a shared multi-node format. The nodes are utilising PhotonOS as there Operating System but we don't have the 1:1 Container to VM alignment of VIC. This is important to consider as your sizing of those nodes should reflect your needs. There are default 'Flavors' (remember Flavors define the resource configuration profiles) aligned to the different clusters but they can be overwritten at the API, CLI level when creating a cluster.

This system is built as a multi-tenant service from the ground up so when you create workloads they need to be placed within a Project assigned to a Tenant. As you would expect, a Tenant gets an allotment of Resources (vCPUS, memory, Disk Capacity, VMs, etc) which are then subdivided into Resource Tickets (think Gold , Silver Bronze classes or whatever). A Project is then assigned a Resource Ticket which it then further carves out a resource reservation /limit out of. All the way down this logical structure you can add Access Controls via the integration with VMware Lightwave. 

So importantly as a tenant, I can create my own cluster service to host workloads or just create VMs directly based on virtual appliances or virtual disks imported into the system.  I can then control my VM workloads via the CLI and API into Photon Platforms Controller VM as you would of normally via vCenter, or use the cluster manager of choice to control the workloads within them.

As a final statement on the installation of the clusters what I can say is this. I have built all 3 flavours using simple sandbox methods as well as going through the full manual processes (more then once as I like punishment) and there is a highly compelling value proposition here. To be able to create these services as you require them  on a production class platform so simply is huge. There will be challenges with the way it is done now such as how do they stay close to release parity (hence the compelling part of it being Open Source), but this is a new era platform that brings together the traditional on-premise controls to the new cloud native era's service requirements in the one stack. This is great stuff for a version 0.8 release!

Some Notes On Cluster Enablement:

Be Wary Of Resource Stinginess

My nature is to give less then more when allocating resources. This sent my on a dance to the requirements of the various clusters as I did not have enough resources aligned to my Project.  Just be aware of the sizing needs (they vary of course depending on your own configuration requirements) and if you are resource constrained, use custom 'Flavors'. I prefer to bang my head on the wall so got very use to the 'Photon Cluster Create' operation being closely followed by the 'Photon Cluster Delete' operation to delete the failure and started again (side note, it didn't like me trying to recreate deleted clusters with the same name, didn't ping this down but got into the habit of using unique names each attempt) :)

Cluster Creation IP Address Requirements

Within the instructions it is emphasised that DHCP needs to be disabled / not present in the management network which is where the clusters are installed. Ironically, if DHCP is disabled the cluster deployment fails with a 'Unable to Obtain an IP Address' error. To progress forward I re-enabled DHCP in the management network (the one that the Photon Controller is installed into). The more accurate requirement description is to ensure the static IP addresses you provide (i.e. for the 'zookeeper' servers in Mesos and 'etcd' servers in Kubernetes and Swarm) are not also served within the active DHCP scopes.

Those Damn Certificates

Maybe you lucky that your company provides an open network and all traffic is created equal. That is not the case in mine, we trust no-one and inject ourselves into every certificate coming into the network. Any problems with this is mitigated by adding our own certificates into the trusted root but this is difficult for those environments that are created dynamically as these clusters are.

When you create a cluster PhotonOS based VMs are created corresponding to your configurations requirements (i.e. how many slaves, how many provide the interconnect service) and are then configured by templates contained on the Controller. Where the certificates being trusted becomes critical is the fact that the services run as containers within the VMs and as such the images need to be downloaded as part of the initial execution. The end result is the cluster creation operations fail with 'Time Exceeded' errors.

Anyway its an easy fix! I edited the template files contained in the controller directory '/usr/lib/esxcloud/deployer/scripts/clusters' with descriptive names such as 'kubernetes-master-user-data-template' to inject our certificates into the VMs. This at least made that problem go away and you of course could do this to apply any other customisations that may be required. It would be nice to see some option in the future to inject certificates into the process. I also provided this same feedback for 'vSphere Integrated Containers' as had the same problem there!

VMware Photon Platform, Notes from the Field - Entry One

First, some overview ramblings:

Ok, first of all let me say this, I am excited about the changing landscape in the industry. This shift to containerisation is exciting to me in two ways. Firstly, as an automation focused engineer, this just seems like the natural progression / evolution of the current platform (incoming alert, Unikernels). Secondly, the technology landscape is very exciting, lots of new tools (or is that toys) to play with!

That said I have been fortunate to be working with VMware (shout out to Roman Tarnavski @romant) for a while now on their rapidly evolving 'Cloud Native' program. With this a couple of recent announcements have there two primary offerings 'vSphere Integrated Containers' and 'Photon Platform' are both now Opensource on github so now everyone can have a go. A key point of differentiation between the two is that VIC leverages your existing VMware vSphere platform to host container centric workloads on a 1:1 basis on a Virtual Machine, the other, Photon Controller still leverages ESXi (today anyway) but has it's own control plane.

Photon Controller is a replacement to vCenter focused on the operational requirements of the container workloads, not virtual machines. Why you may ask? An easy way to think about it is the rate of change and scale that us usually associated with a container aligned environment versus the long term service life and different scale models associated with traditional Virtual Machines drives a different model. 

If you are all in for containers, or alternatively, have a substantial container footprint today (or will soon) you have some considerations when assessing VIC versus Photon Platform:

• Will you be limited by the sizing maximums of vSphere. Think vCenter, is 10,000 VMs enough when 1 VM equals 1 container how about the change rate of those VMs.
• You also may be questioning the commercials around vSphere in a container world, do you really need those advanced availability services (e.g. HA/DRS) in a container world. I would say no, the idea of containers is pretty simple, let them scale out and have availability controls north of the service such as with NLB (I know it is an over simplification) or south with stateful data services (should be scale out themselves but that is for another day).

Neither of these statements cover the why VMware anyway? Isn't this new world order a shift away from the vendors we typically associated with infrastructure, doesn't the consumer in this new world not care about infrastructure? All I would say there is that there is a world of difference between developing in a sandbox environment and having your workload run in a production class platform with all the controls, monitoring and reporting thats in place within organisations today for traditional workloads. I also do a lot of testing in the Cloud and directly on my laptop, tat is the beauty of containers, you can just shift around. But for it to go production, there are certain requirements that a lot of organisations need to comply to. This may be due to regulatory demands, data sovereignty requirements, cost controls, to name a few.  

The idea of either offering is to be able to provide the application developers, operator, <insert whoever else here> the same user experience, with the same toolsets but aimed at a different platform. This platform then is able to have the same operators (VMware Admins) look after it in the same / similar way that they do there vSphere environment. Everyone wins :)

Anyway, next I will look at the install and configuration stage. There is some great content at github and by other bloggers such as William Lam. But I am finding some differences with my own experiences to what others have documented. 

Some quick notes:

Controller Install VIB Upload fail

I kept getting file upload errors when the installer tried to upload the Photon VIB to the Management ESX host (see these issues in the log /var/log/esxcloud/deployer/deployer.log) . I still have not isolated where the issue is and not been able to isolate it as yet in the source code. What I did do though is simplified my naming convections and the issue went away. I noticed all the blog samples where pushing into default bare ESX hosts where I aligned mine to my own pseudo naming standards. In the end I simplified my environment by taking hyphens out of my port group and datastore naming and was now able to install. 

YAML Install File with CLI

The examples provided and the exported file provide the 'image_datastores' field within the 'deployment' section as an array. I ended up getting errors around the conversion of the field to an array when attempting to deploy through the cli. This occurred regardless if I had or or more values assigned. As such changed the field to a straight string and it worked. Change shown below:

Uploading Images getting HTTP 413 Error
To enable management clusters (schedulers) you need to upload the corresponding dish images into the Controller (Mesos, Kubernetes or Swarm). I found that regardless of the image I was getting a NGINX error of 413 'Request Entity To Large'. You don't have to be a google ninja to identify the error quickly with the reported resolution to set the 'client_max_body_size' value in the 'nginx.conf' file.On the controller VM the NGINX service runs as docker container called 'ManagementUi' so it was easy to then pull the configuration file down with a 'docker cp' command to have a look, low and behold the setting is not there!

OK, now I have just a rat down he drainpipe I can say that I was on the wrong track. NGINX looks after the User Interface but not the API. Looking at the docker containers there is one running HAPROXY which is the loadbalancer service, called you guessed it 'LoadBalancer'. Jumping into this VM and looking att he file 'haproxy.cfg' you can see that the API is served by ports 28080 and 9000.  By switching the photon cli target to either these ports let the images upload successfully (e.g. photon target set http://10.63.251.150:9000).

I took the long way but got there in the end, time to start playing!!!

Unbox, Power On, Watch the RackHD Magic Begin!

What am I saying, you are probably asking? I have had great fun over the last week getting down and dirty with one of the headline EMC {code} projects, RackHD. This is a very cool solution for taking care of your low level activities for bare metal infrastructure. Think configuration management for all those physical pieces in a rack in the data center. This is a way to manage the firmware through to the personas of these devices, very good stuff and it is Open Source!

I guess first up I should give some context around what I am referring to with EMC {code}. At EMC we can be perceived to be a player in the more traditional areas of IT Infrastructure, EMC {code} is one example (of many) where this is a long way from the current day EMC that I work for. EMC Code is the landing place for developer enablement and open source projects for EMC.

This is your one stop shop to find any open source projects supported by EMC, community projects by EMC staff, partners and customers. It also includes training content and projects aligned to helping enable the developer community on wide spectrum of those new Mode 2, Platform 3 Cloud Native, SDDC technologies (you get the idea). I strongly encourage anyone to have a look and join in. There are some very smart people in this community and very accessible via tools such as Git and Slack.

Anyway back to RackHD, what is it really. So, we know that there is a number of configuration management solutions out there today such as Puppet, Ansible, Salt, Chef, etc but these tend to have a common trait, they look after the nodes/hosts/clients through remote agentless access (SSH, WBEM, etc) or via agents that are installed in the target device. What this of course requires is that the device is ready to accept remote requests or have agents installed. in other words they take control and configuration management of the platform once it is operational. A few such as Puppet with Razor have the ability to control the physical world but not as an all inclusive service with mulitple action workflow smarts.

Looking at RackHD you have a solution that provides:

• Bare metal configuration management across the physical infrastructure stack. So not just with the compute but all the bits that go in a rack (hence the name) including:
• The compute
• The Network
• The Storage
• The enclosures that may contain the nodes
• The Racks themselves and PDUs (remember, just plug the thing in)
• A strong but intuitive Restful API
• Aligns to the 'Infrastructure as Code' model. Allows node definition, associated workflows and SKUs to be fed in as JSON files via the API (or UI if thats your preference)
• Fully self contained service providing all the mechanisms required to control a physical environment such as DHCP, PXE,TFTP, HTTP, etc
• A scale-out architecture that can grow to your environment needs
• Full support of dynamic discovery and physical control through interactions with hardware via physical interaction standards like IPMI, SNMP, BMC, DMI
• Ongoing low level configuration tracking and management through Pollers
• Provides that one stop shop for physical telemetry data and alerts

This stuff is cool and to watch something be discovered and then have a profile assigned and provisioning actions kicked off is very cool. It does not matter if is using Zerotouch for Network switches or building out a Docker Cluster via Kickstart scripts, Ansible modules and Docker-Machine (did I mention that there is a Docker-Machine driver), it is great to watch.

The process of discovery and workflow execution

The best way to learn about tools like this is to start playing with it and luckily the EMC {code} have mode that very easy for all of us with a fully functional Vagrant Demo setup that leverages VirtualBox off your laptop. I highly recommend anyone to give this a go as I definitely have had some fun with it. The guys have also written a docker-machine driver that can also be tested with RackHD within Vagrant, get it from GitHub now in under an hour you would have your first workload up and going! 

If you want to see this in action, Kendrick Coleman did a great demo video on YouTube.

http://bit.ly/rackhd-docker

Docker-machine and VMware vSphere Clusters, Lessons Learned

I have been doing a bit of work with various flavours of containers lately which comes with the highs and lows of being a technologist. One thing that was interesting is that with the recent updates to docker-machine, my setup for binding and deploying to our vSphere environment no longer worked.

This issue presented itself when attempting to provision a new instance with the following error:

Running pre-create checks...

Error with pre-create check: "default host resolves to multiple instances, please specify"

Digging deeper and I found out that where I was using the cli argument / variable 'VSPHERE_COMPUTE_IP' to specify a ESXi host to bind to, you know need to use 'VSPHERE_HOSTSYSTEM' (or as a cli argument '--vmwarevsphere_hostsystem'). As this is new documentation is fairly light as are real examples. What the documentation does provide though is two syntax examples as follows:

for cluster -                      VSPHERE_HOSTSYSTEM=<Cluster Name>/
for stand-alone host         VSPHERE_HOSTSYSTEM=<Cluster Name>/*

Using those two syntax examples in the lab I first attempted 'VSPHERE_HOSTSYSTEM="UCS8#General Use/' but this resulted in the same error message. It was then also notable that examples were pushing to single node clusters, the test cluster I was targeting has 4 nodes.

Anyway to cut straight to it, the documentation is a little incomplete. You do need to still target a specific host in the cluster and docker-machine will not do that for you. As such in a cluster with multiple hosts the actual syntax is:

VSPHERE_HOSTSYSTEM=<Cluster Name>/<Host Name>

 so for my setup it looks like this

VSPHERE_HOSTSYSTEM="UCS8#General Use/esxrack01.vce.asc" 

I guess while I am here another hint does not hurt. The other piece of lovely syntax requirements is the optional variable for targeting a specific ResourcePool, to do this you use the variable VSPHERE_POOL. The syntax for this is

VSPHERE_POOL="/<DataCentre>/host/<Cluster>/Resources/<Resource Pool>/..."

So again in our lab setup the variable looks like this:

VSPHERE_POOL="/ASC/host/UCS8#General Use/Resources/Containers" 

Docker-Machine and vSphere together are a great team and well worth a play so hope this helps. 

I am also fortunate with doing a bit of work with VMware's vSphere Integrated Containers (VIC) which takes these issues away and provides a very strong alternative to other scheduling/clustering services such as DOCKER SWARM within your vSphere environment but ore on that in another post later.

What is VMware VSAN 6.2 Erasure Coding?

I have been asked what is the 'Erasure Coding' support in VSAN all about as it gets referenced in the same sentences as the new support for RAID5 and RAID6 configurations (only available in all flash configurations).

VMware is stating Erasure Coding as a term to reference for any data partitioning scheme that allows data to be fragmented but recoverable in case of failure even if some parts  are missing (use of parity). In general reference they are referring to the support of Erasure Coding to represent the fact that they have moved from a mirroring scheme to support both RAID 5 (support for 3+1) and RAID 6 (Support for 4+2) to increase the level of usable data from RAW Capacity across nodes.  

Prior releases only supported a 1+1+witness or worst (1+n where n=failure tolerance but each n was a mirror copy) scheme hence you always needed a minimum 3 nodes to start with but only really got 50% usable capacity at best. This is why the Erasure Code support is getting such big focus as it makes VSAN more efficient with physical resources.

Worth noting that Erasure Coding is not referring to a disk configuration as with RAID but how the fragments are distributed between hosts. It is for this reason that depending on the choice mode also drives the minimum number of hosts required (3 for standard setup, 4 for R5 and 6 for R6).

So Who Does Use Linked vCenters?

OK, I am sure that most onboard and I put up my hand, I personally am a fan. When I talk to a lot of clients though, generally I find that most aren't.

Two linked vCenters in Web Client

In the old days we linked the instances together with the linked mode utility (or unlinked it with the same tool). Nice and easy and all of a sudden security and licensing information is shared between vCenter instances. The big gain though was that through the C# client (oh I do miss your usefulness sometimes in this depleting role world) I had a one stop shop to access up to 12 vCenters. Now it is even easier, just have the vCenters in a common Platform Service Controller (PSC) SSO implementation (common site/ multi site doesn't matter).

This in turn enables some very cool features beyond the original benefits. The one that drive me to link mine up was the ability to enable cross vcenter migrations. Being someone that is fortunate to have access to a well equiped lab I have enabled this with VPLEX for the stretched storage and use both SRM 6.1 and vMotion in vCenter to control the migrations (more on that in another blog post I think). 

This blog is not to tell you how to link, that is covered in a lot of other blogs written by very smart people. More I want to draw attention to issues that surface a lot with the newer Enhanced Link Mode with those add-on services that use the web client.

As I have just experienced an issue again I am writing this while it is fresh in the mind. What I see is that with a lot of product teams that do include Web Client plug-ins, there seems to be quite an absence of testing vCenters in Enhanced Linked Mode. Have you ever had this style of conversation:

me)              I am <inset problem description here> in the web client 

developer)   what version are you running

me)              The one listed in requirements

developer)   Are you using the Windows Version of vCenter or the Appliance?

me)              I am using the (whatever) in linked mode

developer)   ohh, we haven't tested that

Anyway some gotchas I seem to experience include:

• Only one vCenter is supported in the linked set for the plug-in
• Only the first vCenter is visible to the plug-in
• Plug-in is unable to differentiate (filter) objects out that are only present within the aligned vCenter (e.g. all datastores across all vCenters are displayed)

So the point of this entry beyond the rant is just to bring attention to the fact that there are some considerations when writing a plug-in for linked vCenters. If you are running into issues and you have got linked vCenters I would suggest that the first step should be test against a stand-alone vCenter. This helps isolate the issue and can help when you are talking to the support/product/vendor teams.

Welcome VxRail, VCE's Entry into the HCIA Market

You want the ease of a Hyperconverged appliance but still have all those advanced capabilities that service enterprise data centres, then the new VCE VxRail is for! This is not so much an evolution of the VMware EVO:Rail appliances but more a revolution.

Why do I say that? The Achilles heal of EVO:Rail was the lack of flexibility, all you could do was chose between two RAM configurations (no variant in disk or CPU), 128GB or 196GB which for some reason had the 196GB model referred to as the desktop service model... never quite got my head around that one, why wouldn't other services want a larger memory footprint? The other side was the fact that you could only grow your appliance service out by adding additional appliances, start with a 4 node 2U appliance and then grow in increments of 4 from there till you hit the 64 node ceiling (aligned to the maximum size for a HA / VSAN cluster).

Now think the VCE VxRail appliance here we have flexibility that can meet a huge range of use cases, your choices now include:

• Starting point is now 3 nodes (today) and growth can then be in single node increments up to the 64 node maximum
• CPU model options provide a range of 6 cores (single socket) through to 28 cores (dual socket) per node
• Memory configurations from 64GB through to 512GB
• Disk configurations of Hybrid (Flash + Spinning HDD) or all flash ranging from 3.6TB through to 19TB per node 
• Even networking options will support 1G connections on the low end nodes with the standard being dual 10G 

You can also now mix different model configurations and EVO:Rail appliances under the same management. Phew... all that flexibility but with the ease of use that comes with the VxRail appliance.

Some other cool features of note are things such as:

• All the coolness of VSAN 6.2 with Dedupe, compression, flash optimisation, erasure coding, snapshots, cross site HA (yes you can link two remote appliances into a vSphere Metro Cluster)
• VM Level replication with EMC RecoverPoint/VM (yep included) 
• Cloud based storage tiering EMC Cloud Array (yep also included)
• VCE Vision software for service health and interoperabilty compliance monitoring and reporting
• vRealize LogInsight for lag aggregation and analytics
• vSphere Data Protection (EMC Avamar)
• Appliance Market Place, your VxRail App Store for easy request / enablement of additional services

And that is not the end of it, too much to list :)

If you want to have a look for yourself have a look at this demo of the VxRail Manager I recorded!

More information is available at http://www.vce.com/products/hyper-converged/vxrail

Phew, vForum Day One Done!

Well just a quick one as thought it was time for a new entry and having just done day one at vForum at Luna Park it was a good chance for an update. I had the great oppurtunity to present in the EMC session today with Matt Zwolenski, our ANZ SE leader on some of our cool software based solutions.

This 40 minute section covered 4 demos (maybe to many for 40 minutes) that we produced out of the local solution center. I think it could be a week before my sleep catches up for that one. It is interesting just how much time filling a 40 minute segment can take up in preparation. This is amplified by my belief that if I am going to show it, I need to build it and prove it first. All demos I show I also buil and produce myself. Maybe that explains the dodgy Camtasia call-outs.

That said we showed:

• The ViPR Data services
• This is very relevant for VMware as it is now providing the Object Storage Services and the snpashot repository for the Database-as-a-Service offerings in vCloud Air
• VVols with the EMC VNXe
• A demo showing the 4 parts of implementing Virtial Volumes 1) Protocol Endpoint 2) Storage Provider, 3) Storage Containers 4) Storage Policies
• Recoverpoint for Virtual Machines
• Very cool replication technology now available at a VM granular level
• The VMware + EMC Hybrid Cloud
• Showed the vRealize suite servicing Virtual Machines,catalogues (vRealize Automation), VM mobility (vCloud Connector), Cost transparency (vRealize Business),  Storage-as-a-Service (vRealize Automation + ViPR)

As a techie I really do enjoy building this stuff and the technology from both VMware and EMC is very cool. It is hard to put a prep time on the work to produce the demonstrations but it was stretched over weeks. With 3 of the 4 solutions shown, done with release levels still in beta or earlier it did bring on some interesting challenges.

It does go a long way to show how far these technologies have come. The old days of any service management solution being an endless drag on professional services are definitely starting to move behind us.

Anyway day one done, one speaking slot and a live vBronwbag podcast completed. Once the vForum roadshow completes in ANZ I will publish the 4 demos into Youtube.

 

Cool Solution# Hadoop File Services with EMC Isilon

I'm Sold!

First let me say, by my own admittance, I am an infrastructure guy! That said,  I have been lucky enough recently to be given the chance to dive into the world of Big Data and PaaS. As a techie the extensive technology options in this area are very impressive and as a consequence, I have had a lot of fun combined with late nights and heavy learning.

I have always worked on the KISS principle 'Keep It Simple Stupid!' and Hadoop first into that with the exception of its file services HDFS. By having that layer of abstraction to the file system the ability to manage and populate the file system is non existent without specific tools written for the task.

I know DAS and scale-out through the data nodes is great and builds a big pond for your big data by combining the compute and disk resources of a 1000+ server nodes . Yay all good! But putting my EMC and Infrastructure hat on for a moment what about the following:

1. How do I backup my HDFS based data
2. How do you use those other cool storage capabilities such as snapshots, auto-tierring, etc
3. How do I get real-time analysis on data without having to move it into HDFS
4. How could you share the data from within HDFS
5. What if you need more compute resources in your cluster but not storage, or the other way around. 

This is where EMCs Isilon comes to the rescue through its OneFS. The flexible open access into its large single name space is invaluable. All the joys you expect from a true scale-out, enterprise class storage array are there as well as the capability of accessing the same file system in multiple ways at the same time. Think about a web site logging to an NFS mount that is part of the HDFS file system allowing for realtime analytics against it! 

Want to see this in action, look at this demo http://www.youtube.com/watch?v=Qx9BMzZa8UI

 

Further information can be found at http://www.emc.com/domains/isilon/index.htm

EMC ESI with Microsoft Applications

EMCs ESI, it's FREE!!!!

A few weeks back I recorded a series of short videos that showcase the ease of which you can extend your applications direct from EMCs FREE Storage Integration Suite (ESI). These are short sub 90 second videos that each focus on only one particular feature provided through the ESI console.  As part of our application stack value prop this is a great example of what EMC can provide for a customer and of course, it is FREE!

ESI is like EMCs glue into the Microsoft ecosystem and includes:

• Simple user console for storage, replication and application management
• Support for Windows, HyperV, vSphere and XenServer
• Powershell library with almost 200 cmdlets
• System Center Orchestrator Integration Pack
• System Center Operations Manager Management Pack
• Hyper-V VSS Provider and associated PowerShell library
• Support for Exchange (up to 2013) including Native and Third party replication (enabled by RecoverPoint)
• Support for Sharepoint and SQL Server (SQL AlwaysOn and FC coming in a few weeks)

Storage Provisioning with ESI

http://www.youtube.com/watch?v=9effRPY7uLU

Microsoft Exchange Discovery with ESI

http://www.youtube.com/watch?v=CveZDDBf6pE

Microsoft Exchange Database Provisioning with ESI

http://www.youtube.com/watch?v=VVdOA7LsZw0

Microsoft Exchange Database Replication with ESI

http://www.youtube.com/watch?v=0Vy3uNlAbsc

Microsoft SharePoint Content Provisioning with ESI

http://www.youtube.com/watch?v=_22OiKxsV_Y

You can download it from http://www.emc.com/data-center-management/storage-integrator-for-windows-suite.htm

It's Not Just About Backup!

Why Have Different Levels of Protection?

I like the saying that comes when people refer to an old classic car 'they don't make them like that anymore'. The thing is, the statement should be followed with the statement 'thank goodness'. Can you imagine putting up with that level of reliability, quality, safety, comfort level with a car you purchased from a dealer today that you would have got in years gone by?

Same goes with technology. I still wake-up some nights and shudder when remembering the old tape backup routine. This use to go in two ways, the first was the nightly scheduled tape shuffle which thankfully I was not tasked with. The other was the pre-rollout backup we would run before pushing any new service out. This use to comprise us kicking the job off, then trotting down the road to a golf driving range to knock a few buckets of balls into no mans land to kill the time (usually hours).

So looking at that, we had two different use cases but only one method at our disposal, the inglorious, forever unreliable tape backup. Fast forward 15 years (maybe more but I keep that quiet) and we have many different forms of protection at our disposal including storage replication, snapshotting, tape backup (but hopefully VTL, not those horrible tapes).

All these options serve a purpose and work together nicely. If we were to classify the different use cases into buckets they could be:

Business Continuity (DR) = Remote Replication, constant protection driven by RPO and RTO requirements

Operational Protection = Local Snapshotting (or continuous protection as is provided with RecoverPoint) typically done by set interval (storage level) or self service (VM level)

Long Term Retention = Traditional Backup style typically run once a day

Each of these has there purpose and specific business requirements that drive their implementation. Interestingly I see a lot of the BCM and Long Term Retention being positioned but little of the Operational Protection being catered for. Back to my scenario at the start, if I could of had snapshotting at my disposal (or if I was real lucky, EMC RecoverPoint Continuous Data Protection) my golf practice (aka time wasting) would not have happened as we could have snapshotted the services that we were updating and started the rollout straight away.

Self service snapshotting is easily available these days thanks to two things:

1. Virtualisation with the VM level snapshotting (Checkpointing in a HyperV world)
2. Automation tools for storage (EMCs Storage Integration Suite is a good example)

So that is all cool for those known events but what about those unknown such as data corruption. That is what scheduled snapshots protect you against, providing a much more granular way to protect your systems beyond the typical 24 hour cycle of Long Term Retention. You may only retain a short set of snapshots such as 1-7 days but they can provide good peace of mind for those services deemed worthy.

I do realise that replication can also provide a way of rolling back but typically it is 'to the last change' or committed IO operation, so a corruption could easily by on the remote site as well as the source. Also replication would require that traffic to come back down the wire which adds time to the recovery / rollback process.

Another benefit of Operational Protection is that it can provide an easy / quick way for copies of  datasets such as those within a database to be presented to an alternate location such as a from production to a test/dev instance.

Anyway, I got that of my chest so I feel better. Operation Protection = Good!

Just as a last note on this, I did not include High Availability (HA) in this as I am more looking at where the old functions we used tape for have evolved. There is some real cool stuff that can be done with stretched high availability that spans physical locations as is supported with vSphere Metro Storage Clusters and Microsofts Failover Clustering with products such as EMCs VPLEX, but that is a big enough topic on its own.