VMware Photon Platform, Notes from the Field - Entry One

First, some overview ramblings:

Ok, first of all let me say this, I am excited about the changing landscape in the industry. This shift to containerisation is exciting to me in two ways. Firstly, as an automation focused engineer, this just seems like the natural progression / evolution of the current platform (incoming alert, Unikernels). Secondly, the technology landscape is very exciting, lots of new tools (or is that toys) to play with!

That said I have been fortunate to be working with VMware (shout out to Roman Tarnavski @romant) for a while now on their rapidly evolving 'Cloud Native' program. With this a couple of recent announcements have there two primary offerings 'vSphere Integrated Containers' and 'Photon Platform' are both now Opensource on github so now everyone can have a go. A key point of differentiation between the two is that VIC leverages your existing VMware vSphere platform to host container centric workloads on a 1:1 basis on a Virtual Machine, the other, Photon Controller still leverages ESXi (today anyway) but has it's own control plane.

Photon Controller is a replacement to vCenter focused on the operational requirements of the container workloads, not virtual machines. Why you may ask? An easy way to think about it is the rate of change and scale that us usually associated with a container aligned environment versus the long term service life and different scale models associated with traditional Virtual Machines drives a different model. 

If you are all in for containers, or alternatively, have a substantial container footprint today (or will soon) you have some considerations when assessing VIC versus Photon Platform:

• Will you be limited by the sizing maximums of vSphere. Think vCenter, is 10,000 VMs enough when 1 VM equals 1 container how about the change rate of those VMs.
• You also may be questioning the commercials around vSphere in a container world, do you really need those advanced availability services (e.g. HA/DRS) in a container world. I would say no, the idea of containers is pretty simple, let them scale out and have availability controls north of the service such as with NLB (I know it is an over simplification) or south with stateful data services (should be scale out themselves but that is for another day).

Neither of these statements cover the why VMware anyway? Isn't this new world order a shift away from the vendors we typically associated with infrastructure, doesn't the consumer in this new world not care about infrastructure? All I would say there is that there is a world of difference between developing in a sandbox environment and having your workload run in a production class platform with all the controls, monitoring and reporting thats in place within organisations today for traditional workloads. I also do a lot of testing in the Cloud and directly on my laptop, tat is the beauty of containers, you can just shift around. But for it to go production, there are certain requirements that a lot of organisations need to comply to. This may be due to regulatory demands, data sovereignty requirements, cost controls, to name a few.  

The idea of either offering is to be able to provide the application developers, operator, <insert whoever else here> the same user experience, with the same toolsets but aimed at a different platform. This platform then is able to have the same operators (VMware Admins) look after it in the same / similar way that they do there vSphere environment. Everyone wins :)

Anyway, next I will look at the install and configuration stage. There is some great content at github and by other bloggers such as William Lam. But I am finding some differences with my own experiences to what others have documented. 

Some quick notes:

Controller Install VIB Upload fail

I kept getting file upload errors when the installer tried to upload the Photon VIB to the Management ESX host (see these issues in the log /var/log/esxcloud/deployer/deployer.log) . I still have not isolated where the issue is and not been able to isolate it as yet in the source code. What I did do though is simplified my naming convections and the issue went away. I noticed all the blog samples where pushing into default bare ESX hosts where I aligned mine to my own pseudo naming standards. In the end I simplified my environment by taking hyphens out of my port group and datastore naming and was now able to install. 

YAML Install File with CLI

The examples provided and the exported file provide the 'image_datastores' field within the 'deployment' section as an array. I ended up getting errors around the conversion of the field to an array when attempting to deploy through the cli. This occurred regardless if I had or or more values assigned. As such changed the field to a straight string and it worked. Change shown below:

Uploading Images getting HTTP 413 Error
To enable management clusters (schedulers) you need to upload the corresponding dish images into the Controller (Mesos, Kubernetes or Swarm). I found that regardless of the image I was getting a NGINX error of 413 'Request Entity To Large'. You don't have to be a google ninja to identify the error quickly with the reported resolution to set the 'client_max_body_size' value in the 'nginx.conf' file.On the controller VM the NGINX service runs as docker container called 'ManagementUi' so it was easy to then pull the configuration file down with a 'docker cp' command to have a look, low and behold the setting is not there!

OK, now I have just a rat down he drainpipe I can say that I was on the wrong track. NGINX looks after the User Interface but not the API. Looking at the docker containers there is one running HAPROXY which is the loadbalancer service, called you guessed it 'LoadBalancer'. Jumping into this VM and looking att he file 'haproxy.cfg' you can see that the API is served by ports 28080 and 9000.  By switching the photon cli target to either these ports let the images upload successfully (e.g. photon target set http://10.63.251.150:9000).

I took the long way but got there in the end, time to start playing!!!

Docker-machine and VMware vSphere Clusters, Lessons Learned

I have been doing a bit of work with various flavours of containers lately which comes with the highs and lows of being a technologist. One thing that was interesting is that with the recent updates to docker-machine, my setup for binding and deploying to our vSphere environment no longer worked.

This issue presented itself when attempting to provision a new instance with the following error:

Running pre-create checks...

Error with pre-create check: "default host resolves to multiple instances, please specify"

Digging deeper and I found out that where I was using the cli argument / variable 'VSPHERE_COMPUTE_IP' to specify a ESXi host to bind to, you know need to use 'VSPHERE_HOSTSYSTEM' (or as a cli argument '--vmwarevsphere_hostsystem'). As this is new documentation is fairly light as are real examples. What the documentation does provide though is two syntax examples as follows:

for cluster -                      VSPHERE_HOSTSYSTEM=<Cluster Name>/
for stand-alone host         VSPHERE_HOSTSYSTEM=<Cluster Name>/*

Using those two syntax examples in the lab I first attempted 'VSPHERE_HOSTSYSTEM="UCS8#General Use/' but this resulted in the same error message. It was then also notable that examples were pushing to single node clusters, the test cluster I was targeting has 4 nodes.

Anyway to cut straight to it, the documentation is a little incomplete. You do need to still target a specific host in the cluster and docker-machine will not do that for you. As such in a cluster with multiple hosts the actual syntax is:

VSPHERE_HOSTSYSTEM=<Cluster Name>/<Host Name>

 so for my setup it looks like this

VSPHERE_HOSTSYSTEM="UCS8#General Use/esxrack01.vce.asc" 

I guess while I am here another hint does not hurt. The other piece of lovely syntax requirements is the optional variable for targeting a specific ResourcePool, to do this you use the variable VSPHERE_POOL. The syntax for this is

VSPHERE_POOL="/<DataCentre>/host/<Cluster>/Resources/<Resource Pool>/..."

So again in our lab setup the variable looks like this:

VSPHERE_POOL="/ASC/host/UCS8#General Use/Resources/Containers" 

Docker-Machine and vSphere together are a great team and well worth a play so hope this helps. 

I am also fortunate with doing a bit of work with VMware's vSphere Integrated Containers (VIC) which takes these issues away and provides a very strong alternative to other scheduling/clustering services such as DOCKER SWARM within your vSphere environment but ore on that in another post later.

What is VMware VSAN 6.2 Erasure Coding?

I have been asked what is the 'Erasure Coding' support in VSAN all about as it gets referenced in the same sentences as the new support for RAID5 and RAID6 configurations (only available in all flash configurations).

VMware is stating Erasure Coding as a term to reference for any data partitioning scheme that allows data to be fragmented but recoverable in case of failure even if some parts  are missing (use of parity). In general reference they are referring to the support of Erasure Coding to represent the fact that they have moved from a mirroring scheme to support both RAID 5 (support for 3+1) and RAID 6 (Support for 4+2) to increase the level of usable data from RAW Capacity across nodes.  

Prior releases only supported a 1+1+witness or worst (1+n where n=failure tolerance but each n was a mirror copy) scheme hence you always needed a minimum 3 nodes to start with but only really got 50% usable capacity at best. This is why the Erasure Code support is getting such big focus as it makes VSAN more efficient with physical resources.

Worth noting that Erasure Coding is not referring to a disk configuration as with RAID but how the fragments are distributed between hosts. It is for this reason that depending on the choice mode also drives the minimum number of hosts required (3 for standard setup, 4 for R5 and 6 for R6).

So Who Does Use Linked vCenters?

OK, I am sure that most onboard and I put up my hand, I personally am a fan. When I talk to a lot of clients though, generally I find that most aren't.

Two linked vCenters in Web Client

In the old days we linked the instances together with the linked mode utility (or unlinked it with the same tool). Nice and easy and all of a sudden security and licensing information is shared between vCenter instances. The big gain though was that through the C# client (oh I do miss your usefulness sometimes in this depleting role world) I had a one stop shop to access up to 12 vCenters. Now it is even easier, just have the vCenters in a common Platform Service Controller (PSC) SSO implementation (common site/ multi site doesn't matter).

This in turn enables some very cool features beyond the original benefits. The one that drive me to link mine up was the ability to enable cross vcenter migrations. Being someone that is fortunate to have access to a well equiped lab I have enabled this with VPLEX for the stretched storage and use both SRM 6.1 and vMotion in vCenter to control the migrations (more on that in another blog post I think). 

This blog is not to tell you how to link, that is covered in a lot of other blogs written by very smart people. More I want to draw attention to issues that surface a lot with the newer Enhanced Link Mode with those add-on services that use the web client.

As I have just experienced an issue again I am writing this while it is fresh in the mind. What I see is that with a lot of product teams that do include Web Client plug-ins, there seems to be quite an absence of testing vCenters in Enhanced Linked Mode. Have you ever had this style of conversation:

me)              I am <inset problem description here> in the web client 

developer)   what version are you running

me)              The one listed in requirements

developer)   Are you using the Windows Version of vCenter or the Appliance?

me)              I am using the (whatever) in linked mode

developer)   ohh, we haven't tested that

Anyway some gotchas I seem to experience include:

• Only one vCenter is supported in the linked set for the plug-in
• Only the first vCenter is visible to the plug-in
• Plug-in is unable to differentiate (filter) objects out that are only present within the aligned vCenter (e.g. all datastores across all vCenters are displayed)

So the point of this entry beyond the rant is just to bring attention to the fact that there are some considerations when writing a plug-in for linked vCenters. If you are running into issues and you have got linked vCenters I would suggest that the first step should be test against a stand-alone vCenter. This helps isolate the issue and can help when you are talking to the support/product/vendor teams.

It's Not Just About Backup!

Why Have Different Levels of Protection?

I like the saying that comes when people refer to an old classic car 'they don't make them like that anymore'. The thing is, the statement should be followed with the statement 'thank goodness'. Can you imagine putting up with that level of reliability, quality, safety, comfort level with a car you purchased from a dealer today that you would have got in years gone by?

Same goes with technology. I still wake-up some nights and shudder when remembering the old tape backup routine. This use to go in two ways, the first was the nightly scheduled tape shuffle which thankfully I was not tasked with. The other was the pre-rollout backup we would run before pushing any new service out. This use to comprise us kicking the job off, then trotting down the road to a golf driving range to knock a few buckets of balls into no mans land to kill the time (usually hours).

So looking at that, we had two different use cases but only one method at our disposal, the inglorious, forever unreliable tape backup. Fast forward 15 years (maybe more but I keep that quiet) and we have many different forms of protection at our disposal including storage replication, snapshotting, tape backup (but hopefully VTL, not those horrible tapes).

All these options serve a purpose and work together nicely. If we were to classify the different use cases into buckets they could be:

Business Continuity (DR) = Remote Replication, constant protection driven by RPO and RTO requirements

Operational Protection = Local Snapshotting (or continuous protection as is provided with RecoverPoint) typically done by set interval (storage level) or self service (VM level)

Long Term Retention = Traditional Backup style typically run once a day

Each of these has there purpose and specific business requirements that drive their implementation. Interestingly I see a lot of the BCM and Long Term Retention being positioned but little of the Operational Protection being catered for. Back to my scenario at the start, if I could of had snapshotting at my disposal (or if I was real lucky, EMC RecoverPoint Continuous Data Protection) my golf practice (aka time wasting) would not have happened as we could have snapshotted the services that we were updating and started the rollout straight away.

Self service snapshotting is easily available these days thanks to two things:

1. Virtualisation with the VM level snapshotting (Checkpointing in a HyperV world)
2. Automation tools for storage (EMCs Storage Integration Suite is a good example)

So that is all cool for those known events but what about those unknown such as data corruption. That is what scheduled snapshots protect you against, providing a much more granular way to protect your systems beyond the typical 24 hour cycle of Long Term Retention. You may only retain a short set of snapshots such as 1-7 days but they can provide good peace of mind for those services deemed worthy.

I do realise that replication can also provide a way of rolling back but typically it is 'to the last change' or committed IO operation, so a corruption could easily by on the remote site as well as the source. Also replication would require that traffic to come back down the wire which adds time to the recovery / rollback process.

Another benefit of Operational Protection is that it can provide an easy / quick way for copies of  datasets such as those within a database to be presented to an alternate location such as a from production to a test/dev instance.

Anyway, I got that of my chest so I feel better. Operation Protection = Good!

Just as a last note on this, I did not include High Availability (HA) in this as I am more looking at where the old functions we used tape for have evolved. There is some real cool stuff that can be done with stretched high availability that spans physical locations as is supported with vSphere Metro Storage Clusters and Microsofts Failover Clustering with products such as EMCs VPLEX, but that is a big enough topic on its own.

Thin Provisioning with VAAI and VASA in vSphere Working Together

That Damn Pesky Thin Provisioned Threshold Alarm in vCenter

Have you ever had those alarms go off in vCenter telling you that your thin LUN has exceeded its consumed space threshold? This is an interesting repercussion of VASA (storage awareness reporting)   feeding its view of the thin provisioned LUN back to vCenter and then it reacting to it.

So first let's have a look at the issue. Thin provisioning in a block storage world tends to start small and then continue to grow until the configured upper limit is reached. Makes sense and it is what you would expect, as data matures the efficiencies are reduced with Thin Provisioning. In the wonderful world of virtualisation where VMs tend to be fairly fluid due to provisioning, deletion, moving activities a LUN can be full one moment and half empty the next. Add vSphere Storage Clusters and SDRS and it does not even need manual intervention.

Anyway back to the problem at hand, the thin datastore all of a sudden looks full so you shuffle stuff around and clean up space. All good at the datastore level but you are still getting alarms off the datastore, specifically the 'Thin-provisioned volume capacity threshold exceeded' alarm. The reason behind this is that once a block has been written to the array does not know how it is being utilised so reports it as consumed.

This is what VAAI Unmap is all about, giving the host a way of clearing a previously written to block in a way that the array can act on. The drawback on this is that there VAAI unmap is still not an automated process so needs to be executed through the CLI (esxcli storage vmfs unman <datastore>) or via PowerShell as a manual process.

Be aware though as this only returns space back to the array that has been released at the datastore level such as what would happen if you deleted a VM. There is also the in-guest space issue, a thin provisioned VMDK will also grow in much the same way that a datastore does. You can also return space back that has been deleted to within the guest through tools that need to be run within the guest such as Microsoft's SDELETE (http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx) or VMware's own Guest-Reclaim tool (https://labs.vmware.com/flings/guest-reclaim).

By getting both your in-guest and ESXi host level space reclaim strategy in place you can ensure that VAAI + VASA = Happy House and you are getting the most efficiency out of your storage (at the space level anyway).