Federation certificate expiration notifications

You may have recently received an automated email warning about the imminent expiry of a federation certificate in your virtual datacentre, reading as follows:

The federation certificate expiration is [DATE] [TIME]. An expired certificate may disable federation with the identity provider setup with your organization. The certificate can be regenerated from the Federation Settings page.

The federation certificate is a part of the SAML Identity Provider process introduced in vCloud Director 5.1, to which we upgraded last year. SAML can be used within vCloud organisations as a basis for user authentication to their vDC (i.e. for authentication to your virtual datacentre through the vCloud portal account that you use to gain access to the consoles of your VMs). Most customers use either the inbuilt vCloud Director accounts or local Active Directories for this – if you are using SAML you should already know, as you will have had to explicitly set it up. Unfortunately vCloud Director does not actually check whether or not this message is relevant to you. It just sends out the warning regardless.

Federation certificates are valid for one year and warning e-mails go out a week before they expire. The update process is fortunately very straightforward:

1. Log into your vCloud environment and click on the Administration tab.

2. On the left hand side of this menu, choose the Federation link under the Settings menu. You should see the following screen:

saml1a

3. The “Use SAML Identity Provider” box should be unchecked. If it is, continue with this procedure. If it is checked, then please stop here. Contact NSMS and we will advise further.

4. Scroll to the bottom of the screen to the Certificate section, displaying the expiration details underneath. Click the Regenerate button to create a new certificate.

5. The system will prompt you with the following message:

saml2

Provided the “Use SAML Identity Provider” box from the previous steps is unchecked, it is safe to click on OK and regenerate the certificate. The new certificate will be generated and the new expiration date displayed (it will be valid for one year).

Posted in Uncategorized | Leave a comment

The University Private cloud and the HeartBleed Vulnerability

I’m sure you will have all seen in the media, messages regarding the heart bleed vulnerability in the openssl software used to secure many websites and services across the internet, as well as local announcements from OxCERT and other university IT providers.

The cloud team would like to reassure our customers that all except one component of the cloud service was not vulnerable to this issue. All public facing interfaces, consoles and the underlying infrastructure were not affected. One component, that which provided the edge device gateways for a small number of customers of the virtual datacenter service who have deployed vshield edge devices, was affected. The impact of these however was extremely low, and the appropriate patching and mitigation steps were take as soon as a patch became available from VMware.

If you have any concerns please do not hesitate to contact us (nsms@it.ox.ac.uk)

For an excellent blog post from the information security office see

http://blogs.it.ox.ac.uk/oxcert/2014/04/14/open-heartbleed-surgery/

The local OxCERt bulletin on this can be found at

http://www.oucs.ox.ac.uk/network/security/bulletins/osb2014-054.xml

whilst the VMware documentation on patch availability and product vulnerability can be located at

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2076225

 

Posted in Uncategorized | Leave a comment

Disaster recovery of the University’s private cloud infrastructure

The University’s private cloud operates over two sites – 13 Banbury Road (former OUCS) and the University Shared Data Centre on Parks Road. Half the service is delivered from each site, with “Campus” virtual datacentres hosted at OUCS and “Datacentre” vDCs hosted at USDC.

(The difference between “Campus” and “Datacentre” vDCs will be covered in more depth in a future blog post, but is basically in how networking is presented. If you want your local network – and hence your existing IP range – to be available in your virtual datacentre, then you’ll want a Campus vDC. If this is not a concern for you and you are happy to use the far more resilient data centre networks, then we will deploy a “Datacentre” vDC. On reflection we should have named “Datacentre” vDCs something a little more illuminating, but hey).

As well as delivering half the service, each site also acts as a failover location for the other half of the service. In the event of a disaster at one of the sites, we will fail over your virtual datacentre to the alternate site (so for example if the OUCS site were to suffer a complete power outage, we would fail all “Campus” vDCs over to the USDC site). There are spare ESXi hosts ready at each site ready to accommodate the extra VM workloads in the event of a site failover.

The Oxford Cloud DR solution was developed jointly by VMware and the University and is believed to be the first of its kind in the world. It is based upon a combination of VMware’s Site Recovery Manager product and a series of PowerCLI scripts and aims to automate the process of disaster recovery as far as is practical. The technical design has been the subject of several seminars and presentations, most notably at VMworld 2012 (session NF-BCO2155) and the UK VMware User Group (VMUG) User Conference in November last year. Presentations are available online (although it appears that a subscription is required for access to VMworld presentations for non-attendees) and a case study detailing the components of the Oxford solution has recently been released. This technical paper is entitled “VMware vCloud Director Infrastructure Resiliency Case Study” and is available from VMware’s site at http://www.vmware.com/resources/techresources/10355.

We will likely present on this at this year’s ICTF conference, but in the meantime feel free to contact us if you are interested in hearing more about the disaster recovery solution.

Posted in Uncategorized | Leave a comment

Rescanning your SCSI bus to see new storage

If you have added new storage to a running VM, you probably won’t see it. This is because the SCSI bus to which the storage devices are connected needs to be rescanned to make the new hardware visible.

How you rescan the SCSI bus depends on the operating system your Virtual Machine is running.
Instructions for common supported operating systems are as follows, these assume you have root or suitable privileges.

Linux, when adding a new disc

First find your host bus number

grep mpt /sys/class/scsi_host/host?/proc_name

Which should return a line like

/sys/class/scsi_host/host0/proc_name:mptspi

where host0 is the relevant field.

use this to rescan the bus with the following command

echo "- - -" > /sys/class/scsi_host/host0/scan

In the above command the the hyphens represent controller,channel,lun, so – – – indicates all controllers, all channels and all luns should be scanned.

Linux, when expanding an existing disc

Assuming you know the device name of the disc you have expanded (eg /dev/sda) then you can simply issue the following command to force the rereading of the disk geometry. NOte that this is not 100% reliable with LVM on older kernels but should work with current kernels

echo 1 > /sys/class/scsi_device/device/rescan

Windows Server

Windows should pickup the change in discs regardless of you adding or expanding discs however if not you can either use the gui or the command line to force this. Note that expanding the boot device will typically always require a reboot, and will require you to boot from an alternative boot device (a boot CD image) to actually expand the windows boot partition. All these method assume you are an administrator on the system.

GUI method – Server 2003

Open Computer Management (Local).
In the console tree, click Computer Management (Local), click Storage, and then click Disk Management.
Click Action, and then click Rescan Disks.

GUI method server 2008

Open Server Manager.
In the tree pane, double-click the Storage node, and select Disk Management.
Right-click Disk Management and select Rescan Disks.

Command line method

as an administrator, start a command prompt and type

diskpart

at the diskpart prompt, type

rescan
Posted in Uncategorized | 4 Comments

Options for installing VMware Tools into Linux VMs

There are two versions of VMware Tools – the full commercial package shipped with VMware products, and an open source version based on development snapshots of the commercial packages, called the Open VM Tools. The latter are not supported by VMware, but are available in some distributions’ default repositories (e.g. Ubuntu and OpenSuSE) as the ‘open-vm-tools’ package.

However, VMware also provide the commercial VMTools as Operating System Specific Packages (OSPs) – which are packaged versions (.deb or .rpm) of the commercial VMware Tools. These are fully supported and can be installed using standard Linux package management systems (e.g. apt or yum).

The OSPs and the Open VM Tools are named similarly and therefore fairly easily confused with each other. However, mostly because they are unsupported development snapshots, we do not recommend installing the Open VM Tools.

To install the commercial VMware Tools, there are three options:

  • Install from vCenter
  • Upload and install from a .tgz or .iso
  • Install from the VMware Operating System Specific Packages (OSP) repositories

The last of these integrates with the guest’s package management system and so may be preferable for some Linux administrators.

Installing VMware Tools from vCenter

Follow the instructions at http://www.oucs.ox.ac.uk/sis/cloud/using/installVMTools.xml (only covers Debian-based systems but should be easy enough to extrapolate for others). This is the standard (and recommended) approach. However, note that this is not an option on Debian 7.0 systems with kernel version 3.2.35-2 (the latest as at the time of writing) – there appears to be a bug in this version of the kernel that causes a VM to lock whenever a virtual CD drive is mounted. So for Debian 7 the next option is favourite.

Installing from a compressed tarball / ISO

The VMware Tools are stored on ESXi servers as an ISO file, which is generally presented via the vSphere client or vCD portal to the VM as a virtual CD. However there is nothing to stop you downloading the ISO from the ESXi host and extracting the compressed tarball from it. Each ESXi host stores its VMware Tools ISOs in the /usr/lib/vmware/isoimages directory. The file required is linux.iso.

Another option is to upload the ISO to your VM and mount it via the loopback device:

mount -o loop -t iso9660 /tmp/linux.iso /mnt

For ESXi 5.0 Update 2, the compressed tarball is file VMwareTools-8.6.10-913593.tar.gz

Once uploaded and mounted, follow the instructions at http://www.oucs.ox.ac.uk/sis/cloud/using/installVMTools.xml to install.

Installing from the OSP repository

RHEL / CentOS / Scientific Linux

1. Obtain and import the VMware Packaging Public GPG key:

wget http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

2. Import the key:

rpm --import VMWARE-PACKAGING-GPG-RSA-KEY.pub

3. Add the repository by creating the file /etc/yum.repos.d/vmware-tools.repo and adding the following:

[vmware-tools]
name=VMware Tools for CentOS (or OEL) $releasever - $basearch
baseurl=http://packages.vmware.com/tools/esx/5.0u2/rhel6/x86_64
enabled=1
gpgcheck=1
gpgkey=file:///srv/crypto/VMWARE-PACKAGING-GPG-RSA-KEY.pub

Edit the gpgkey line for wherever you stored the public key. Edit the baseurl line for appropriate version of ESXi, distribution and architecture.

4. Install the VMware Tools

yum install vmware-tools-esx

or, for no graphical support:

yum install vmware-tools-esx-nox

Ubuntu

1. Obtain and import the VMware Packaging Public GPG key:

wget http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

2. Import the key:

apt-key add VMWARE-PACKAGING-GPG-RSA-KEY.pub

3. Add the repository by editing the file /etc/apt/sources.lst and adding the following:

deb http://packages.vmware.com/tools/esx/5.0u2/ubuntu precise main

Make sure you customise this line for your distribution – this example is for Ubuntu 12.04 LTS (Precise Pangolin).

4. Refresh the local cache and then install kernel modules

apt-get update
apt-get install vmware-tools-esx-kmods-kernel_release

kernel_release will be one of “generic“, “server“, “virtual” or “generic-pae” based on your kernel type. Check which by running uname -r.

5. Install remaining components

apt-get install vmware-tools-esx

or, for no graphical support:

apt-get install vmware-tools-esx-nox

Debian

There are no OSPs for Debian. While in theory you could use the Ubuntu repositories, the safest option is to use one of the previous methods – install from vCenter or install from a .tgz/ISO. Follow the instructions at http://www.oucs.ox.ac.uk/sis/cloud/using/installVMTools.xml as described previously.

Posted in Uncategorized | 2 Comments

Setting up Kerberos and Active Directory authentication for vCloud Director

Logging in to your vCloud Director organisation via the web portal can be achieved in a number of ways. You can use a database local to vCloud Director, your Active Directory, or another LDAP v3 compliant directory service (e.g. eDirectory), for authentication and group membership lookup.

(At the present time using your Oxford SSO Kerberos credentials is not supported. As it happens, vCloud Director itself does support Kerberos authentication against an external Kerberos realm and we have successfully completed proof-of-concept testing, but some reconfiguration of the Oxford KDCs is required before we can roll this out in production. This is coming soon and we will document the procedure when the necessary work has been completed).

The rest of this blog post will focus on secure Active Directory authentication. I’ve seen a few articles explaining how to set up AD connectivity in a lab environment, but they tend to suggest using unencrypted simple LDAP binds. These are a Very Bad Idea, especially in a production environment, as they pass the AD user credentials across the network in plain text. Simple LDAP binds can (and should) be encrypted using SSL. This is better (and your only other option if you are using eDirectory), but it is still preferable not to send the password at all, encrypted or not. To achieve this we can use Kerberos authentication. With Kerberos, no passwords cross the wire – just encrypted Kerberos tickets with a limited lifespan (ten hours by default).

1. Configuring the domain controllers

The first step is to turn on LDAP over SSL on your domain controllers. Even though we will use Kerberos for authentication, vCloud Director needs to look up group information in the AD. Again we don’t want these LDAP queries to be sent across the network in plain text, so we need to enable LDAP over SSL (LDAPS).

To do this, follow the steps in Microsoft Knowledgebase article KB 321051 (http://support.microsoft.com/kb/321051). When you create the CSR, make sure that the key length is at least 2048 bits and that the “Subject” line contains all the information required by the signing authority (e.g. Subject=”CN=myserver.oucs.ox.ac.uk, OU=IT Services, O=University of Oxford, L=Oxford, S=Oxfordshire, C=GB”). Ideally you should request a certificate for all your domain controllers, although vCloud Director can use only one.

Secondly, a service account must be set up in your AD. This account needs to have sufficient rights to read group memberships of user accounts in the Active Directory. By default, the Domain Users group has this right, so no special privileges are required in the AD.

Once your domain controllers accept LDAPS connections and the service account is set up, we (IT Services) will configure your vCloud Director organisation to authenticate against the AD using Kerberos.

As an organisation administrator, you won’t be able to see this in your organisation’s config, so the rest of this blog post is really for information.

2. Configuring vCloud Director for Kerberos authentication

From the screenshot you’ll see that we can only enter one domain controller, which is a bit of a flaw. Potentially we can fix this with round robin DNS, but we’ve not investigated this.

We would prefer to import the certificate that you used to enable LDAPS on your domain controller, rather than succumbing to the “Accept all certificates” temptation (perhaps not a great security risk, but let’s do it properly). So we’ll ask you to make this available to us.

(We leave “Use external Kerberos” unchecked, but when the OX.AC.UK KDCs are ready it’s this checkbox and some extra config that will allow us to use Oxford SSO).

The Edit All Realms button is necessary to define DNS to realm mappings, as well as the KDC for each realm. For AD, the KDCs are your domain controllers. So in the Realm tab, we add the Active Directory realm (this is the same as the DNS name, but Kerberos realms are in upper case to distinguish them from DNS domains) and the domain controllers for that realm:

We can leave “Allow lower-case realms” ticked – it doesn’t much matter whether this is on or off. Note that because we can enter multiple domain controllers here, we have redundancy in terms of authentication. However the main page from the first screenshot still only gives us the option to use one server to do LDAP lookups (i.e. the one listed in the Server field). So we still have a dependency on one domain controller.

In the DNS tab of the Edit Realms screen, we also define the DNS to realm mappings:

Back on the main screen (the first screenshot), we enter the AD service account username and password. Note that we must have the realm entered in upper case – this is a Kerberos convention, defining a realm principal. Put it in lower case and it won’t work.

That’s it – at this point, you can import users from your Active Directory into your vCloud Director organisation and set up their access rights. Once that’s done, they can log in to the portal using their AD credentials.

One last thing of note is that when vCloud Director imports users from AD, it maps certain attributes to locally displayed data (for example, it will display the AD “sn” attribute for a user account in the vCD “Surname” field. For the user name, it is worth noting that when Kerberos is used, the mapping is set to be from the AD userPrincipalName attribute (e.g. jsmith@oucs-test.ox.ac.uk) rather than the sAMAccountName (jsmith). This means that users will need to log in at the vCD portal with their full UPN (jsmith@oucs-test.ox.ac.uk) rather than the shorter version (jsmith). As such, it is preferable to edit the mappings so that the vCD User name attribute maps to the AD sAMAccountName attribute:

and then users can log in with their short name.

That’s pretty much it. There is a VMware Knowledgebase article that briefly covers Kerberos authentication to vCD (KB 2015986) but it doesn’t have much in the way of explanation as to what’s going on. It’s also worth mentioning that the setspn command mentioned in step 12 of that article isn’t necessary.

Posted in Uncategorized | 2 Comments

Browser support for the OxCloud vCloud director console.

Firstly we need to be clear, pretty much any browser will work with the general cloud interface, assuming it supports flash and Java. The problems start when we get to opening the console of a vApp or VM.

This console uses a version of the VMware console plugin that has been supplied for the web interface for vSphere for a while now. It comes in two flavours – a Windows executable and a Linux version. Underneath all this though it’s a Java and flash application with all the joy that brings.

A summary of our findings is shown in the table below, we’d tried to test more recent browsers than are listed by VMware.  Where things are listed as not working this typically means that they either have never worked for us , or are so unreliable that we would not consider using them.

OS Browser Supported ? Tested ? Works ?
Windows XP – 32bit IE7 Yes Yes Yes
IE8 Yes Yes Yes
FireFox 10 Yes Yes Yes
Windows 7 – 32bit IE8 Yes Yes Yes
IE9 Yes Yes Yes
FireFox 10 Yes Yes Yes
Windows XP – 64bit IE7 Yes* Yes Yes
IE8 Yes* Yes Yes
FireFox 10 Yes* Yes Yes
Windows 7 – 64bit IE8 Yes* Yes No
IE9 Yes* Yes No
FireFox 10 Yes* Yes No
Centos 6.3 – 32 bit FireFox 11 No Yes Yes
FireFox 13 No Yes Yes
Ubuntu 11.10 – 32 bit FireFox 15 No Yes Yes

*32 bit version only supported.

The official line :

VMware knowledge-base article

When 32 is greater than 64!

Officially the console is only supported with 32bit browsers, even on 64 bit OS’s. Unofficially your milage may vary, which is a phrase I’ll be using a fair bit when talking about this topic.

On Windows 32bit, IE upto and including IE9 and Firefox up to and including 10.01 work fairly reliably, however Firefox can produce an interesting side effect whereby it only works in full screen mode, especially on versions after 10. OnWindows 64bit  most of the time we have had problems, typically consoles not connecting. We have also found that connecting a console from a 32bit systems whilst a 64 bit is connected can “wake up” the 64bit console connection, at which point this will work until it is disconnected.

This seems to vary from update to update, so recently, people in the office who had working installs were cursing the update to Firefox 13 which meant that only full screen mode worked., even on 32 bit Windows. Hopefully a future update will fix this, but in this time of auto-updating web browsers against vulnerabilities, it would be nice to see a more stable interface.

VMware tools and the console.

VMware tools play an increasing role in cloud director, as they are the mechanism by which vApp guest customisation happens, enabling IP reconfiguration and the like. However another area of weakness for the console, especially for linux guests seems to be the VMware  tools integration.  Even with the latest tools installed and running, its often the case that this has no effect on the console, leaving the user with a very small useable screen area and no support for mouse swapping between windows. I’ll be investigating this further and reporting back in a future post

The good news is that browser support has improved a little in the latest release. Watch our blog for more information when we upgrade.

 

UPDATE : April 2013

We’ve had reports that recent versions of Firefox ESR (version 17.0.x) also only work if you full screen the console. This certainly seems to be the case on a short test within the team.

Posted in Uncategorized | Tagged , , , | Leave a comment

Welcome

Welcome to the OxCloud blog. We’re intending it to be a collection of technical notes on the University private cloud, general musings on cloud computing, as well as a commentary of what we like (and don’t like) about the various technologies underlying the private cloud infrastructure we’ve implemented. Probably we will include the odd ill-thought out rant, too, because that’s what blogs are all about (or so I’ve heard).

So firstly, a quick introduction to the Oxford private cloud. Full documentation is available from http://www.oucs.ox.ac.uk/sis/cloud/ – but in brief, there are three services offered to the University:

1. Individual VMs (the “Hosted VM” service)

Under the Hosted VM service, we deploy a Windows or Linux VM and hand it over to the customer. Hosting and the underlying infrastructure are looked after by the Oxcloud team, but the administration of the VM itself (patching, application installation and maintenance etc) is the responsibility of the customer.

2. Infrastructure-as-a-service (the “Virtual Datacentre” service)

Under this service we offer a pool of resource (CPU, RAM and storage) that the customer can assign to virtual machines as he or she sees fit. This concept is called a “virtual datacentre”. For the VMware-savvy, in essence what we do is create a resource pool in vSphere and hand it over to you to deploy VMs into (although that’s a bit of a watered-down description – there’s a bit more to it than that). Within this service there are two options regarding how your virtual datacentre is networked:

  • it can go on the “Campus” cluster, which allows you to bring in the VLAN (and hence the IP range) that you are using in your college or department,
  • it can go on the “DataCentre” cluster, which means a new VLAN – and so any VMs you deploy will be on a different IP range to the VLAN that you use in your college or department.

There are advantages and disadvantages to both approaches, which will be the subject of a further blog post.

3. Database-as-a-service (the “ORDS” service)

The third service is the Oxford Online Research Database Service (ORDS). This allows research projects with staff at Oxford to swiftly create relational databases that can be edited, searched, and shared online. It also allows existing databases to be imported, so that they may be shared or further developed. Full documentation on ORDS will be forthcoming shortly on the OUCS website.

The infrastructure underlying these three services is based on Dell hardware (Dell BladeCenters with Dell Compellent storage), running VMware’s vSphere and vCloud Director virtualisation products. The infrastructure is spread across two sites (OUCS on Banbury Road and the University Shared Data Centre on Parks Road) with half the service delivered from each site. Should it be necessary, we have the ability to fail one site over to the other and deliver the entire service from one site – either in a planned migration, or in the event of a disaster that takes one site offline. We’ll have more on this in a later blog post.

So, for now – welcome, and we hope you will find the content of this blog to be of interest. We welcome feedback: if on the blog, please leave a comment, and if on the cloud service, feel free to get in touch via one of the mechanisms at http://www.oucs.ox.ac.uk/sis/about/contact.xml

Posted in Uncategorized | Tagged , , | Leave a comment