Project MADDOX Workshops

In September Project MADDOX invited ITSS to attend a 2-hour workshop entitled “Active Directory – Single Sign-On Integration”. The workshop content was geared towards ITSS who operate Active Directory environments and who want those environments to be integrated with the University’s SSO infrastructure.  Around 40 ITSS signed up across two workshops, with approximately equal representation from across the University’s divisions and colleges.

The workshop was planned as an interactive session to foster audience participation, with the following format:

  1. Introduction (5 minutes)
  2. Break-out Session: “How do you use AD?” (15 minutes)
  3. Presentation (30 minutes)
  4. Break-out Session: “How does a central AD help you?” (25 minutes)
  5. Break (5 minutes)
  6. Break-out Session: “Every Service has Benefits and Costs” (25 minutes)
  7. Next Steps and Questions (15 minutes)

After an introduction to the workshop we started with a break-out session, entitled “How do you use AD?”, in which we asked attendees to split into pairs and describe what kinds of usage their current AD deployments are put to. Responses to this question were wide-ranging, with not only descriptions of the expected use cases such as workstation user authentication and authorisation, group policy, controlling access to shares and printers, web services, database services, and third-party appliances, but also for VPN authN/authZ, access for Macs, and many other rarer, more specialised third-party applications.

Adrian Parks and Nigel Brown then gave a presentation on Project MADDOX to the attendees. We began by describing the background of AD and SSO within the University, from 2000 onwards, through the work done in the run-up to the Nexus implementation, up to the current day. Next we provided an update on the current OUCS project that is investigating ways of offering an enhanced level of support for AD-SSO integration: Project MADDOX. The project’s investigations were carried out into the potential of an Active Directory integrated with central SSO for authentication only. This is because authorisation in a federated environment is best devolved locally (local ITSS will know their authorisation requirements best). We described the centralised AD support scenarios that we investigated as a part of the MADDOX project (Native AD, Indirect Realm Trust, and Direct Trust). We briefly explained the reasons for the scenario selections, and why some other scenarios were ruled out. Finally we described the tests that we ran and the results. The presentation is available for ITSS to review at http://projects.oucs.ox.ac.uk/maddox/resources/MaddoxWorkshop-v1.8.ppt.

Following the presentation we introduced another break-out session, entitled “How does a central AD help you?”, in which attendees were asked to break out into small groups, to discuss the perceived benefits of the feasible scenarios described in the presentation, as well as what would characterise their ideal AD solution. The responses were wide-ranging again, but commonly focused on:

  • easier account provisioning and deprovisioning
  • an improvement in the user experience (the convenience of fewer passwords)
  • benefits for group policy based deployment
  • a desire for a better level of documentation and support, with ITSS recognising the potential for cross-unit shares and other central services that a wide take-up of SSO  would offer

There was also some discussion about the authorisation difficulties that some units faced, many of which the Core User Directory (CUD) project should help to ease. Another important point raised was that while 3rd party applications may appear to “support AD”, they may not support Microsoft’s recommended Kerberos authentication mechanism (in cases such as these it would be beneficial to ask vendors for appropriate support).

At this point in the workshop we took a break for attendees to stretch their legs.

After returning to the session we embarked on the final break-out session of the workshop, “Every Service has Benefits and Costs”, in which ITSS were asked to identify the benefits of enhanced central support for AD authentication, and consider any costs. In an attempt to estimate the worth of a proposed service some good points were made for both benefits and costs. To summarise:

  • Some ITSS recognised that time savings were possible, from the point of view of reduced calls to the Help Desk, reduced pressure on local ITSS time, a reduction in duplicated effort, and improvements in reputation and customer satisfaction
  • Others postulated that the value of the service compared to the local cost of integrating it would be negligible.
  • Where ITSS had already invested in their own account provisioning and deprovisioning, the benefits were not seen to be significant
  • For those that did not have account management processes in place there were increased benefits in, or improved support for, centrally-provided account management.
  • Many of the benefits were seen as difficult to equate with a monetary value.

This session also touched on the question of whether a service like this could be chargeable, and what units would be prepared to pay for such a service. Many (but not all) College and Department ITSS were/would be reluctant to pay for a centralised authentication-only AD service, and the general feeling appeared to be that University Departments already pay for SSO services through the infrastructure charge. However, the range of responses to this was large. Most attendees from Departments estimated the savings to be minimal, with the exception of one Departmental attendee estimating a saving of up to 3 people’s workload per year, and attendees from Colleges largely estimating a negligible saving, with a few College ITSS esimating a saving of up to 50 hours per year. In cases where the move to a centralised service was estimated to incur net costs, it was considered that this was likely to be a one-off migration cost, with the cost gradually offset by future benefits.

Following the final break-out session we described the next steps: to assess the options in the light of ITSS feedback, cost-effectiveness, and key stakeholder concerns; to select an appropriate solution and implement it. Once the choice of a preferred solution has been made, we plan to communicate this to ITSS. We then took questions from attendees. One question was about whether any of the existing AD SSO solutions would be removed as a result of decisions taken during the MADDOX Project (it should be stressed that the currently available AD SSO solutions will remain, whatever choices are made as a result of the MADDOX Project). Other questions were concerned with the technical possibilities around using cached logins and shadow accounts with a central domain. Someone else posed the question of why we can’t simply use OUs (or individual domains) in a single central AD (because there could be no local-specific settings, would be too many admins, problems in maintaining intra-forest security boundaries etc), or set up password synchronisation to local ADs (password synchronisation has technical limitations that would make this entirely unsupportable, quite apart from the security risks inherent in widely distributing SSO credentials).

Finally we wrapped up the session, and as I did then, again I would like to take this opportunity to thank all the ITSS that attended the workshops, who will have helped inform the choices to be made during the MADDOX Project.

Posted in Uncategorized | Comments Off on Project MADDOX Workshops

UAS Conference 2011

Tuesday saw this year’s UAS Conference taking place in the University Examination Schools. This annual event brings together staff from around the University with the aim of exchanging information about things happening in the University Administration and Services Division. OUCS officially joined UAS in January of this year (moving from ASUC), so although we have been invited to present / attend in the past, there was perhaps a sense in which our presence was cast in a different light this time around.

The UAS Conference and ICTF Conference are quite different affairs. Where the ICTF Conference has a backbone of plenaries with workshops woven around them, the UAS Conference programme is much more open with up to 11 concurrent workshops (either 30 minutes or hour-long). These are organised in four hour-long slots, with an opening plenary (this year given by the Registrar under whom UAS sits), and a closing panel Q&A session.

Workshop topics are quite diverse, as you might expect given the diversity of operations within UAS itself. Some, such as “Using assertiveness to improve communication in the workplace”, are bite-sized training sessions, whilst others such as “X5 Project: an update”, aim to communicate about current activities and future plans.

I made it along to 5 sessions, interspersed with lots of ad hoc discussions as I bumped into colleagues from all parts of the University.

Toward Coordinated Central ICT provided an update on work to bring together BSP, ICTST, and OUCS. The presentation was given by Anne Trefethen who is leading the project, and outlined key features of the project and current thinking. The project’s objectives are to create a more coordinated service for users, and to increase efficiency and effectiveness. A high-level schedule was outlined: identify opportunies 0 – 6 months; quick wins, planning, revised governance model, and initial consolidation 6 – 12 months; implementation of plans 12 – 24 months; completing implementation of new organisational structure 24 – 36 months, and we were given an indication of what the revised governance model might look like.

Understanding the JRAM and the Infrastructure Charge – second time round for me – showed how incoming funds such as student fees are divided and distributed amongst the academic divisions, departments, and colleges, and how a portion of this is then claimed back from divisions to fund central University operations. Some fine animated visuals made it easy to understand what is in fact quite a complex model (the JRAM). The workshop went on to describe how the 123 Infrastructure Charge then determines how much should be pulled back from academic departments, and on this I will attempt to provide a summary.

The 123 Infrastructure Charge works on the basis that the cost of centrally provided services are related to academic departments in one of 3 main ways. Type 1 services (e.g. divisional office) are consumed by specific divisions, so the cost is essentially a direct cost to that division. Type 2 services (e.g. student admin, estates) are shared by several departments, and the cost is distributed amongst these according to an appropriate measure of “level of use” which may vary depending on the service (e.g. student numbers, floor space, annual turnover, etc). Type 3 services do not have a usage-based link to divisions, and are subdivided into those for which the divisions can reasonably be expected to pay (e.g. VC/Registrar, PRAS), and those for which divisions cannot be expected to pay (e.g. University Parks, museums). In a slightly Douglas Adams-esque fashion, the 123 Infrastructure Charge includes a fourth category of “service to service” charge where the costs of a service which is only consumed indirectly by divisions (such as BSP) is redistributed across the directly consumed services prior to the apportionment calculation.

The Worst Website in the World made extensive – almost exclusive – use of audience participation to describe the most annoying web site we could cumulatively imagine. By suggesting how to achieve the opposite of each annoyance we then discovered how to create something which isn’t the worst website in the world. The notes that we typed up will be published soon on a …website… near you!

What’s Religion and Belief got to do with me? This workshop, offered by the Equality and Diversity Unit, raised awareness of legislation changes introduced in the Equality Act 2010, and ongoing work to explore perceptions of how well the University understands accommodates those with particular religions and belief systems. An interesting session from which I took away two points: one of the areas most commonly raised as a problem by survey respondents was college food, and the Equality and Diversity Office is an excellent source of advice and assistance.

Sharing Services: what do you think we can do? was another facilitated discussion, with the audience encouraged to think about the opportunities for greater sharing of services within the University, what could be stopped in order to release resources to do this, and how to get the ball rolling. What made this particularly interesting for me – coming from OUCS – was hearing about the opportunities for non-IT services to be shared.

The day was rounded off with a panel Q&A session. The questions were the sort of thing you might expect – “What do you see as the biggest challenges / opportunities facing us”, “How can a democratic governance ensure that every voice is heard”, … However it was two of the panels responses that I will remember.

The first was in response to a question about colleges and the university working together – the push and pull of centralisation. Tim Gardam provided one of the best expressions of the value in Oxford’s organisational (un)structure. He said that the University structure is two-dimensional – it could be viewed as a vertical structure within the divisions, cross-cut by the colleges. This is a very modern (matrix) structure, and provides great resilience because a vulnerability to current conditions in one structure can be bolstered by strength in the other. He went on to say that this also enables the Collegiate University to provide students, tutors, and researchers, with an experience that is both “big and small” – vast yet personal.

The second was from a panel member who joined the University just 7 weeks ago. Her comment was that Oxford “is like no other place”. Yes indeed, there’s no cookie-cutter for Oxford, despite what some consultants would have us believe – long live innovation!

Posted in Conferences | Comments Off on UAS Conference 2011

Using RT at OUCS

Request Tracker is an open source issue ticketing software which we use extensively in OUCS, and Sysdev are responsible for managing it. In recent weeks, I found myself discussing some of the ways we use RT with people from the RT user and developer community, and I thought it might be interesting to share some details here.

I’m not going to provide a review or detailed description of RT as such, but the basis is a database-driven application which creates and manages tickets (the basic unit which which is analagous to a bug report, user enquiry, incident, as you prefer). The primary interface for users is incoming and outgoing email, and a web interface is provided to privileged (staff) users of the system. (It also offers a self-service web interface to end-users, which we haven’t deployed.)

RT has been around for a relatively long time, and OUCS began using it relatively early, too. The first ticket still in our main RT installation dates from September 2001, not far off 10 years ago. Initially, RT was deployed as a way for the help desk to track incoming enquiries, but since then its use has grown to all sorts of other areas, with various systems teams including ourselves using it for internal incident management, and so on. RT now handles a fairly significant volume of email, critical to OUCS’s business (providing IT services to the University).

When I first started working at OUCS, the RT system was already well-established, using RT 2, which was by that point fairly long in the tooth (the first release of RT 3 was four years previously in 2003). It had already handled more then a million tickets.

As RT is implemented in Perl, and Sysdev has always been fairly comfortable with hacking Perl, various customisations were implemented to handle specific requirements. Unfortunately, those changes were not in general made in collaboration with the developer community, largely because RT had moved on so much (some “interesting” hacks were made in order to overcome some severe performance problems, for example). When we finally managed to get time to bring our installation up to date around two years ago, a fair bit of software archaeology was required to find out what changes we had, and why!

One of the pleasant side-effects of Sysdev working on RT has been the chance to work on, and improve the packaging of RT in Debian (our Linux distribution of choice for deploying services on). Since I am a registered Debian Developer, I was in a good position to join the packaging team in Debian which had suffered from lack of sustained interest. This provided an excellent excuse to get stuck into updating the RT packages to 3.8, which I did, based on preparatory work by others. Interestingly, I wasn’t the first Sysdev team member to be involved in RT packaging for Debian; inspection of the older changelogs reveals work done by one of my predecessors. Being involved in the official packaging project has also meant that I’ve become reasonably familiar with RT4, which has been out for a few months now, even though we’re a little way off deploying that at OUCS.

Once we started to analyse how an upgrade to RT 3.8 would look at OUCS, and discussed the system with others around the department, we found that many of the tweaks added either weren’t important, or had been superceded by other changes in RT, but there were still quite a few customisations required. Since the migration, I’ve tried hard to make sure that issues we face are tracked in the upstream bug tracking system, and that patches we write are both manageable (so that they don’t present such an obstacle to future upgrades) and where possible, general enough to be fed back upstream. Inevitably this doesn’t always happen, of course — at the time of writing, we have 13 patches to the core (on top of 18 applied by Debian). These cover trivial things like adding extra convenience links to ticket display pages, to bugfixes which only arise because of some of the “interesting” ways we use RT, to a patch set originally taken from the “QueueDeactivatedScrips” extension.

One example of the “interesting” ways we use RT is in our handling of bounces; RT3, unlike RT2, doesn’t by default support routing bounce messages relating to tickets back into RT, but because of the range of internal users of the system, we needed to be able to continue to support this functionality. What we’ve ended up with is an exim filter rule, together with a scrip based on this one from the RT wiki (another very useful community resource) with some additions which email our staff users to let them know about the incoming bounce, and with a patch to ensure that bounces get routed to the right place.

It’s worth mentioning that RT supports quite a powerful overlays system which allows for plugins to cleanly implement a certain amount of new functionality without having to patch the core codebase. Some of our work is done this way, but a few more changes have been made directly, as discussed above. Nowadays we have much better ways for managing patchsets on top of official Debian packages, which makes this a less worrying prospect than it was in our RT2 days.

Our migration project also involved a fair bit of work to get the rt2tort3 scripts up and running with RT3.8 (I suspect we were one of the first to do so — after all, most people hopefully got round to the migration long ago!). That work is available as a forked github project.

The nice thing about working on RT at OUCS is that it’s an important tool used by a significant proportion of OUCS staff for their daily work, all day every day. This results in a large number of detailed change requests, which we have been diligently collating and assessing for implementation. Due to other workloads we haven’t been as responsive to those requests as we’d like, but a team internal ‘hackfest’ is planned for next year in order to hopefully address some of those (and probably also to move to RT4).

Another nice thing about RT is working with a great bunch of individuals both at Best Practical, who are the core maintainers of RT, and the external user and developer communities, both on IRC, mailing lists and wikis. This has made it especially rewarding working on the official Debian packages, as well as deploying RT here at OUCS. It’s an excellent example of Free/Libre Software working for all parties, with the users contributing valuable code back to the project, and I’m pleased to be able to participate in the free exchange of ideas and code, with the support of the OUCS OSS policy, which makes releasing code under Free/Open licences fairly trouble-free.

Coincidentally, I was able to meet up with the Best Practical team in Cambridge, MA, USA last year whilst attending the MIT Kerberos Conference; those sort of face-to-face meetings with collaborators in free software are an excellent way of getting to know the people at the other end of an IRC client or mailing list.

Crucially, they recommended a bar serving excellent beer.

Posted in Uncategorized | Comments Off on Using RT at OUCS

Project MADDOX: Oxford Active Directory

OUCS has initiated a project to investigate a central Active Directory for Oxford University. This is intended to bring enhanced support for integration of Microsoft AD based environments with central identity and access management services. At a very basic level it is hoped that this will offer a Microsoft-supported mechanism for providing single sign-on to systems running a wide variety of Microsoft products.

The ability to offer initial sign-on to Microsoft workstations (through the Welcome screen / GINA) and seamless subsequent access to domain-based resources has been available for several years (see https://wiki.oucs.ox.ac.uk/itss/KerberosADTrust). Active Directory uses the Kerberos (version 5) protocol for user authentication  – this has been the case since Active Directory was launched in Windows 2000 (see, e.g. http://technet.microsoft.com/en-us/library/bb742516.aspx), and Microsoft recommends Kerberos rather than NTLM (http://msdn.microsoft.com/en-us/library/aa378749%28VS.85%29.aspx).

Microsoft’s Kerberos implementation makes use of some optional features – worthy of note is the inclusion of a Privilege Account Certificate (PAC) in the service ticket to convey information about SIDs and group memberships – but is still fully compatible with other popular Kerberos implementations such as MIT-Kerberos. As such, it is relatively straight forward to configure a “cross-realm trust” so that users can login using their credentials in the Oxford Kerberos realm (OX.AC.UK – used for Oxford SSO authentication) to login to an Active Directory domain. There are currently around 35 Active Directory domains registered for cross-realm trust.

Technically then, a working solution is already in place – applications that are designed to run in an Active Directory environment appear to work smoothly in this configuration and the user gets the single sign-on experience that they want. There are, however two issues that crop up:

  • There is very little up-to-date information about the bigger picture in which these trust relationships are implemented – an end-to-end analysis needs to look at server roles & applications, client systems, and the federated nature of IT service provision at Oxford.
  • Although Microsoft officially supports Kerberos trust relationships between two Microsoft Active Directory domains, they are far less clear about their position regarding trust relationships with other Kerberos implementations. There is a risk that the cross-realm trust could stop working following a hotfix, service pack, or platform upgrade.

Project MADDOX is a two-stage project designed to address both of these issues.

The first stage of the project involves building numerous environments incorporating Active Directory, server, and client components in differing configurations to find out what does, and does not, work. Tests will compare the results of Active Directory to MIT-Kerberos cross-realm trusts against Active Directory to Active Directory trusts (which are fully supported by Microsoft). It is expected that this stage will be completed towards the end of August 2011.

Following this, it is expected that a top-level Active Directory “authentication” domain will be created to support user authentication to other Active Directory domains deployed around the University (see http://technet.microsoft.com/en-us/library/cc757352%28WS.10%29.aspx for an outline of domain trust models). This will form an addition to the suite of identity and access management services offered by OUCS, offered initially as a pilot to establish demand, expected to be available towards the middle of Michaelmas Term 2011.

Posted in Uncategorized | Comments Off on Project MADDOX: Oxford Active Directory

Service Level Descriptions Updated

Service Level Descriptions for sysdev-owned services have been updated on the OUCS web site. This is an annual activity, typically falling in May/June, when SLDs for all OUCS services are reviewed, updated, and published. It is an opportunity for us to ensure that any changes to the service level offered are reflected in the SLD so that users and potential users of the service can assess its suitability for any given purpose.

Summary of Changes to Existing SLDs

Oak LDAP · Webauth · Shibboleth · OxFile · Web Publishing

This year we have focussed on making the SLDs more useful to end-users. With this aim we have improved several of the introductions/overviews to clearly state the purpose (supported business process) that the service is designed for, and to clarify the user support arrangements. The introductions typically include a link to a more thorough description of the service, and identifies (in our case) sysdev as the service owner.

Most of our SLDs now sport a clearer statement of the “Hours of Operation” to make it clear when the service is intended to be available for use, when technical support is available to deal with faults, and the usual schedule for any maintenance work.

Information about major outages is now contained in a “Disaster Recovery” section which provides two key details: (1) the state in which the service will be once it has been recovered (i.e. what data will have been restored), and (2) how long this might take. We hope that these will prove more useful than some dry technical details about what RAID levels and hardware models have been used to deliver this.

New SLDs

Maillist · News · Mirror · GNU/Linux Computing

Service Levels Descriptions have been written for a number of services that have actually been available from OUCS for quite some time, but until now have not been decorated with the appropriate documentation. In some cases this was because the facility had not actually be recognised as a service, in others it was a simply case of priorities leaving this work far down the list of to-dos.

Service Descriptions

We have also started to provide a more complete description of each service. The Service Description and Service Level Description documents are complementary, with the first providing information about the purpose of the service and what it offers, while the second  details the warranty under which the service is offered so that you can assess whether it is fit to use.

Further Improvements

Our work here is not yet complete. During the updates of the SLDs this year we identified further areas for improvement, and there are still some inconsistencies that we are yet to iron out. Although we have written service descriptions for several of our more popular services, some are still missing. Perhaps the largest change is that we are planning to present the suite of services that support identity and access management as just that – a suite of IAM Services – rather than a piecemeal listing of the various technologies on which these sit.

Posted in Uncategorized | Comments Off on Service Level Descriptions Updated