Roger Clarke's Web-Site
© Xamax Consultancy Pty Ltd, 1995-2021
|Identity Matters||Other Topics||Waltzing Matilda||What's New|
Version of 13 September 2012
Roger Clarke **
© Xamax Consultancy Pty Ltd, 2012
Available under an AEShareNet licence or a Creative Commons licence.
This document is at http://www.rogerclarke.com/II/SAH-12.html
A sysadmin was, and to some extent still is, a person who administered a complex of computing infrastructure. It emerged from the Unix-enabled explosion in mini-computer capabilities in the mid-to-late 1970s, peaked 1980 to 2005, and has been in decline as outsourcing of enterprise platforms took hold, and organisations ceded control of their information infrastructure and put themselves at the mercy of service-providers.
During its lifetime, the occupation matured and professionalised, as indicated by the SAGE-AU Body of Knowledge. However, it also splintered, with continual change in the focus of technology - from minis to micros, from star networks to LANs and WANs and VPNs, from terminals to workstations to PCs to handhelds, from uniprocessors to multi-processor configs and back to uniprocessors and back to multi-processing (now 'virtualised') devices. Throughout, the brands and flavours multiplied and diversified.
All trades and professions have a life-cycle, reflecting the long-term waves of overlapping technologies. As archery was replaced by blunderbusses, bowyers and fletchers gave way to armourers and powder-monkeys. As horses yielded right of way to motorised transport, farriers gave way to, well, eventually, tyre-shops. But whereas once blacksmiths begat blacksmiths who begat blacksmiths, the length of a profession's life-cycle has been reduced from generations down to less than a working-life.
The decline of the sysadmin has been hastened by reduction of the magic of a guild-member, via the lustre of a profession, and the respectability of a trade, all the way down to the blind following of structured clerical recipes. Even more depressing is the way that IT costs have spiralled, as the savings that arise from paying low wages for dumbed-down jobs have been eaten up by the need for vastly greater numbers of people who see little of the whole, and understand less than they see. It's distressing to see no genuine 'full-scope' and 'whole of life' costing, and instead mindless acceptance of marketing spin.
These reflections on the history of sysadmins were stimulated by a thread on on the link list on 13 September 2012.
Date: Thu, 13 Sep 2012 12:53:00 +1000 (EST)
From: firstname.lastname@example.org (Rachel Polanskis)
To: Marghanita da Cruz <email@example.com>
Subject: Re: [LINK] Roles and Responsibilities of System Administrators
I have been a Sysadmin for <some_uni> since about 1995. I have noticed that the job has changed a lot, at least for me. The most noticable things about the change for me can be lumped into 3 or 4 categories:
All of the above take the (UNIX) Systems Administrator out of the manifold processes we used to be responsible for. I can address each one of these and explain the impact. I do not know how relevant these are for other organisations, but this is the path I have been through in recent years/months.
This sounds like a rant, but it is not. It is however quite soul destroying and depressing to see this in action, up close. Originally, what I would call "Layer 4" services, such as FTP, Email, HTTP, LDAP and other similar services were traditionally hosted on UNIX systems and administered by a UNIX admin.
It doesn't matter whether it is Linux or a commercial UNIX platform.
Email services were part of a UNIX sysadmin's role and in my case, keeping the mail flowing was about 10% of my daily time and it was a simple process and even when there were outages it was not fraught with too many difficulties that could not be overcome by more resilient storage, bigger, faster processors.
Today, in my case, email is run on a cluster of MS systems, spread across terabytes of redundant disks and admin'd full time by two dedicated email administrators. Where once any administrivia was managed by the helpdesk these admins now deal with the work as it has become too complex for the helpdesk to resolve. It can be argued that the new modern portable devices make the move to MS Exchange more practical, there seemed to be a complete ignorance to the fact that the UNIX people could supply a similar solution much cheaper, with most of the rough edges worn off and it would still only be a small percentage of our time. But this introduction to MS was to be the wedge of much bigger slices into our work. Along with Exchange,you need Calendars so that comes into the fold, although we had a Calendar solution, it did not work well Blackberries in the early 00's so MS got its foot in the door via that. So once you have Calendars and Email, the trick is to subsume Directories. So along comes a Calendar/Sharepoint Admin, plus 2 AD Admins. Now you have an MS infrustructre on 12+ disparate boxes, physical and virtual and 5 dedicated admins.
We can't fight this, so a big chunk of UNIX administration is offloaded onto these "dedicated" administrators with all the costs and factors that go with that. A commensurate amount of knowledge of "Sendmail" internals and deep Directory knowledge is suddenly reduced to "trivia".
This is great. I love Lights Out. It means I do not have to visit the noisy cold datacentre and can do 90% of all sysadmin functions that do not require a physical presence without leaving my desk. I was a big fan of this and got on board right away. But little did I comprehend how this would further reduce my actual time doing "Systems Administration". In the past,I was appreciated for and utilised constantly, for my ability to diagnose a fault, raise a solution and repair it, all by myself. But the rationale now is, if we do not have to attend the DataCentre, why should we? Those sysadmins are wasted if they have to be in the DC pulling a CPU module or fitting a replacement disk. So, we get a contract with the vendor to do "on site support". Suddenly, another component of my job is taken away.
Skills I studied and trained for are all of a sudden redundant and an element of something I actually enjoyed is out the door. The need to attend the DC is reduced to the point where the managers and bean counters start thinking in terms of "why do we even have a DC?", "Why can't we virtualise the platforms and lease the hardware from a vendor who will do all the support?". 50% of a UNIX admin's job description - gone!
Once something is standardised, it can be virtualised. Once it is virtualised, you can do that to the whole DataCentre. UNIX admins begone! We don't need your intense knowledge of SCSI reflections and how to replace the DIMM's in a blade chassis. We'll let the vendor deal with that.....
I am aware of two methodologies in use where I work. One is something called "ITIL" which I think stands for "IT Is Losing" or "Idiots Training Idiots Legitimately" but I think it actually means "Process comes before Productivity". The second one I am aware of is something called PRINCE2, or "The Methodology formerly known as PRINCE, but didn't work right".
Both these methodologies are supposed "enhance the dialogue between IT and the stakeholders", or "standardised the processes used to conduct business within the IT unit". To me, they are unmitigated Productivity Killers. Having to go through a CAB for 4 months while a File System endlessly corrupts itself when it could have been a 15 minute fix.
Waiting 6 months to confirm the installation of Security Patches onto a customer facing vulnerable system". Having an urgent Change Request rejected because the "Subject Matter Expert" was not the one raising the change. Conducting an "Emergency Change Meeting" instead of getting the problem fixed... And so on. Spending money meanwhile, on consultants and training in these methodologies, while missing out on improving Technical staff through getting them trained on actual technical stuff and not hiring technical people to assist when there is a surge in a particular area.
These methodologies do nothing to assist the sysadmin in their job. While I understand the requirement for change managment processes, so you do not set fire to your databases, they have become the job itself. Paralysed by the inability to perform actual maintenance, these methodologies often lead to systems in entropy while the methodology states "if it is running do not change it". A much bigger, more panicked disaster later is the result.
I have worked in an environment where nothing was documented and everything was mapped in the single most important person's in the building (the sysadmin's) head. This was great if he relinquished that information to others, but he was difficult.
So, this is not a good place to be. But, conversely this person was capable of so much. The email, directories, the web, the network security, the provisioning of hardware/OS/apps. This guy can do it all. Other UNIX admins want to be him, some just want to have his job. It is on the brains and typing skills of admins like this that so many IT depts have been built. The sysadmin was respected, well paid and if not too overworked loved what they did. Today, I see my own role reduced to a series of mouse-clicks, filling in stuff for a Consultant so they can do what I was once tasked for. Handing over work that I used to enjoy, that I had possibly carved a niche for myself doing. Creating bespoke applications to merge/demerge print queues, or creating web farms or compiling GNU apps where required. All of it requires experience and knowledge that takes years to learn properly. All of it now useless as individual consultants come in and do a bit of this part of the job, hire another to do that bit....
Apparently all that is too hard for us now, as it is moved into the realms of Buy Not Build methodologies, that ignore the in house skills of the admins, reduced to just watching on the sidelines where once we would attend a meeting to provide input and even offer a solution, we now only find out at the last minute that something is implemented (in some cases we could have simply replicate the thing ourselves) and our role is to give the consultant an access to do the thing, or to attend "training" on what mouse clicks need to be done in what order.
In short, depending on what organisation or business you are in, I think the role of systems admin is changing. In my case, we are being corporatised and Change Managed so that while all the KPI's look good for the Project Managers, we need to constantly find a way to justify ourselves, as we are overlooked and considered less than godly as we once were.
Meanwhile, deskilling increases as more tasks are taken away and "standardised" with no reference to how things are actually done. Outsourcing and Buy Not Build returns the sysadmin to being dumbed down. The solutions I am told are something like "Suck it up" or "Get another job".
All of this means when they go through the hire process for sysadmins of the future, their skills will not mean as much, their role in the organisation becomes greatly diminished and the dumbing down and denerdification will eventually mean organisations that do things this way will ultimately become inflexible to urgent changes in technological directions, incapable of self-innovation in the face of requirements and a capacity to ignore the brilliant (sometimes flawed) ability of their admins to provide solutions that are elegant, thrifty and useful.
I would like to be a Sysadmin who isn't just playing clicky-clicky with the mouse. Cripes, even my Sun Workstation has been replaced by a Mac (I am not really complaining about that...). I would love to be asked to be technical lead on some project, or to come up with a solution to a problem that involved coding or doing something useful with my brain.
But I think it is too late. We have a PM for that, or Consultants.
I have no control over my career anymore. I thought I could do this for the rest of my working life, but I think the role of the sysadmin is not as respected or as relied upon as it once was. Like the old lawn mower that used to do a great job as it was pushed around, we are just old smelly dinosaurs in the garage of modern IT....
From: Darrell Burkey <firstname.lastname@example.org>
Cc: Link list <email@example.com>
Subject: Re: [LINK] Roles and Responsibilities of System Administrators
Thanks for taking the time to write this very useful post. I have just left the ANU, where I was hired as a Unix Sys Admin, for the very reasons you have outlined. There appears to be little to no understanding of what Sys Admin provides and how much money it can save an organisation.
Instead they buy it from outside and hire cheaper Windows techos to click icons. I thought that a Uni would be the last place that such a thing would happen but once Microsoft fooled everyone in to thinking you had to have Active Directory they started putting in multiples of Windows servers to replace what one Unix box was doing much better. And you get to pay for that.
In the case of the ANU, it's especially difficult to watch given it's the birthplace of SAMBA. Not to mention SAMBA 4 can do what AD does better than AD does. But if it's not Windows, they don't care. Ignorance is bliss I guess.
Don't forget to look at the great Sys Admin books by Tom Limoncelli http://www.amazon.com/Practice-System-Network-Administration-Edition/dp/0321492668.
Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in the Cyberspace Law & Policy Centre at the University of N.S.W., and a Visiting Professor in the Research School of Computer Science at the Australian National University. Among other things, he's researched and published on various aspects of the history of IT.
He was a beneficiary of sysadmins / wizards throughout his career 1970-2020, but was himself only ever very modestly capable of using command-line environments, and was not competent to make even one machine balance on its ear and then stand it up straight again, let alone juggle many at once.
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax.
From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 65 million in early 2021.
Sponsored by the Gallery, Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer
Xamax Consultancy Pty Ltd
ACN: 002 360 456
78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Tel: +61 2 6288 6916
Created: 13 September 2012 - Last Amended: 13 September 2012 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/II/SAH-12.html