Broken teams can’t react quickly.
June 5, 2007
As I jump between different organizations on consulting engagements I find lots of different dynamics which effect how well employees are able to perform their various functions. It is certainly an educational experience and over the years I’ve started to identify some common forces which leads individuals to become ineffective.
I blogged recently about our internal DevCentre but in reality that is probably an unusual scenario that isn’t repeated often in other organizations but I wanted to explore the impact of something as simple as the political forces between teams can have of productivity, and therefore costs.
Recently I came across a situation in one company where the development team was being hampered because they couldn’t set up a piece of critical infrastructure in a timely fashion.
The infrastructure component really needed to be managed by the development team because the infrastructure teams within the organizations didn’t have the skills to do it themselves and probably wouldn’t have been able to respond to configuration change requests in a timely enough fashion – and when they were executed, it would need to be a developer that did it anyway because of the nature of the beast.
Unfortunately for the organization in question this simple requirement causes a fair bit of consternation simply because it isn’t normal practice from an IT management point of view to give users the keys to the kingdom.
But what is this about “users”? Well if you are a developer reading this blog then you will doubtless know that there is a massive divide in IT between the “systems guys” and the “code guys”. To over simplify the reasoning the “systems guys” are responsible for making sure that the environment is stable and generally available for all standard business users and the “code guys” are responsible for creating new stuff and changing existing stuff – two goals that are seemingly at odds with each other.
This creates a natural political division that has been reinforced in most organizations by having two branches of IT management, one for projects (often software development related, but not always) and another for operations.
I’m not sure what the solution to this gridlock is but I suspect that developers need to start being realized as “non-users” in the say way that the “systems guys” are and given a level of access and systems management responsibility that a user wouldn’t normally get.
I’m not suggesting that developers should have administrative rights across the domain (although I have seen this work to varying degrees of success depending on the quality of team members), but they do need to be given the management responsibility for the servers that they use day to day, even to the point where they should be able to put on a custom server build if they want to.
In the scenario that I am thinking of this would allow the development team to provision a system that was instantly more useful to them and allow a very costly resource to be more productive. The fix for the broken team is realize developers and systems guys are fighting for the same cause and ultimately developers share some of the operational skills that system administrators have.
June 6, 2007 at 10:13 am
The larger the organisation the bigger the divide! When our team was small we built the boxes, now we have to fit in with another group who now has this responsibility.
We have recently commenced using virtual environments for development and I’ve almost convinced the infrastructure guys that they can safely let the developers manage one of those VMs. How much damage can we do if they can undo the damage just by copying a vhd?
June 6, 2007 at 2:37 pm
[...] just wrote a blog entry on software teams that can’t react quickly. In the scenario that I am thinking of this would allow the development team to provision a [...]
June 7, 2007 at 11:19 am
Hi Mitch,
I posted a reply here:
http://www.adopenstatic.com/cs/blogs/ken/archive/2007/06/05/6428.aspx
but I’m not sure how to make trackbacks appear on your blog (sorry for being such a bloggin n00b)
Cheers
Ken
September 18, 2007 at 5:42 pm
As a ’system guy’, I am find your opinion to be very short sighted. While dev guys, need access to dev and test their products, they do not need access to production systems. Sure you can have a good system guy ths is also a developer but it is my experience that many dev guys now little about the underlying systems on which they develop. As well, with the rapid change of technology, many do not completely understand their dev platforms before they are off trying to master a new one.
As a system guy, developers work on single web applications but systems guys are responsbile for 10’s/100’s on the same site. If the dev guy does not the inter action between their applications and other applications — we can have a system failure and who will get blamed? The system guy.
Dev guys never admit to any mistakes, because of their God complex.
Systems guys usually admit their mistakes because they understand technology.
September 18, 2007 at 5:44 pm
Sorry for all the typo’s, I hit submit before I meant to. Please accept my regrets. Paul
September 18, 2007 at 9:13 pm
Hi Paul,
I guess the point that I am making is that developer != standard user (if there is such a thing anymore). Ultimately developers will do what they have to do to do their jobs, their job is to create and ship software. They will put up with artificial barriers for a while in the interests of “team spirit”, but their #1 motivator is shipping that code.
So whilst all the arguments about why developers shouldn’t touch production systems are actually irrelevant, because if they need to (in their eyes) to get the job done – they will.
September 20, 2008 at 11:44 am
[...] Mitch talks about the problems encountered when SysAdmins struggle to provision the infrastructure necessary to support developers – and I entirely see his point-of-view.In my opinion, it is rare for an organisation that has SysAdmins that are already skilled up on the issues required to deploy new technologies when the development teams are cranking out brand new products. Most SysAdmins are busy keeping the existing infrastructure working properly and trying to optimise it – not out learning new technologies that aren't being used yet. That's not necessarily a fault of the IT staff (though I think it would be much better if more IT staff showed initiative and self-drive) but sometimes the culture of the organisation.on the other hand, most developers are woefully equipped in terms of understanding the implications of what they are doing. The last thing an enterprise needs is rogue DHCP servers being setup that network build/provisioning systems, or rogue IIS boxes that can be compromised (NIMDA/Code Red anyone), or Exchange servers that become open relays. I've seen first hand, developers struggle for weeks to get Exchange, MOSS, DCs etc all functioning well together. And a typical solution to solving a problem is to open access to everyone, not follow best practices. In general developers seem to care about getting the thing working to spec – not worry to much about the side effects!So how do we address the issue? In two ways I think.1. Multiple EnvironmentsFirstly we can create various environments within the organisation. The less impact that the environment has on production, the more control can be handed to developers and end users. In the most extreme case, you can have a completely segmented development environment, where developers can install whatever the want, configure it exactly how they want, and reconfigure it without needing to tell anyone else.As we move closer to larger impacts on the production network, we need to increase the level of change control measures and the involvement of other parts of the business (namely the sys admins, the network guys, the security team and so on). And when we finally come to deploy the application into production, we have the full set of change control procedures (and division of responsibility) that applies to everyone else (it's not just developers that have to jump through hoops to get Active Directory changed – all the rest of the IT staff have to as well!). This change control process means that all relevant stakeholders know what is happening in the production environment, and that the production environment moves from one known state, to another known state.2. Better Project Management (which is a prerequisite for agile IT)good project managers know when to engage other parts of the IT organisation. This allows SysAdmins to research the product and its optimal configuration, work how how to provision the product, and even get a delegation tool in place to allow developers to manage select parts of the infrastructure. The project plan should look something like:Unfortunately, in a lot of cases, it seems to look like this:And that type of poor project planning results in low agility IT. Filed under: Other [...]