I have been thinking a lot about Continuous Integration, DevOps and related topics where we are building more and more tools to help take the variability out of building systems. By building systems I am talking about the phase of the SDLC where we begin begin thinking about writing the actual code and the time we deploy that code to production. When I think of the lifecycle of an application this loop stands out to me as one that gets executed many, many (!!) times. So it makes sense to 1. have this as efficient as possible and 2. increase the accuracy as much as possible.
We are doing this is couple different ways which I am not sure I can go into much detail around because of company policies. But suffice it to say that we use a third party build/test tool for our code that checks everything out of source control, modifies configuration, builds, runs unit tests and deploys the code (and documentation).
This year we have a new requirement that has me rubbing my chin quite a bit. The requirement from the auditors is that (according to SOX) those people in the development role cannot have write access to production bits and those in the deployment role cannot have access to the development bits. When I say bits I am talking at the run time/deployabe level (not the source code). The rationale behind this (so I am told) is that this prevents anyone from introducing changes into the process where code is promoted from lesser test environments to production. Reserving any commentary about how I feel about this policy it is something that we are being required to do. And given the sensitivity regulators have around the investment management industry and the enterprise approach the parent company takes - it doesn't really matter how I feel.
My anti-strategy is not use humans separation to implement this. Having a team just to press a button that I asked them to press seems like a waste. Sure nothing should go wrong if we are doing this right, but something WILL go wrong. Can you just imagine the conversation between the deployer and the developer when this happens....
Deployer: "The package failed"
Developer: "What was the error"
Deployer: "Some really big negative number"
Developer: "Can you send me the logs?"
Deployer: "Where are they?"
Developer: "Try this new package"
Deployer: "I don't see approval from your manager for this"
Developer: "It is 2am my manager is sleeping"
The other issue I see with the human solution to this is that of production support. When in a time critical break/fix scenario - the last thing you want to have to do is get through a process that does not understand your systems, your business and therefore the context in which they are performing. Sure you have separated the roles, but at what cost?
My strategy is that by leveraging our continuous integration tool I should be able to accomplish much of this since the deployer is a system/application itself and as long as I can show accountability then we should be all set. I will admit I am getting some initial resistance to this; but I hoping through a partnership with the auditors we can figure out a reasonable way to do this.
One of the interesting topics around this new requirement is the DBAs - since they inherently break the developer/deployer separation by having access to everything in every environment. The database seems to me like the perfect place to be doing something that is "not on the up and up". Interestingly, the DBAs are considered out of scope for requirement. Biting my tongue. Is this implying that developers are inherently less trust worthy than a DBA. Or is that no DBA is savvy enough to possibly change a sproc (aka code) to do something devious. Maybe is it that the DBA lobby is that much better than the developer lobby.
No comments:
Post a Comment