We have been using a three-staged system and have multiple administrators making configuration changes concurrently on the DEV server and promoting to TEST server for customer testing and finally to PROD server for production use. The system does not tie to specific issues reported by customers so we cannot promote changes on the basis of specific customer requests. There is no notion of a set of configuration changes related to a specific issue/request.
As a result, we frequently run into cases where we want to promote changes for a specific customer request but there are changes in the same system artifact (a TYPE, say) that correspond to multiple customer requests. In that case, we cannot promote one change without also promoting the other. In a large environment with many users and changes going on continually, this leads to sometimes chaotic behavior.
What kind of things are being done by you all out there in the community to help minimize risks associated with this kind of environment?
We have all user issues/requests logged on our production server as a particular item type. The workflow for that item type handles approvals, design, testing and promotion to the production server.
On that item type, we've added a multiselect pick field with all the item types in the system as well as a longtext field to mention the exact details. Administrators fill out those fields while working on the change for the specific item.
Before we promise a deploy date for a particular change, we look to those impact fields to analyze which changes are in flight and to determine if multiple changes need to be deployed simultaneously because they affect the same object.
Even though it is a manual process to both document and analyze the changes, we've found its better than just promoting from memory or based on the description of the change.
We've also found that by grouping the related changes up front (for example bundling several changes to a specific Type) then having those changes tested and cleared to deployed to production as a bundle, we end up having fewer collisions when it comes time to actually do the migration.
Of course sometimes critical things happen and in those cases, we back out the non critical changes to the object, deploy to production, then add the non-critical changes back in. Documenting the changes as we go is definitely helpful here, otherwise it ends up being poring through the object's history and hoping to not forget anything.
PS, if there is any RFC for tying admin changes to an item along the same lines of how change packages tie to an item, and then migrating from staging to production along those lines, please add me to the RFC.
There is actually an Request for Change for this. I strongly believe neither of you are on it.
Please contact Integrity Support at your convenience to add yourselves to this RFC.
Thanks for posting this, Charles.
This sounds very similar to what another customer of ours has done. I made a new blog post highlighting some of the high-level points and RFCs which may be of interest to anyone who wants better auditing and control of their admin migration process.