In-Depth
Gaining Control Through Enterprise Process
How new policies and procedures can help you gain control over processes, ranging from patch management to topology changes.
- By Greg Shields
- 07/01/2004
“Can you get another multi-site Exchange organization built … by tomorrow …?”
“Rebooting the file server will fix that problem. I know there are
users on it, but no one will know I did it.”
“I know no one else is using Visual Basic .NET, but I really want
the new features. Can’t you install it for me?”
Life as an IT person is filled with these little temptations from users,
managers and ourselves. They come with the best of intentions, and sometimes
we make inappropriate changes just to get people off our backs. It’s not
until months down the road that we find the accumulation of all these
undocumented changes creates a major headache when something breaks.
When
Microsoft Certified Professional Magazine named me their
Editor for a Day, they asked, “Bring us the issues from down in the trenches.
Give us a topic our users need to hear about.” So, rather than focus on
the next best technology from Microsoft or next new server platform, I
thought it best to step back and take a sweeping view of the process and
procedures that govern how we do our daily jobs.
Working in the IT department for Raytheon Company in Aurora, Colorado
for nearly five years, I’ve experienced some great successes. I have also
been a part of some spectacular failures in technology deployment driven
from inappropriate schedules, lack of management buy-in, or poorly written
documentation.
Throughout these five years I’ve played a minor part in redesigning our
core IT processes, taking us from an environment of constant firefighting
and uncoordinated changes to one where systems are maintained to a known
state, and where nothing changes without group agreement.
Why Enterprise Process?
As a general concept, “enterprise process” is difficult to envision; but
the goals it aims to achieve are easy to understand when related to the
tasks that make up your work day. Think for a minute about what you do:
- When you bring a new system online, are the steps to get it up repeatable?
- Is the knowledge required to fix today’s problem easily transferable
to new teammates?
- Are you comfortable that today’s changes to your environment won’t
affect tomorrow’s uptime?
From a management perspective, the establishment of enterprise processes
ensures that expensive technology lessons are learned only once and that
each employee’s daily tasks are completed in a consistent, repeatable
fashion from which others can learn.
From the perspective of the IT trenches, establishment of common processes
agreed upon by both management and IT ensures that our environment, servers,
workstations and applications won’t change from a baselined state without
our consent. It also protects us from an often incessant stream of inappropriate
requests brought about by over-zealous management or unreasonable time
schedules.
Sounds like a great idea, huh? How do you start? The first step is to
baseline your environment.
Bring It to Baseline
Not long ago Microsoft released Internet Explorer patch MS03-048 with
four different versions. This patch used a different installation file
depending on whether the system was running IE Version 5, Version 5.5,
Version 6 Service Pack 1, or Version 6 on Windows Server 2003.
| Keeping
Track of “Builds” |
|
In my program area at Raytheon, we started with the
concepts of server and workstation baselining firmly
ingrained into our user’s minds. Office users,
administrative assistants and management types are given
our “standard build.” This build includes
Microsoft Office, Adobe Acrobat, Lotus Notes and a few
other miscellaneous tools needed by everyone in the
program.
As we’re primarily a software development shop,
our developers have a different build. Our “Dev
Build” is a superset of the “Standard Build”
with additional tools for software developers.
All software that makes up our servers—our standard
build, dev build and additional software installations—is
stored in our software tracking database in Microsoft
Access. We use this database to report software licensing
metrics to management and locate machines where patches
or software updates might have conflicts prior to deployment.
|
|
|
At that time, the Raytheon network in Aurora encompassed more than 1,200
desktops and dozens of servers and laptops, yet large sections of the
network hadn’t agreed to undergo a baseline control process, so the installation
of this critical patch took more than a month to complete, as the IT department
scrambled to identify which machines needed which patch. Many of the workstations
were patched manually by techs in the field rather than through an automated
process.
Raytheon Aurora is made up of a number of separate software development
programs. Fortunately, my program agreed early on to complete a computer
baselining activity where nearly 400 workstation and server configurations
were captured into a central database. All workstations are imaged with
standard software builds, and any individual changes to that build are
captured to the database.
Because of our baseline, it took just 48 hours to patch more than 95
percent of our systems. Only one of the four patch variants needed to
be deployed to our computers, and zero software conflicts were experienced.
 |
| Figure 1. Enterprise Process Decision-Making
Flowchart. |
Start with the Servers
Bringing your computers to baseline sounds like an ominous task. But the
process of baselining your environment needn’t occur overnight. Since
there are fewer of them, start with your servers. Documenting every detail
of their hardware configuration and installed software can involve little
more than Microsoft Word and a screen capture utility.
If you’re running Windows 2000 Server or Windows Server 2003 and your
company security policy allows, use Terminal Services to connect to your
servers and take screen shots of every system configuration screen. By
pasting these screen shots into a Word document, even the most complicated
configurations need take a only a few hours to complete.
Compile your screen shots along with other pertinent information like
hardware composition, installed software and patches, and software installation
location into a central file location on your network—this is your “baseline.”
For larger environments, tools such as Systems Management Server automate
the process of ascertaining the software and hardware spec for each piece
of server equipment in your environment.
Next, the Workstations
At Raytheon Aurora, we’ve created our own “build tank.” To this location,
new workstations arriving from HP as well as previously used ones needing
re-imaging are delivered. Once there, each computer’s software configuration
is wiped clean and one of two “standard images” are squirted onto the
system using Symantec Ghost.
Each and every computer leaves our build tank with a known configuration,
including known and tested software and drivers. Because of this baselining
of software and drivers on each system, our number of help-desk tickets
has been significantly reduced.
Once servers are completed, the process of baselining workstations can
begin. Standardize on a workstation image and use Ghost or HP Altiris
to replicate this image to your workstations in a consistent and repeatable
fashion.
If you’ve just started this process, keep “swap” machines available to
immediately deliver a fresh machine to each user as you pull their old
machine for re-imaging. Once the machine’s re-image is complete, it becomes
the swap machine for the next user. For machines with special software
or patch requirements, start a database that details how each machine’s
software build deviates from your baseline, and information about all
non-standard software products.
If your environment contains major groupings of applications, consolidate
on as few images as possible. Maintenance of as few workstation images
as possible means less work to update your “golden images” as you get
new patches or software updates.
Your build tank need not be complicated or expensive. At Raytheon Aurora,
our build tank is nothing more than a three-shelf metal rack rescued from
the recycle bin. Zip-tied to this rack are 14 individual cable drops that
connect back to a central KVM and network switch. The entire rack sits
on its own subnet with a file server that hosts our Ghost server and the
image files. The entire assembly was built for less than $3,000.
Admin Rights: Use ‘Em or Lose ‘Em
Once you’ve obtained a workstation baseline, it’s reasonable to consider
granting admin rights to your users as the end of your baseline. Notwithstanding
who sits in front of the computer all day, the ultimate owner of that
workstation is the company, and the ultimate reason for that workstation’s
existence is for the benefit of the company.
Our users used to view admin rights as a privilege granted to them along
with their medical and dental benefits. I can’t tell you how many times
I’d go to a user’s desk to find the reason for their system crash was
a recently installed MP3 player.
It’s worth saying again: The widespread use of admin rights is the death
knell of workstation stability.
To maintain stability, you’re going to need management buy-in. Involve
management in your baselining plan. Discuss in terms of dollars and cents
the value of lost company productivity and increased workstation downtime
attributed to workstation mismanagement. Management should understand
the additional cost involved when users retain admin rights they likely
don’t even need.
In our case, we obtained upper management buy-in on our process early
on. The process is as follows:
- Initially, no one gets admin rights.
- To obtain admin rights, an employee is required to review a short
training presentation with their manager that details the responsibilities
required of a person with admin rights.
- The individual and their manager then send an e-mail to a special
mailbox agreeing to the policy and certifying that they won’t make unauthorized
changes to their workstation.
- Once we receive and file the e-mail, we grant them admin rights.
We explain in the training presentation that admin rights can and will
be revoked by IT should evidence be found of rule-breaking. Removal of
admin rights means that the user can’t perform their job function; the
employee’s made aware of the results of not being able to complete their
job.
In the three years this policy’s been in effect, we’ve had virtually
no issues requiring us to revoke a person’s admin rights.
Decision-Making by Committee: The Change Control
Board
After agreeing on a firm baseline, you next need a process to modify that
baseline. Enter the Change Control Board (CCB). In parallel with the baselining
of your environment, you should develop a plan for establishment of a
regular CCB meeting where changes to baseline configurations are discussed
and approved by a committee of peers.
At Raytheon Aurora, many Windows system administrators (SAs)—including
myself for a time—initially saw the establishment of the CCB as a slap
in the face to our own capabilities as talented SAs. However, as I saw—and
you’ll see—there are real benefits in having a “support group” that shares
the responsibility for each change to your environment.
Because of the size and complexity of the IT department at my company,
the process of establishing the CCB, defining the rules, determining what
should and shouldn’t be brought to the CCB, and who should be in attendance
was a multi-year process. It involved a lot of arguing and nearly resulted
in the loss of some very talented SAs.
Most of the time involved with setting up our CCB was spent dealing with
political issues and calming the nerves of SAs who were used to making
independent changes to the environment, often without anyone else’s knowledge.
Once our group realized how useful the CCB could be in ensuring that
every change was agreed upon through group consensus, and how our jobs
were somewhat protected as part of that group, we grew to appreciate its
usefulness.
Your CCB can meet as often as necessary, but the meeting times should
be regularly scheduled and rarely, if ever, modified. The establishment
of set times for your CCB meeting ensures that individuals with political
agendas can’t modify the meeting time in order to approve a change they
haven’t clearly thought through in the first place.
Check out “CCB Rules of Engagement” for some
guidelines you can use to setup the rules for your own CCB.
| 12
Change Control Board Rules of Engagement |
|
1. The CCB should be composed of regular representatives
from your major IT groupings: technical support, systems
administrators, networks, network security, and IT management.
The CCB should also be open to other interested parties,
including those not from IT.
2. Changes should be submitted in writing, using a
standard Change Request Form sufficiently prior to the
CCB date and time that your peers can review the change
and provide comments.
3. Completed Change Request Forms should be e-mailed
to CCB participants when completed. Any interested party
within or outside the IT department should have the
ability to receive these Change Request Form notifications.
4. Issues with any particular change shouldn’t
be brought up during the CCB, but instead worked out
prior to the meeting. Minor edits or clarifications,
though, can be brought up in the meeting.
5. For changes where issues can’t be resolved
through peer communication, the issue should be elevated
to the Technical Interchange Board (discussed later).
6. The CCB isn’t intended as an approval body
for major project plans. Large projects and their accompanying
test plans, implementation plans, schedules, and user
training guides should first be approved by the Engineering
Review Board (discussed later).
7. Once the ERB has approved a major project plan,
the individual changes to your environment and their
timeframe for implementation should then be approved
by the CCB.
8. Consider inviting one non-technical member of management
to the CCB to represent their concerns such as customer
impact and risk mitigation. The presence of this management
liaison ensures that CCB decisions are carried to your
company management team through the most direct route.
9. A list of minor changes not requiring CCB approval
should be agreed upon and documented prior to the first
CCB meeting.
10. Change Request Forms should be stored in electronic
form. The ongoing record of change requests will serve
as the beginning of a searchable repository of IT knowledge.
If your company uses a work order tracking system to
coordinate help desk tickets, use this system so all
your IT knowledge is consolidated in a single searchable
database.
11. Some provision for emergency changes should be
made, because even the best-laid plans go to waste in
an emergency. Your process for Change Request approval
should include some method to short-circuit the process
during crises.
12. When a crisis does occur, have a standard Incident
Report Form similar to your Change Request Form. This
form is brought to the CCB for review at the conclusion
of the crisis. The sum total of your Incident Report
documents will populate your database with a history
of lessons learned.
|
|
|
Setting up a CCB
The CCB is the committee equivalent of “the buck stops here.” No change
occurs in your computing environment without approval of your CCB.
Any change to your enterprise relevant enough to be approved by the CCB
should be documented into a standardized Change Request Form. This Change
Request Form is submitted to the CCB members through some medium (typically
e-mail) sufficiently prior to the CCB meeting time that your fellow SAs
have time to review the change and reply with comments.
In order for a Change Request to be approved by the CCB, all comments
or issues with the change must be adjudicated prior to the CCB meeting.
At the CCB, each individual Change Request Form is reviewed by the members
in attendance and approved or “rubber-stamped” by a process of acclimation.
At Raytheon Aurora, we found it easiest to automatically approve all
changes unless one of the following three conditions aren’t met:
- The CCB Chair determines that not enough pre-approval due diligence
has been completed on the part of the submitter. This could be missing
documentation, lack of communication with fellow SAs, the network team,
company management, or missing user documentation.
- A member of the CCB determines that the time for the change is inappropriate.
The individual wishing the change in implementation date should to provide
an appropriate alternate date and time so no Change Requests are left
permanently in limbo.
- A member of the CCB flags the Change Request as having a remaining
issue not adjudicated by the Change owner.
As any member of the CCB can flag a Change Request and delay the implementation
of the change, the onus of responsibility lies on the Change owner to
ensure that:
- The change is truly necessary, and therefore good for the business.
- They’ve researched and documented what they believe is the best possible
solution to the problem or issue at hand.
- They’ve coordinated with their peers to resolve any possible conflicts
that could come out of the change.
These three points are the key to why your organization needs a CCB.
Lacking a CCB, a company is missing a system of checks and balances on
administrators. Since your SAs typically have advanced rights and privileges
in your Windows domain, and since SAs could make changes without fully
thinking them through, the “peer pressure” of an authoritative body of
your peers ensures that changes are researched, documented, and implemented
carefully and thoroughly.
When we first setup the CCB at Raytheon Aurora, we set up some skeleton
rules but made the decision to work on the “details” later as we learned
what was appropriate and what wasn’t for Change Control.
This process of organizationally learning what was and was not pertinent
was a challenging one. For months, the individuals involved with our CCB
struggled with issues such as:
- What changes need to be brought before the board?
- In an emergency, what should I do first? Should I first work on the
documentation, get approval, or simply begin working on the emergency?
- If I can’t get consensus on a change I want to implement, where do
I turn?
The answers to these questions will be different depending on your company’s
size, tolerance of downtime, and your user culture. You should, however,
have some understanding of the impact of these issues as you begin writing
the document that defines your CCB.
Find Help and Avoid Group-Think: The Technical
Exchange Board
Setting up an Active Directory at Raytheon Aurora was—and is—a very hot
topic. With our AD implementation touching the daily workflow of nearly
everyone in our IT department, every person in that department seemed
to have an opinion on how it should be implemented.
The actual task of implementing AD was handed to an engineer in our Operations
department. The engineer had opinions on how the AD should be implemented,
as did many others. The wide range of opinions virtually ensured that
no Change Request for its implementation would ever be approved by the
CCB.
For situations like this, we have an established process where conflicting
opinions can be resolved in a timely manner using a formal structure.
That process takes place in the form of our Technical Exchange Board.
Unlike the Change Control Board, where people come together every week
to rubber-stamp agreed-upon changes to the infrastructure, the Technical
Exchange Board uses a less formal approach to resolving sticking points
among the greater IT group.
Your Technical Exchange Board should come together on an as-needed basis
and remain solely a gathering of peers for the purposes of mediation.
Its purpose is to provide a structure for open and candid conversation
between technically-minded peers, with the intention of coming to a group
consensus on a most-appropriate solution for the technical problem at
hand. The chair of the board should be responsible solely for maintaining
productivity within the group and guiding the conversation to the most
appropriate conclusion.
The Technical Interchange Board should be a tool for mediation and cross-IT
education; not an approval body. This ensures that decisions made there
are done by common group agreement in preparation for CCB or Engineering
Review Board approval.
At Raytheon Aurora, we’ve convened the Technical Exchange Board for topics
ranging from implementation of Microsoft Software Update Services using
SMS, to deployment of a new DNS structure for the company, to the aforementioned
AD implementation. In each case, everyone in the room eventually came
to an agreement on a best-possible solution—even when that solution required
repeated meetings.
Reaching Process Consensus: The Engineering Review
Board
The final component of a successful enterprise process deployment is the
establishment of an Engineering Review Board (ERB). This board, comprised
of a small group of high-level systems engineers and members of management,
is responsible for the approval of new processes and the review and subsequent
approval of large projects.
Within your process documents is where the various review boards discussed
earlier are created, where they’re given their authority, and where the
procedures that guide your workflow are defined. The ERB is responsible
for the review and approval of these process documents.
Typical membership on an ERB includes individuals with understanding
of the “big picture.” They’re the individuals who understand how the approval
of process documents will affect the performance of you, your computer
resources, and ultimately the business as a whole.
In addition to creating and approving process, these individuals are
also typically the body of authority for large projects and their associated
project schedules and documentation. These long-term projects drastically
change your computer environment and users’ experience. It’s here that
large project schedules, impacts, risks, costs, and benefits are evaluated
and approved.
At Raytheon Aurora, we’ve brought dozens of documents before our ERB
for approval. We’ve sent documents detailing the process of requesting
and obtaining domain admin rights and SMS rights, for establishing the
formal process for obtaining call metrics for our help desk, and for detailing
our formal Service Level Agreement between IT and the programs in our
company.
Each of these process documents was reviewed by the members of the ERB,
analyzed to determine its relevance to our core business, and approved
for use. The ERB and its authority in approving each of our processes
brings legitimacy to our organization.
Worth the Time and Effort
It took us a few years to get all this process structure up and functioning
the way it is today. It took just as long to get buy-in from the IT people
who found the process a hindrance to “getting work done”. However, I can
honestly say our company has benefited from the experience.
Even though our approval process has a tendency to slow down change implementation,
our records have proven that more inappropriate or inherently risky changes
have been prevented than time-critical ones have been slowed down. In
our old working environment, our management would expect us to jump to
deploy new technology that may or may not jibe with everything else on
the network. In our new working environment, all requests for new network
services require pre-implementation due diligence and an plan of action
before they’re green-lighted.
Enterprise process may be a check on your virtually unlimited power as
a domain admin, but it will save you when you need time and the peer support
to ensure your job is done correctly.