Years of observing hundreds of network managers has revealed 7 habits that differentiate the effective network manager from the others. In this white paper, we will identify those 7 habits and discuss how these habits contribute to providing a well performing and reliable network. When problems do occur, these problems can be resolved in a timely manner, reducing the impact on the business, while at the same time, maintaining the credibility of the networking department.
- Validation Testing
- Pre-deployment Testing
The highly effective network manager has found that in order to provide a reliable network to the business, a number of things need to be in place. Without these, applications will run slow, intermittent problems will plague the user community, and the networking engineering staff will spend much of their time chasing problems and not working on new projects.
Providing the staff with the training, tools, and a common approach ensures they have the resources necessary to attack problems as they occur. Pre-deployment testing and validation guarantees that the network and applications will work as intended, before being put into production. Ongoing monitoring and documentation assists in detecting problems at the moment they occur and isolating the fault domain quickly and efficiently. In this paper, we will explore each of these habits and how they contribute to a reliable network.
Would a doctor be allowed to operate on a person without the proper training? Or a pilot allowed to fly a new aircraft, without being properly trained and checked out? It is clearly understood that allowing either of these would place both lives and property in danger. However, in most corporate environments, new technologies are implemented on a daily basis, without providing the necessary training for the network engineers. When these new technologies fail or perform in a sub-optimal manner, those that are responsible for troubleshooting and resolving the problems, lack the necessary knowledge to troubleshoot the problems.
Over the years many excuses have been provided for lack of proper training. Managers have been afraid that trained employees will take the newly acquired knowledge and seek other employment. Lack of budget to pay for training and travel to the training. Too short staffed to allow someone to leave for a week. As with all excuses, some of these have merit, while others stem from not knowing the importance of proper training.
The effective network manager understands the importance of providing the necessary training for his staff. He knows that providing the training prior to implementation ensures the network engineers will understand how the technology is designed to operate, as well as the step required to troubleshoot problems. Trained network engineers are less likely to make mistakes, when changes are made to the system.
As with making investments in upgrading the network infrastructure, the effective network manager understands it is important to invest in the people that support the network. This investment demonstrates they are a valued resource that is as critical to the network operation as the equipment they are supporting.
Training assists in getting everyone on the same page and dispels myths and misunderstandings when it comes to how the technologies operate. This common understanding allows the team to work together, instead of at odds with each other. Working as a team allows the network manager to focus on the technical aspects of the problem, not the personnel issues surrounding differing opinions.
The effective network manager does not see the right tools as an option. He has seen result of a lack of tools or the wrong tool for the job. His network engineers do not have the visibility required to quickly diagnose network problems, resulting in extended network outages that impact the business.
Acquiring the proper tools and the associated training are part of the network upgrade process. As new technologies are brought into the enterprise, it is important to ensure the tools needed to troubleshoot these technologies are brought in at the same time. Having the necessary tools during implementation allows the network engineer to validate the implementation as well as develop a baseline of operation. This baseline can then be used for comparison when troubleshooting network problems.
A lack of tools reduces visibility, increases the time required to resolve network problems, and prevents the network engineer from performing the necessary validation. The effective network manager knows these tools are as important as the switches, routers, firewalls, and physical infrastructure that makes up the network itself.
In many cases, each network engineer within an organization has a unique approach to solving problems. This approach is the result of a variety of factors. These include the types of technologies supported, managers, education, and curiosity level. While this approach may work well for the individual network engineer, it does not always work well within an organization.
The effective network manager works to combine the strengths of each of these approaches into a common, documented approach that is used within the organization. He has seen what happens when each person takes a different approach. Problems take longer to resolve or one thing gets fixed and another is broken.
Another benefit of this common approach is the reduction in time when bringing new people on board. All too often we see that the existing staff does not have the time to train the new person. When this is the case, it can take a long time for the new network engineer to get up to speed. When a common documented approach is used, the new network engineer is provided the documented approach and the necessary tools to begin resolving network problems. This allows him to become an effective troubleshooter in a shorter period of time.
It is important to note that the troubleshooting approach is a living thing. As new technologies are implemented and new applications are put on the network, this approach must be updated. This ensures the processes and procedures are in place when issues arise with these new technologies.
The word “should” is a dangerous word. All too often in the network industry we hear this word used when implementing new networks. “That should be in the right VLAN”, or “We should be getting gig speeds out of that link”. Statements such as these are not based on measurements, but instead based on what the network engineer believes.
The effective network manager does not accept “should” statements. He is looking for fact based measurements that show things are working as designed. This is where validation testing comes into play.
A physical cabling plant would never be accepted without being properly tested using copper and fiber certification equipment. However, we see that active equipment such as routers, switches, load balancers, and firewalls are installed with little to no testing. This leaves the opportunity for problems when the production traffic is introduced.
As part of the network design and implementation process, it is important to include validation. This entails identifying the expected performance of the network and how that performance is to be measured. As part of implementation, measurements are to be taken and compared with the expected results. Should the measurements meet or exceed the expected results, the measurements are documented and the network is accepted. If the measures fail, the network engineer uses these measurements to determine what is wrong and re-measures after the necessary changes are made.
After the network has been designed, implemented, and validated, it is important to monitor the network in production. The effective network manager uses long term, ongoing monitoring to look for trends and outages. Through the use of thresholds and alerts, his team will be notified of problems, before they are detected by the end users.
In many cases, what differentiates the highly effective network manager from the others is where the monitoring is performed. Most organizations monitor from the center out. The monitoring tools are installed at the datacenter and monitor resources in the data center.
The application world has changed and centralized monitoring is not adequate to monitor the variety of applications in today’s environment. The key is to monitor both from centralized location and from the end user perspective. The centralized monitoring allows the network manager to monitor traffic patterns, utilization levels, and look for errors. The decentralized monitoring provides visibility to the response time and availability for cloud based applications and to get the end user’s response time.
As with tools, training, and the approach to troubleshooting, network monitoring must be a part of the implementation of new applications and infrastructure. The effective network manager works with the business to ensure that new applications are monitored, whether they are implemented by the IT group or by the business. In the end, it is the network manager’s responsibility to provide a reliable, well performing path between the user and the application server, regardless of whether it is an internally maintained application or a service provider.
Prior to deployment, each application must be evaluated for a number of reasons. The most important is to determine how the application will perform on the organization's network. When applications are deployed, without being evaluated and problems arise, it is always blamed on the network. The highly effective network manager knows that it is better to discover problems during pre-deployment testing than during a production cutover.
Another reason to perform the pre-deployment testing is to determine the application dependencies. These dependencies will be used to create troubleshooting procedures that will be integrated into the troubleshooting approach. Without knowledge of the servers and infrastructure on which an application depends, it becomes very difficult and time consuming to troubleshoot the application.
With applications such as voice over IP, the network manager must understand the thresholds used for validation and ongoing monitoring. During the pre-deployment phase, network conditions can be varied in a lab environment to determine the applications ability to handle latency, packet loss, and jitter. These values can then be used to validate that each of the sites where the application will be deployed will be able to support the new application.
Everyone hates doing documentation. This is evidenced by the fact that very few organizations have good, current network and application documentation. The effective network manager knows from experience that without the documentation, problems will take longer than necessary to resolve.
Good network documentation includes:
Validation test results – The results from the post implementation validation tests are used as a baseline when troubleshooting problems. The acceptable results are compared with the results collected while troubleshooting. The difference between these results will help the network engineer isolate the problem.
As-built diagrams and tables – There is often a gap between the way things are designed and how they are implemented. By carefully documenting the network and traffic flows as the implementation occurs, the network manager is assured that the documentation accurately reflects the network as it is, not as it should be.
Addressing and naming schemes – As new network engineers join the team, it is easy for naming conventions and numbering schemes to change. As with a lack of documentation, inconsistent use of addresses and names will increase the time it takes to troubleshoot problems, when they occur.
In addition to demonstrating a professional approach to managing a network, the effective network manager knows that this documentation will reduce the time it takes to bring new engineers on board. When key aspects of the network are kept in engineers' heads, it becomes very difficult for new team members to get up to speed and become effective troubleshooters. In the end, accurate network documentation benefits the business as well as the networking group.
Highly efficient networks don’t just happen. They are the result of a concerted effort on the part of those that are responsible for the network. While these efforts will not eliminate all network and application problems, they will significantly reduce the time it takes to resolve those problems. The highly effective network manager understands that these do not all happen at once. It takes time to get buy-in from management to support these efforts, both in budgetary support and recognizing how these efforts will result in an overall reduction in cost to the business.
Network engineering groups are often times set in their ways and resistant to changing the way they have done things in the past. They have found the techniques they have used are successful. As they receive the training and tools necessary, validate network installs, and document the network, they will find that time can be focused on working on new and interesting projects, instead of troubleshooting problems.