Internet Rover 3.0 Introduction PRELIMINARY - BETA TEST VERSION - DO NOT DISTRIBUTE The Merit Internet Rover package contains many NOC tools that have been used by the Merit NOC to manage the National Science Foundation Network (NSFNET), the Michigan State wide regional (MIchNet), the University of Michigan Campus Network (UMNet).. These tools have evolved over the past four years, and are in use in many Network Operations Centers (NOCs) across the country and around the world. This paper will describe the collection of tools individually, and then collectively, describing how, as a system, these tools have worked together to manage the networks. This release represents the next generation of Internet Rovers including graphical as well as text monitors, statistics collection tools, a fast SNMP API library, and some preliminary cuts of some network discovery code. As always, all source code is provided with hopes that we can all leverage the improvements made by others. This document will attempt to show the pieces of the rover system, the file formats, and the necessary information in order to help you get started applying these tools to your environment. It is important to note that not all pieces of this code will be applicable to your environment. Several sub-systems in this code (e.g. CircuitCheck) were designed specifically for the NSFNET architecture, and will probably not be useable in their current form. For the sake of simplicity, and because the code may be modified to apply to your environment, the code is being released as is. If you don't have the Motif libraries, simply don't build the graphical tools and the rest of the code will work fine. If you can't monitor the CSU/DSUs in your network, simply don't build the CircuitCheck applications. You will notice that all interprocess communication is done via flat files. This was done on purpose. You will notice that the rover code is composed of many independent pieces that can be put together in a variety of ways. This was also done on purpose. Over the last five years, we have grown to appreciate flexibility. In our environment, we have needed to split the pieces across physical machine, hardware platform, communication media, and finally, administrative boundaries. If things can get confusing, they will. Being able to debug problems by catting flat files is a plus, and much easier than debugging various other IPC mechanisms. This has been my experience anyway, and I've been the one who has had to muck with both situations in times of calamity! It also makes it easier for the novice to poke around and figure out how things work. I hope you can use the code. We have been fairly happy with it, and it has served us in the 24-hour 7-day a week environment here at the Merit NOC for the past 4 years since I started at Merit. Bill Internet Rover 3.0 Installation and Setup 1. Create the /etc/mib_desc file - copy the one in the distribution etc is needed. This file contains the mib description required for the fast snmp api routines. 2. Decide where you want your data directory (herein called $PINGKYDIR) and edit the Makefile for your configuration. The data directory will hold all of the critical rover data files and can be placed on a NFS-mountable partition for easy multi-workstation access. 3. Assure that the bin directory in the rover package is in your path, OR you have a copy of mkdep in your path somewhere. 4. Type make clean depend all install ( This has worked cleanly on our RS6K, Sun and Ultrix machines. If you don't have the Motif libraries, then you will run into problems with the graphical display code. In this case, you will need to muck with the makefiles to avoid the building the xmap application. ) 5. Configure the $PINGKYDIR/hostfile with a list of hosts you wish to monitor and tests you wish to be performed on those hosts. 6. Make sure that the pingd program has the setuid bit on, and is owned by root. One way to do this is to become root, and type make root-install which does all of the right things (copies the binary into the bin dir, sets permissions and ownership). 7. Type InetRoverd & pingd & This will start up the InternetRover code along with the pingd which handles all PING() tests in the hostfile. 8. For the graphical displays, you will need to create a LINKS file. The LINKS file describes the links in your network and which nodes and interface #'s are on either end of each link. See the LINKS File Format section for details... 9. Create a Rover Daemon for your network - use the existing xxxxRoverd scripts as templates. This script will invoke the xxxxpoller program which, using your LINKS file, will poll the interface status, assign the nodes and links a state, and finally store this information in network status file *xxxx.STATUS ). You will also need to make a symbolic link (or copy the poller) ( Example: cp MeritRoverd xxxxRoverd; ln -s meritpoller xxxxpoller ) 10. Start up the xxxxRoverd. ( xxxxRoverd & ) 11. Create a symlink for the graphical map, and bring up the graphical map by typing your network name xxxx. ( ln -s xmap xxxx; xxxx & ) Note: You will need to bring up the graphical map twice before it will work ; bring up the map (it will be a small rectangular box with no nodes) and select the file menu bar and the quit option under that bar. Bring up the map a second time, and the nodes should appear somewhere onthe screen. You can move the nodes around as you like (we'll iscuss that later ). 12. Probably the easiest way to proceed is to now make a copy of what I call an executaion environment script. We have one for each network we manage. See NSFNET, CICNET, MERIT, MICHNET for examples. These scripts contain all the necessary commands and environmental variables necessary for end users to run the programs, including the resources! This has been a pain in the past, because user's can unknowingly override the rover resources, so we created a executaion environment that prevents this. ( cp NSFNET XXXX ) 13. Edit your execution environment script and try it out. You will notice the reference to the $PINGKYDIR/XXXX.resources file. This is where you define all of your menus, actions, bitmaps for nodes, background bitmaps, etc. Again, peruse the examples. You can do things like assign double mouse clicks on nodes to execute a particular script, a different script for tripple mouse clicks, a script for hitting the programmable function key 1 (PF1), shift-F1, etc. Basically, you can assign arbitrary actions to X-events on your graphical map (that's why we call it a programmable graphical map!) 14. Create additional useful scripts, assign them to menus or function keys, and tell folks on the rover-list@merit.edu about the cool new scripts/programs you have assigned to your map and make them available to your fellow network management chums. Internet Rover 3.0 InetRover Chapter 1. InetRoverd ******************************************** * * graphics -- Cannot represent as text. * ******************************************** 1.1 Hostfile Format The InetRoverd task performs tests on hosts, and adds and deletes problems from the PROBLEM.FILE based on the success or failure of the tests. This information is kept in a file called the hostfile. The hostfile has the following fields: #hostname uniqidentifier HelpFileName PrimaryTest Test .... Test CIC-OSU-LAN 192.68.143.1 Help/CIC.OSU PING() TELNET() ******************************************** * * raster -- Cannot represent as text. * ******************************************** The hostname can be any contiguos string of characters and corresponds to a recognizable host name. We use the name of the host to indicate network, location, and interface, whenever reasonable. The uniqidentifier can be any contiguos string of characters that uniquely identify this node. This field is passed to the tests defined in this hostfile entry, and is also used as matching criteria when problems are added and deleted from the PROBLEM.FILE The help file name will be discussed in the Text Display section, and provides the name of a file containing useful information (contact names, support hours, etc.) for the host. For each entry there exists a primary test, and any number of secondary tests. The primary test is used to determine if the host is reachable. If the primary test fails, then an entry is added to the PROBLEM.FILE indicating that this test has failed, and all secondary tests are removed from the PROBLEM.FILE, if they exist. If the primary test succeeds, then this primary entry is removed from the PROBLEM.FILE (if it exists) and all secondary tests are performed. 1.2 PROBLEM.FILE Format The PROBLEM.FILE is a text file containing any number of lines of alerts. Each alert is formatted as follows: TimeStamp HostName UniqueID Test StatusLine where: TimeStamp is the unix long representing the start time of the alert HostName is the name of the host as specified in the hostfile UniqueID is the unique identifier for this host as specified in the hostfile Test is the test that failed StatusLine is descriptive text associated with the problem entry. Example PROBLEM.FILE: ******************************************** * * raster -- Cannot represent as text. * ******************************************** 1.3 InetRoverd Tests In the Internet Rover scheme, tests have the obvious sematics; either the test succeeds or fails. Based on the result, a problem entry is deleted or added. Any test that can be coded to return a boolean value can be added to Internet rover. To avoid performing tests on nodes that are clearly not accessible, Internet Rover takes a two-tiered approach: primary tests and secondary tests. In practice, the primary test is typically the PING() test. The secondary tests may include the following tests: TELNET(), FTP(), NAMED(), SMTP(), TROUBLE(), DIXIE(), or any generic test. Generic tests are ones that are performed outside of InetRover. Generic tests are scripts or programs that perform some test, and create or remove files based on the failure or success of the test. InetRover will look for the existance or nonexistance of the file to determine if an alert should be generated or removed. In this way, InetRover can be extended to monitor just about anything. One example of a generic test is our T1ASChecker. A script runs periodically that queries the T1 NSFNET backbone for a list of autonomous systems (ASs) announced to the backbone. If an AS used to be announced, but is no longer announced, a file is created with a name like AS###. Otherwise, if the file already exists, and the AS is announced, the file is removed. hostfile entry: AS231 231 Help/AS.HLP ASDOWN(AS/AS231) When InetRover processes this hostfile entry, it won't recocognize the ASDOWN test, and assume it is a generic test. The generic test returns a boolean based on the existence of the file specified in the parens. In this case, if the file AS/AS231 exists, the test fails, and an alert goes into the PROBLEM.FILE. Otherwise, the alert is deleted from the PROBLEM.FILE. Internet Rover 3.0 Ping Daemon ******************************************** * * graphics -- Cannot represent as text. * ******************************************** The most common test performed by rover is the PING() test. This test determines if the not is reachable or not by sending an ICMP echo request packet and awaiting an ICMP echo response. The problem is that it takes time to determine that a node is not responding to ping. In practice, this causes network management tools to work least efficiently during times of network calamity, and most efficiently during times of network stability. Obviously, this is not the desired result. An additional problem arises as well. During network routing information exchanges associated with metric changes and other routing instabilities, it is likely that seconds can elapsed before packets once again become routed. The original ping test sent ICMP echo request packets one second apart until a response was received, or five seconds had elapsed. The network instability caused by network route exchange would cause a situation where nodes would incorrectly be marked unreachable. The next cycle through would indicate thate the nodes were once again reachable. Which nodes were marked unreachable was dependent upon which nodes were being polled during the routing exchange. These false alerts caused the alerts authenticity to be questioned. The ping daemon was written to solve this problems. By changing the implementation of the PING() test from sequential ICMP echo requests to parallel ICMP echo requests, there is an upper bound on the elapsed time before a node outage is detected. The pingd reads from the same hostfile to determine which nodes need to be ping'd. pingd maintains a table of nodes to ping, when the next ping is to be sent, the current state of the node, along with configuration information (number of retries, and time between retries). Initially, all pings are sent out in parallel. As responses are received, the next ping times are calculated, and the state of the node is set to up. Since the responses are received at different times, there is a natural skewing of the next ping times. There is a default number of retries and a default time between retries. Users may edit the hostfile and alter the default timeout,retry, and cache timeout parameters on the PING() test. The syntax is: PING( retries, timeout, cachetimeout) where: retries = the maximum number of retries before marking the host DOWN timeout = the time in seconds between retries cachetimeout=the time to hold the node state before restarting the pings This facility solves the previous problems in two ways. First, it works equally well during outages and network steady states, because there is now a cap on how long a node is non-responsive before it is marked down. Secondly, short duration outages caused by routing transitions no longer cause alerts, because the user can adjust the sensitivity of the test. (Kind of like a squelch knob on a citizen's band radio.) Internet Rover 3.0 Discovery Rover Chapter 1. Discovery Rover One other problem with the InetRoverd is that all of the nodes and tests need to be manually configured into the hostfile. Whenever humans are required to manually update configuration files, there is a non-zero likelihood that the configuration may be incorrect. Also, we have seen and exponential growth in network traffic, and a proportionate growth in the number of network equipment that required monitoring. Combine the two situations and you have the potential for a configuration mess. An additional problem has to do with alert the alert monitoring showing side affects of problems as well as the cause of the problem. For example, if a network partition occurs, the cause of the outage may a broken link, and the effect may be several nodes becoming not reachable. If you used the PING() test to monitor these nodes, all of the side affects of the outage would show up as alerts. Perhaps the link being down would cause an alert as well, but this "real" problem may buried in a series of side affect alerts. The desired alert display only contains a list of problems that the operators need to work on, and filter out the side affects. The discovery rover solves these problems by only reporting what is "known" to be a problem. This is accomplished by letting nodes and links occupy one of four states: UP, BUSY, Not Reachable (NR) and DOWN. ******************************************** * * graphics -- Cannot represent as text. * ******************************************** Example: All nodes responded to neighbor table poll, each reporting all links in the UP State ******************************************** * * graphics -- Cannot represent as text. * ******************************************** Example 2: A link goes down, and the algorithm correctly identifies the broken component. ******************************************** * * graphics -- Cannot represent as text. * ******************************************** Example 3: A network partition occurs, and the algorithm correctly identifies the broken components. ******************************************** * * graphics -- Cannot represent as text. * ******************************************** Example 4: A node not responding to queries, but seen as "UP" by any adjacent node is marked "BUSY". Luckily, on the NSFNET backbone, the SNMP provides easy access to information that allows applications to "discover" the state and topology of the backbone. Using Merit's Fast SNMP API, which we will discuss later, we are able to discover the state and topology of the backbone in thirty seconds or less. The discovery code queries each node for a list of neighbors and link states, and uses this information to maintain a "Network Status File". As new nodes are added to the backbone, the discovery rover automatically adds them to the Network Status File. As nodes and links transition between states, the network status file is updated to reflect this. Two programs use this file; SortStatus and the graphical rover map. SortStatus is invoked after the poller discovers the network, and adds or removes problems from the PROBLEM.FILE appropriately. This program prvides the "Glue" between the Discovery Rover and the PROBLEM.FILE-based display programs. Many available network technologies do provide instrumentation denoting adjacencies. For these networks, we split the topology discovery and state determination into separate pieces. The discovery code will dump the routing tables to discover ajacencies. Often this involves hundreds of queries. The output of the discovery code will be a "LINKS" file that contains detailed information about the network links, including interface addresses and interface numbers. 1.1 The Graphical Rover Display The graphical rover map provides a graphical reflection of the network status file. As nodes and links change state, the color of the nodes and links on the map change color. As new nodes and links are added to the network, these nodes and links are also added to the backbone map. New nodes are placed interactively on the map, and facilities exist for each user to create their own network map. Also associated with each graphical rover map is a set of actions. In their .Xdefaults file, user define what actions should be executed when they click on a node or a link. They can define their own menus of actions for nodes and links. Function key assignments can also be assigned to invoke actions when the mouse is positioned over a node or a link. This kind of configurability make the graphical rover map extremely powerful. Each map is started up by specifying the name of the network, where the network name is a symbolic link to the xmap program. You can do this on unix boxes by a command like: ln -s xmap nsfnett3 . The xmap will dtermine from the name of the program, what the network Class is, and from that determine which user resources to get from the resource manager. Assume the network is called nsfnett3, and the user has a .Xdefaults file like this: Nsfnett3*draw.XmPushButton.Translations: #augment \ F1: system( viewisis ) \n\ ,,,: system( openping ) \n\ : PostNodeMenu \n\ Ctrl : MoveIt() Nsfnett3*draw.Translations: #override \ Ctrl: PlaceIt() \n\ : PostLinkMenu() \n\ Nsfnett3.nodeMenuItem1: openping Nsfnett3.nodeMenuItem2: opentelnet Nsfnett3.nodeMenuItem3: opentelnet OUT-OF-BAND Nsfnett3.linkMenuItem1: DIagnoseProblem Nsfnett3.linkMenuItem2: CheckDSUs In the first section of this user's .Xdefaults file, the F1 function key, when clicked on a node, will invoke the viewisis script. All programs invoked by the graphical display will receive environmental variables detailing which node or link was selected, so programming the graphical display becomes simply a matter of defining what actions are to occur when the user performs key clicks or other X actions. The second translation in the first section shows how a sequence of mouse events can be used to trigger a script. In this example, a mouse double click triggers an openping script, which opens up a window pinging the node. The last two translations are necessary intrinsics for the application. The PostNodeMenu action will be discussed later, and provides a user-defined popup menu for network nodes. The MoveIt() action allows a node to be selected for interactive placement. The programmable graphical display also supports up to 20 menu items for nodes and links. The second section shows an example of how this might be configured. In this example, there are three node menu items shown to the user when a node is selected with mouse button 3, and two menu items when a link is clicked upon with mouse button 3. What we have done administratively is to use a "Execution Environment script" to assure that all users get the same map, menus, actions, etc. This script sets $XENVIRONMENT to a resource file that contains the resources for that network, and then invokes the xmap application specifying the correct community name. Users can always choose to create their own .Xdefaults file and invoke the program independent of this effort, but for people just starting, this works out very nicely. It also lets individuals network "owners" maintain the appearance of their maps independently of one another. Sample network Resource File: ******************************************** * * raster -- Cannot represent as text. * ******************************************** 1.2 Graphical Rover Display Actions We have created scripts that automatically diagnose common network faults such as circuit problems. NOC operators can click on a link and see a graph showing input and output packets over the previous 24 hours, which is especially useful for planning scheduled outages. Additionally, simple scripts that present historical information with regards to node and link outages have been written, along with scripts that access the trouble ticket database for more information. The real power of the map is being able to see the state of the network, and lauching tasks to query the network or locally stored information. All of the power and flexibility of the unix operating environment is available to script and program writers. This is an important difference, as many available commercial packages require users to write scripts in sometimes arcane proprietary languages, typically with no process control, programming tools or debugging environment. Just code up the script or program and add it to the .Xdefaults file and you are off. Chapter 2. MAP File Format The MAP file exists for each network, and performs to functions. First, it maps UniqueID from the STATUS file to a Name that will appear on the map. Secondly, it provides the x,y location on hte screen for the node. The nodes can be moved around and placed interactively on the screen. The file looks like this: ******************************************** * * raster -- Cannot represent as text. * ******************************************** Chapter 3. LINKS File Format linktype TimeStamp UniqueID IPAddr ifNum ifNum IPAddr UniqueID TimeStamp x y Where: linktype = use the ifType numbers from the rfc 1213 (MIB II) (use 0 otherwise) TimeStamp = The time this side of the link was validated (use 0 otherwise) UniqueID = a single queryable address of the box. For this box, there must be one and only one UniqueID - this is the name that describes the box. IPAddr = the IPAddress on this UniqueID that terminates this link. ifNum = the interface number - this is the interface rover will poll to determine the state of this link. x, y = don't care - this was used for the Circuit Trouble Ticket Sub-System and is probably not applicable for general use. See the CircuitCheck code if you have questions about this and think you may have the ability to detect circuit transmission and reception problems by querying your tranmission units (CSU,DSU,modems, etc.) Here is an example LINKS file. John Vollbrecht (Merit) wrote a script to generate this LINKS file from the cisco configuration files. ******************************************** * * raster -- Cannot represent as text. * ******************************************** Chapter 4. Dynamic Link width on the graphical display The width of the links on the graphical display can be displayed as dynamic. It is up to the user to define the metrics for link width and invoke the routines to create the information necessary for the map application. One such program is included in the package and is called makelinkwidth. The output of this program will be a linkwidth file (.LW) placed in $PINGKYDIR, and will be automatically detected by the xmap application. The LinkWidth Menu will present to the user all of the possible LinkWidth files. Internet Rover 3.0 SNMP Applications Chapter 1. SNMP Applications 1.1 snmpstats2 - SNMP statistics collection program The snmpstats2 application collects network statistics using SNMP and stores the data using the /rtdata format described below. The /rtdata format is a set of "data rules" describing how the data files are named, and how the data is stored within the data files. This set of rules is uniform across all types of data we collect, and allows us to build a complete statistics collection systems with all of the following characteristics: any type of data any number of networks any number of nodes to query any number of objects to collect from each node a naming convention allowing the use of standard unix utilities to synch,skulk and archive The data is stored in files named: /rtdata////. where: network is the name of the network data type if the type of the data - in our case snmp nodename is the DNS name or a queryable IP address of the node dataname is the name of the collected object in this file datestr is the day that the data was collected in the form YYMMDD Sample ifInUcastPkts.921203 file: ******************************************** * * raster -- Cannot represent as text. * ******************************************** 1.2 xdview4 - SNMP statistics viewing program Given the data rules defined in the above section, xdview4 provides an easy mechanism to view SNMP collected data. The xdview4 application can be viewed as a SNMP statistics browser, allowing you to select any number of nodes, from any network you monitor, and view the data of any period of time you wish. xdview4 uses the naming convention described above to determine the number of hosts in a particular network, and the types of SNMP objects available from the node. 1.3 snmptable2 - dump any number of tables from any number of hosts useing SNMP. syntax: snmptable2 -h [ -h ... ] -c ... snmptable2 does a series of get-next requests to retrieve the specified MIB tables from any number of nodes. The underlying code deals with the retransmissions and timeouts, and presents the output to stdout. Example: ******************************************** * * raster -- Cannot represent as text. * ******************************************** Chapter 2. The Fast SNMP API The Fast SNMP API uses the CMU SNMP code as a base, and provides an interface that allows applications to specify many hosts and many objects to retrieve in parallel. Multiple objects are packed into single SNMP packets for efficiency. The code allows the application writer to specify the number of retries, and the timeout between retries. This code is used by the discovery rover program called getneighbors2, the Merit statistcis program called snmpstats2, the SNMP table dump utility called snmptable2, along with several other ad hoc network management programs. Internet Rover 3.0 Environment Chapter 1. Interesting Operating Environments The rover system is entirely open. Also, the simple flat-file nature of the IPC used makes application of rover to network management tasks relatively straight forward. Further, the rover system allows use of the standard unix utilities and programming and operating system environment. This allows the use of the remote execution commands, X-windows toolkits, scripts, debugging tools, communications systems including NFS and RPC, etc. This flexibility has led to several interesting rover operating environments. 1.1 Rover Satellite System We had the desire to allow any of our "customers" to run graphical and text displays, but did not necessarily want to burden the data collection machines with additional tasks. Further, security folks demanded accountability, so providing one userid for NOC folks, local engineers, foreign engineers, VPs etc. was deemed not acceptable. The solution was to rdist the data files to a "satellite" machine. This machine was administered by locals, has its own set of binaries, and has plenty of horsepower to run many many displays. There is, of course, a 3 minute delay before the data is disted to the satellite system, but this limitation was deemed acceptable for passive viewing applications. This system is currently in production at Merit. ******************************************** * * graphics -- Cannot represent as text. * ******************************************** 1.2 Rover over NFS Another mechanism that we experimented with for a while was to have the data collection machine "share" the data files by making them available over an NFS mount. This way, any machines that mount the partition can have free and transparent access to the data. The down side ended up being reliability of NFS, and the on the machine supporting the NFS partition. ******************************************** * * graphics -- Cannot represent as text. * ******************************************** 1.3 Multiple-userid NOC The folks at Rice are using multiple user ids running the DIsplays. We have adopted some of Rice's code in this release for this purpose. The benefits include being able to view the log files and see who updated problems. I believe the mechanism they use involves the creation of a rover group, and the operators are members of that group. ******************************************** * * graphics -- Cannot represent as text. * ******************************************** 1.4 Separate but equal NOC In this scenerio, the requirements are that separate groups of rover collectors are logically separated. This can easily be accomplished by setting the $PINGKYDIR environmental variable to one data directory before starting a select set of rovers, and then change the $PINGKYDIR to point elsewhere before starting the next set of rovers. This strategy has several advantages. First off, it allows similiar hardware/software to be clustered together on one text alert screen without affecting the other screens. This strategy can be extended to n screens. This also provides a political firewall - if the rover breaks in one place, it doesn't affect the other rovers/displays. Secondly, these rovers and displays can run on other machines. As long as the NOC can get to the rover or display machine, they can display alerts. This is particularly important if there is CPU resource contention. It ******************************************** * * graphics -- Cannot represent as text. * ******************************************** 1.5 One collector - multiple status files This environment is designed to allow a total network map view as well as a breakdown by area. The roverd can be altered (it is a script after all) to, after polling, grep out certain portions of the status file to create another STATUS file. These other STATUS files can then be used with their own MAP file. Finally, you can configure the menus such that when you click on a node, the execution of another xmap application would result, where this invocation will be displaying a sub-net. Internet Rover 3.0 Flow Diagram ******************************************** * * graphics -- Cannot represent as text. * ******************************************** This code is copyrighted but freely available via anonymous ftp from merit.edu (35.42.1.42) in ~ftp/pub/inetrover.. .