What Can I Do With Probes?

Pat: Welcome, everyone, to today's webinar on Intermapper Probes. What are they and what can you do with them? My name is Pat Cameron. I have been working with Intermapper for the past year. I have been with HelpSystems, which is the parent company of Intermapper, for almost 15 years, and working with Intermapper since we acquired it in December of last year. I would like to welcome Rich Brown, who is with me today, as our presenter. I'm just hosting this webinar. Rich is one of the creators of the Intermapper product and one of the founders of Dartware, which was the company that he started for InterMapper. I am really happy that he is able to join us today and talk with us about probes. What are they? What can you do with them? I love listening to your passion about the product, Rich.

Rich: Thank you, Pat. Good morning to everyone. Were you going to run the poll, Pat?

Pat: I am. Before we get started with our presentation, I've got some polling questions. We just would like to get a feel for our audience. I'm going to go ahead and open up the poll. You should see a box pop up and I've just got a couple of questions to ask about your experience with Intermapper. Are you brand new to the product? Have you worked with it for a while? Is this the first time that you have ever seen it? Then, if you have worked with Intermapper, a little bit about your level of experience. Are you brand new to the product, again? Have you worked with it for a while? Are you an expert that's creating your own probes? We would just like to get an idea of the experience that our attendees have.

Then the third question is about IPV version 6. Are you ready for it? Are you thinking about it? Have you not started thinking about it yet? Just get a little idea of where people are with that. I am going to keep this poll open for just a minute. We have quite a few attendees today, so I will give you a chance to go ahead and answer.

A little bit of housekeeping then before we get started also. We do have a chat window and you are welcome to send us a chat message if you have any questions throughout the presentation. I am going to be monitoring the chat while Rich is doing his presentation. In order to bring that chat window down, if you just take your cursor, put it up at the top of the screen and you will get a little dropdown box from WebEx. Go ahead and click on that chat icon and that will bring that chat window up on your screen. You are welcome to send your questions to the host and presenter. It would be good to send it to both of us, I guess. Send it to the host and presenter or you could send it to all attendees; that would be great. We will try and answer your questions today if we are able. If we run out of time or are unable to, then we will follow up after today's webinar with the answers to your questions.

I also want you to know that I will be recording the session today and after this session you will be receiving a follow-up email with some information about Intermapper and probes, as well as a link to the recording. I am going to go ahead and open up the poll questions, and let me share those with everyone. You can see who we have got with us today. Can you see the answers, Rich?

Rich: No, I don't see them.

Pat: We practiced it and it came up just fine.

Rich: Okay.

Pat: Hang on a minute. Maybe there is some kind of a delay.

Rich: There are so many people that the computer is having trouble counting it.

Pat: Well, I can see some of the results and it looks like we have a nice mix of people. We have a few that are just trailing Intermapper. We have about 20 that have used it for a couple of years, 35 that have used it for more than five years, and a few that this is the first time that they have seen it. We’ve got a nice mix that have used Intermapper. We have some newbies, some novices, six experts, so be ready. Those will be great questions. Most people just haven't started thinking about IP version 6.

Rich: Okay. All right, very good. Well, thank you, Pat. Let me jump into the presentation today. I wanted to talk about Intermapper probes, what they are, what you can do with them. I want to spend a fair amount of time showing off some probes and thereby also showing some of the things that you can do with Intermapper. When I get to the demonstration, I have divided them into three groups. The important probes, they are the ones that you are going to use all the time when you are setting up your maps to watch the network. I have a sense that we have a number of less well-known probes. I wanted to call those out because they are really useful, and something that you could get good advantage of. Then there are a bunch of cool probes. Our customers have made a lot of probes for Intermapper, and I want to show some of those off.

What is a probe? A probe is a software plugin for Intermapper and the probe specifies how Intermapper is supposed to test the device. A probe specifies whether to use SNMP queries or to make a TCP connection to the device or to run some kind of a script, to retrieve data from the device and then compare that to thresholds that you set and then that determines the device's status. There are well over a hundred built-in probes built into InterMapper. We have a lot more on the Intermapper.com website that customers have contributed. As I mentioned, you could build your own.

Intermapper probes monitor just about anything. Anything that has got an IP address, you can probe somehow: network equipment, obviously the routers and switches that you can see, the interfaces and the traffic and the errors on all those interfaces. You can look at applications, webmail, FTP, all kinds of applications that you run. There is a whole grab bag of other kinds of equipment, and I will touch base on some of these.

Basically, Intermapper either has a built-in probe or some other probe contributed that will… let us just see into the performance of most anything. I am going to jump in and just go on to demo mode here. These, as I said, are the important built-in probes. They are the backbone of what you are going to use. I am going to just switch over to an Intermapper map. This is all live data. I'm just going to go through some of these devices that are on this map. As I said, this is live data. I was looking at a couple of pieces of equipment at the HelpSystems office.

We are looking here at this device. I am right clicking to open the status window on this device, and the status window shows the DNS name and its IT address and its uptime, how long Intermapper has seen it. The probe type is Ping Echo. What Intermapper is doing is sending a ping and looking for a response. Its availability is 100%, packet loss is 0% and short-term packet loss is also 0. The difference between the packet loss and short-term packet loss is that packet loss is overall time; the server has been running for about 20 hours now. There have been 998 total pings sent with 0% loss.

Short-term packet loss is looking at the last 100 pings that have been sent so that Intermapper remembers the last 100 pings to show what's happened recently. The response time on this device is very good, it's zero millisecond. It's under a millisecond. That's the ping device. If you can't test a device any other way, you can usually ping it. In fact, pinging is a very powerful technique. Actually I want to say one more thing about it. On your local network, your packet loss should be 0%. If you are seeing packet loss on the local net, something is wrong, and you need to track it down. Obviously, if you are on a Wide Area Network, packet loss can be higher, but you still don't want it to get much more than a couple of percent because your users and customers are going to notice problems.

The next probe I want to talk about is the SNMP Traffic Probe. Again, I'm going to open up the status window here, and what this does is it looks at the device that uses SNMP to retrieve information about it. That tells about the interfaces, about the traffic on those interfaces, and all. Oh I didn't reset my demo. Excuse me. Pay no attention to the man behind the curtain. I'm going to unlock the map here. I'm going to set info. It is probably worth showing you the “Set Probe” window.

Every icon on an Intermapper map has one or more probes associated with it. The basic set of probes is ping echo and SNMP traffic. I will talk about map status in a minute. Automatic actually determines whether the device speaks SNMP, in which case it tests with an SNMP Probe or otherwise it pings it. I am going to set it to SNMP traffic for just a moment. So here we are. This is the real SNMP traffic window. The probe type is SNMP traffic. It has got the same packet loss statistics and short-term packet loss. If I had shown all the interfaces on this, you would see all the traffic and marching and I am pretty sure you are familiar with that. I am going to change this now to the “Set Info, Set Probe” and I'm going to change it to the SNMP Table Viewer Probe.

This shows a lot more information about the device, and so, opening the status window now, we see the same stuff at the top of the window. This is name, this is contact availability and all, but there are also a number of underlined links here. Each of these opens up a table that gives you a lot of information about this device. Here we are looking at the IF table. If you're familiar with SNMP, you’ll know that is the interface table. Here are all the interfaces, their index, basically their row number, their description, the link type, link speed, their MAC address, and whether they are up or not. There are 36 rows in this table. Some devices have an extended IF Table. You can look at the traffic counters.

A lot of the other tables in device—the IP adder table is routing information—so we can make this a little bit wider and see what interfaces traffic is set out. This is the SNMP Table Viewer Probe. I actually almost never use the SNMP traffic; I only use the SNMP table viewer just because it provides so much more information. It doesn't add any traffic to the network or to the polling load on the device because Intermapper only fills in the information for these tables when you click and open the table. Intermapper makes the query, fills in the table, and then goes back to its normal polling. That's the Table Viewer Probe.

Intermapper has a bunch of probes for monitoring web servers: HTTP, HTTPS, HTTP does a GET. We do have an HTTP post and probe. What Intermapper does is it sends an HTTP GET request to the host asking for this page. If the response contains this string, then the page is fine. What is happening is this probe is sending a query to InterMapper.com and it's yellow. How come? The query is being sent to the Intermapper.com host. The reason code down here says that it's getting a 301. I know what's going on. Let me go back to the Set Probe Window. If you're familiar with HTTP headers, we’re requesting InterMapper.com. Well, we have set up the InterMapper.com site to redirect any queries for InterMapper.com to www.InterMapper.com. Let me change this to be www.InterMapper.com. Click okay and it turns green.

What's happening is Intermapper is continually querying this webserver and, in fact, the data that comes back contains the HTML response. Intermapper has a lot of probes for mail servers. Here I'll talk about how I got three of them together but we have a probe that’s checking the SNTP server at Google, at Gmail. It’s connecting on port 25, requesting, logging in and asking if, in fact, the Post Master exists. It's a Post Master. Similarly it has a ping echo probe attached to it and you can see the ping stats are there and we're also making an IMAP SSL connection to it. All of these probes are going against that single device to see if it's working. Intermapper has what we call a Post Resources Probe; it is an SNMP Probe. What it does is it looks into a lot of the host performance. The top of the window is the same but you can see the system up time is two days, the date is here. Number of user sessions, number of processes running, installed memory. You can look at the disk storage and say: “Here are the disks. The A drive, the C drive, the D drive, and virtual and physical memory.” Look at their percent; how full they are.

Pat: I think we have an issue on the D drive there.

Rich: Maybe. That's what SNMP is reporting. I don't know anything about this particular system. This is on the demo map. This is really good because network managers can then come in and say, "Is this what I expect or not?” The Host Resources Probe uses SNMP to retrieve this information.

The last one I am going to talk about are the really important ones: Intermapper has a whole set of DNS Probes. What does it mean to test a DNS server? Well, the DNS server is probably working fine but the really important question is if I ask you what's the IP address of some server? You really want to know that the DNS server is returning the right answer. I’ve configured this probe to ask specifically what the IP address is of www.Intermapper.com and I have also configured into it. I'll show you that in a minute. The right answer is 50.56.155.193. Fortunately, the good news is Google is returning the right address. Let me show you how I configured this.

In the server standard, there’s a section on DNS queries. We have an A record query. I specified this is the domain name to request. This is the right answer. You can check for mail servers. You can check up MX [SP]. You can say, “What’s the MX record for Intermapper.com?” You can specify that, or the main server, the reverse lookup or the TXT but I'm going to leave it at the A record. Basically this a good tool for just making sure that your domain names are out there and that they are correct. It is also convenient if you are moving a server. You can make two probes: one with the proper address right now and make another Probe with the address of the server after you make the switch. Once you get in trouble, and the other should be good, and then after you made the switch, they should swap. That's the DNS Probe. I think that was the set of really important probes that you always want to use.

Pat: Those are all probes that we ship with the product?

Rich: They come with the product.

Pat: Standard?

Rich: Um-hum.

Pat: Perfect. We have a couple of questions. Otto is asking about the HTTP calls. Does it constantly perform DSN hits on the probe times? Are you adding any burden to DNS servers?

Rich: Intermapper uses the underlying OS-DNS cache. The glib answer to the question is no. When Intermapper of course makes the first query to that device, it forces the underlying OS to do the lookup and that result is cached as long as the TTL comes back and responds.

Pat: Roy is asking where to find the hosts resources probe. Do you want to bring that up and show it?

Rich: Yes I will.

Pat: I bet it is in your list.

Rich: Yeah. I am going to go to “Set Probe.” It's in server's standard host resources.

Pat: Open up your server standard. For those of you that joined late, if you want to send us a chat message, go up to the top. Put your cursor at the top screen. You will get a little dropdown box from WebEx and then you can just click on that chat icon and it will drop down that chat window for you.

Rich: I actually thought of a follow up answer to that DNS query. The DNS Probes do actually send DNS queries each time because we are testing the server itself—not the host operating system’s ability to cache the answers. For the things like the HTTP queries and even the SNMP devices, Intermapper uses cached answers but for the DNS queries, for the DNS Probes, Intermapper always sends those queries directly. You don't want the cached answer. You want the current answer.

Pat: One more question about the host resources. Dave is asking if he can set different limits for different drives, and absolutely you can. Just open up that probe and you can…I think down at the bottom there is a window there and you can set whatever limit you want for any other drive.

Rich: One of the coolest parts about Intermapper is what we call the map status probe. I'll come around the other way. One of the coolest parts about Intermapper is the ability to have sub maps. On this map here we see two sub maps: the Seattle site and the Chicago site. If I double click on the Seattle site, up comes another map that shows all the equipment at the Seattle site. You can see that the most serious condition here is these devices that are orange, that are in alarm. If I click on them, I can look at the status window and oh that's not good. There is pretty high packet loss to that device. Actually these devices are down, aren't they? No these are in critical. These have very high packet loss. This is not my network so I don't really know what these devices are, but obviously if you see a device with high packet loss, that’s worth investigating.

The status of the icon here reflects the most serious condition on the sub map. What I'm going to do is I'm going to acknowledge this. I’m going to right click, chose “acknowledge.” That's odd. Put my initials there. Okay, now you will see what happened is that the icon turned blue. Acknowledging a device means that it's no longer a really hot color—red, yellow, or orange—but it’s turned blue. It's still in the same condition, but the other thing you’ll see is that instead of being a red icon, Seattle now is orange with a triangle. Chicago site same thing, I am going to acknowledge this as well, my initials there. It's acknowledged and I will acknowledge this one while I am here.

Pat: Clean that up, would you, Rich?

Rich: Yeah, right. Another way to acknowledge is Ctrl + quote. It’s a nice keyboard shortcut. Do that. I am going to close this window and what you see is that the Chicago site icon is blue. That means everything on that has been acknowledged, presumably somebody is taking care of it, but you see the little badge here still says that it’s yellow. The most serious condition on that is yellow. This is cool. These sub maps are really nice. How do we make a sub map? I'm going to ask you to watch very carefully because it's so easy, you'll miss it.

I'm going to the Map List window. I am going to take the item here for the New Hampshire site and I am going to drag it into the window. Double click. There is the equipment at the New Hampshire site. That's it. Now this doesn’t look the same so I am going to change the label and I happen to know that that’s 24- point, bold, there we go. I am going to right click, choose “attach to” and you get this line. I am going to click it and there we go. There’s some more cleanup I can do but now this map has an icon for the New Hampshire site. Double clicking it lets me see what's there. You could go to the “Set Probe” window and set the Map Status Probe on it. You will have to fill in all these tricky fields. It’s just much easier to drag from the map list into the map.

Pat: And then Intermapper just does the right thing?

Rich: Then Intermapper does the right thing. This actually works fine if you have multiple Intermapper servers and you have a map on server A that you would like to be a sub map on server B. Let me open this up. I didn't brief like this so it may not work, never mind. It would work if I was able to connect. I'm just going to cancel that for now. That's the map status probe. It's a nice way that you can build a hierarchy; you are going to have a national map and then drill down to various regions.

Pat: Oh.

Rich: Okay. Let's see if we can reconnect. Can you still see my screen?

Pat: I can and it looks like we are back.

Rich: We are back. Not sure what that was about. Another feature of Intermapper is the probe group. A lot of times, you want to have multiple probes monitoring a single device. You can do that by having a bunch of icons; one probe per icon. You might fill up your map with a bunch of those icons all talking to a single device. The probe group allows you to put them into a single icon. Let me show you this probe group here. I am going to open the status window and you can see the member probes are a host resources probe and an SNMP Traffic Probe. If I click the twisty triangle, I get the status window for the SNMP Traffic Probe. I can close that and also look at the SNMP Host Resources Probe, and this is the host resources information.

A probe group is pretty easy to create. I am going to ungroup them to start. I’ve got two icons here both going to the same host. I am going to click and select them both and then choose “group” and there we go. They are put into a single group and if I ungroup them, they split up, but I like them together so I’m going to leave them that way. Probe groups are a very convenient way to see multiple devices perform multiple tests on a single device. If you were watching carefully you’ll notice that this Gmail.com was also a probe group with three member probes. There is no architectural limit to the number of probes you can put into a probe group, but I’ve certainly seen several dozen.

Another useful probe is the MIB Viewer Probe. One of the things customers often ask us is, "Hey I got the MIB from my vendor. Can you make me a probe?" The answer is yes we can. There is a tool which I will talk about, at the end, that will do that, but the MIB Viewer Probe lets you see all the statistics of a device that come out of a particular MIB. I don’t know if you’re familiar with the RFC1213, also known as the MIB-II Probe. It has a number of variables and a number of tables. These tables might be familiar because they are also in the SNMP Table Viewer probe. This probe was generated entirely automatically. It takes about two seconds to generate it. If I click on the disclosure triangle, we see that this is the system description, this is the object ID; you see all this good stuff that’s out of MIB-II. Here’s the interface’s group IF number, the IP group. You can see all the statistics that come out of this device.

If you wanted to make your own probe, typically what you do is start with the MIB Viewer Probe and then prune out the ones that I don't care about. Again, clicking on the view the IF table link opens up the IF table as you saw before. Here is another MIB Viewer Probe. This is for a printer MIB. The printer MIB doesn’t have any scaler variables but it has many, many tables. Who knew that a printer would have this much information to talk about? There are input and output trays, how much paper is in each one. I happen to know that the PRT marker supplies table tells you how much toner is available. It’s unfortunate that the MIB is not very informative. It says that if the level is three, it means there is some left. That's all the information you get, but this is better than knowing nothing.

Pat: Better than waiting until it is totally empty.

Rich: That’s right. The next probe that I want to talk about is the NetFlow Exporter Probe. Intermapper Flows is a component of Intermapper. What is does is it listens for incoming flow information that is sent by routers and switches. NetFlow and S flow are two techniques for finding out who is sending the traffic on the network, how much they are sending, and to whom they are sending it. It’s a really good tool for figuring out… well, yeah our link to the internet is pegged. It is 95% used. The next question you want to answer is who is sending that? The nice part about NetFlow and S flow is you can get that information, and that’s the whole thing that Intermapper Flows does.

The problem is you have to configure the device to send the flow data and then hope that it remembers to do that. How can Intermapper be aware of the flows data arriving? What Intermapper will do is it listens for those updates for the flow records arriving and we see that we’re seeing 322,000 updates an hour; we have received several gazillion packets. The last one was 16 seconds ago. Intermapper is currently receiving data and we’ve been recording since the middle of last week. If we stopped hearing data, the icon would turn yellow and that could of course trigger alerts to let you know there’s trouble.

I saw a question flash up: does Intermapper work with an ASA9? I believe it does. We made a change, I don't remember how long ago. I believe that we can now handle data from ASA firewalls, but that is a good question we can check into.

The last probe in this group is the NRPE. Intermapper, by and large, is agentless. Intermapper doesn’t require you to install an agent or some remote device to monitor it. It uses the facilities that are already inherent in the device to monitor it. However, we are not opposed to using agents and you may be familiar with the Nagios monitoring system. Nagios has a huge, vibrant ecosystem of people building plugins for Nagios that test specific facilities. If Intermapper doesn’t have a probe to monitor something but there is a Nagios plugin, it’s pretty straightforward to install a Nagios plugin on a device and then monitor it. What I am doing here is I have a Nagios plugin running on another computer and InterMapper is using the NRPE protocol—the Nagios Remote Procedure Execution protocol—to send a command to this other computer to say run the plugin called Check Users, and tell me the answer. On this computer, there are currently five users logged in.

Let me show you how to configure this. I will go to “Set Probe.” It says connect to this device whose IP address is… and then tell Nagios on that remote computer to run the Check Users plugin and get the answer back. I am going to change this to the Check Load plugin. It comes up and the status window shows that the load is 1.6 and the other load 5 and load 15 are very low. Nagios is a good way to use plugins to monitor things that Intermapper doesn’t monitor. There is a plugin that we use to look at the age of our SSL certificates on our online store so that you get a 30-day warning to say that your certificate is going to expire in 30 days. That is plenty of time to get with Thawte or VeriSign or somebody and update your current certificate. Those are kind of the lesser known probes. Are there any questions there?

Pat: We have a lot of questions about NetFlow, which is a topic for another day. We will get some information to those people to talk about charting bandwidth, etc., and yes, we can do those things. Yes, NetFlow is an add-on. Question about setting… to get an alarm. On that Host Resources Probe, can I set an alarm at any disk usage percentage? Maybe we can just show them the alarms in the morning and when you can set those.

Rich: I believe it is not per disk. There are thresholds here for the processor load alarm and warning values. If it gets above 80%, put it in warning, which is our name for yellow. If it gets above 90%, put it in alarm, which is the orange color, and the same we have thresholds for 75% and 90% for warning and alarm, memory usage alarm. We also have load average alarms. This section down here is a list of storage units, discs that should be ignored. If I check that, it means that Intermapper will just ignore it. I think it won't even collect the statistics from it. That is how you set the thresholds in the host resources.

Pat: Do we have any Cisco specific Probes?

Rich: First off, let me say that in the network devices there are several probes built in — IPSLA — that are there. If we go to the Intermapper website in the support section there is a probes page. This shows all the contributed probes. These are probes that our customers have come up with. There’s a very long section on switch firewall router monitoring probes and I believe there is even a page for Cisco. There’s a long list of Cisco probes here in addition to other probes for lots of other vendors. These probes here… using a search is your friend here. Does that answer that question?

Pat: That does, I believe.

Rich: The last thing I wanted to talk about is what I call Cool Probes. They are beyond the meat and potatoes; these are the exciting things. All the examples I am going to talk about today have been drawn from the Intermapper Probe Challenge that we ran. This is just a sampling of what's there. I don't have the equipment available to test against, and so these are just going to be screenshots that people provided for the various devices.

We’ve got a probe here to look at ENVIROMUX Mini. It’s somebody that makes an environmental monitor that shows the temperature and humidity. It has also got context to look for things like water on the floor. Wouldn't that be interesting to know about? This is the probe for the Centurion Environmental Monitor. It shows temperature and humidity. It also shows light levels so that if you have a lights out operation and the lights come on in the middle of the night, that might be interesting to know about. NetBotz comes from APC. It’s a very common environmental monitoring tool.

There are a bunch of cool probes for wireless gear. The Aruba Probe here came from Christopher Howard. He was the randomly selected winner of the probe challenge. His probe just looks at the number of access points and the number of authenticated users on the wireless network at UTC. I think that’s University of Tennessee at Chattanooga. He has got this great tool that's… I think it even has a threshold to say that if the number of users drops by a certain percentage, let’s have an alert. If you lose a bunch of people, that's an important thing to know about. Another status window down here shows that this is a Trango [SP] device, just looking at the receive signal level and the bit error rates to let you know how that device is working.

This is a canopy CMM micro. It has a bunch of ports; it is like a switch, but it also has the capability of syncing all these devices with GPS satellites. One of the things that they track is the number of satellites that are visible and the number of satellites that are tracked. They have down to at least microsecond accuracy on their clocks. I think that’s very cool. Network equipment; not only can Intermapper monitor stuff like the traffic in and out of an interface or all the interfaces on a device, people made Probes for the HA proxy load balancer. I believe it’s an open source load balancer and it can let you know when your front ends and back ends are up and running or worse, when they are not.

The problem with the load balancer is that it hands off a load and spreads the load across all the devices that you have available. If one of them goes offline, no sweat. It just spreads the load across the remaining one. You need a tool that is watching how many back ends do I have available. What if I only have one back end available and that fails? Suddenly I have zero. That is not a good number to have. This probe continuously watches for the number of machines in the load balanced set.

In the lower left corner there is a SteelHead WAN Optimizer, Wide Area Network Optimizer. I think they do compression on the interfaces and they can tell how much of a data reduction or bandwidth improvement you are getting, using their box. Intermapper can keep an eye on that. MailFoundry, this just kind of shows good statistics about how long it has been up, what the dates are for the spam and virus updates, and then queue lengths. I think MailFoundry is a mail filtering spam tool where it makes an intrusion detection system. One of our customers made a probe that looked at the key statistics that came out of this box.

The last set of cool probes is the ones that you get excited about and that you make. There are lots of pieces of gear out there. Intermapper can’t monitor them all; we can’t be experts in monitoring them all. One of the things that is very exciting is to see the results of the probe challenge. We had 42 entries. Pat is putting this note out here but here is the URL, it’s bit.ly/InterMapperMIBViewer. We will definitely make sure to mail that out. Basically you paste a MIB file here and the bottom of the window fills with a probe that you can import into Intermapper. That’s the probes. If you need us, let us know and we can help. Thank you for joining.

Pat: Thank you, everyone. There are a number of questions that are specific to issues that you are having and I will make sure that we’ll get back to you with those answers; also some questions about specific probes that are available or not. We will get back to all of you with answers to your questions. Thank you for joining us everyone. Thank you, Rich. I love listening to you talk about Intermapper.

Rich: Okay. Thank you, everybody.

Pat: All right, you all have a good day.