Friday, October 21, 2011

Network Monitoring - It's Your Livelihood

What are your daily duties as the VoIP network operations director? You have to keep your network up and running. You have to answer calls which may relate to situations like “X location is down” or “Y location is slow”.  My we suggest that you proactively monitor your network as described below and perform tasks like:

  1. Monitor your network and take actions with respect to situations like device and line failures.
  2. Analyze line/physical facility utilization, errors on the facility and be sure about network performance and conformance to SLAs.
  3. Be aware of what "talks to what" and when?  Be sure how much bandwidth is needed for every single application riding your network (and the networks you traverse.)
  4. Know your exact data flows over your networks.
If you have all this information at your disposal, people will think twice before they point finger at you. 

But how can you achieve this?

You need a phased approach to understand network monitoring. I am not talking about network layers, but network monitoring layers. We have to involve deeply to monitoring layers before deciding about network monitoring software needs. A simple summary could include these:

  • Preconditions of network monitoring.
  • Up/Down monitoring
  • Performance Monitoring / SNMP monitoring
  • Who talks with whom? / Netflow monitoring
  • Data capture / Data sniffing
Preconditions of Network Monitoring
Network documentation is essential to monitor a network. Trying to set up network monitoring tools before going through the documentation is a complete waste of time. You will see everything green on the screen, but this maybe due to one of the redundant lines that are down. You will sit staring without knowing what is happening. Always remember, documentation comes first and everything follows.
Suggested Network documentation tools: Powerpoint/Visio, NetViz

Up/Down monitoring
Design a map in which you can see some red and green lights glowing. Green means up and red means down. It is simple yet powerful. You will immediately come to know that there is some problem if the red light glows.This is based on ping. Almost every IP devices support echo/echo reply. So, you can monitor all IP devices in your network by using ping.  Go one step further by monitoring one application at a time present on a device instead of whole device. All network applications utilize TCP/UDP ports. You can monitor the applications by trying to access with telnet to its TCP/UDP ports. The port being open suggests that the application is running
Suggested monitoring tools: WhatsupGold, nmap

Performance monitoring / SNMP monitoring
The lines are up, the devices are up, but life is not perfect. People may complain about the performance of data lines, but are they saturated or do they have plenty of spare bandwidth?  Is there packet loss on the lines? Are routers running out of memory? We need SNMP to monitor the heart beat of the network.
Suggested monitoring tools: MRTG, Solarwinds Orion, PRTG

What is "talking" with what? / Netflow monitoring
You may realize that the line is full, but is someone or some applications increasing traffic load enormously. Who are they? Is it necessary traffic? In some devices, by using “ip accounting” command you can get an idea of current traffic sources and destinations. Nevertheless, to analyze and to optimize the traffic we need flow monitoring. We need to know source and destination IP addresses and TCP/UDP ports and number of packages/bytes.

Everyone blames the network speed until you publish an network usage report that clearly shows only 15% of the traffic is ERP traffic and rest comes from Internet access.You should know that flow monitoring tools requires more server resources, since they collect enormous amount of data.
Suggested monitoring tools: Fluke Netflow monitor, Paasler

Data capture / RMON – Sniffer tools
Sometimes you need to observe the exact data flow on the line and not just information about it. Just have a look at this sample scenario. After you find out that the web service causes inappropriately high network traffic, the owner of the application just can say “No, we are not pushing this much of data to network. We just respond Yes or No in this web service and it is just 100 bytes”. Therefore, you should sniff the data flow on the line. Maybe, you will find that web service responds yes or no (100 bytes) and with the definition of web service (6 kilobytes).
Suggested monitoring tools: Wireshark, Palladion

You can have a look at Network Monitoring Tools in Stanford University web site for a superb list of network monitoring tools. You can find another tidy list at Network Traffic Monitoring in Alan Kennington’s

Monday, October 17, 2011

IP PBX Toll Fraud - Everyone Is At Risk

It seems no one is safe. Every IP Telephony Service Provider [ITSP] I have talked to has suffered an IP PBX hack. End Users of VoIP phones most commonly do not remember or know how to change the password on their PBX or voicemail account. Default passwords for a PBX are usually the last four digits of the phone extension so hackers can easily cycle though to determine a weak or discoverable password. Once into the PBX, they can originate calls from anywhere on the Internet and pump traffic volume to numbers that will realize them fraudulent revenues

Voicemail can be configured to dial out eg when you hear the greeting “please wait while we attempt to reach your party”, of course the voicemail system is making an outbound call which is setup by the hacker to redirect to their intended destination.

The fraudster can resell such phone capacity. One of the nice advantages of VoIP is its built in features for Moves, Adds and Changes. When the time comes to move office, you can just pack you VoIP phone and take it with you. Plug it into the internet or your companies IP cloud and it will register with your IP PBX and you can make calls. If you travel, you can take your VoIP phone with you and plug it into the internet in your hotel room and makes calls as if from your desk. So The PBX has no concept of your physical location or who is using the phone and so fraudsters, once they hack your PBX can make calls from anywhere, using your account.

The Telecommunications industry has annual revenues of $2.1 Trillion. Telecom fraud is calculated to cost the telecom industry $40 Billion each year.

The calls come out of your IP PBX, to your ITSP, offering a SIP Trunking service (i.e. a service that routes calls from a VoIP environment to the PSTN and therefore to any expensive international destination offered. So your ITSP receives an invoice from the International carrier for all these international calls. Fraudsters often choose weekends or other times outside business hours to attack. This way, the attack goes unnoticed and the account is beaten to death whilst no one notices or cuts it off.

“How to lose your year’s profits in 15 minutes!” was the way one of our customers described Toll Fraud. The ITSP receives this rather large bill from the international carrier. If he demands payment from the enterprise which has allowed their IP PBX to be hacked, he will surely loose a customer. Small companies who typically have small telecom bills can afford suddenly to pay up for a $20,000 to $30,000 bill. Many events run into the hundreds of thousands.

Although the ITSP caught in the middle may be to share some of the costs with their customer and the international or long distance carrier, they cannot afford to lose customers and will not often impose the charges on their client. So this is a loss the ITSP usually has to take on the chin.

How can we detect it in real-time and turn it off in real-time to prevent the cost leakage? Advanced monitoring systems are now available which will not only detect the fraud attack in real-time but will also turn if off.