By Ron Gula

April 1999

http://www.network-defense.com

Copyright 1999 Network Defense Consulting

This paper is loosely organized into a list of rules that can be applied to your network operation in the event of an external network scan. Each rule has several examples of what can go wrong and what can go right for a given situation. The rules are also in the order they should be applied in a network security situation. The last section discusses how to handle internal security events at a high level. Please use this paper as a guide. Think about it and what it means for your particular network. It may make the difference between deterring a network attack and having to respond to a network compromise.

Even though network security events are reported in the media and they are a very serious threat, they are not likely to occur to you on a daily basis. (If they are occurring to you on a daily basis you must be pretty good at dealing with them by now and probably don't need this paper.) Most network organizations that suffer multiple attacks and probes experience them in groups. They are not sporadic events.

With this in mind we can roughly classify network probes into two categories. First, the security event could actually be the result of a non-security event. This is known as a 'false positive'. In this case there is nothing to worry about. Second, the security event could have been a probe that tested your site for something. Tests could include determining the type of operating system of a server or even sweeping the network for open ports. If the probe turns up negative, there is a good chance that 'they' won't be coming back. With this situation, there is also little to worry about. At your leisure, you can pursue those responsible for the probe. If the probe seems to have found something that is vulnerable, then you may have returning visitors. Regardless of the outcome of the probe, it requires careful analysis and some judgement calls to determine its nature. That's what the rest of this paper is about.

When presented with a security event, all you really know is that further investigation is required. However, knowing that these things don't happen that often shouldn't cause you to put the analysis of the event off to long. Timely analysis of any security event is the key to quickly finding the real ones from the false positives.

Panic or at least an adrenaline rush is experienced by many network administrators when security incidents occur. Here are some rules of thumb to keep in mind when handling the situation.

Initially, only tell people about the security event on a need to know basis

Telling one person who tells another can cause any office or operations center to quickly fill with people who are not the right ones to deal with the problem. They may also get in the way. Discretion is highly recommended.

When any network event occurs, there is a tendency for normal people to rise to the challenge and work long hours to see the event through. Security events are no different. If an event occurs at the end of a normal duty day or a shift, people working extra hours may suffer from fatigue, irritability and hunger. All of these can impair the judgement of any person. It may also endanger them for their ride home. Later on we discuss the importance of documentation. Documentation and tracking of a security event can make change over between employees much easier.

Don't let people jump to conclusions

During high pressure situations, some people may feel the need to blame others in an attempt to find answers. It's important to downplay any of these comments until all of the facts are considered.

Get second opinions on 'rash' decisions

When conducting a security investigation, it is very important to get input from peers and even management about your current theories and plans. For example, it may seem like a good idea to take the corporate web server offline for analysis, but a second opinion might ask you to stand up a replacement. If probes are occurring in real time, it is also temping to take certain courses of action such as 'hack backs', setting up of honey-pots or even trying to slow the scanning down. All of these actions may have unintended repercussions on your company or network.

Focus on any obvious explanations

It may not seem obvious, but suspicious activity can be explained away most of the time with normal network events. Consider recent network changes or scheduled network testing when analyzing a security event.

Like it or not, as the network security guru, you are also in a position of leadership during security events. If you are not sure of yourself, panic stricken or exhibiting a high level of stress, these traits can easily spread to other people. In ideal situations, the network security staff (if there are more than one of you) is regularly involved in network operations. Knowing your co-workers and making sure that they know you will reduce stress and panic during security events.

When any network security event occurs, you should document it. It doesn't matter how you record the information as long as it is secure, accurate and can be stored for some time. I like to keep a log book, but log books can be lost. Some network shops record suspicious logs directly to CD-ROM. More and more network shops are using ticketing systems to track security events. These have the advantage of existing in a database which can easily be searched for event correlation. You need a solution which is right for you.

Why document? There are several reasons:

People forget, even security gurus

Having a physical record of security events from days gone past is an immense help when analyzing security events of today and tomorrow. Being able to answer questions like, "Has this IP address ever scanned us before?", or "How many other IMAP probes have we had this year?" can only be answered by reviewing historical logs.

Security systems may not keep logs forever

Some security programs do not keep logs forever. They delete old logs to make room for new ones. If you have a system such as this, then those security events from last month might not be in your security system any more. Keeping logs separate from the production systems ensures a greater level of data protection also. What happens if you have a hard drive crash? Many security systems do not use back-ups for data integrity.

It might be evidence in court

If you keep good logs (and good is subject to lawyer's interpretation), then those logs may be used as evidence in a court case. It is very important to keep a 'chain of evidence' with any log system. This means you need to have proper access control on the log information. If the security or integrity of the log can be questioned, then the log may not be admissible in court. Paper print outs and CD-ROMs tend to be more believable than electronic media. Even logging of DNS names instead of IP addresses may be an issue, since DNS name resolution can be corrupted.

It helps you sell security

If you are like most companies, network security is viewed as an important but expensive thing. Firewalls, intrusion detection systems and conferences to Black-Hat are all expensive. Keeping logs can help show management that there is indeed a threat and they are getting their money's worth from you and the fancy security equipment.

Final thoughts:

During the heat of the moment is when it is most important to write down or record any information about a security event. Don't forget to record the people involved, their phone numbers and what they have said. Recording all of the information allows for more efficient processing of the data once it is assembled together.

Later on, when things are less hectic, it is good practice to write up the security event in a one or two page paper. Sharing this paper with any lessons learned can have a very positive effect on your entire network staff. Keeping records of these reports can also help you and possibly your successor.

If this is a network security event, then you are probably obtaining network data from an intrusion detection system, a firewall or some other network element. In most cases you know the source and destination IP addresses involved. This is the 'who' and there are a wide variety of tools that can be used to glean information about the owners of the addresses.

This tool is available on NT and UNIX. The command can convert IP addresses to DNS names and vice versa. It is usually a good place to start to get some quick information about a suspicious IP address. Some care should be taken though. DNS information can be spoofed. One of the neatest hacks is to modify DNS answers to throw network security investigators off of your trail. DNS queries may also be observed by the network attacker. This may alert them that their scanning has been discovered. Many ISPs have taken to naming their dial-up PPP or SLIP addresses with the word 'dial', 'ppp', 'modem' or 'slip'. If you see these, there is a good chance that the source of the scan is a modem user. Similar assumptions can be made for 'home.com' names in which those IP addresses are assigned to cable modems. Consider obtaining a shell account well away from your corporate network to conduct DNS lookups. This may not alert the scanner that you have detected their activity.

Once you have a DNS name, you can query the 'whois' database to find out contact information for that domain. When domains are registered, they usually require some sort of contact information to include a phone number, an email address and a name. Don't underestimate that the name in the 'whois' database could be the fellow doing the scanning. It's unlikely, but it has been known to happen. There are a variety of web interfaces to the 'whois' database and UNIX clients have a built in 'whois' command. Here is example 'whois' output for network-defense.com which has been slightly modified:

It can be seen here that there is a lot of information that can be used for documentation and contact purposes. There is a home address in case you want to launch Tomahawk missiles and a telephone number for voice contact. There is also an email address and a listing of which root level DNS servers 'take care of' the queried domain.

The ARIN [1] database is publicly available at http://www.arin.net/whois/arinwhois.html and allows one to find out the owners of a particular IP address. When DNS has not offered an answer to what a search, doing a reverse IP lookup at ARIN can still provide useful information. Here is an example ARIN search output for 24.3.17.92 which is a home.com IP address:

This query provides us with some interesting information. First, we know that the IP address is part of the home.com (@Home) ISP. A lot of hackers get cable modems. A lot of hackers hack people with cable modems. The second pieces of information tells us that @Home is using the Comcast Cable company for local access. One could even venture a bet that the 'MD' referred to Maryland and the 'HWRD' referred to Howard county. It's dangerous to assume things like this, but in some cases it makes sense. For instance, not every net block that has a 'TX' in it is from Texas. Use good judgement.

The Web

If you know the IP address of a network probe source, you may want to try to look for web servers on that network. An easy way to do this is through guess work. Let's say a target's DNS name resolves to name1.place.com for an example. Attempting to go to the web server at www.place.com may provide you with the network's public web server. In some cases, using a tool such as Nmap[2] to scan a class C network for any web server can be fruitful. This should only be done as a last resort. In ISP networks, scanning a class C network may not bring you any closer to information about the scanner as you connect with one unrelated user's web page after another. The goal here of course is to discover some sort of contact for you to voice your distress over the network scanning.

It is also very important to understand exactly 'who' locally is involved in the scanning. Recording IP addresses is not enough. What is the second level relationship between them. Is this a sequential scan? Are these systems mail systems? Do they run IRC daemons? Are they all DNS servers? You get the picture. Figuring this out may allow you to understand what drew the network scanners to your network in the first place.

Determining what a network scanner is probing for can be a very difficult activity. I offer some general guidelines to analyze suspicious activity, but no one can be certain exactly what the intention of a particular scan is without asking the person doing the scanning. More importantly, we discuss a variety of scanning techniques and the type of information that the scan returns. Using this to analyze activity on your network can be interesting. We do not consider techniques for direct vulnerability probing because this is a very cumbersome topic and a lot has been written about it already.

Much suspicious network activity concerns discovery of target network and systems by malicious individuals. Sophisticated attempts exist that can map out a network. Attackers equipped with knowledge of a network's hierarchy can plan effective attacks. Here are some common and not so common topology scanning techniques that you may observe on your networks.

Another piece of information that network probes attempt to discern is the type of operating system of a given IP address. There are a variety of methods that exist to do this and we discuss them here. These scanning techniques are very popular in the hacker community.

Many network probes are attempts to discover active services on a network or on a particular server. Typically, these scans are automated and connect with ranges of ports for a given IP address. They then report the ports that were open or active. Network attackers can then select exploits based on the active ports found. Here are some different port scanning techniques that you may encounter when analyzing network probes.

Some scanning is only focused on one thing, finding vulnerable systems. This type of scanning is subjective, but basically involves a lot of automation. There are different strategies used by network scanners to find vulnerable systems. Here are a few of them:

Normal RPC scanning (rpcinfo -p) sends a query to the rpcbind program which is more commonly known as the portmapper. This query can ask for a list of all other RPC programs on the server. RPC programs historically have always existed around port 1024, or usually below that. Several years ago, Solaris and many other flavors of UNIX started to run RPC programs around port 32771. Many packet filter and firewall designers were unaware of this situation and deployed access control lists that did not prevent connections to these ports.

In the case of 'normal' RPC behavior, it is possible for an RPC program to be assigned a port slightly above port 1024. If the firewall rules do not prevent this sort of connection, then the RPC service is directly accessible from external IP addresses. The same goes for RPC programs that are assigned ports above port 32771. These are a problem because they may be directly connected to.

Network scanners may search for these 'high' RPC ports by using port scanning to identify ports and then conduct RPC queries to any ports that are open. If an RPC service is identified, it may have a vulnerability that can be exploited.

Figuring out 'why' a particular suspicious network event occurred can be a very challenging and daunting task. The important thing to realize (and sometimes this only comes through experience) is that some things can't be explained. When an explanation seems to elude you or your staff, don't let it consume more and more resources. Prioritize your investigation and don't let it be hindered by not understanding exactly why something happened.

For example, a server may have been rebooting sporadically and your staff suspects that a new denial of service attack is being used. Sniffing, intrusion detection and system security analysis indicate otherwise. Finally, a maintenance person discovers a faulty power supply. This example may seem trivial, but occurs many times on modern networks.

Another example of a 'why' explanation is to consider the source of the security information. Many network management intrusion detection systems are complex and have many threshold levels for causing alarms to trip. If these threshold levels are drastically changed, then all of a sudden there may be many system alerts and possible intrusions. Try to identify any recent system changes that could attribute such activity.

Unfortunately, all suspicious network activity can not be ruled out. Network probes should be considered hostile until you know exactly what you are dealing with. Why would someone want to break into your network? You tell me! There are many reasons for breaking into networks. Are they looking to deface a web page? Do you have political enemies that are network savvy? The good news is that you may have detected their early network probing efforts and now you can act accordingly.

Knowing when something happened allows you to construct a timeline or order of events. The order of events may be very crucial for determining the exact steps taken by hackers using network probes. Did they attempt a DNS zone transfer before we deployed the new DNS server? Did the scan on the internal web server occur after the firewall changes? These are example questions that depend on accurate time reporting.

One key to determining the order of events is to use a common time source. The Network Time Protocol (NTP) is very reliable, accurate and resistant to a variety of attacks. NTP can be used to synchronize all network and server nodes with a very accurate and uniform time source. With all of them synchronized, log analysis becomes a lot easier. I also recommend trying to keep all of your systems on the same time zone clock. If you are unlucky enough to have servers in multiple time zones all keeping local time, then log analyses can become very cumbersome. With one unified time clock, it may be easier to detect network wide probes of your systems.

Having a good and reliable time system is also crucial if you want to enter any of the logs into a court case as evidence.

When analyzing network probes, the question of 'where' is often overlooked. We are referring to the physical and electronic location of all of the computers involved, including the security systems. Identification of system locations is important to help understand and analyze recorded data. It may also explain why some information is missing or inaccurate.

Confirmation of physical and logical location is often necessary when conducting an investigation. Your network maps may not be up to date. There have been cases when a simple network connection mistake has placed a vulnerable server outside of a protecting firewall. It may be helpful (and surprising) to obtain a remote account and conduct network mapping probes to see how things are connected.

The use of Ethernet addresses can also be of great use when identifying the sources of spoofed IP packets. Although Ethernet packets can be trivially spoofed locally, they can't be spoofed across the Internet. This can help you determine which router or switch interface spoofed packets may have originated from.

Every network security program should conduct regular vulnerability testing using commercial and free network security tools. It is one of the best ways to determine if you are at risk to common network attacks. But what if someone is probing you on a port that you have never heard of? What if they are probing you on a port that you thought had no security problems? There are several things you can do.

The first thing to do is to identify the port if it is not well known to you. Set up a network analyzer and collect some traffic on that port. Analysis of the traffic may help identify it. There are a variety of proprietary network protocols such as PC Anywhere. These may seem very strange and unfamiliar to you. Another technique you could try is to take an inventory of all of your network software. This is unrealistic in a short time frame, but usually identifies a variety of programs (and protocols) that could cause the traffic you are looking for. If all else fails, consult the Internet news groups. There is a lot of open discussion about network protocols and you may be lucky enough to find one about yours.

For example, when I first got my cable modem, I saw a variety of traffic to my computer that was UDP and port 22. All of the packets contained two bytes of ASCII data that were 'NQ'. I had no luck finding out what the protocol was until I stumbled onto a firewalls discussion list that was talking about PC Anywhere.

Once you know the target, try to determine the threat. Again, this is very subjective. I try to look for recent vulnerabilities described in the various network security groups on the Internet. Recently, I've begun to observe scanning for port 21 on a variety of firewalls and intrusion detection platforms I have access to. Prior to this about three weeks ago, there were several posts that could remotely exploit the Washington University FTP daemon. It makes sense that people are looking for FTP servers to attack because this is a new vulnerability.

Of course, the only way to know that you are vulnerable is to actually test the problem. You can be safe and disable the service if you don't need it, but not everyone has that luxury. Getting your hands on the latest exploits is usually not a problem. There are a wide variety of full disclosure security mailing lists and web sites. There are also a wide variety of consultants available to help you with this sort of thing. Testing your network lets you know what hackers and network scanners may see or already have seen.

Don't believe that you may not be vulnerable to an attack just because an exploit has not be created for your particular operating systems and platforms. This is a trap that many system administrators fall into. For example, there are all sorts of remote buffer overflows written for the LINUX platform. Just because you are running a SPARC station, does not mean you are safe and out of harm's way. Ports of exploits to new systems can appear at any moment and any hacker worth his or her salt should be able to covert exploits between systems.

If you are vulnerable, then you may have a problem. First of all, you've seen scanning and you think you are vulnerable. It would be wise to approach the box with caution as it may have been compromised. Second, you need to figure out what sort of impact that box has on the network so you can decide what actions you want to take to secure it. Can you down the box to make some patches? Is the box a single point of failure for the network? Can you protect the box by making a firewall rule change someplace else? The important thing here is to mitigate any negative impact on your network.

It may seem surprising, but many security problems and events do not get reported for a variety of reasons. These reasons are a combination of technical and political factors that prevent a clear reporting system. We will discuss some of the specific reasons that security incidents don’t get reported.

Many system and network administrators do not report security events because they believe it will reflect poorly on them. An administrator may have previously boasted that "no one" can break into their network. Managers need to realize that these claims are ludicrous and should expect to see monthly security reports detailing suspicious network activity.

Chain of Command

Who should a security event get reported to? If an organization has not stated how to handle security events, then when one happens it will get handled ad hoc. Most people correctly assume they should tell their immediate supervisor. This is true, but if a security department exists that has been trained to handle security situations, then they may be kept out of the loop. It may also be unclear what your supervisor may do with the information. (They should inform their supervisor and so on.)

Management Indifference

It may be difficult to explain exactly why a specific type of network activity is 'bad' to inexperience management. Technology marches on and it is difficult to keep up with it for some people. However, this should not prevent an employee from reporting a possible security situation. Don't be afraid to use analogies to get your points across about network probes. Be prepared to demonstrate how an attacker could be probing your network to gain proprietary information.

Management Overreaction

The opposite of the above example is true. If a network manager is not accustomed to experiencing network probes and scans, then the first time they occur may be traumatic. The best way to handle this is to be up front about the threat to your networks when you talk with your manager or supervisor. Be wary of any drastic measures taken by your management such as calling the FBI or the newspapers. That may not be the best course of action for the given situation.

Once security probes have been noticed, the most important thing that a network administrator can do to their network is to make sure it is monitored. Depending on you network, you may be able to leverage your operations center. You may also have to place extra intrusion detection sensors to monitor uncovered network sections. Some people may even wish to deploy packet analyzers. Regardless of your technique, it is important to keep a watchful eye for any suspicious activity. The heightened state of alert exists because your network has been probed and you have made a determination that a network attack may be imminent.

When leveraging any operations center, its important that you give them as much information on how to contact you and to accurately describe the security situation. In security situations, most operations personnel will contact you when anything suspicious occurs. Unfortunately, this may include normal network performance problems such as routers failing and Windows NT machines blue screening for no reason. Giving an operations center access to the intrusion detection systems is also a good thing. Hopefully, this has already occurred on your network. Leveraging quality 24x7 people can only be effective if they are properly trained and have well planned security response procedures.

If you choose to deploy extra intrusion detection infrastructure, then you really have two choices. You can change the current rules used by the IDS to be more sensitive or you can deploy completely new systems. All IDS products are 'tuned' to a particular network. Thresholds need to be set and when they are exceeded, alerts and events are recorded. In a hostile environment where an attack may be imminent, lowering these thresholds may record extra probing from an attacker. It will also record more false positives. Deploying more IDS platforms may also be an option. If you have portions of your network that are not covered by the current set of IDS platforms, then deploying additional sensors may expose further attacks or network probe activity.

Some security gurus may also wish to deploy a set of packet analyzer filters. The most common of these is TCPDUMP. These packet analyzers are deployed with similar strategies to packet based IDS platforms. You want to expose them to as much traffic as possible. In a switched environment, there is a possibility that you may want to run TCPDUMP on a web server or some other isolated production server. If you do this, keep in mind that you should try to obscure the sniffing program. Most network attackers are very familiar with network packet monitors. If they compromise a server that has a network analyzer running, they may find these logs and delete or modify them.

When constructing packet filters to monitor suspicious traffic there are several strategies. You can log by destination port, destination address, source address or for a specific packet signature. Watching for specific destination ports can be very effective if you are in an environment where those ports are not active. Consider monitoring the network for all IMAP traffic which may be a service that a network scanner is targeting. If you do not have any IMAP servers, then scan attempts for this service will stand out. Sniffing all HTTP traffic in a web environment would not be a good idea unless you had some sophisticated analysis tools to deal with the barrage of recorded data. The same rules apply to destination addresses. If you have busy server, then logging all traffic to it may not be a good idea. But if the server is not heavily loaded, then recording all traffic may be an option. Sniffing based on the source IP address or source IP network may also find interesting activity. And finally, if you know or suspect the scanning techniques in use by a hacker, consider writing specific packet capture rules. A very easy example of this type would be to record any FIN-SYN packets. Hopefully your IDS is doing that already.

When you choose to contact another network organization that is part of the Internet, you must keep a lot of things in mind. The most important thing you can be is organized and present your information clearly. Separate your conclusions from your solid data. Allow the person you are contacting to think about the situation for a moment. Don't demand immediate action. Here are some other guidelines to use when making contact:

Other security people are just like you. They get sick, go to lunch and take vacations. It is not unreasonable to contact an organization and find your security point of contact unavailable. In this case, you may have to deal with someone who is less skilled in security terminology or network administration. When this happens I like to ask for anyone who operates the servers or routers. Usually these people are very capable and can help analyze security events.

If you attempt to contact foreign countries to chase down suspicious events, you may encounter someone who does not speak your language. English is very common worldwide, but there is no guarantee that you can use it to tell someone about a security incident. Some cultures can read English better than listening to it so consider sending emails or even fax communications. If you have any non-English speaking assets in your organization, it may be a good opportunity to tap them for translator duty.

Most commercial networks have some sort of 24x7 operations, but security gurus still seem to keep normal daylight hours. The person you are calling may be asleep. Realizing this, you may be lucky enough to get an organization that realizes the importance of your call and follows a procedure to wake the key individuals up for some analysis and hopefully some answers. On the other hand, you may also call and get an answering machine. In this situation you should leave concise information and multiple points of contact on your end. Remember, they may call back when you are asleep.

In some cases, the person listed as the security point of contact may actually be responsible for the scanning of your network. Many hackers get jobs at ISP and other commercial network organizations. Many of them even get jobs in network security capacities. It is not unthinkable that these individuals may abuse their powers. Some telltale signs of this may include apathy to the situation, denial of the events and even open hostility. If nobody asks you for your name, telephone number or email address then something may be amiss. If the person has any knowledge of the scanning that you did not volunteer, then this may also be an indicator they are hiding something.

Some network organizations are very busy. In rare cases, the people you contact won't do anything. Some ISP networks have an attitude that they provide unrestricted connectivity for their users and aren't responsible for their actions. Other organizations will help you, but are so busy with other security events that it may take some time before they can handle your complaint. Providing clear and concise facts about the incident can make their job easier and get quicker results for yourself.

Depending on what the situation is, your information could get some people fired, kicked out from the ISP and even sent to jail. I've had at least one ISP tell me they have "black listed" an individual for hacking. Some security events are honest mistakes, but it is possible for your point of contact to overreact and take some action that you are not comfortable with. For instance, an ISP may place a rule in their outbound packet filters that prevents any traffic to your network from theirs. This is very secure, but now nobody from that network can get to your web servers, send email or play Quake.

During many security situations, the people you are contacting may have suffered a security compromise. If this is the case then either they know it or they don't. If they know about the break-in, then they may be forth coming with all sorts of information. They may also be deceptive and try to hide the incident. If they do not indicate that they have been compromised, but you think they have, it's very important to try and get this information to the correct people. Telling a receptionist isn't going to help the problem get fixed unless you need his or her help to find the correct people. Taking a chance and calling a webmaster, the sales line or even customer support will usually get you within one or two phone calls of the correct person to talk with. They may ask you to work with them in analyzing and tracking down hackers. Be carefully not to discus any proprietary network information or security invents that do not directly involve your point of contact. This will shield you from inadvertently compromising someone's right to privacy.

It's been said before, but I'll say it again. All of the conversations, intrusion detection events, logs, packet dumps and reports that you or your staff are involved in may be admissible in a court case. Be careful who you give logs to. You may or may not want your logs used in a court case or even have you or members of your security staff called as witnesses in a trial. I'm not going to preach moral values of network security, but it might not be in your best interests to assist some other network organization half way across the country to prosecute a hacker. Of course, the corollary to this is true also. Wouldn't you like it if another security expert was willing to testify that they detected probes from the individual(s) on trial for breaking into your network? Another aspect to keep in mind is to only give out log information to people who truly need access to it. The data can contain a variety of privacy information such as passwords and network usage. Only give the data to people you feel have a need to know and can effect responsible changes to prevent further network abuse.

One last comment is to think twice before sending security event information over email. Email can be easily compromised. However, monitoring a large volume of email traffic may be outside the scope of a hacker. I like to email points of contact that have multiple accounts and choose one that is not on the possibly compromised network. Also consider the use of encryption.

Final thoughts:

Be professional when you handle yourself with other people outside of your organization. If you are in the business of selling Internet access, professionalism and common courtesy can take you a long way. The security community is very small and you may be surprised how often you'll run into other security people from different organizations.

When security events occur, you may want to consider getting the word out to other Internet groups. There are a variety of outlets for this sort of information. What is right for you depends on the situation. Consider telling other organizations through the following means:

The FBI and other authorities are continually getting better at tracking down hackers. The United States military also has an excellent program to track down suspicious network events. Regardless, the security events that you wish to bring to the attention of these organizations should be something new or serious activity. What is this sort of activity? I would say any activity that spans multiple networks or organizations. An example of this could be organized targeting of web servers that accepted credit card data at multiple locations at multiple companies. Another reason to contact the authorities is if you have information that could be useful to an ongoing investigation. Many security events are covered in the media and your information could be related and prove useful.

Computer Emergency Responses Teams track network security events and in some cases, will directly respond to them. Most people have heard of CERT advisories. Some of these advisories do not have vulnerability information, but warn of hacker activity. Information that you report to CERT can help produce better and more timely advisories. If you think that network scanners are looking for a new vulnerability, then CERT can use this information when issuing new vulnerability advisories. There are many different CERT organizations. If you are in the United States military, you should report security incidents to your branch's CERT agency. There are many CERT organizations that span international and corporate entities. Choosing which CERT agency to report an incident is sometimes a difficult task, but can get you more focused support.

In some cases, reporting scanning activity directly to a security mailing list can be beneficial for a variety of reasons. First, the information is very timely. Readers of the mailing lists are there to exchange information about new security vulnerabilities and your report of network scanning may spawn discussion as to what it could mean. Second, other people that have had similar network security events may come forward with other information. They may post to the mailing list or contact you directly. One word of caution though, hackers and malicious individuals also have access to these mailing lists. Don't disclose any information about vulnerabilities in your network or unique information about your network such as IP addresses or domain names. You may even want to get a second shell account just to send email to these mailing lists. Free web based email services such as Hotmail are very useful for this sort of activity.

There are also a wide variety of security web servers on the Internet today. These sites track vulnerabilities and hacker activity. Writing a story or submitting some content about recent scanning activity may help find a solution to the problems. Follow the same sanitation rules with your content to prevent drawing undue attention to your network.

In some cases, it may be appropriate to raise the level of concern at your company. If more money or resources should be involved with network security, then telling Vice-Presidents and other upper corporate management may get you some results and raise awareness. In some cases, CEOs and CIOs may have misconceptions about the current level of network security. There is nothing like a security event to bring a little reality to the situation.

Bringing the media into a security situation is usually a recipe for disaster. Most security experts agree that the last thing you should do is to tell other people how secure you are or tell them that you are under network attack. This is a magnet for hackers. You might as well hang a sign on your web page that says, "Hack Me!". Realistically though, the media can be used to tell your customers about your network security if it is done right. If you have firewalls and intrusion detection systems, then there is really nothing wrong with telling people that you have firewalls and intrusion detection systems. Just don't make the leap and state that you have "impenetrable" security. With security incidents, care must be taken to present a positive image about your company or network without reflecting poorly on yourself. Press releases are an excellent tool for this purpose.

Any investigation of suspicious activity should attempt to conclude what attracted the network scanning in the first place. By attempting to discover this information, you may prevent some scanning in the future. Hackers tend to be opportunistic. Unless they have a bone to grind with you, most malicious hackers are simply looking for vulnerable servers to compromise. Here are some things that opportunistic hackers look for:

One of the easiest ways to quickly assess the security of a network is to visit each of its web servers and see how many of them display the default Internet Information Server or Apache splash pages. These can usually indicate recently deployed systems, development networks or systems that are not maintained that well. All of these may have security problems. The combination of the security possibilities and the lack of maintenance can be a green light to most hackers.

If your DNS server has names for network nodes such as 'test-linux-1', 'guest' or even 'web-development', then these names could attract a hacker's attention. Naming things by their function ('router1', 'firewall-3', etc.) tells your staff and the entire world exactly what the operation of a particular network node is. If an attacker is looking for a particular exploit to try, they may consult your DNS server to find a target. In some cases, poorly maintained DNS entries can also indicate a poorly administered network. Many hackers (correctly) assume this also means poor security. A good example of a DNS problem is the advertisement of RFC 1918 addresses through normal DNS queries.

Many hackers will conduct a quick limited probe to find out how secure or unsecured a given network is. What they may be looking for is a group of computers not protected by a firewall and offering a variety of services such as DNS, TELNET, FTP, SMTP, HTTP and RPC. For example, if a hacker does conduct a quick port scan of a web server which results in only port 80 being active, then the hacker may conclude that the site is reasonably secured. Compare this to a port scan of the web server that discovers twelve active ports. Each situation represents a different level of perceived security posture.

Nothing draws attention to your site like the media. If your company or clients to your company have media exposure, then it follows that some hackers will get the idea to try and scan or hack you. At least one ISP I am aware of was dismayed when one of its customers actually challenged hackers to break into their web server. The ISP suffered fratricide when attacks on the web server spilled over to the operations center.

Running IRC servers is an easy way to attract hackers. Many hackers stereotypically spend hours on end chatting about a variety of topics. When hackers disagree with each other, they will use a variety of denial of service attacks to disable the other person's network access. In some cases direct attacks on the other person's network will also ensue. Having network users that spend time in IRC chat discussions is also an advertisement to hackers. The debate to allow or not to allow IRC access from your network may be abated by providing free access to a shell accounts outside of your network, or by placing an IRC server outside of your firewall.

Nothing attracts hordes of hackers like a hacker web page. Many of these pages suffer network attacks. Many of these pages are also seen as 'trophy' pages that various hacker groups would like to claim credit for hacking. If you run a network that has this sort of information, you may be experiencing a certain level of security events simply because people are poking around your network looking for ways to break into the hacker web page.

If you have a hacker working for you or at your company, then the suspicious probes you may see could be coming from the hacker during off hours or from the hacker's friends (or enemies).

Not all hacking events are random acts of opportunity on a hacker's part. The suspicious events could indicate a coordinated effort to compromise the security of your network to obtain some level of information. This information could include email, trade secrets, plans, spread sheets or any number of other proprietary pieces of information.

If you suspect that an internal person in your company is conducting network probes or doing something nefarious, then you should follow some loose guidelines when analyzing and presenting evidence.

Do not spread rumors or false accusations about any individual suspected of abusing their network access. This could be viewed as slander in a court case. When presenting suspicious internal security information to other individuals, do not associate any suspected individuals with the data. Try to sanitize the data before presenting it to any group of people. Once someone is branded a hacker, they have a tough time loosing that image even if they are shown to be innocent. Also keep in mind that other people may not have the same opinions about security as you do. Depending upon their position and yours, you may have to work out an agreement or get an interpretation of your network security policy.

Try to save as much information about the suspicious events as possible. In most cases, these events are more powerful and damaging (and damning) than your word alone. Make sure that the records are in a secured place where they cannot be tampered with. Physical media such as CD-ROMs are ideal for this sort of application. These records will become very crucial in any decisions to terminate an individual. They may also play a role in any police or FBI investigations.

Most companies have an HR department that deals with personnel issues. Some companies also have a corporate security group that handles internal security issues. Involving both of these organizations can help protect your interests as well as the company's . They may also have other information that you don't have. Consider the example where an employee has submitted their two week notice and has started to hack the network form the inside. This is a very serious situation which you may have only perceived as a minor network nuisance. If you are the technical subject matter expert on security, you may be asked by these groups to assist in an investigation.

We are all human. Chances are you would not like your actions to be monitored on a regular basis. Small acts such as taking office supplies for home use seem much more serious when an investigation is underway. The same is true for network security investigations. For example, just because a person is visiting hacker web sites does not mean they are a hacker. Focusing to much on a person's network activities can provide a myopic view of the big picture.

Once the proper authorities are involved and there is agreement that the individual is doing something strange on the network, the individual should be challenged. If this is done correctly, then the individual will be surprised and may give themselves away through statements they make. That is of course if they are hiding something. You should carefully use these meetings to find out what the individual was doing. Ask direct questions about network usage and try to find anything that can explain the suspicious network traffic. If you hit a brick wall, you may want to tell the individual what sort of traffic was observed. Stereotypically, this is when the person confides that they let Bruce from the accounting department use the computer during lunch time.

You should also have a physical inspection of the computer performed while the meeting is taking place. These may seem like very extreme measures, but if an individual is to be terminated immediately, the chance to discover Trojan horse programs or logic bombs needs to be addressed. Also a physical inspection of their computer may provide contradicting evidence to their testimony or explanation of their activity.

Final thoughts:

Computer analysis should best be left to computer experts. However, since security events involve people, the best chance of catching an internal hooligan may come from an ex-police officer or an ex-investigator. People with these backgrounds and some good computer knowledge are becoming very popular to handle computer crime investigation. If your company has these assets, then I strongly urge you to involve them in computer security incidents.

Following the steps outlined here may save you a lot of time and headaches when dealing with network probes and security events. I urge you to discuss these concepts with your staff and management. You should also discuss this information with any other people in your organization associated with network operations.

The one thread that holds all of these rules together is 'common sense'. Analysis and termination of network probes and security events occurs in the human world. It does not occur in the computer world. Dealing with other people can be tricky. Be nice. Be courteous. There are some people at your company and at other networks that won't be easy to work with, but this is a fact of life that usually shows itself during a security event. Act professional and stick to the facts. For a variety of reasons, many people tend to forget these when placed under pressure and in high tension situations. Good luck!

References:

1. ARIN is the American Registry for Internet Numbers. http://www.arin.net

 

2. Nmap is a network and port scanning tool written by Fyodor (fyodor@dhp.com). It is available at http://www.insecure.org/nmap/index.html

 

3. QueSO can fingerprint operating systems remotely using a variety of techniques. It is available at http://www.apostols.org/projectz/queso/

 

4. Klaxon is a tool that detects port scans. Available at the COAST security archive. ftp://coast.cs.purdue.edu/pub/tools/unix/klaxon/klaxon.tar.gz

 

5. Van Jacobson, traceroute documentation and source code, Lawrence Berkeley National Laboratory

 

6. David Goldsmith and Michael Schiffman, "Firewalking: A Traceroute-Like Analysis of IP Packet Responses to Determine Gateway Access Control Lists", Cambridge Technology Partners