Chapter 1, "Network Management and Troubleshooting" lists several reasons why you might not know the details of your network and the computers on it. This chapter assumes that you are faced with a networked computer and need to determine or reconstruct its configuration. It should be obvious that if you don't understand how a system is configured, you will not be able to change its configuration or correct misconfigurations. The tools described in this chapter can be used to discover or change a host's configuration.
As discussed in Chapter 1, "Network Management and Troubleshooting", if you have documentation for the system, begin with it. The assumption here is that such documentation does not exist or that it is incomplete. The primary focus is network configuration, but many of the techniques can easily be generalized.
If you have inherited a multiuser system that has been in service for several years with many undocumented customizations, reconstructing its configuration can be an extremely involved and extended process. If your system has been compromised, the intruder has taken steps to hide her activity, and you aren't running an integrity checker like tripwire, it may be virtually impossible to discover all her customizations. (tripwire is discussed briefly in Chapter 11, "Miscellaneous Tools".) While it may not be feasible, you should at least consider reinstalling the system from scratch. While this may seem draconian, it may ultimately be much less work than fighting the same battles over and over, as often happens with compromised systems. The best way to do this is to set up a replacement system in parallel and then move everyone over. This, of course, requires a second system.
If rebuilding the system is not feasible, or if your situation isn't as extreme as that just described, then you can use the techniques described in this chapter to reconstruct the system's configuration.
Whatever your original motivation, you should examine your system's configuration on a regular basis. If for no other reason, this will help you remember how your system is configured. But there are other reasons as well. As you learn more, you will undoubtedly want to revisit your configuration to correct problems, improve security, and optimize performance. Reviewing configurations is a necessary step to ensure that your system hasn't been compromised. And, if you share management of a system, you may be forced to examine the configuration whenever communications falter.
Keep a set of notes for each system, giving both the configuration and directions for changing the configuration. Usually the best place to start is by constructing a list of what can be found where in the vendor documentation you have. This may seem pointless since this information is in the documentation. But the information you need will be spread throughout this documentation. You won't want to plow through everything every time you need to check or change something. You must create your own list. I frequently write key page numbers inside the front covers of manuals and specifics in the margins throughout the manual. For example, I'll add device names to the manpages for the mount command, something I always seem to need but often can't remember. (Be warned that this has the disadvantage of tying manuals to specific hardware, which could create other problems.)
When reconstructing a host's configuration, there are two basic approaches. One is to examine the system's configuration files. This can be a very protracted approach. It works well when you know what you are looking for and when you are looking for a specific detail. But it can be difficult to impossible to find all the details of the system, particularly if someone has taken steps to hide them. And some parameters are set dynamically and simply can't be discovered just from configuration files.
The alternative is to use utilities designed to give snapshots of the current state of the system. Typically, these focus on one aspect of the system, for example, listing all open files. Collectively, these utilities can give you a fairly complete picture. They tend to be easy to use and give answers quickly. But, because they may focus on only one aspect of the system, they may not provide all the information you need if used in isolation.
Clearly, by itself, neither approach is totally adequate. Where you start will depend in part on how quickly you must be up to speed and what specific problems you are facing. Each approach will be described in turn.
Even if you plan to jump into the configuration files, you will probably want a quick overview of the current state of the system before you begin. For this reason, we will examine status and configuration utilities first. This approach has the advantage of being pretty much the same from one version of Unix to the next. With configuration files, the differences among the various flavors of Unix can be staggering. Even when the files have the same functionality and syntax, they can go by different names or be in different directories. Certainly, using these utilities is much simpler than looking at kernel configuration files.
TIP: The output provided by these utilities may vary considerably from system to system and will depend heavily on which options are used. In practice, this should present no real problem. Don't be alarmed if the output on your system is formatted differently.
In this example, the first and last columns are the most interesting since they give the owners and the processes, along with their arguments. In this example, the lines, and consequently the arguments, have been truncated, but this is easily avoided. Running processes of interest include portmap, inetd, sendmail, telnetd, and chargen.bsd4# ps -aux USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 6590 22.0 2.1 924 616 ?? R 11:14AM 0:09.80 inetd: chargen [2 root 1 0.0 0.6 496 168 ?? Ss Fri09AM 0:00.03 /sbin/init -- root 2 0.0 0.0 0 0 ?? DL Fri09AM 0:00.52 (pagedaemon) root 3 0.0 0.0 0 0 ?? DL Fri09AM 0:00.00 (vmdaemon) root 4 0.0 0.0 0 0 ?? DL Fri09AM 0:44.05 (syncer) root 100 0.0 1.7 820 484 ?? Ss Fri09AM 0:02.14 syslogd daemon 109 0.0 1.5 828 436 ?? Is Fri09AM 0:00.02 /usr/sbin/portmap root 141 0.0 2.1 924 616 ?? Ss Fri09AM 0:00.51 inetd root 144 0.0 1.7 980 500 ?? Is Fri09AM 0:03.14 cron root 150 0.0 2.8 1304 804 ?? Is Fri09AM 0:02.59 sendmail: accepti root 173 0.0 1.3 788 368 ?? Is Fri09AM 0:01.84 moused -p /dev/ps root 213 0.0 1.8 824 508 v1 Is+ Fri09AM 0:00.02 /usr/libexec/gett root 214 0.0 1.8 824 508 v2 Is+ Fri09AM 0:00.02 /usr/libexec/gett root 457 0.0 1.8 824 516 v0 Is+ Fri10AM 0:00.02 /usr/libexec/gett root 6167 0.0 2.4 1108 712 ?? Ss 4:10AM 0:00.48 telnetd jsloan 6168 0.0 0.9 504 252 p0 Is 4:10AM 0:00.09 -sh (sh) root 6171 0.0 1.1 464 320 p0 S 4:10AM 0:00.14 -su (csh) root 0 0.0 0.0 0 0 ?? DLs Fri09AM 0:00.17 (swapper) root 6597 0.0 0.8 388 232 p0 R+ 11:15AM 0:00.00 ps -aux
There are a number of options available to ps, although they vary from implementation to implementation. In this example, run under FreeBSD, the parameters used were -aux. This combination shows all users' processes (-a), including those without controlling terminals (-x), in considerable detail (-u). The options -ax will provide fewer details but show more of the command-line arguments. Alternately, you can use the -w option to extend the displayed information to 132 columns. With AT&T-derived systems, the options -ef do pretty much the same thing. Interestingly, Linux supports both sets of options. You will need to precede AT&T-style options with a hyphen. This isn't required for BSD options. You can do it either way with Solaris. /usr/bin/ps follows the AT&T conventions, while /usr/ucb/ps supports the BSD options.
While ps quickly reveals individual processes, it gives a somewhat incomplete picture if interpreted naively. For example, the inetd daemon is one source of confusion. inetd is used to automatically start services on a system as they are needed. Rather than start a separate process for each service that might eventually be run, the inetd daemon runs on their behalf. When a connection request arrives, inetd will start the requested service. Since some network services like ftp, telnet, and finger are usually started this way, ps will show processes for them only when they are currently running. If ps doesn't list them, it doesn't mean they aren't available; they just aren't currently running.
For example, in the previous listing, chargen was started by inetd. We can see chargen in this instance because it was a running process when ps was run. But, this particular test system was configured to run a number of additional services via inetd (as determined by the /etc/inetd.conf configuration file). None of these other services show up under ps because, technically, they aren't currently running. Yet, these other services will be started automatically by inetd, so they are available services.
In addition to showing what is running, ps is a useful diagnostic tool. It quickly reveals defunct processes or multiple instances of the same process, thereby pointing out configuration problems and similar issues. %MEM and %CPU can tell you a lot about resource usage and can provide crucial information if you have resource starvation. Or you can use ps to identify rogue processes that are spawning other processes by looking at processes that share a common PPID. Once you are comfortable with the usual uses, it is certainly worth revisiting ps periodically to learn more about its other capabilities, as this brief discussion just scratches the surface of ps.
Output is interrupted with a q or a Ctrl-C. Sometimes system administrators will leave top running on the console when the console is not otherwise in use. Of course, this should be done only in a physically secure setting.15 processes: 2 running, 13 sleeping CPU states: 0.8% user, 0.0% nice, 7.4% system, 7.8% interrupt, 84.0% idle Mem: 6676K Active, 12M Inact, 7120K Wired, 2568K Cache, 3395K Buf, 1228K Free Swap: 100M Total, 100M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 6590 root 35 0 924K 616K RUN 0:15 21.20% 20.75% inetd 144 root 10 0 980K 500K nanslp 0:03 0.00% 0.00% cron 150 root 2 0 1304K 804K select 0:03 0.00% 0.00% sendmail 100 root 2 0 820K 484K select 0:02 0.00% 0.00% syslogd 173 root 2 0 788K 368K select 0:02 0.00% 0.00% moused 141 root 2 0 924K 616K select 0:01 0.00% 0.00% inetd 6167 root 2 0 1108K 712K select 0:00 0.00% 0.00% telnetd 6171 root 18 0 464K 320K pause 0:00 0.00% 0.00% csh 6168 jsloan 10 0 504K 252K wait 0:00 0.00% 0.00% sh 6598 root 28 0 1556K 844K RUN 0:00 0.00% 0.00% top 1 root 10 0 496K 168K wait 0:00 0.00% 0.00% init 457 root 3 0 824K 516K ttyin 0:00 0.00% 0.00% getty 214 root 3 0 824K 508K ttyin 0:00 0.00% 0.00% getty 213 root 3 0 824K 508K ttyin 0:00 0.00% 0.00% getty 109 daemon 2 0 828K 436K select 0:00 0.00% 0.00% portmap
In a sense, ps is a more general top since it gives you all running processes. The advantage to top is that it focuses your attention on resource hogs, and it provides a repetitive update. top has a large number of options and can provide a wide range of information. For more information, consult its Unix manpage.
One use of netstat is to display the connections and services available on a host. For example, this is the output for the system we just looked at:
The first column gives the protocol. The next two columns give the sizes of the send and receive queues. These should be 0 or near 0. Otherwise, you may have a problem with that particular service. The next two columns give the socket or IP address and port number for each end of a connection. This socket pair uniquely identifies one connection. The socket is presented in the form hostname.service. Finally, the state of the connection is given in the last column for TCP services. This is blank for UDP since it is connectionless. The most common states are ESTABLISHED for current connections, LISTEN for services awaiting a connection, and TIME_WAIT for recently terminated connections. Any of the TCP states could show up, but you should rarely see the others. An excessive number of SYN_RECEIVED, for example, is an indication of a problem (possibly a denial-of-service attack). You can safely ignore the last few lines of this listing.bsd4# netstat -a Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp 0 0 bsd4.telnet 22.214.171.124.3473 TIME_WAIT tcp 0 17458 bsd4.chargen sloan.1244 ESTABLISHED tcp 0 0 *.chargen *.* LISTEN tcp 0 0 *.discard *.* LISTEN tcp 0 0 *.echo *.* LISTEN tcp 0 0 *.time *.* LISTEN tcp 0 0 *.daytime *.* LISTEN tcp 0 0 *.finger *.* LISTEN tcp 0 2 bsd4.telnet sloan.1082 ESTABLISHED tcp 0 0 *.smtp *.* LISTEN tcp 0 0 *.login *.* LISTEN tcp 0 0 *.shell *.* LISTEN tcp 0 0 *.telnet *.* LISTEN tcp 0 0 *.ftp *.* LISTEN tcp 0 0 *.sunrpc *.* LISTEN udp 0 0 *.1075 *.* udp 0 0 *.1074 *.* udp 0 0 *.1073 *.* udp 0 0 *.1072 *.* udp 0 0 *.1071 *.* udp 0 0 *.1070 *.* udp 0 0 *.chargen *.* udp 0 0 *.discard *.* udp 0 0 *.echo *.* udp 0 0 *.time *.* udp 0 0 *.daytime *.* udp 0 0 *.sunrpc *.* udp 0 0 *.syslog *.* Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr c3378e80 dgram 0 0 0 c336efc0 0 c3378f80 c3378f80 dgram 0 0 0 c336efc0 0 c3378fc0 c3378fc0 dgram 0 0 0 c336efc0 0 0 c336efc0 dgram 0 0 c336db00 0 c3378e80 0 /var/run/log
A couple of examples should clarify this output. The following line shows a Telnet connection between bsd4 and sloan using port 1082 on sloan:
The next line shows that there was a second connection to sloan that was recently terminated:tcp 0 2 bsd4.telnet sloan.1082 ESTABLISHED
Terminated connections remain in this state for a couple of minutes, during which time the socket pair cannot be reused.tcp 0 0 bsd4.telnet 126.96.36.199.3473 TIME_WAIT
Name resolution can be suppressed with the -n option if you would rather see numeric entries. There are a couple of reasons you might want to do this. Typically, netstat will run much faster without name resolution. This is particularly true if you are having name resolution problems and have to wait for requests to time out. This option can help you avoid confusion if your /etc/services or /etc/hosts files are inaccurate.
The remaining TCP entries in the LISTEN state are services waiting for a connection request. Since a request could come over any available interface, its IP address is not known in advance. The * in the entry *.echo acts as a placeholder for the unknown IP address. (Since multiple addresses may be associated with a host, the local address is unknown until a connection is actually made.) The *.* entries indicate that both the remote address and port are unknown. As you can see, this shows a number of additional services that ps was not designed to display. In particular, all the services that are under the control of inetd are shown.
Another use of netstat is to list the routing table. This may be essential information in resolving routing problems, e.g., when you discover that a host or a network is unreachable. Although it may be too long or volatile on many systems to be very helpful, the routing table is sometimes useful in getting a quick idea of what networks are communicating with yours. Displaying the routing table requires the -r option.
There are four main ways entries can be added to the routing table -- by the ifconfig command when an interface is configured, by the route command, by an ICMP redirect, or through an update from a dynamic protocol like RIP or OSPF. If dynamic protocols are used, the routing table is an example of a dynamic structure that can't be discovered by looking at configuration files.
Here is an example of a routing table from a FreeBSD system:
At first glance, output from other systems may be organized differently, but usually the same basic information is present. In this example, the -n option was used to suppress name resolution.bsd1# netstat -rn Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 188.8.131.52 UGSc 0 0 xl0 127.0.0.1 127.0.0.1 UH 0 0 lo0 172.16.1/24 172.16.2.1 UGSc 0 7 xl1 172.16.2/24 link#2 UC 0 0 xl1 172.16.2.1 0:10:7b:66:f7:62 UHLW 2 0 xl1 913 172.16.2.255 ff:ff:ff:ff:ff:ff UHLWb 0 18 xl1 172.16.3/24 172.16.2.1 UGSc 0 2 xl1 205.153.60 link#1 UC 0 0 xl0 184.108.40.206 0:0:a2:c6:e:42 UHLW 4 0 xl0 906 220.127.116.11 link#1 UHLW 1 0 xl0 18.104.22.168 0:90:27:9c:2d:c6 UHLW 0 34 xl0 987 22.214.171.124 ff:ff:ff:ff:ff:ff UHLWb 1 18 xl0 205.153.61 126.96.36.199 UGSc 0 0 xl0 205.153.62 188.8.131.52 UGSc 0 0 xl0 205.153.63 184.108.40.206 UGSc 2 0 xl0
The first column gives the destination, while the second gives the interface or next hop to that destination. The third column gives the flags. These are often helpful in interpreting the first two columns. A U indicates the path is up or available, an H indicates the destination is a host rather than a network, and a G indicates a gateway or router. These are the most useful. Others shown in this table include b, indicating a broadcast address; S, indicating a static or manual addition; and W and c, indicating a route that was generated as a result of cloning. (These and other possibilities are described in detail in the Unix manpage for some versions of netstat.) The fourth column gives a reference count, i.e., the number of active uses for each of the routes. This is incremented each time a connection is built over the route (e.g., a Telnet connection is made using the route) and decremented when the connection is torn down. The fifth column gives the number of packets sent using this entry. The last entry is the interface that will be used.
If you are familiar with the basics of routing, you have seen these tables before. If not, an explanation of the first few lines of the table should help. The first entry indicates the default route. This was added statically at startup. The second entry is the loopback address for the machine. The third entry is for a remotely attached network. The destination network is a subnet from a Class B address space. The /24 is the subnet mask. Traffic to this network must go through 172.16.2.1, a gateway that is defined with the next two entries. The fourth entry indicates that the network gateway, 172.16.2.1, is on a network that has a direct attachment through the second interface xl1. The entry that follows gives the specifics, including the Ethernet address of the gateway's interface.
In general, it helps to have an idea of the interfaces and how they are configured before you get too deeply involved in routing tables. There are two quick ways to get this information -- use the -i option with netstat or use the ifconfig command. Here is the output for the interfaces that netstat generates. This corresponds to the routing table just examined.
For our purposes, we are interested in only the first four entries. (The other interfaces include the loop-back, lo0, and unused interfaces like ppp0*, the PPP interface.) The first two entries give the Ethernet address and IP address for the xl0 interface. The next two are for xl1. Notice that this also gives the number of input and output packets and errors as well. You can expect to see very large numbers for these. The very low numbers indicate that the system was recently restarted.bsd1# netstat -i Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll xl0 1500 <Link> 00.10.5a.e3.37.0c 2123 0 612 0 0 xl0 1500 205.153.60 220.127.116.11 2123 0 612 0 0 xl1 1500 <Link> 00.60.97.92.4a.7b 478 0 36 0 0 xl1 1500 172.16.2/24 172.16.2.13 478 0 36 0 0 lp0* 1500 <Link> 0 0 0 0 0 tun0* 1500 <Link> 0 0 0 0 0 sl0* 552 <Link> 0 0 0 0 0 ppp0* 1500 <Link> 0 0 0 0 0 lo0 16384 <Link> 6 0 6 0 0 lo0 16384 127 localhost 6 0 6 0 0
The format of the output may vary from system to system, but all will provide the same basic information. There is a lot more to netstat than this introduction shows. For example, netstat can be run periodically like top. We will return to netstat in future chapters.
The program is available for a staggering variety of Unix systems, often in both source and binary formats. Although I will limit this discussion to networking related tasks, lsof is more properly an operating system tool than a networking tool. You may want to learn more about lsof than described here.
In its simplest form, lsof produces a list of all open files. You'll probably be quite surprised at the number of files that are open on a quiescent system. For example, on a FreeBSD system with no one else logged on, lsof listed 564 open files.
Here is an example of the first few lines of output from lsof:
The most useful fields are the obvious ones, including the first three -- the name of the command, the process ID, and its owner. The other fields and codes used in the fields are explained in the manpage for lsof, which runs about 30 pages.bsd2# lsof COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME swapper 0 root cwd VDIR 116,131072 512 2 / swapper 0 root rtd VDIR 116,131072 512 2 / init 1 root cwd VDIR 116,131072 512 2 / init 1 root rtd VDIR 116,131072 512 2 / init 1 root txt VREG 116,131072 255940 157 /sbin/init ...
It might seem that lsof returns too much information to be useful. Fortunately, it provides a number of options that will allow you to tailor the output to your needs. You can use lsof with the -p option to specify a specific process number or with the -c option to specify the name of a process. For example, the command lsof -csendmail will list all the files opened by sendmail. You only need to give enough of the name to uniquely identify the process. The -N option can be used to list files opened for the local computer on an NFS server. That is, when run on an NFS client, lsof shows files opened by the client. When run on a server, lsof will not show the files the server is providing to clients.
As you can see, this is not unlike the -a option with netstat. Apart from the obvious differences in the details reported, the big difference is that lsof will not report connections that do not have files open. For example, if a connection is being torn down, all files may already be closed. netstat will still report this connection while lsof won't. The preferred behavior will depend on what information you need.bsd2# lsof -i COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME syslogd 105 root 4u IPv4 0xc3dd8f00 0t0 UDP *:syslog portmap 108 daemon 3u IPv4 0xc3dd8e40 0t0 UDP *:sunrpc portmap 108 daemon 4u IPv4 0xc3e09d80 0t0 TCP *:sunrpc (LISTEN) inetd 126 root 4u IPv4 0xc3e0ad80 0t0 TCP *:ftp (LISTEN) inetd 126 root 5u IPv4 0xc3e0ab60 0t0 TCP *:telnet (LISTEN) inetd 126 root 6u IPv4 0xc3e0a940 0t0 TCP *:shell (LISTEN) inetd 126 root 7u IPv4 0xc3e0a720 0t0 TCP *:login (LISTEN) inetd 126 root 8u IPv4 0xc3e0a500 0t0 TCP *:finger (LISTEN) inetd 126 root 9u IPv4 0xc3dd8d80 0t0 UDP *:biff inetd 126 root 10u IPv4 0xc3dd8cc0 0t0 UDP *:ntalk inetd 126 root 11u IPv6 0xc3e0a2e0 0t0 TCP *:ftp inetd 126 root 12u IPv6 0xc3e0bd80 0t0 TCP *:telnet inetd 126 root 13u IPv6 0xc3e0bb60 0t0 TCP *:shell inetd 126 root 14u IPv6 0xc3e0b940 0t0 TCP *:login inetd 126 root 15u IPv6 0xc3e0b720 0t0 TCP *:finger lpd 131 root 6u IPv4 0xc3e0b500 0t0 TCP *:printer (LISTEN) sendmail 137 root 4u IPv4 0xc3e0b2e0 0t0 TCP *:smtp (LISTEN) httpd 185 root 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 198 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 199 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 200 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 201 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 202 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 10408 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 10409 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 10410 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 25233 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) httpd 25236 nobody 16u IPv4 0xc3e0b0c0 0t0 TCP *:http (LISTEN) telnetd 58326 root 0u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telne t->sloan.lander.edu:1184 (ESTABLISHED) telnetd 58326 root 1u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telne t->sloan.lander.edu:1184 (ESTABLISHED) telnetd 58326 root 2u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telne t->sloan.lander.edu:1184 (ESTABLISHED) perl 68936 root 4u IPv4 0xc3dd8c00 0t0 UDP *:eicon-x25 ping 81206 nobody 3u IPv4 0xc3e98f00 0t0 ICMP *:*
If you specify an address, then only those files related to the address will be listed:
One minor problem with this output is the identification of the telnet user as root -- a consequence of root owning telnetd, the server's daemon. On some systems, you can use the PID with the -p option to track down the device entry and then use lsof on the device to discover the owner. Unfortunately, this won't work on many systems.bsd2# lsof -email@example.com COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME telnetd 73825 root 0u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telnet-> sloan.lander.edu:1177 (ESTABLISHED) telnetd 73825 root 1u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telnet-> sloan.lander.edu:1177 (ESTABLISHED) telnetd 73825 root 2u IPv4 0xc3e0eb60 0t0 TCP bsd2.lander.edu:telnet-> sloan.lander.edu:1177 (ESTABLISHED)
You can also use lsof to track an FTP transfer. You might want to do this to see if a transfer is making progress. You would use the -p option to see which files are open to the process. You can then use -ad to specify the device file descriptor along with -r to specify repeat mode. lsof will be run repeatedly, and you can see if the size of the file is changing.
Here are the results for the system we just looked at:
You can see that for the interfaces xl0 and xl1, we are given a general status report. UP indicates that the interface is operational. If UP is missing, the interface is down and will not process packets. For Ethernet, the combination of BROADCAST, SIMPLEX, and MULTICAST is not surprising. The mtu is the largest frame size the interface will handle. Next, we have the IP number, address mask, and broadcast address. The Ethernet address comes next, although some systems (Solaris, for example) will suppress this if you aren't running the program as root. Finally, we see information about the physical interface connections.bsd1# ifconfig -a xl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 inet 18.104.22.168 netmask 0xffffff00 broadcast 22.214.171.124 ether 00:10:5a:e3:37:0c media: 10baseT/UTP <half-duplex> supported media: autoselect 100baseTX <full-duplex> 100baseTX <half-dupl ex> 100baseTX 10baseT/UTP <full-duplex> 10baseT/UTP <half-duplex> 10baseT/UTP xl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 inet 172.16.2.13 netmask 0xffffff00 broadcast 172.16.2.255 ether 00:60:97:92:4a:7b media: 10baseT/UTP <half-duplex> supported media: autoselect 100baseTX <full-duplex> 100baseTX <half-dupl ex> 100baseTX 10baseT/UTP <full-duplex> 10baseT/UTP 10baseT/UTP <half-duplex> lp0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> mtu 1500 tun0: flags=8010<POINTOPOINT,MULTICAST> mtu 1500 sl0: flags=c010<POINTOPOINT,LINK2,MULTICAST> mtu 552 ppp0: flags=8010<POINTOPOINT,MULTICAST> mtu 1500 lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 inet 127.0.0.1 netmask 0xff000000
You can ignore the entries for lp0, tun0, sl0, and ppp0. In fact, if you don't want to see these, you can use the combination -au to list just the interfaces that are up. Similarly, -d is used to list just the interfaces that are down.
While netstat allows you to get basic information on the interfaces, if your goal is configuration information, ifconfig is a better choice. First, as you can see, ifconfig supplies more of that sort of information. Second, on some systems, netstat may skip interfaces that haven't been configured. Finally, ifconfig also allows you to change parameters such as the IP addresses and masks. In particular, ifconfig is frequently used to shut down an interface. This is roughly equivalent to disconnecting the interface from the network. To shut down an interface, you use the down option. For example, ifconfig xl1 down will shut down the interface xl1, and ifconfig xl1 up will bring it back up. Of course, you must have root privileges to use ifconfig to change configurations.
Since ifconfig is used to configure interfaces, it is typically run automatically by one of the startup scripts when the system is booted. This is something to look for when you examine startup scripts. The use of ifconfig is discussed in detail in Craig Hunt's TCP/IP Network Administration.
Typically, addresses are added or removed automatically. If your system needs to communicate with another system on the local network whose MAC address is unknown, your system sends an ARP request, a broadcast packet with the destination's IP address. If the system is accessible, it will respond with an ARP reply that includes its MAC address. Your system adds this to its ARP table and then uses this information to send packets directly to the destination. (A simple way to add an entry for a directly connected device to the ARP table is to ping the device you want added. ping is discussed in detail in Chapter 3, "Connectivity Testing".) Most systems are configured to drop entries from the ARP table if they aren't being used, although the length of the timeout varies from system to system.
At times, you may want to examine or even change entries in the ARP table. The arp command allows you to do this. When arp is invoked with the -a option, it reports the current contents of the ARP table. Here is an example from a Solaris system:
The format or details may vary from system to system, but the same basic information should be provided.sol1# arp -a Net to Media Table Device IP Address Mask Flags Phys Addr ------ -------------------- --------------- ----- --------------- elxl0 126.96.36.199 255.255.255.255 00:00:a2:c6:0e:42 elxl0 188.8.131.52 255.255.255.255 00:e0:29:21:3c:0b elxl0 184.108.40.206 255.255.255.255 00:90:27:43:72:70 elxl0 mail.lander.edu 255.255.255.255 00:90:27:9c:2d:c6 elxl0 sol1 255.255.255.255 SP 00:60:97:58:71:b7 elxl0 pm3.lander.edu 255.255.255.255 00:c0:05:04:2d:78 elxl0 BASE-ADDRESS.MCAST.NET 240.0.0.0 SM 01:00:5e:00:00:00
For Solaris, the first column gives the interface for the connection. The next two are the IP address and its mask. (You can get just IP numbers by using the -n option.) There are four possible flags that may appear in the flags column. An S indicates a static entry, one that has been manually set rather than discovered. A P indicates an address that will be published. That is, this machine will provide this address should it receive an ARP request. In this case, the P flag is for the local machine, so it is natural that the machine would respond with this information. The flags U and M are used for unresolved and multicast addresses, respectively. The final column is the actual Ethernet address.
This information can be useful in several ways. It can be used to determine the Ethernet hardware in this computer, as well as the hardware in directly connected devices. The IEEE assigns to the manufacturers of Ethernet adapters unique identifiers to be used as the first three bytes of their Ethernet addresses. These addresses, known as Organizationally Unique Identifiers (OUI), can be found at the IEEE web page at http://standards.ieee.org/regauth/oui/index.html. In other words, the first three bytes of an Ethernet address identify the manufacturer. In this case, by entering on this web page 00 60 97, i.e., the first three bytes of the address 00 60 97 58 71 b7, we find that the host sol1 has a 3COM Ethernet adapter. In the same manner we can discover that the host 220.127.116.11 is Bay Networks equipment.
WARNING: OUI designations are not foolproof. The MAC address of a device may have been changed and may not have the manufacturer's OUI. And even if you can identify the manufacturer, in today's world of merger mania and takeovers, you may see an OUI of an acquired company that you don't recognize.If some machines on your network are reachable but others aren't, or connectivity comes and goes, ARP problems may be the cause. (For an example of an ARP problem, see Chapter 12, "Troubleshooting Strategies".) If you think you might have a problem with IP-to-Ethernet address resolution on your local network, arp is the logical tool to use to diagnose the problem. First, look to see if there is an entry for the destination and if it is correct. If it is missing, you can attempt to add it using the -s option. (You must be root.) If the entry is incorrect, you must first delete it with the -d option. Entries added with the -s option will not time out but will be lost on reboot. If you want to permanently add an entry, you can create a startup script to do this. In particular, in a script, arp can use the -f option to read entries from a file.
The usual reason for an incorrect entry in an arp table is a duplicated IP address somewhere on your network. Sometimes this is a typing mistake. Sometimes when setting up their computers, people will copy the configuration from other computers, including the supposedly unique IP number. A rogue DHCP server is another possibility. If you suspect one of your hosts is experiencing problems caused by a duplicate IP number on the network, you can shut down the interface on that computer or unplug it from the network. (This is less drastic than shutting down the computer, but that will also work.) Then you can ping the IP address in question from a second computer. If you get an answer, some other computer is using your IP address. Your arp table should give you the Ethernet address of the offending machine. Using its OUI will tell you the type of hardware. This usually won't completely locate the problem machine, but it is a start, particularly for unusual hardware.
One way of detecting this is to use a port scanner to see which ports are active on your system. You could choose to do this from the compromised system, but you are probably better off doing this from a remote system known to be secure. This assumes, however, that the attacker hasn't installed a trapdoor on the compromised host that is masquerading as a legitimate service on a legitimate port.
There are a large number of freely available port scanners. These include programs like gtkportscan, nessus, portscan, and strobe, to name just a few. They generally work by generating a connection request for each port number in the range being tested. If they receive a reply from the port, they add it to their list of open ports. Here is an example using portscan:
The arguments are the destination address and beginning and ending port numbers. The result is a list of port numbers and service names for ports that answered.bsd1# portscan 18.104.22.168 1 10000 -vv This is a portscanner - Rafael Barrero, Jr. Email me at firstname.lastname@example.org For further information. Enjoy! Port: 7 --> echo Port: 9 --> discard Port: 13 --> daytime Port: 19 --> chargen Port: 21 --> ftp Port: 23 --> telnet Port: 25 --> smtp Port: 37 --> time Port: 79 --> finger Port: 111 --> sunrpc Port: 513 --> login Port: 514 --> shell
Figure 2-1 shows another example of a port scanner running under Windows NT. This particular scanner is from Mentor Technologies, Inc., and can be freely downloaded from http://www.mentortech.com/learn/tools/tools.shtml. It is written in Java, so it can be run on both Windows and Unix machines but will require a Java runtime environment. It can also be run in command-line mode. Beware, this scanner is very slow when used with Windows.
One last word about these tools. Don't get caught up in using tools and overlook simpler tests. For example, you can check to see if sendmail is running by trying to connect to the SMTP port using telnet. In this example, the test not only tells me that sendmail is running, but it also tells me what version of sendmail is running:
In the same spirit:lnx1# telnet 22.214.171.124 25 Trying 126.96.36.199... Connected to 188.8.131.52. Escape character is '^]'. 220 bsd4.lander.edu ESMTP Sendmail 8.9.3/8.9.3; Wed, 8 Mar 2000 09:38:02 -0500 (EST) quit 221 bsd4.lander.edu closing connection Connection closed by foreign host.
clearly shows ipfw is not running on this system. All I did was try to use it. This type of application-specific testing is discussed in greater detail in Chapter 10, "Application-Level Tools".bsd1# ipfw list ipfw: getsockopt(IP_FW_GET): Protocol not available
|1.3. Troubleshooting and Management||2.2. System Configuration Files|
Copyright © 2002 O'Reilly & Associates. All rights reserved.