Performance Measurements Commands

netstat -ta     show current intenet services/connections
        -a  : show (a)ll  (include listening port process)
        -n  : ip (n)umber only (no dns lookup)
        -r  : (r)outing table   (change with route cmd)
        -i  : show stat for diff nic (i)nterfaces
        -k ce0 : lot of interface specific info, ce NIC will have duplex stat.



vmstat -a   : all 
       -n   : 
       -p   : process owning port

iostat


sar


date; mkfile 1000m test; date			 	# create a 1 GB file (filled with 0)
date; dd if=/dev/urandom of=test bs=1024 count=100000 	# same, file has random data.

SE Toolkit

setup env:

export PATH=$PATH:/opt/RICHPse/bin
export SEPATH=/opt/RICHPse/examples:/opt/RICHPse/toptool

interactive tools:

se zoom.se		# gui, summary status for all components.  Main Window. 
se multimeter.se	# gui, cpu, cache, vm and locks meter

se toptool.se		# gui, just like top
se xload.se		# gui, just like xload, show hostname :)
se infotool.se		# gui, menu to lot of sys info (cpu, net, disk, etc)
se xit			# gui  wrap on text disk stat dump (xiostat.se)

se -DWIDE pea.se 10	# text, dump top like info to stdout every 10 sec
se disks.se		# text, dump lot of disk usage info

se webtune.se		# display current, min and max values for perf params

se virtual_adrain.se &	# text, dump warning to stdout if perf problem found 
			# run cli in background, non permanent, only output to
			# login screen; process end, all cleared.

-------------------------

# install:
# pkgrm RICHPse
# gunzip RICHPse.tar.gz
# tar xf RICHPse.tar
# pkgadd -d . RICHPse
# edit /opt/RICHPse/etc/se_defines, enable "disk nfs"

# alt, can just copy to network drive, and set PATH and SEPATH
# at least for the interactive tools above

# always run monitor:
/opt/RICHPse/etc/init.d/vader start     # init.d script to start vader
se /opt/RICHPse/examples/vader.se       # the "Virtual Adrian Daemon", 
                                        # start on host to be monitored

se /opt/RICHPse/examples/darth.se -h remotehost # gui, start on client.
	# This gui is the front end of the bg monitor

#!/bin/sh

# setoolkit-install.sh
# quick script to setup  and start se toolkit

cd /mnt/sa/share/software/SEtoolkit

pkgadd -d . RICHPse.331


(cd /opt/RICHPse/etc; tar cf - *.d) | (cd /etc ; tar xvf - )

# /etc/init.d/mon_cm start
/etc/init.d/monlog start
/etc/init.d/percol start
/etc/init.d/va_monitor start
/etc/init.d/vader start

Network Tracing

tcpdump

tcpdump parameters
-n: ip number, do no resolve hostname
-e: ethernet (?)
-i: interface
-w [FILE]: write output to file (capture use, more info than redirect output)

host IP-or-NAME		capture info only related to the specified host

operators accepted:
&&	= and
||	= or
!	= not

eg cmd of tcpdump [expression]  :

tcpdump host 10.0.71.165
tcpdump src  10.0.71.165
tcpdump 'dst net 128.3'
tcpdump 'src or dst port ftp-data'   
tcpdump 'ether host 0:d0:b7:a9:c9:5a'

Sample trace output

showmount -e 192.168.209.30 # VIP
tcpdump -n host 172.24.51.182  # misconfigured NAT
18:49:41.964873 eth0 < 172.24.51.182 > tin-linux.zambeel.com: icmp: 172.24.51.182 udp port sunrpc unreachable [tos 0xc0] 
18:56:24.677264 eth0 < 172.24.51.182 > 10.0.15.11: icmp: 172.24.51.182 udp port sunrpc unreachable [tos 0xc0] 
18:56:24.679401 eth0 < 172.24.51.182 > 10.0.15.11: icmp: 172.24.51.182 udp port sunrpc unreachable [tos 0xc0] 
timestamp     src-if ?   source ip     destination prtl  err message

tcpdump -n port sunrpc
18:54:31.055821 eth0 > 10.0.15.11.1388 > 192.168.209.30.sunrpc: udp 56
              src-if ? source  ip.port ? dest        ip.port  : protocol + port


   [z-00D0B7A873CE] # tcpdump -e port sunrpc
18:15:55.628675 eth2 < 0:e0:52:d:7e:18 0:0:0:0:0:1 ip 74: 10.0.15.11.2499 > 172.24.51.182.sunrpc: S 4260207884:4260207884(0) win 32120  (DF)
time            if   ? src mac         dst-mac(host)      src ip.port            dest ip.port    TCP SYN and other protocol info
18:15:55.628696 eth2 > 0:0:0:0:0:0 0:2:e3:0:3b:9d ip 54: 172.24.51.182.sunrpc > 10.0.15.11.2499: R 0:0(0) ack 4260207885 win 0
time            if   ? src mac         dst-mac(host)      src ip.port            dest ip.port    TCP SYN and other protocol info

Here is an example of messed up translation.
Note that source & dest mac-address is rewritten on each router hop.


   [z-00D0B7A871DF] # tcpdump -n | egrep '10\.0\.15\.11|192\.168'
19:02:43.964206 eth2 > 172.24.51.12.telnet >   10.0.15.11.2411:   P 2646085534:2646085754(220) ack 2623622447 win 32120 {nop,nop,timestamp 2624922 80719743} (DF)
19:02:43.982115 eth2 < 10.0.15.11.2411     > 172.24.51.12.telnet: . 1:1(0) ack 220 win 31856 {nop,nop,timestamp 80720053 2624922} (DF)
19:02:45.277592 eth2 B 172.24.51.1.route   > 172.24.51.255.route: rip-resp 25: {192.168.13.0/255.255.255.0}(2) {192.168.14.0/255.255.255.0}(2) {192.168.15.0/255.255.255.0}(2) {192.168.16.0/255.255.255.0}(2) {192.168.17.0/255.255.255.0}(2)[|rip]

snoop

snoop host [IP]			# traffic with a given host (as src or dst)
snoop -r port 25		# all traffic in port 25 (smtp), 
				# do not resolve ip to dns names
-s 	= sniplet length (def is whole packet)
	= 80 ip hdr only, 120 = nfs header only

-V	= layer info
-v	= more verbose than -V, lot of info.


from cli :

Usage:  snoop
        [ -a ]                  # Listen to packets on audio
        [ -d device ]           # settable to le?, ie?, bf?, tr?
        [ -s snaplen ]          # Truncate packets
        [ -c count ]            # Quit after count packets
        [ -P ]                  # Turn OFF promiscuous mode
        [ -D ]                  # Report dropped packets
        [ -S ]                  # Report packet size
        [ -i file ]             # Read previously captured packets
        [ -o file ]             # Capture packets in file
        [ -n file ]             # Load addr-to-name table from file
        [ -N ]                  # Create addr-to-name table
        [ -t  r|a|d ]           # Time: Relative, Absolute or Delta
        [ -v ]                  # Verbose packet display
        [ -V ]                  # Show all summary lines
        [ -p first[,last] ]     # Select packet(s) to display
        [ -x offset[,length] ]  # Hex dump from offset for length
        [ -C ]                  # Print packet filter code

Sample snoop

Capture traffic on NIC hme0 specific to a host, capture up 8K of the packet, 
and dump result to an output file:
snoop -d hme0 -s 8192 -o /tmp/snoop.out host 10.215.55.211

Read input file back.  May wish to use ethereal to read this file for easier access.
snoop -i /tmp/snoop.out		


snoop -s 120 port 25 host 211.196.53.194

titaniumleg.com  mail server traffic monitor
snoop -r -D -P -s 1500 -c 100000 -o /export/tmp/smtp01.20030122.snoop port 25

snoop -n /dev/null  -D -P -s 1500 -c 100000 -o /export/tmp/smtp01.20030122.snoop port 25
snoop -D -s 9000 -c 100000 -o jumpstartclient.snoop host jumpstartclient
-r = do not resolve hostname  # not in sol 7 snoop
-D = display num of dropped packets
-P = non promiscuous mode capture   (don't use in troubleshooting jumpstart problems).
-s snipplet length
-c count num of backets to capture
-o output file



###
### more explanations TBA
###

Ethereal

ethereal (GUI)
tethereal (CLI)

most flags work for both.



snoop-like behaviour (mostly for ethereal):
-l	: scroll capture 
-S 	: update as capture is in progress.
-k 	: start capture immediately  (disable interaction?)

--

-i [IF] : specify interface, eg eth0, hme0
-n 	: no dns resolution, use ip Number


-V 	: more verbose output, captured data displayed in tree mode instead of 1 line per packet.


-f 	: capture filter expression  (tcpdump notation needed), eg:

	>	tcp port 23 and host 10.0.0.5
	>   src net 10.0.15.0/24
	>   dst net 10.0.15.0 mask 255.255.255.0
	>> 	[src|dst] host 
	>>	ether [src|dst] host 00:E0:2B:DE:0E:00
	>> 	[tcp|udp] [src|dst] port 

	host 10.215.20.152 || host 10.215.2.21 || host 10.215.19.73



------------------------------------------------------------

ethereal view filter expression 
[ work in GUI filter box when viewing, 
NOT as capture filter (which is tcpdump format ]

operatos:
           eq, ==    Equal
           ne, !=    Not equal
           gt, >     Greater than
           lt, <     Less Than
           ge, >=    Greater than or Equal to
           le, <=    Less than or Equal to

           and, &&   Logical AND
           or, ||    Logical OR
           not, !    Logical NOT

boolean: true (1) or false (0)

some commonly used filter fields:

           eth.src == aa-aa-aa-aa-aa-aa
           ip.dst eq www.mit.edu
           ip.src == 192.168.1.1
           ip.addr == 129.111.0.0/16
           eth.src == aa-aa-aa-aa-aa-aa
           eth.src[0:3] == 00:00:83			# filter by vendor by use of slide
           tcp.port == 80 and ip.src == 192.168.2.1
		   ip.addr is for both src or dest, these multiple ocurring field is a bit confusing for packet filtering.

for generic filter dealing with a specific host, but not necessary filtering by tcp/udp/icmp
ip.dst
ip.src
ip.addr

udp
udp.port
udp.dstport
udp.srcport

tcp
tcp.port
tcp.dstport
tcp.srcport
tcp.seq

icmp


bootp.dhcp==true		: frame is dhcp
bootp.hw.addr

smb.cmd==(unsigned 8 bit int)	: smb protocol command number
smb.cmd == 0x06  		: cmd is smb unlink
smb.status != 0x0000	: Error code, 4 bytes aka status, lot of items.
smb.errcls != 0x0		: error class, 1 byte represent the categories
              0x0       = Success
              0x1       = DOS Error
              0x2       = Server Error
              0x3 	= hardware error
              0x4	= not a smb cmd
			Note, netBench Fail code 32 maybe in Dos or Hrd.
smb.pid
smb.mid		(multiplex id)
smb.uid		(user id, maybe per process)
nfs.*
nfs.fh.version != 3		= not sure what this is, not nfs protocol version!
rpc.programversion != 3		= all packet that are rpc program nfs version 3.

lot of higher level protocol stuff available, including vlan on switches, etc.
see the man page on ethereal or tethereal (very long!)


GUI version, filter can just enter a protocol type.  eg: smb
That means smb protocol is present.  A protocol in the filter w/o any comparison operator means filter packets where such field is present in the packet.  
eg: smb.errcls  filter packet that contain smb error class.




Network trace capture with tcpdump or snoop, save to file for viewing with ethereal

tcpdump -i [interface] -s 1500 -w [some-file]
tcpdump -s 8192 -w netuse.tcpdump 'host 10.0.71.232 or host 10.0.71.15'
snoop -d hme0  -o /tmp/snoop.out host 10.215.55.211

editcap can be used to trim captured file, or convert between formats
(tcpdump, ethereal, snoop, ms netmon, etc).

Network Scanner

nmap

nmap: network scanner
nmapfe: w/ gui front end, supposed to need gtk, but worked anyway.

nmap -sT -O -PI -PT 172.27.31.0/24	# scan whole class C vlan 31, with os identification.  long output.

Network Testing

ttcp, speed performance test for tcp & udp

receiving comptuer:
java ttcp -r
java ttcp -r -l 4096 -n 100     # 4096 bytes buffer, 100 of them.
java ttcp -r -l 32768 -n 4096

Sending computer:
java ttcp -t 10.215.2.124


args: (try these in receiving computer)
-l 		= buffer size, 			def 8192, try 32768
-n 		= num of buffer to xfer, 	def 2048, try  4096  ==> gives 128 MB xfer.

java version doesn't seems to suppport these:
-u		= udp test
-b 		= change system buffer size.
-v		= verbose, more stat
-d 		= dbg

----

various port avail.
linux rh come with a package
but seems rather old and no central org support.

http://www.netcordia.com/network-services.html