Re: sflowtool manipulation

Thanks Kevin,

FYI, the latest sflowtool includes a "-l" option to print flow and
counter samples as line-by-line output. It doesn't replace what you
have done here, but it might be useful for a subset of your
applications.

regards,
neil

On Mar 28, 2005, at 2:28 PM, Kevin Kawaguchi wrote:

> Greetings fellow sflow fans!
>
> I use sflowtool quite a bit when we have slowdowns and am usually able
> to notice patterns in traffic flowing by and figure out there is a DoS
> or DDoS going on.
>
> If I use sflowtool -t -p PortNum | tcpdump -r - -e -n -nn, I get a
> pretty readable output that scrolls along one line per sampled frame
> (including ARP and whatever tcpdump knows about). However, there are
> none of the sflow extras such as device, port, and VLAN which we use
> to track down the source of a problem.
>
> If I use plain sflowtool, I get multiline output that scrolls by too
> fast to see a pattern (but the device and port information is there).
>
> Looking at the InMon awk scripts, I got the idea to look through
> sflowtool output and gather what I needed and make it look similar to
> tcpdump and still have the device ip address and port number so I can
> track down a misbehaving host. I also self imposed a restriction that
> it needs to fit in 132 columns since that is all secureCRT seems to
> do.
>
> This is some sample output....
> [cckevin@theOracle cckevin]$ sflowtool -p 6400 | ./sflowcrunch.pl
> date:time:agent:inputPort:outputPort:in_vlan:srcMAC:dstMAC:srcIP:
> dstIP:IPProto:srcPort:dstPort:moreInfo
> 20050328:141411:10.107.7.7:25:24:559:00065bed6f96:00e04cb13d91:
> 10.0.0.8:10.0.0.185:tcp:445:4757:.A....
> 20050328:141411:10.107.7.7:20:20:560:00e04cb13d91:00065bed6f96:
> 10.0.0.185:10.0.0.8:tcp:4757:445:.A....
> 20050328:141411:10.199.6.15:1:multiple:714:0004809a1000:01005e7ffffd:
> xxx.yyy.97.2:239.255.255.253:udp:49180:427:
>
> This is only for IP packets not including ARP which I don't believe is
> decoded by sflowtool. If you wish to search for tcp flags, just pipe
> this output through grep with capital letter representations of the
> flags (all other letters are lower case).
>
> I'll include the source for anybody interested. Hope you find this
> useful. My disclaimer is that it is quick and dirty and I probably
> won't have time to work on it in the near future. If somebody out
> there comes up with something inspiring, that could change.
>
> Kevin Kawaguchi
> UC Davis Network Operations Center
>
> -----------------------------------------------------------------------
> -------
>
> [cckevin@theOracle cckevin]$ cat sflowcrunch.pl
> #!/usr/bin/perl
>
> # PURPOSE
> # take sflowtool output and make tcpdump-like output but include
> agent, port, and vlan
> # to help track down source of traffic seen in sflow samples.
> Especially useful for
> # homing in on DDoS since switch and port info is shown. TCP flags
> are the only CAPs
> # so grepping for SYN or FIN is really simple. sCRT has column limits
> of 132 so each
> # line should not exceed 132 columns (at least not very often).
>
> # VERSION and HISTORY
> # 2004-11-04 Version 0.1 Started and Completed
> # 2004-11-05 Version 0.1 Make more readable and add some documentation
>
> # TYPICAL USAGE
> # sflowtool -p 6400 | ./sflowcrunch.pl
> # sflowtool -p 6400 | ./sflowcrunch.pl | egrep "S|F"
> # sflowtool -p 6400 | ./sflowcrunch.pl | head -500
>
> # HOW IT'S DONE:
> # look for startDatagram, then agent, then startSample
> # if sample FLOWSAMPLE (rather than COUNTERSAMPLE)
> # look for inputPort, outputPort,
> # in_vlan, out_vlan,
> # dstMAC, srcMAC,
> # srcIP, dstIP, IPProtocol
> # ICMPType, ICMPCode
> # TCPSrcPort, TCPDstPort, TCPFlags
> # UDPSrcPort, UDPDstPort
> # at endSample, print based upon IPProtocol
>
> # CAVEATS
> # ARP aka ether type 0806 not distinguished in sflowtool other than
> -08-06- in hex header
> # Not interested in name resolution or port type, I'm a tcpdump -r -
> -e -n -nn kind of guy
> # Depends on sflowtool output not changing too much
>
> # CUSTOMIZING available
> # set $fs in sub init to change delimiter when printing (i use colon a
> lot)
> # choose date and time format in sub printSample (i dont use colon in
> timestamp...see above)
>
> # WANTED
> # use getops for count, delimiter, etc
> # implement count option
>
> # CONTACT
> # Kevin Kawaguchi
> # Network Operations Center
> # University of California, Davis
> # ktkawaguchi@ucdavis.edu
> # http://noc.ucdavis.edu
>
> ################## I N I T ######################
> sub init {
> $numPrinted="";
> #$fs=","; #field separator used in printing (space, colon, comma,
> etc)
> #$fs=" "; #field separator used in printing (space, colon, comma,
> etc)
> $fs=":"; #field separator used in printing (space, colon, comma,
> etc)
>
> #set up array for tcp FLAG for textual printing
> my $sample="uaprsf";
> for($i=0;$i<=63;$i++) { #just do all combinations...
> $flags[$i]=$sample;
> #flags in uAPrsf format for a Push ACK
> #if ($i & 1) { $flags[$i] =~ s/f/F/ }
> #if ($i & 2) { $flags[$i] =~ s/s/S/ }
> #if ($i & 4) { $flags[$i] =~ s/r/R/ }
> #if ($i & 8) { $flags[$i] =~ s/p/P/ }
> #if ($i & 16) { $flags[$i] =~ s/a/A/ }
> #if ($i & 32) { $flags[$i] =~ s/u/U/ }
> #flags in .AP... format for a Push ACK
> if ($i & 1) { $flags[$i] =~ s/f/F/ } else { $flags[$i] =~ s/f/./
> }
> if ($i & 2) { $flags[$i] =~ s/s/S/ } else { $flags[$i] =~ s/s/./
> }
> if ($i & 4) { $flags[$i] =~ s/r/R/ } else { $flags[$i] =~ s/r/./
> }
> if ($i & 8) { $flags[$i] =~ s/p/P/ } else { $flags[$i] =~ s/p/./
> }
> if ($i & 16) { $flags[$i] =~ s/a/A/ } else { $flags[$i] =~
> s/a/./ }
> if ($i & 32) { $flags[$i] =~ s/u/U/ } else { $flags[$i] =~
> s/u/./ }
> }
>
> # set up ICMP type descriptions
> $ICMPDesc[0]="(echo reply)";
> $ICMPDesc[3]="(unreachable)";
> $ICMPDesc[4]="(source quench)";
> $ICMPDesc[5]="(redirect)";
> $ICMPDesc[8]="(echo request)";
> $ICMPDesc[11]="(time exceeded)";
> $ICMPDesc[12]="(parameter problem)";
> $ICMPDesc[13]="(timestamp request)";
> $ICMPDesc[14]="(timestamp reply)";
> $ICMPDesc[15]="(info request)";
> $ICMPDesc[16]="(info reply)";
>
> clearDatagramVar;
>
> # print header for column headers if imported someplace
> print
> "date",$fs,"time",$fs,"agent",$fs,"inputPort",$fs,"outputPort",$fs,"in_
> vlan",$fs,"srcMAC",$fs,"dstMAC",$fs,"srcIP",$fs,"dstIP",$fs,"IPProto",$
> fs,"srcPort",$fs,"dstPort",$fs,"moreInfo","\n";
> return 0;
> }
>
> ######### C L E A R S A M P L E V A R S #########
> sub clearSampleVars {
> $inputPort="";
> $outputPort="";
> $in_vlan="";
> $out_vlan="";
> $dstMAC="";
> $srcMAC="";
> $srcIP="";
> $dstIP="";
> $IPProtocol="";
> $ICMPType="";
> $ICMPCode="";
> $TCPSrcPort="";
> $TCPDstPort="";
> $TCPFlags="";
> $UDPSrcPort="";
> $UDPDstPort="";
> return 0;
> }
>
> ####### C L E A R D A T A G R A M V A R S #######
> sub clearDatagramVars {
> $agent="";
> $samplesInPkt="";
> clearSampleVar;
> return 0;
> }
>
> ############ P R I N T S A M P L E ##############
> sub printsample {
> my $localmonth = $month+1; #returned as 0..11 not 1..12
> my $localyear = $year;
> if ($year < 2004) {$localyear+=1900}; #returned as almost 2 digit
> number...104 for 2004
>
> #date with no separators
> $date=$localyear;
> if ($localmonth < 10) {$date.="0" . $localmonth} else
> {$date.=$localmonth}
> if ($day < 10) {$date.="0" . $day} else {$date.=$day}
>
> #date with dashes between year month day, could easily be slashes /
> :.+1,.+3s!-!/!g
> #$date=$localyear . "-";
> #if ($localmonth < 10) {$date.="0" . $localmonth . "-"} else
> {$date.=$localmonth . "-"}
> #if ($day < 10) {$date.="0" . $day} else {$date.=$day}
>
> #time with no separators
> if ($hour < 10) {$timestamp="0" . $hour} else {$timestamp = $hour}
> if ($min < 10) {$timestamp.="0" . $min} else {$timestamp.=$min}
> if ($sec < 10) {$timestamp.="0" . $sec} else {$timestamp.=$sec}
>
> #time with colon separators (careful when using colon delimter in
> $fs)
> #if ($hour < 10) {$timestamp="0" . $hour . ":"} else {$timestamp =
> $hour . ":"}
> #if ($min < 10) {$timestamp.="0" . $min . ":"} else
> {$timestamp.=$min . ":"}
> #if ($sec < 10) {$timestamp.="0" . $sec} else {$timestamp.=$sec}
>
> #protocol 1 -> ICMP
> if ($IPProtocol == 1) {
> print
> "$date$fs$timestamp$fs$agent$fs$inputPort$fs$outputPort$fs$in_vlan$fs$s
> rcMAC$fs$dstMAC$fs$srcIP$fs$dstIP$fs\x69cmp$fs\x74ype
> $ICMPType$fs\x63ode $ICMPCode$fs$ICMPDesc[$ICMPType]\n";
> $numPrinted+=1;
> }
>
> #protocol 6 -> TCP
> if ($IPProtocol == 6) {
> print
> "$date$fs$timestamp$fs$agent$fs$inputPort$fs$outputPort$fs$in_vlan$fs$s
> rcMAC$fs$dstMAC$fs$srcIP$fs$dstIP$fs\x74cp$fs$TCPSrcPort$fs$TCPDstPort$
> fs$flags[$TCPFlags]\n";
> $numPrinted+=1;
> }
>
> #protocol 17 -> UDP
> if ($IPProtocol == 17) {
> print
> "$date$fs$timestamp$fs$agent$fs$inputPort$fs$outputPort$fs$in_vlan$fs$s
> rcMAC$fs$dstMAC$fs$srcIP$fs$dstIP$fs\x75dp$fs$UDPSrcPort$fs$UDPDstPort$
> fs\n";
> $numPrinted+=1;
> }
> return 0;
> }
>
> #################################################
> # M A I N #
> #################################################
>
> init;
>
> while (defined($line=<STDIN>)) {
> chomp($line);
> SWITCH: {
> $line =~ /^startDatagram/ && do {
> clearDatagramVars;
> ($sec,$min,$hour,$day,$month,$year) = localtime(time);
> last SWITCH;
> };
> $line =~ /^agent/ && do {
> $agent=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^samplesInPkt/ && do {
> $samplesInPkt=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^startSample/ && do {
> clearSampleVars;
> last SWITCH;
> };
> $line =~ /^sampleType/ && do {
> $sampleType=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^inputPort/ && do {
> $inputPort=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^outputPort/ && do {
> $outputPort=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^in_vlan/ && do {
> $in_vlan=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^out_vlan/ && do {
> $out_vlan=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^dstMAC/ && do {
> $dstMAC=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^srcMAC/ && do {
> $srcMAC=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^srcIP/ && do {
> $srcIP=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^dstIP/ && do {
> $dstIP=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^IPProtocol/ && do {
> $IPProtocol=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^ICMPType/ && do {
> $ICMPType=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^ICMPCode/ && do {
> $ICMPCode=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^TCPSrcPort/ && do {
> $TCPSrcPort=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^TCPDstPort/ && do {
> $TCPDstPort=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^TCPFlags/ && do {
> $TCPFlags=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^UDPSrcPort/ && do {
> $UDPSrcPort=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^UDPDstPort/ && do {
> $UDPDstPort=(split / /, $line)[1];
> last SWITCH;
> };
> $line =~ /^endSample/ && do {
> printsample;
> clearSampleVars;
> last SWITCH;
> };
> $line =~ /^endDatagram/ && do {
> clearDatagramVars;
> last SWITCH;
> };
> } #SWITCH
> } #while STDIN
>
>
----------
Neil McKee
http://www.inmon.com
Received on Tue Mar 29 04:06:57 2005