#!/usr/bin/perl -w # -*- perl -*- =head1 NAME snmp__if_combined - SNMP card plugin to monitor the network interfaces of any networked equipment. =head1 APPLICABLE SYSTEMS Any SNMP capable networked computer equipment. Using a command such as munin-node-configure --snmp switch.langfeldt.net --snmpversion 2c --snmpcommunity public | sh -x should auto-detect all applicable interfaces. On a typical switch you will get one plugin pr. ethernet port. On a router you might get one plugin per VLAN interface. =head2 NOTABLE DIFFERENCE WITH SNMP__IF_MULTI The `smnp__if_multi` plugin records data in _bytes_ per second, and shows them as bps by using CDEFs when graphing. This plugin performs the multiplication upon reporting the value, and therefore doesn't have a CDEF in the graphing path. This is important when migration from `snmp__if_multi`, for at least two aspects: - Reusing RRDs: it is not possible to just rename the old RRD files from `smnp__if_multi`, as historical data will be graphed as 8 times smaller than in reality. The stored values would need to be adjusted. - Data loaning: borrowing graphs no longer need to include a CDEF to convert bytes to bits. Conversely, they do now need a CDEF to convert bits to bytes. To reflect this aspect explicitely, the root graph's totals are named `recv_bits` and `send_bits`. =head1 CONFIGURATION As a rule SNMP plugins need site specific configuration. The default configuration (shown here) will only work on insecure sites/devices: [snmp_*] env.version 2 env.community public env.ifTypeOnly ethernetCsmacd env.stackedRoot 1 env.stackedMax 0 In general SNMP is not very secure at all unless you use SNMP version 3 which supports authentication and privacy (encryption). But in any case the community string for your devices should not be "public". Please see 'perldoc Munin::Plugin::SNMP' for further configuration information. The ifTypeOnly is a space separated list of media types to show. To see what interface types your switch, router, or other net thing uses you can use this command: snmpwalk -c public -v 2c switch 1.3.6.1.2.1.2.2.1.3 It may show something like this: IF-MIB::ifType.1 = INTEGER: ethernetCsmacd(6) IF-MIB::ifType.4 = INTEGER: mplsTunnel(150) IF-MIB::ifType.5 = INTEGER: other(1) IF-MIB::ifType.6 = INTEGER: softwareLoopback(24) IF-MIB::ifType.8 = INTEGER: tunnel(131) IF-MIB::ifType.13 = INTEGER: propVirtual(53) IF-MIB::ifType.123 = INTEGER: l2vlan(135) propVirtual or l2vlan is usually used for VLAN interfaces. Tunnel would normally be for VPNs. A minor horde of different interface types are supposted, please see IANAifType-MIB (on your system or find with Google) for a full list. =head2 MULTIPLE INSTANCES It is possible to run multiple instances of the plugin for the same host. This is most useful to generate separate views for groups of ifTypes (e.g., ethernetCsmacd, ieee82011, l2vlan, ...). New instances should be symlinked to the script with an underscore-separated suffix. A matching configuration file should be present to specify the required ifTypes (or any other relevant parameters). $ readlink /etc/munin/plugins/snmp_192.2.0.1_if_combined_vlan /path/to/munin-contrib/plugins/snmp/snmp__if_combined $ sudo cat /etc/munin/plugin-conf.d/snmp_192.2.0.1_if_combined_vlan env.ifTypeOnly l2vlan The suffix will be appended to the graph base name, e.g., `snmp__if_combined_` will generate the `snmp_if_combined_` multigraph and associated series for the selected ``. =head2 STACKED ROOT GRAPH The `stackedRoot` option determines whether the root summary graph shows the traffic on each interface separately, or stacks them on top of one another to show the total traffic through the device. =head1 INTERPRETATION A single graph is generated with all the interfaces overlaid one over the other: incoming traffic is received on the interface from the connected device, outgoing is sent to it instead. With the `stackedRoot` option, traffic is plotted as stacked areas instead, and two additional series are calculated to show the total usage of the device. Those series are useful for loaning into higher-level summary graphs (but see BUGS below). Sub-graphs are created, one per interface, akin to `snmp__if_multi` plugin. =head1 MIB INFORMATION This plugin requires the IF-MIB the standard IETF MIB for network interfaces. It reports the contents of the IF-MIB::ifHCInOctets/IF-MIB::ifHCOutOctets if available, IF-MIB::ifInOctets/IF-MIB::ifOutOctets if not. The former are 64 bit counters only available with SNMP 2 and later. The later are 32 bit counters (see FEATURES below). =head1 MAGIC MARKERS #%# family=snmpauto #%# capabilities=snmpconf =head1 VERSION $Id: snmp__if_.in 1818 2009-01-03 19:29:30Z janl $ =head1 BUGS =head2 Missing indexing options Should support indexing by - ifIndex - ifName - ifDescr - ifAlias - mac-address (and anything else MRTG can index by) in addition to OID-index as now. =head2 Total traffic oddities in stackedRoot mode. When using the `stackedRoot` option, the total lines (`send_bits` and `recv_bits`) may sometimes jump to nonsensical values. This is a bug shared with the original `snmp__if_multi`. This seems to be due to the way those series are calculated, by summing the current values of the other counters. The first issue is that of summing 64-bit values into a single variable that is likely to also be 64 bits. The second issue (compounded by the first) is that of wraparound. When one of the counters wraps around, the sum jumps backwards. With a `min` set to 0, and other counters having kept increasing, this looks like a huge increase the total counter. Depending on the `max` (which should match the backplane bandwidth), this may not be correctly recognised as a spurious value, just recorded as valid. As a workaround, the `stackedMax` option is available. It will be set as the max values for the `send_bits` and `recv_bits` sum series, allowing to prevent overshoot. It should be set to around the expected maximum given the monitored network, rather than the sum of the theoretical maxes of the interfaces. This is a blunt tool that is not going to be very precise, but it should get rid of the largest outliers, keeping the graphs useful. Barring that, it should be possible to salvage the data after the fact. This can be done my either clipping all values beyond a maximum (e.g., the known use of the switch, rather that its full backplane bandwith). RANGE='[0-9.]\+e+\(0[789]\|[1-9][0-9]\)' # Anything >=1e07 rrdtool dump ${RRD_FILE} \ | sed "s/${RANGE}/NaN/" \ | rrdtool restore - ${RRD_FILE}.new mv ${RRD_FILE}.new ${RRD_FILE} # Make sure you have a backup! Alternatively, if you can determine the time at whiche the spurious value was recorded, you can just make that value a NaN. TIMESTAMP="1623492000" rrdtool dump "${RRD_FILE}" \ | sed "/${TIMESTAMP}/s/.*<\/v>/NaN<\/v>/" \ | rrdtool restore - "${RRD_FILE}.new" mv ${RRD_FILE}.new ${RRD_FILE} # Make sure you have a backup! =head1 FEATURES You may get strange results if you use SNMPv1, or SNMPv2 on switches that do not support 64 bit byte counters. If the interface traffic exceeds about 50Mbps a 32 bit byte counter will wrap around in less than 5 minutes making the graph for the interface show random results. If you have a switch/device that supports 64 bit byte counters this plugin will use them and the graph will be fine. The graph information will inform about this. You must use SNMPv2c or SNMPv3 to be able to use 64 bit counters - if the device supports them. This problem is a feature of the device SNMP implementation or your usage of it, it is nothing the plugin can fix. In the future Munin may be able to run the plugin more often than the counter wraps around. =head1 AUTHORS Copyright (C) 2012 Diego Elio Pettenò. Original snmp__if_multi: Copyright (C) 2004-2010 Jimmy Olsen, Dagfinn Ilmari Mannsåker, Nicolai Langfeldt, Redpill Linpro AS and others. Original snmp__if_ plugin: Copyright (C) 2004-2009 Jimmy Olsen, Dagfinn Ilmari Mannsåker. Initial SNMPv3 support by "Confusedhacker". Documentation, porting to Munin::Plugin::SNMP and further grooming by Nicolai Langfeldt. Reworked to snmp__if_multi by Nicolai Langfeldt. Reworked to snmp__if_combined by Diego Elio Pettenò. Stacked option by Olivier Mehani =head1 LICENSE GPLv2 =cut use strict; use Munin::Plugin; use Munin::Plugin::SNMP; my $scriptname='snmp_if_combined'; my @scriptname_components= split(/_/,$0); my $instance = 'All interfaces'; if ($scriptname_components[-1] ne 'combined') { $instance = $scriptname_components[-1]; $scriptname .= "_${instance}"; } my $response; my $iface; # This is the snmpwalk: # .1.3.6.1.2.1.2.1.0 = INTEGER: 2 # .1.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1 # .1.3.6.1.2.1.2.2.1.1.65539 = INTEGER: 65539 # .1.3.6.1.2.1.2.2.1.2.1 = STRING: MS TCP Loopback interface # .1.3.6.1.2.1.2.2.1.2.65539 = STRING: Broadcom NetXtreme Gigabit Ethernet # .1.3.6.1.2.1.2.2.1.3.1 = INTEGER: softwareLoopback(24) # .1.3.6.1.2.1.2.2.1.3.65539 = INTEGER: ethernetCsmacd(6) # .1.3.6.1.2.1.2.2.1.4.1 = INTEGER: 1520 # .1.3.6.1.2.1.2.2.1.4.65539 = INTEGER: 1500 # .1.3.6.1.2.1.2.2.1.5.1 = Gauge32: 10000000 # .1.3.6.1.2.1.2.2.1.5.65539 = Gauge32: 1000000000 # .1.3.6.1.2.1.2.2.1.6.1 = STRING: # .1.3.6.1.2.1.2.2.1.6.65539 = STRING: 0:30:48:75:65:5e # .1.3.6.1.2.1.2.2.1.7.1 = INTEGER: up(1) # .1.3.6.1.2.1.2.2.1.7.65539 = INTEGER: up(1) # .1.3.6.1.2.1.2.2.1.8.1 = INTEGER: up(1) # .1.3.6.1.2.1.2.2.1.8.65539 = INTEGER: up(1) # # 64 bit counters: # .1.3.6.1.2.1.31.1.1.1.6. Counter64 ifHCInOctets # .1.3.6.1.2.1.31.1.1.1.10. Counter64 ifHCOutOctets if (defined $ARGV[0] and $ARGV[0] eq "snmpconf") { print "require 1.3.6.1.2.1.2.2.1.1.\n"; exit 0; } my $stackedRoot = 0; if (exists $ENV{'stackedRoot'}) { $stackedRoot = $ENV{'stackedRoot'}; } my $stackedMax = 0; if (exists $ENV{'stackedMax'}) { $stackedMax = $ENV{'stackedMax'}; } my $sysDescr = '1.3.6.1.2.1.1.1.0'; my $sysLocation = '1.3.6.1.2.1.1.6.0'; my $sysContact = '1.3.6.1.2.1.1.4.0'; my $sysName = '1.3.6.1.2.1.1.5.0'; my $ifOIDBase = "1.3.6.1.2.1.2.2.1"; # ifEntry my $ifv2OIDBase = "1.3.6.1.2.1.31.1.1.1"; # ifXEntry my %ifTypes = ( other => 1, regular1822 => 2, hdh1822 => 3, ddnX25 => 4, rfc877x25 => 5, ethernetCsmacd => 6, iso88023Csmacd => 7, iso88024TokenBus => 8, iso88025TokenRing => 9, iso88026Man => 10, starLan => 11, proteon10Mbit => 12, proteon80Mbit => 13, hyperchannel => 14, fddi => 15, lapb => 16, sdlc => 17, ds1 => 18, e1 => 19, basicISDN => 20, primaryISDN => 21, propPointToPointSerial => 22, ppp => 23, softwareLoopback => 24, eon => 25, ethernet3Mbit => 26, nsip => 27, slip => 28, ultra => 29, ds3 => 30, sip => 31, frameRelay => 32, rs232 => 33, para => 34, arcnet => 35, arcnetPlus => 36, atm => 37, miox25 => 38, sonet => 39, x25ple => 40, iso88022llc => 41, localTalk => 42, smdsDxi => 43, frameRelayService => 44, v35 => 45, hssi => 46, hippi => 47, modem => 48, aal5 => 49, sonetPath => 50, sonetVT => 51, smdsIcip => 52, propVirtual => 53, propMultiplexor => 54, ieee80212 => 55, fibreChannel => 56, hippiInterface => 57, frameRelayInterconnect => 58, aflane8023 => 59, aflane8025 => 60, cctEmul => 61, fastEther => 62, isdn => 63, v11 => 64, v36 => 65, g703at64k => 66, g703at2mb => 67, qllc => 68, fastEtherFX => 69, channel => 70, ieee80211 => 71, ibm370parChan => 72, escon => 73, dlsw => 74, isdns => 75, isdnu => 76, lapd => 77, ipSwitch => 78, rsrb => 79, atmLogical => 80, ds0 => 81, ds0Bundle => 82, bsc => 83, async => 84, cnr => 85, iso88025Dtr => 86, eplrs => 87, arap => 88, propCnls => 89, hostPad => 90, termPad => 91, frameRelayMPI => 92, x213 => 93, adsl => 94, radsl => 95, sdsl => 96, vdsl => 97, iso88025CRFPInt => 98, myrinet => 99, voiceEM => 100, voiceFXO => 101, voiceFXS => 102, voiceEncap => 103, voiceOverIp => 104, atmDxi => 105, atmFuni => 106, atmIma => 107, pppMultilinkBundle => 108, ipOverCdlc => 109, ipOverClaw => 110, stackToStack => 111, virtualIpAddress => 112, mpc => 113, ipOverAtm => 114, iso88025Fiber => 115, tdlc => 116, gigabitEthernet => 117, hdlc => 118, lapf => 119, v37 => 120, x25mlp => 121, x25huntGroup => 122, trasnpHdlc => 123, interleave => 124, fast => 125, ip => 126, docsCableMaclayer => 127, docsCableDownstream => 128, docsCableUpstream => 129, a12MppSwitch => 130, tunnel => 131, coffee => 132, ces => 133, atmSubInterface => 134, l2vlan => 135, l3ipvlan => 136, l3ipxvlan => 137, digitalPowerline => 138, mediaMailOverIp => 139, dtm => 140, dcn => 141, ipForward => 142, msdsl => 143, ieee1394 => 144, 'if-gsn' => 145, dvbRccMacLayer => 146, dvbRccDownstream => 147, dvbRccUpstream => 148, atmVirtual => 149, mplsTunnel => 150, srp => 151, voiceOverAtm => 152, voiceOverFrameRelay => 153, idsl => 154, compositeLink => 155, ss7SigLink => 156, propWirelessP2P => 157, frForward => 158, rfc1483 => 159, usb => 160, ieee8023adLag => 161, bgppolicyaccounting => 162, frf16MfrBundle => 163, h323Gatekeeper => 164, h323Proxy => 165, mpls => 166, mfSigLink => 167, hdsl2 => 168, shdsl => 169, ds1FDL => 170, pos => 171, dvbAsiIn => 172, dvbAsiOut => 173, plc => 174, nfas => 175, tr008 => 176, gr303RDT => 177, gr303IDT => 178, isup => 179, propDocsWirelessMaclayer => 180, propDocsWirelessDownstream => 181, propDocsWirelessUpstream => 182, hiperlan2 => 183, propBWAp2Mp => 184, sonetOverheadChannel => 185, digitalWrapperOverheadChannel => 186, aal2 => 187, radioMAC => 188, atmRadio => 189, imt => 190, mvl => 191, reachDSL => 192, frDlciEndPt => 193, atmVciEndPt => 194, opticalChannel => 195, opticalTransport => 196, propAtm => 197, voiceOverCable => 198, infiniband => 199, teLink => 200, q2931 => 201, virtualTg => 202, sipTg => 203, sipSig => 204, docsCableUpstreamChannel => 205, econet => 206, pon155 => 207, pon622 => 208, bridge => 209, linegroup => 210, voiceEMFGD => 211, voiceFGDEANA => 212, voiceDID => 213, mpegTransport => 214, sixToFour => 215, gtp => 216, pdnEtherLoop1 => 217, pdnEtherLoop2 => 218, opticalChannelGroup => 219, homepna => 220, gfp => 221, ciscoISLvlan => 222, actelisMetaLOOP => 223, fcipLink => 224, rpr => 225, qam => 226, lmp => 227, cblVectaStar => 228, docsCableMCmtsDownstream => 229, adsl2 => 230, macSecControlledIF => 231, macSecUncontrolledIF => 232, aviciOpticalEther => 233, atmbond => 234, voiceFGDOS => 235, mocaVersion1 => 236, ieee80216WMAN => 237, adsl2plus => 238, dvbRcsMacLayer => 239, dvbTdm => 240, dvbRcsTdma => 241, x86Laps => 242, wwanPP => 243, wwanPP2 => 244 ); my %ifTypeByNum = map { $ifTypes{$_} => $_; } keys %ifTypes; # Needed as globals my $snmpinfo; my $snmpinfoX; sub do_collect { # Collect information from SNMP agent my $session = Munin::Plugin::SNMP->session(); $snmpinfo = $session->get_hash( -baseoid => $ifOIDBase, -cols => { 2 => 'ifDescr', # Type: See above 3 => 'ifType', 5 => 'ifSpeed', # Oper: 1) up 2) down 3) testing # 4) unknown, 5) dormant 6) not present # 7) lowerLayerDown 8 => 'ifOperStatus', 10 => 'ifInOctets', 13 => 'ifInDiscards', 14 => 'ifInErrors', 16 => 'ifOutOctets', 19 => 'ifOutDiscards', 20 => 'ifOutErrors', }); # ifXEntry - SNMP v2 and up only - on some devices $snmpinfoX = $session->get_hash( -baseoid => $ifv2OIDBase, -cols => { 6 => 'ifHCInOctets', 10 => 'ifHCOutOctets', 15 => 'ifHighSpeed', 18 => 'ifAlias', }); } sub do_preprocess_if { my ($mediatype, $if) = @_; my $response = $snmpinfo->{$if}->{ifType} || 1; if (defined($mediatype)) { if (defined($mediatype->{$response})) { # This is one of the interesting media types } else { # This media type is not interesting. Remove. delete $snmpinfo->{$if} if exists $snmpinfo->{$if}; delete $snmpinfoX->{$if} if exists $snmpinfoX->{$if}; return; } } if (defined ($response = $snmpinfo->{$if}->{ifOperStatus})) { # 1 = up, 2 = down, 7 = lowerLayerDown if ($response == 2 or $response == 7) { # Fold recognized down states into one. $response = $snmpinfo->{$if}->{ifOperStatus} = 2; } elsif ($response != 1) { # This interface is fishy, remove and forget. delete $snmpinfo->{$if} if exists $snmpinfo->{$if}; delete $snmpinfoX->{$if} if exists $snmpinfoX->{$if}; return; } } } sub do_preprocess { my $mediatypes = 'ALL'; if (exists( $ENV{'ifTypeOnly'} )) { $mediatypes = $ENV{'ifTypeOnly'}; } my @mediatypes = split(/[ ,]+/,$mediatypes); # Hash of numerical media types the user finds interesting. my $mediatype={}; if ($mediatypes eq 'ALL') { $mediatype = undef; } else { foreach my $type (@mediatypes) { if (exists($ifTypes{$type})) { $mediatype->{$ifTypes{$type}} = 1; } else { die "Unknown media type '$type' specified in ifTypeOnly\n"; } } } foreach my $if (keys %{$snmpinfo}) { do_preprocess_if($mediatype, $if); } } sub do_config_root { my ($host, $version) = @_; print < $b} keys %{$snmpinfo}) { print " recv$if=$scriptname.$if.recv send$if=$scriptname.$if.send"; push @ifs, "$scriptname.$if"; } if ($stackedRoot) { print " recv_bits send_bits"; } print "\n"; # XXX: For some reason, Munin doesn't resolve AREASTACK properly in this case... my $area_or_stack = "AREA"; foreach my $if (sort {$a <=> $b} keys %{$snmpinfo}) { my $alias = $snmpinfoX->{$if}->{ifAlias} || $snmpinfo->{$if}->{ifDescr} || "Interface $if"; if (! ($alias =~ /\d+/) ) { # If there are no numbers in the $alias add the if index $alias .=" (if $if)"; } print < 0) { print < $b} keys %{$snmpinfo}) { print " recv$if=${scriptname}_err.$if.total_in send$if=${scriptname}_err.$if.total_out"; } print "\n"; foreach my $if (sort {$a <=> $b} keys %{$snmpinfo}) { my $alias = $snmpinfoX->{$if}->{ifAlias} || $snmpinfo->{$if}->{ifDescr} || "Interface $if"; if (! ($alias =~ /\d+/) ) { # If there are no numbers in the $alias add the if index $alias .=" (if $if)"; } print <{$if}->{ifAlias} || $snmpinfo->{$if}->{ifDescr} || "Interface $if"; if (! ($alias =~ /\d+/) ) { # If there are no numbers in the $alias add the if index $alias .=" (if $if)"; } my $warn = undef; my $speed = 0; my $extrainfo=""; if (defined ($snmpinfoX->{$if}->{ifHCInOctets})) { # If we get an answer at the 64 bit OID then this switch # supports the extended MIB $extrainfo .= " This switch supports 64 bit byte counters and these are used by this plugin."; } else { # If not we only have a 32 bit counter and are lost. $extrainfo .= " NOTE! This switch supports only 32 bit byte counters which makes the plugin unreliable and unsuitable for most 100Mb (or faster) interfaces, where bursts are expected to exceed 50Mbps. This means that for interfaces where much traffic is sent this plugin will report false throughputs and cannot be trusted."; # unless perhaps the operator can get us snmp version 2c or 3? $extrainfo .= " I notice that you use SNMP version 1 which does not support 64 bit quantities. You may get better results if you switch to SNMP version 2c or 3. Please refer to the plugin documentation." if $version == 1; } if (defined ($speed = $snmpinfoX->{$if}->{ifHighSpeed}) and $speed) { # Speed in 1,000,000 bits per second $speed = $speed * 1000000; $warn = $speed / 75 * 100; my $textspeed = scaleNumber($speed,'bps','', 'The interface speed is %.1f%s%s.'); $extrainfo .= ' '.$textspeed if $textspeed; } elsif (defined ($speed = $snmpinfo->{$if}->{ifSpeed}) and $speed) { # Speed in bits pr. second $warn = $speed*100/75; my $textspeed = scaleNumber($speed,'bps','', 'The interface speed is %.1f%s%s.'); $extrainfo .= " ".$textspeed if $textspeed; } $response = $snmpinfo->{$if}->{ifType}; $extrainfo .= " This is a '".$ifTypeByNum{$response}."' interface."; print < 0) { printf("recv.max %s\nsend.max %s\n", $speed, $speed); } if ( defined($warn) ) { printf("recv.warning %s\nsend.warning %s\n", $warn, $warn); } print <{$if}->{ifOperStatus} || 1; # 1 means up # 2 means set to down # Everything else we ignore. if ($status != 1) { # Interface is down print <{$if}->{ifHCInOctets} || $snmpinfo->{$if}->{ifInOctets}; $recv = defined($response) ? ($response * 8) : undef; printf("recv.value %s\n", $recv || "U"); $response = $snmpinfoX->{$if}->{ifHCOutOctets} || $snmpinfo->{$if}->{ifOutOctets}; $send = defined($response) ? ($response * 8) : undef; printf("send.value %s\n", $send || "U"); print "multigraph ${scriptname}_err.$if\n"; my $errors = $snmpinfo->{$if}->{ifInErrors}; my $discards = $snmpinfo->{$if}->{ifInDiscards}; printf("errors_in.value %s\n". "discards_in.value %s\n". "total_in.value %s\n", $errors || "U", $discards || "U", ($errors || 0) + ($discards || 0) ); $errors = $snmpinfo->{$if}->{ifOutErrors}; $discards = $snmpinfo->{$if}->{ifOutDiscards}; printf("errors_out.value %s\n". "discards_out.value %s\n". "total_out.value %s\n", $errors || "U", $discards || "U", ($errors || 0) + ($discards || 0) ); return ($recv, $send); } sub do_config { my ($host,undef,$version) = Munin::Plugin::SNMP->config_session(); print "host_name $host\n" unless $host eq 'localhost'; foreach my $if (sort {$a <=> $b} keys %{$snmpinfo}) { do_config_if($host,$version,$if); } do_config_root($host,$version); } # ############################## MAIN ################################ do_collect; do_preprocess; if ($ARGV[0] and $ARGV[0] eq "config") { do_config(); exit 0; } my $recv = 0; my $send = 0, my $recv_if; my $send_if; foreach my $if (sort {$a <=> $b} keys %{$snmpinfo}) { ($recv_if, $send_if) = do_fetch_if($if); $recv += ($recv_if || 0); $send += ($send_if || 0); } if ($stackedMax > 0) { if ($recv > $stackedMax) { $recv = 'U'; } if ($send > $stackedMax) { $send = 'U'; } } if ($stackedRoot) { print "multigraph $scriptname\n"; print "recv_bits.value $recv\n"; print "send_bits.value $send\n"; }