P1Fanatic

Got a customer whose running a full E1 onto a router and they are getting spurious packet loss. We are seeing no problems with the telco circuit. The only thing I can see is that at just over 1.2mb throughput the cpu util is 65%. I was thinking of trying out a 2600 instead (seeing as the 2500 is a bit long in the tooth) but thought Id see if theres anything else I can try config wise to improve the situation e.g. diff queing strategy. IP traffic seems to be the main culprit

xxxxxxx#sh proc cpu
CPU utilization for five seconds: 64%/39%; one minute: 49%; five minutes: 46%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 65268 257870 253 0.00% 0.00% 0.00% 0 Load Meter
2 336 469 716 0.00% 0.00% 0.00% 0 TACACS+
3 2624336 28632 91657 0.00% 0.13% 0.16% 0 Check heaps
4 1028 326 3153 0.00% 0.00% 0.00% 0 Pool Manager
5 0 2 0 0.00% 0.00% 0.00% 0 Timers
6 4 90 44 0.00% 0.00% 0.00% 0 Serial Background
7 79384 53864 1473 0.32% 0.03% 0.00% 0 ARP Input
8 4 2 2000 0.00% 0.00% 0.00% 0 DDR Timers
9 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
10 3024640 899391 3362 13.34% 15.48% 13.34% 0 IP Input
11 23408 129145 181 0.00% 0.00% 0.00% 0 CDP Protocol
12 4 1 4000 0.00% 0.00% 0.00% 0 PERUSER aux
13 116 314 369 0.00% 0.00% 0.00% 0 TCP Timer
14 140 39 3589 0.00% 0.00% 0.00% 0 TCP Protocols
15 0 1 0 0.00% 0.00% 0.00% 0 Probe Input
16 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
17 232 803 288 0.00% 0.00% 0.00% 0 traffic_shape
18 1188028 1310013 906 0.16% 0.10% 0.12% 0 IP Background
19 212960 21472 9918 0.00% 0.01% 0.00% 0 IP Cache Ager
20 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
21 0 1 0 0.00% 0.00% 0.00% 0 ISDN Timer
22 9312 552 16869 11.54% 1.57% 0.97% 2 Virtual Exec
23 0 1 0 0.00% 0.00% 0.00% 0 CallMIB Background
24 0 1 0 0.00% 0.00% 0.00% 0 ISDNMIB Background
25 0 1 0 0.00% 0.00% 0.00% 0 Entity MIB API
26 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
27 380 268 1417 0.00% 0.00% 0.00% 0 Net Background
28 56 60 933 0.00% 0.00% 0.00% 0 Logger
29 274520 1288434 213 0.00% 0.00% 0.00% 0 TTY Background
30 175560 1288446 136 0.00% 0.00% 0.00% 0 Per-Second Jobs
31 353040 1288445 274 0.16% 0.03% 0.01% 0 Net Periodic
32 78720 128574 612 0.00% 0.00% 0.00% 0 Net Input
33 71388 257871 276 0.00% 0.01% 0.00% 0 Compute load avgs
34 1190996 21479 55449 0.00% 0.04% 0.05% 0 Per-minute Jobs
35 185700 1350255 137 0.00% 0.00% 0.00% 0 NTP
36 296 51 5803 0.00% 0.00% 0.00% 0 AAA Accounting
37 586828 21519 27270 0.00% 0.02% 0.00% 0 IP-RT Background
38 0 1 0 0.00% 0.00% 0.00% 0 IP NAT Ager

siftltd01-gw#sh int e0
Ethernet0 is up, line protocol is up
Hardware is Lance, address is 0030.94e6.282e (bia 0030.94e6.282e)
Internet address is xxx.xxx.xxx.xxx/xx
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, rely 255/255, load 14/255
Encapsulation ARPA, loopback not set, keepalive set (10 sec)
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:03, output 00:00:00, output hang never
Last clearing of "show interface" counters 4d04h
Queueing strategy: fifo
Output queue 0/40, 439 drops; input queue 0/75, 37 drops
5 minute input rate 592000 bits/sec, 183 packets/sec
5 minute output rate 569000 bits/sec, 178 packets/sec
58554340 packets input, 1706755581 bytes, 3 no buffer
Received 423 broadcasts, 0 runts, 0 giants, 37 throttles
109 input errors, 0 CRC, 0 frame, 0 overrun, 109 ignored, 0 abort
0 input packets with dribble condition detected
58422020 packets output, 2465953227 bytes, 0 underruns
0 output errors, 1019502 collisions, 1 interface resets
0 babbles, 0 late collision, 966295 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
siftltd01-gw#sh int s1
Serial1 is up, line protocol is up
Hardware is HD64570
Internet address is xxx.xxx.xxx.xxx/xx
MTU 1500 bytes, BW 2048 Kbit, DLY 20000 usec, rely 255/255, load 154/255
Encapsulation HDLC, loopback not set, keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 4d04h
Queueing strategy: fifo
Output queue 0/40, 16190 drops; input queue 8/75, 70 drops
5 minute input rate 1248000 bits/sec, 240 packets/sec
5 minute output rate 1243000 bits/sec, 245 packets/sec
58488597 packets input, 1945719285 bytes, 50 no buffer
Received 36113 broadcasts, 0 runts, 0 giants, 0 throttles
323 input errors, 0 CRC, 0 frame, 161 overrun, 161 ignored, 0 abort
58652352 packets output, 1187912888 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

[Edited by P1Fanatic - 9/27/2002 2:10:29 PM]

Jeff Wiltshire

Err....

I think that you may want to take the IP addresses out of your post...called me paranoid (!)

Seems a lot of collisions on the ethernet segment ???


Jeff

P1Fanatic

Fair point. Was just in a bit of a hurry. I saw the collisions but would that really result in 10% packet loss for a customer when we cant replicate any packet loss from an extended ping using source address e0? Especially when no excessive collisions and low multiple collisions:

#sh controller ethernet
LANCE unit 0, idb 0x891E0, ds 0x8A928, regaddr = 0x2130000, reset_mask 0x2
IB at 0x206E64: mode=0x0000, mcfilter 0000/0000/0100/0000
station address 0030.94e6.282e default station address 0030.94e6.282e
buffer size 1524
RX ring with 16 entries at 0x206EA8
Rxhead = 0x206EE8 (8), Rxp = 0x8A964 (8)
00 pak=0x08D1D8 ds=0x20F71A status=0x80 max_size=1524 pak_size=72
01 pak=0x08AFD0 ds=0x207DE2 status=0x80 max_size=1524 pak_size=206
02 pak=0x08B398 ds=0x208B5A status=0x80 max_size=1524 pak_size=134
03 pak=0x08BB28 ds=0x20A64A status=0x80 max_size=1524 pak_size=70
04 pak=0x08B1B4 ds=0x20849E status=0x80 max_size=1524 pak_size=662
05 pak=0x08C49C ds=0x20C7F6 status=0x80 max_size=1524 pak_size=1486
06 pak=0x08B57C ds=0x209216 status=0x80 max_size=1524 pak_size=70
07 pak=0x08C0D4 ds=0x20BA7E status=0x80 max_size=1524 pak_size=158
08 pak=0x08C864 ds=0x20D56E status=0x80 max_size=1524 pak_size=150
09 pak=0x08D5A0 ds=0x210492 status=0x80 max_size=1524 pak_size=198
10 pak=0x08CA48 ds=0x20DC2A status=0x80 max_size=1524 pak_size=70
11 pak=0x08CFF4 ds=0x20F05E status=0x80 max_size=1524 pak_size=70
12 pak=0x08DF14 ds=0x21263E status=0x80 max_size=1524 pak_size=278
13 pak=0x08E4C0 ds=0x213A72 status=0x80 max_size=1524 pak_size=158
14 pak=0x08CC2C ds=0x20E2E6 status=0x80 max_size=1524 pak_size=182
15 pak=0x08B944 ds=0x209F8E status=0x80 max_size=1524 pak_size=190
TX ring with 4 entries at 0x206F68, tx_count = 0
tx_head = 0x206F80 (3), head_txp = 0x8A9A4 (3)
tx_tail = 0x206F80 (3), tail_txp = 0x8A9A4 (3)
00 pak=0x000000 ds=0x22CE02 status=0x03 status2=0x0000 pak_size=1514
01 pak=0x000000 ds=0x229822 status=0x03 status2=0x0000 pak_size=1514
02 pak=0x000000 ds=0x22AC56 status=0x03 status2=0x0000 pak_size=1514
03 pak=0x000000 ds=0x227D32 status=0x03 status2=0x0000 pak_size=178
6 missed datagrams, 0 overruns
0 transmitter underruns, 0 excessive collisions
5498 single collisions, 466 multiple collisions
0 dma memory errors, 0 CRC errors

0 alignment errors, 0 runts, 0 giants
0 tdr, 0 spurious initialization done interrupts
0 no enp status, 0 buffer errors, 0 overflow errors
0 tx_buff, 91 throttled, 91 enabled
Lance csr0 = 0x73

SiCotty

From Cisco Web Site

Interface Ethernet0 (up/up)
WARNING: The counters are older than 24 hours and may not accurately reflect
the current status of this interface.
TRY THIS: Reset the counters using the 'clear counters' command and wait a few
minutes to resubmit the output to Output Interpreter.

WARNING: The collision rate is 1.7151%, which is greater than 0.1%.
TRY THIS: Look for unterminated or overly long ethernet cables, and/or
malfunctioning transceiver(s). This may require a host-by-host inspection or
the use of a protocol analyzer.
Consider reducing the number of hosts on the segment to reduce the collison
rate, or subdividing the collision domain using switches/bridges.
REFERENCE: For further information see Troubleshooting Ethernet and
Troubleshooting Ethernet Collisions
Optimizing Your Network: Replacing Hubs with Desktop Switches

WARNING: 109 packets have been ignored by the interface because
the interface hardware ran low on internal buffers.
TRY THIS: Monitor the ignored packets over time. If they are increasing,
paste the output from the 'show buffers'
command into Output Interpreter to see if the buffers can be tuned. Also
compare with the 'no buffer' counter and input/output queue drops. Broadcast
storms can cause the 'ignored' counter to increment


As the top one said reset the counters, wait for about five mins and then run the show command again and post the results. I would also consider looking at the number of hosts on the lan etc as the collision rate is very high. A 10MB Ethernet network can only handle about 2-3MBits unit it becoms saturated with traffic. Are their any other devices on the network that exchange large amounts of traffic such as file servers and clients. If the number of clients is relavively small I would recommend upgrading to a switch.

A 2500 should be able to handle 2MB easily which will put them near to 80-90% CPU but this is fine if the router is only doing this one task.

Si



[Edited by SiCotty - 9/27/2002 4:34:39 PM]

P1Fanatic

Ah well turned out to be telco related. They turned up onsite replaced NTU and bang customer says its running fine.