nyxx <~>traceroute jsimpson-h.gsfc.nasa.gov
traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30 hops max, 40
byte packets
1 198.11.22.25 (198.11.22.25) 2 ms 1 ms 1 ms
2 204.131.62.38 (204.131.62.38) 4 ms 3 ms 3 ms
3 gsr-atm0-0-15.dng.vbns.net (204.147.135.178) 27 ms 27 ms 27 ms
4 204.147.128.198 (204.147.128.198) 29 ms 28 ms 29 ms
5 gsfc-chi.nren.nasa.gov (198.10.64.49) 43 ms 43 ms 43 ms
6 192.12.123.42 (192.12.123.42) 44 ms 43 ms 43 ms
7 rtr-900-giga-a.gsfc.nasa.gov (169.154.32.3) 44 ms 44 ms 43 ms
8 jsimpson-h.gsfc.nasa.gov (169.154.161.43) 46 ms 46 ms 46 ms
n
jsimpson% /etc/traceroute nyxx-atm.colorado.edu
traceroute to nyxx-atm.colorado.edu (198.11.22.19), 30 hops max, 96 byte
packets
1 169.154.161.1 (169.154.161.1) 5.709 ms 4.469 ms 4.61 ms (4.0/4.333)
2 rtr-atm-a.gsfc.nasa.gov (169.154.1.1) 4.874 ms 4.851 ms 4.444 ms
(4.0/4.000)
3 192.12.123.41 (192.12.123.41) 5.137 ms 5.717 ms 5.492 ms (5.0/5.000)
4 chi-gsfc.nren.nasa.gov (198.10.64.50) 20.612 ms 20.795 ms 19.593 ms
(20.0/19.667)
5 cs-atm0-0-2006.dng.vbns.net (204.147.128.197) 21.374 ms 21.356 ms
21.437 ms (21.0/21.000)
6 cs-atm0-0-0-24.dnj.vbns.net (204.147.135.177) 45.728 ms 45.649 ms
45.378 ms (45.0/45.000)
7 204.131.62.37 (204.131.62.37) 47.801 ms 48.173 ms 47.38 ms (47.0/47.333)
8 nyxx-atm.Colorado.EDU (198.11.22.19) 47.877 ms 47.459 ms 48.467 ms
(47.0/47.333)
At 11:41 AM -0700 8/24/99, Bill Fink wrote:
>It appears that the Goddard NREN router (192.12.123.41) has a
>static route for the 198.11.22 network, but that this static
>route via the Chicago NGIX NREN router (198.10.64.50) is failing,
>i.e. the Chicago router does not have a route via VBNS to
>198.11.22. Because the route on the Goddard NREN router
>is a static route, our ATM Firewall Router still learns
>the 198.11.22 route via BGP from the Goddard NREN router,
>and thus continues to pass traffic there causing it to
>effectively be blackholed. If it was a dynamic rather than
>static route, our router would fallback to using the NISN
>router to get to 198.11.22, and although this would reduce
>the effective throughput of the path, it would at least still
>work.
>
> -Bill
>
>
>
>> Anne/Dave/Ken/Vishy,
>>
>> Sorry about the late reply. I was out all of last week.
>>
>> This is being CC'd to the NREN engineers as it possibly a routing
>> problem in that part of the network.
>>
>> I did notice in the traceroute (in your message below) that there
>> seems to be a routing problem possibly within the nren network.
>> Instead of the route from 204.147.128.198 going to 198.10.64.49
>> (gsfc-chi.nren.nasa.gov) it went to 198.10.64.34 (jpl-chi.nren.nasa.gov).
>>
>> This is the latest traceroute from jsimpson-h.gsfc.nasa.gov to
>> nyxx-atm.Colorado.EDU (198.11.22.19) which stops just before the hop
>> that should go to gsfc-chi.nren.nasa.gov (198.10.64.50).
>>
>> jsimpson% traceroute 198.11.22.19
>> traceroute to 198.11.22.19 (198.11.22.19), 30 hops max, 96 byte packets
>> 1 169.154.161.1 (169.154.161.1) 4.984 ms 4.298 ms 4.4 ms (4.0/4.000)
>> 2 rtr-atm-a.gsfc.nasa.gov (169.154.1.1) 4.811 ms 4.708 ms 4.505 ms
>>(4.0/4.000)
>> 3 192.12.123.41 (192.12.123.41) 5.073 ms 5.808 ms 5.475 ms (5.0/5.000)
>> 4 * * *
>> 5 * * *
>> --------------------------------------------------------------------------
>> > From hammond@plutus.colorado.edu Wed Aug 18 13:49:23 1999
>> > From: "Anne M. Hammond" <hammond@solarz.Colorado.EDU>
>> > To: lang@gi-joe.gsfc.nasa.gov (Paul Lang)
>> > Subject: Re: route nyxx-atm.colorado.edu <-> jsimpson-h.gsfc.nasa.gov
>> > Cc: nren@plutus.colorado.edu, brummell@plutus.colorado.edu,
>> > hammond@plutus.colorado.edu
>> >
>> > Hi Paul,
>> > Our route from nyxx-atm.colorado.edu (198.11.22.19) is not functioning.
>> > This was just reported this morning:
>> >
>> > nyxx 15% traceroute jsimpson-h.gsfc.nasa.gov
>> > traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30 hops max,
>>40 byte
>> > packets
>> > 1 198.11.22.25 (198.11.22.25) 96 ms 1 ms 1 ms
>> > 2 204.131.62.38 (204.131.62.38) 4 ms 3 ms 3 ms
>> > 3 gsr-atm0-0-15.dng.vbns.net (204.147.135.178) 27 ms 27 ms 27 ms
>> > 4 204.147.128.198 (204.147.128.198) 28 ms 29 ms 28 ms
>> > 5 * jpl-chi.nren.nasa.gov (198.10.64.34) 83 ms *
>> > 6 * * *
>> > 7 * * *
>> > 8 * * *
>> > 9 * * *
>> > 10 * * *
>> >
>> > We can get to it fine from 128.138.188.
>> > Anne
>> >
>> > On Aug 4, 5:40pm, Paul Lang wrote:
>> > > Subject: Re: route nyxx-atm.colorado.edu <-> jsimpson-h.gsfc.nasa.gov
>> > > Anne,
>> > >
>> > > jsimpson-h has been moved to a new hippi switch and the giga router
>> > > will be moved shortly. jsimpson now has a working atm interface but
>> > > we need to transition jsimpson-a (and a number of other systems) to
>> > > a new network number so that we can route for them via our atm router.
>> > > We will send you updates on the above issues as things get worked out.
>> > >
>> > > - Paul
>> > >
>>=========================================================================
>> > > > From dguevara@mail.arc.nasa.gov Wed Aug 4 00:07:51 1999
>> > > > To: bill@wizard.nasa.atd.net
>> > > > From: "Anne M. Hammond" <hammond@solarz.Colorado.EDU> (by way of
>>David A.
>> > > > Guevara)
>> > > > Subject: route nyxx-atm.colorado.edu <-> jsimpson-h.gsfc.nasa.gov
>> > > > Cc: hammond@solarz.Colorado.EDU, lang@gi-joe.gsfc.nasa.gov
>> > > >
>> > > > Bill --
>> > > >
>> > > > See below. I currently can't get to jsimpson-h. I'll try again
>> > > > in the morning. The next hop in Anne's traceroute should
>> > > > be jsimpson. The route looks correct up to our point-to-point
>> > > > peer between NREN and your router.
>> > > >
>> > > > -- david
>> > > >
>> > > > ===========
>> > > >
>> > > > Hi David,
>> > > >
>> > > > The route between nyxx-atm.colorado.edu (198.11.22.19) has
>> > > > gone down in the last "few" days (perhaps over the weekend?
>> > > > Below is a traceroute, and some history.
>> > > >
>> > > > hydra-atm1 is down due to a system crash. But these are
>> > > > on the same physical network.
>> > > >
>> > > > Thanks for your yeoman work in the past. I haven't heard
>> > > > any summary of the differences in data rates; although
>> > > > several people have accounts on our machines and had been
>> > > > looking into it.
>> > > >
>> > > > Thanks for taking a look at this.
>> > > >
>> > > > Anne Hammond
>> > > >
>> > > >
>> > > >
>> > > > nyxx 32% traceroute jsimpson-h.gsfc.nasa.gov
>> > > > traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30 hops
>>max, 40
>> > byte
>> > > > packets
>> > > > 1 198.11.22.25 (198.11.22.25) 2 ms 1 ms 1 ms
>> > > > 2 204.131.62.38 (204.131.62.38) 4 ms 12 ms 3 ms
>> > > > 3 cs-atm0-0-14.dng.vbns.net (204.147.129.185) 147 ms 27 ms 28 ms
>> > > > 4 204.147.128.198 (204.147.128.198) 29 ms 29 ms 29 ms
>> > > > 5 gsfc-chi.nren.nasa.gov (198.10.64.49) 44 ms 44 ms 44 ms
>> > > > 6 192.12.123.42 (192.12.123.42) 44 ms 44 ms 45 ms
>> > > > 7 rtr-900-giga-a.gsfc.nasa.gov (169.154.32.3) 44 ms 44 ms 45 ms
>> > > > 8 * * *
>> > > > 9 * * *
>> > > > 10 * * *
>> > > > 11 * * *
>> > > >
>> > > >
>> > > >
>> > > > On Jun 3, 6:59am, David A. Guevara wrote:
>> > > > > Subject: RE: route from hydra-atm1.colorado.edu ->
>> > jsimpson-h.gsfc.nasa.go
>> > > > > Ron --
>> > > > >
>> > > > > Looks good to me. Thank you for your help.
>> > > > >
>> > > > > -- david
>> > > > >
>> > > > > At 7:18 PM -0700 1/1/88, Ron Bonica wrote:
>> > > > > >Folks,
>> > > > > >
>> > > > > >We believe that this problem was corrected last night at about 10PM
>> > eastern
>> > > > > >time. Please contact us if you
>> > > > > >have any further problems.
>> > > > > >
>> > > > > > Ron
>> > > > > >
>> > > > > >> -----Original Message-----
>> > > > > >> From: Kevin Thompson [mailto:kthomp@mci.net]
>> > > > > >> Sent: Wednesday, June 02, 1999 9:01 PM
>> > > > > >> To: David A. Guevara; mgates@dast.nlanr.net
>> > > > > >> Cc: vbns-eng@mci.net; dast@nlanr.net; bill@wizard.nasa.atd.net;
>> > > > > >> hammond@solarz.Colorado.EDU; sgrant@psc.edu; ncne@ncne.org
>> > > > > >> Subject: Re: route from hydra-atm1.colorado.edu ->
>> > > > > >> jsimpson-h.gsfc.nasa.gov
>> > > > > >>
>> > > > > >>
>> > > > > >> David, Mark, Anne, and Shirl,
>> > > > > >>
>> > > > > >> We verified the routing problems you reported, and are working to
>> > restore
>> > > > > >> several missing ospf adjacancies correlated to our Denver
>>switches.
>> > > > > >>
>> > > > > >> We'll update you in the morning with our progress.
>> > > > > >>
>> > > > > >> Kevin Thompson
>> > > > > >> vBNS Engineering
>> > > > > >> 703-715-7073
>> > > > > >>
>> > > > > >> At 11:02 AM 6/2/99 -0700, David A. Guevara wrote:
>> > > > > >> >mark --
>> > > > > >> >
>> > > > > >> >the last time this connection was "working" was back in late
>> > > > > >> >february, early march. vbns might be able to provide
>> > > > > >> >some input with regards to the additional router in our path.
>> > > > > >> >I'll cc a copy of this message to their engineering group.
>> > > > > >> >
>> > > > > >> >-- david
>> > > > > >> >
>> > > > > >> >At 10:31 AM -0700 6/2/99, Mark Gates wrote:
>> > > > > >> >>Hi Anne and David,
>> > > > > >> >>
>> > > > > >> >>I've been investigating the route that you seem to be
>>using. It is
>> > very
>> > > > > >> >>peculiar to me. Do you know what dates you started having
>>problems
>> > > > > >> >>between the machines?
>> > > > > >> >>
>> > > > > >> >>I would say the problem is not congestion but routing. The two
>> > things
>> > > > > >> >>that looked odd to me are:
>> > > > > >> >>
>> > > > > >> >> 1) The RTO router that is listed on both traceroutes. RTO
>>is Los
>> > > > > >> >>Angeles, and why your packets would be going through there is a
>> > mystery
>> > > > > >> >>to me. They should go directly between Chicago (DNG) and Denver
>> > (DNJ)
>> > > > > >> >>routers.
>> > > > > >> >>
>> > > > > >> >> 2) Looking at various round-trip-time and traffic
>>statistics, it
>> > seems
>> > > > > >> >>there was a definite change in the network on May 20th, 8am
>>UTC. The
>> > > > > >> >>round-trip-time between Denver and Chicago-based
>>institutions jumped
>> > > > > >> >>from 20ms to 40ms. There doesn't appear to be any traffic going
>> > directly
>> > > > > >> >>from the Chicago (DNG) router to the Denver (DNJ) router.
>> > > > > >> >>
>> > > > > >> >>I am going to have the NLANR engineering services group
>>look more at
>> > > > > >> >>this and hopefully discover and fix whatever routing is
>>broken in
>> > the
>> > > > > >> >>network. Someone from there may contact you directly if
>>they need
>> > more
>> > > > > >> >>information. You may still contact me also.
>> > > > > >> >>
>> > > > > >> >> -mark
>> > > > > >> >>
>> > > > > >> >>
>> > > > > >> >>Anne Hammond wrote:
>> > > > > >> >>>
>> > > > > >> >>> Hi,
>> > > > > >> >>>
>> > > > > >> >>> This message below from David Guevara of NASA NREN
>> > > > > >> >>>
>> > > > > >> >>> Would you be able to help in analyzing this slowdown?
>> > > > > >> >>>
>> > > > > >> >>> Anne M. Hammond Anne.Hammond@Colorado.EDU
>> > > > > >> >>> Network/UNIX Admin 303-492-5578 (FAX -5325)
>> > > > > >> >>> JILA / University of Colorado at Boulder
>> > > > > >> >>> Boulder, CO 80309-0440
>> > > > > >> >>>
>> > > > > >> >>> ----- Begin Included Message -----
>> > > > > >> >>>
>> > > > > >> >>> >From dguevara@mail.arc.nasa.gov Tue Jun 1 08:59:02 1999
>> > > > > >> >>> Mime-Version: 1.0
>> > > > > >> >>> Content-Type: text/plain; charset="us-ascii"
>> > > > > >> >>> Date: Tue, 1 Jun 1999 07:58:54 -0700
>> > > > > >> >>> To: "Anne M. Hammond" <hammond@solarz.Colorado.EDU>
>> > > > > >> >>> From: "David A. Guevara" <dguevara@mail.arc.nasa.gov>
>> > > > > >> >>> Subject: Re: route from hydra-atm1.colorado.edu ->
>> > > > > >> jsimpson-h.gsfc.nasa.gov
>> > > > > >> >>> Cc: bill@wizard.nasa.atd.net, mitchell@jila.Colorado.EDU,
>> > > > > >> >>> litvanyi@spot.Colorado.EDU,
>>hammond@hydra.colorado.edu,
>> > > > > >> >>> brummell@hydra.colorado.edu,
>>dguevara@mail.arc.nasa.gov
>> > > > > >> >>> Content-Length: 1929
>> > > > > >> >>>
>> > > > > >> >>> anne --
>> > > > > >> >>>
>> > > > > >> >>> the route between hydra-atm1 and jsimpson-h is back up.
>>there
>> > seems
>> > > > > >> >>> to be a congestion problem. currently our rtt is running
>>about
>> > 104ms
>> > > > > >> >>> (approximately twice the value we had at the beginning of
>> > > > > >> this project).
>> > > > > >> >>> I think the problem lies with vBNS. See below for
>>traceroutes
>> > from
>> > > > > >> >>> hydra-atm1 and jsimpson-h:
>> > > > > >> >>>
>> > > > > >> >>> hydra <~>traceroute jsimpson-h.gsfc.nasa.gov
>> > > > > >> >>> traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30
>> > > > > >> hops max, 40
>> > > > > >> >>> byte pa
>> > > > > >> >>> ckets
>> > > > > >> >>> 1 198.11.22.25 (198.11.22.25) 1 ms 1 ms 1 ms
>> > > > > >> >>> 2 204.131.62.38 (204.131.62.38) 3 ms 3 ms 3 ms
>> > > > > >>
>> > > > > >> >>> 3 cs-atm0-0-14.rto.vbns.net (204.147.130.117) 33 ms
>>33 ms 33
>> > ms
>> > > > > >> >>> 4 cs-atm0-0-13.dng.vbns.net (204.147.129.145) 81 ms
>>204 ms 81
>> > ms
>> > > > > >> >>> 5 204.147.128.198 (204.147.128.198) 82 ms 82 ms 83 ms
>> > > > > >> >>> 6 gsfc-chi.nren.nasa.gov (198.10.64.49) 97 ms 97 ms
>>97 ms
>> > > > > >> >>> 7 192.12.123.42 (192.12.123.42) 98 ms 98 ms 98 ms
>> > > > > >> >>> 8 rtr-900-giga-a.gsfc.nasa.gov (169.154.32.3) 98 ms
>>98 ms 98
>> > ms
>> > > > > >> >>> 9 * jsimpson-h.gsfc.nasa.gov (169.154.161.43) 100 ms
>>100 ms
>> > > > > >> >>>
>> > > > > >> >>> jsimpson% /etc/traceroute hydra-atm1.colorado.edu
>> > > > > >> >>> traceroute to hydra-atm1.colorado.edu (198.11.22.12), 30 hops
>> > > > > >> max, 96 byte
>> > > > > >> >>> packe
>> > > > > >> >>> ts
>> > > > > >> >>> 1 169.154.161.1 (169.154.161.1) 6.397 ms 7.808 ms
>>5.467 ms
>> > > > > >> (6.0/6.000)
>> > > > > >> >>> 2 rtr-atm-a.gsfc.nasa.gov (169.154.1.1) 8.553 ms 8.924 ms
>> > > > > >> 11.023 ms
>> > > > > >> >>> (8.0/9.
>> > > > > >> >>> 000)
>> > > > > >> >>> 3 192.12.123.41 (192.12.123.41) 9.189 ms 5.739 ms 7.83
>> > > > > >> ms (7.0/7.000)
>> > > > > >> >>> 4 chi-gsfc.nren.nasa.gov (198.10.64.50) 20.339 ms 22.183
>> > > > > >> ms 22.424 ms
>> > > > > >> >>> (22.0
>> > > > > >> >>> /21.333)
>> > > > > >> >>> 5 204.147.128.197 (204.147.128.197) 21.977 ms 23.711 ms
>> > 23.756 ms
>> > > > > >> >>> (23.0/22.
>> > > > > >> >>> 333)
>> > > > > >> >>> 6 cs-atm0-0-12.rto.vbns.net (204.147.129.146) 69.767
>>ms 72.649
>> > ms
>> > > > > >> >>> 69.713 ms
>> > > > > >> >>> (69.0/70.000)
>> > > > > >> >>> 7 cs-atm0-0-14.dnj.vbns.net (204.147.130.118) 99.653 ms
>> > 101.246 ms
>> > > > > >> >>>105.729
>> > > > > >> >>> ms (101.0/101.667)
>> > > > > >> >>> 8 204.131.62.37 (204.131.62.37) 103.403 ms 106.348 ms
>>104.908
>> > ms
>> > > > > >> >>> (104.0/104
>> > > > > >> >>> .333)
>> > > > > >> >>> 9 hydra-atm1.Colorado.EDU (198.11.22.12) 103.892 ms
>>104.416 ms
>> > > > > >> 107.273
>> > > > > >> >>> ms (
>> > > > > >> >>> 104.0/104.667)
>> > > > > >> >>>
>> > > > > >> >>> ----- End Included Message -----
>> > > > > >> >>
>> > > > > >> >>--
>> > > > > >> >>Mark Gates 139 ACB
>> > > > > >> >>NLANR applications support 217-244-7959
>> > > > > >> >>http://www.ncsa.uiuc.edu/People/mgates mgates@nlanr.net
>> > > > > >> >
>> > > > > >>
>> > > > > >>
>> > > > >
>> > > > >
>> > > > >
>> > > > >-- End of excerpt from David A. Guevara
>> > > >
>> > >
>> > >-- End of excerpt from Paul Lang