Re: route nyxx-atm.colorado.edu <-> jsimpson-h.gsfc.nasa.gov

Bill Fink (bill@wizard.nasa.atd.net)
Tue, 24 Aug 1999 14:41:15 -0400 (EDT)

It appears that the Goddard NREN router (192.12.123.41) has a
static route for the 198.11.22 network, but that this static
route via the Chicago NGIX NREN router (198.10.64.50) is failing,
i.e. the Chicago router does not have a route via VBNS to
198.11.22. Because the route on the Goddard NREN router
is a static route, our ATM Firewall Router still learns
the 198.11.22 route via BGP from the Goddard NREN router,
and thus continues to pass traffic there causing it to
effectively be blackholed. If it was a dynamic rather than
static route, our router would fallback to using the NISN
router to get to 198.11.22, and although this would reduce
the effective throughput of the path, it would at least still
work.

-Bill

> Anne/Dave/Ken/Vishy,
>
> Sorry about the late reply. I was out all of last week.
>
> This is being CC'd to the NREN engineers as it possibly a routing
> problem in that part of the network.
>
> I did notice in the traceroute (in your message below) that there
> seems to be a routing problem possibly within the nren network.
> Instead of the route from 204.147.128.198 going to 198.10.64.49
> (gsfc-chi.nren.nasa.gov) it went to 198.10.64.34 (jpl-chi.nren.nasa.gov).
>
> This is the latest traceroute from jsimpson-h.gsfc.nasa.gov to
> nyxx-atm.Colorado.EDU (198.11.22.19) which stops just before the hop
> that should go to gsfc-chi.nren.nasa.gov (198.10.64.50).
>
> jsimpson% traceroute 198.11.22.19
> traceroute to 198.11.22.19 (198.11.22.19), 30 hops max, 96 byte packets
> 1 169.154.161.1 (169.154.161.1) 4.984 ms 4.298 ms 4.4 ms (4.0/4.000)
> 2 rtr-atm-a.gsfc.nasa.gov (169.154.1.1) 4.811 ms 4.708 ms 4.505 ms (4.0/4.000)
> 3 192.12.123.41 (192.12.123.41) 5.073 ms 5.808 ms 5.475 ms (5.0/5.000)
> 4 * * *
> 5 * * *
> --------------------------------------------------------------------------
> > From hammond@plutus.colorado.edu Wed Aug 18 13:49:23 1999
> > From: "Anne M. Hammond" <hammond@solarz.Colorado.EDU>
> > To: lang@gi-joe.gsfc.nasa.gov (Paul Lang)
> > Subject: Re: route nyxx-atm.colorado.edu <-> jsimpson-h.gsfc.nasa.gov
> > Cc: nren@plutus.colorado.edu, brummell@plutus.colorado.edu,
> > hammond@plutus.colorado.edu
> >
> > Hi Paul,
> > Our route from nyxx-atm.colorado.edu (198.11.22.19) is not functioning.
> > This was just reported this morning:
> >
> > nyxx 15% traceroute jsimpson-h.gsfc.nasa.gov
> > traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30 hops max, 40 byte
> > packets
> > 1 198.11.22.25 (198.11.22.25) 96 ms 1 ms 1 ms
> > 2 204.131.62.38 (204.131.62.38) 4 ms 3 ms 3 ms
> > 3 gsr-atm0-0-15.dng.vbns.net (204.147.135.178) 27 ms 27 ms 27 ms
> > 4 204.147.128.198 (204.147.128.198) 28 ms 29 ms 28 ms
> > 5 * jpl-chi.nren.nasa.gov (198.10.64.34) 83 ms *
> > 6 * * *
> > 7 * * *
> > 8 * * *
> > 9 * * *
> > 10 * * *
> >
> > We can get to it fine from 128.138.188.
> > Anne
> >
> > On Aug 4, 5:40pm, Paul Lang wrote:
> > > Subject: Re: route nyxx-atm.colorado.edu <-> jsimpson-h.gsfc.nasa.gov
> > > Anne,
> > >
> > > jsimpson-h has been moved to a new hippi switch and the giga router
> > > will be moved shortly. jsimpson now has a working atm interface but
> > > we need to transition jsimpson-a (and a number of other systems) to
> > > a new network number so that we can route for them via our atm router.
> > > We will send you updates on the above issues as things get worked out.
> > >
> > > - Paul
> > > =========================================================================
> > > > From dguevara@mail.arc.nasa.gov Wed Aug 4 00:07:51 1999
> > > > To: bill@wizard.nasa.atd.net
> > > > From: "Anne M. Hammond" <hammond@solarz.Colorado.EDU> (by way of David A.
> > > > Guevara)
> > > > Subject: route nyxx-atm.colorado.edu <-> jsimpson-h.gsfc.nasa.gov
> > > > Cc: hammond@solarz.Colorado.EDU, lang@gi-joe.gsfc.nasa.gov
> > > >
> > > > Bill --
> > > >
> > > > See below. I currently can't get to jsimpson-h. I'll try again
> > > > in the morning. The next hop in Anne's traceroute should
> > > > be jsimpson. The route looks correct up to our point-to-point
> > > > peer between NREN and your router.
> > > >
> > > > -- david
> > > >
> > > > ===========
> > > >
> > > > Hi David,
> > > >
> > > > The route between nyxx-atm.colorado.edu (198.11.22.19) has
> > > > gone down in the last "few" days (perhaps over the weekend?
> > > > Below is a traceroute, and some history.
> > > >
> > > > hydra-atm1 is down due to a system crash. But these are
> > > > on the same physical network.
> > > >
> > > > Thanks for your yeoman work in the past. I haven't heard
> > > > any summary of the differences in data rates; although
> > > > several people have accounts on our machines and had been
> > > > looking into it.
> > > >
> > > > Thanks for taking a look at this.
> > > >
> > > > Anne Hammond
> > > >
> > > >
> > > >
> > > > nyxx 32% traceroute jsimpson-h.gsfc.nasa.gov
> > > > traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30 hops max, 40
> > byte
> > > > packets
> > > > 1 198.11.22.25 (198.11.22.25) 2 ms 1 ms 1 ms
> > > > 2 204.131.62.38 (204.131.62.38) 4 ms 12 ms 3 ms
> > > > 3 cs-atm0-0-14.dng.vbns.net (204.147.129.185) 147 ms 27 ms 28 ms
> > > > 4 204.147.128.198 (204.147.128.198) 29 ms 29 ms 29 ms
> > > > 5 gsfc-chi.nren.nasa.gov (198.10.64.49) 44 ms 44 ms 44 ms
> > > > 6 192.12.123.42 (192.12.123.42) 44 ms 44 ms 45 ms
> > > > 7 rtr-900-giga-a.gsfc.nasa.gov (169.154.32.3) 44 ms 44 ms 45 ms
> > > > 8 * * *
> > > > 9 * * *
> > > > 10 * * *
> > > > 11 * * *
> > > >
> > > >
> > > >
> > > > On Jun 3, 6:59am, David A. Guevara wrote:
> > > > > Subject: RE: route from hydra-atm1.colorado.edu ->
> > jsimpson-h.gsfc.nasa.go
> > > > > Ron --
> > > > >
> > > > > Looks good to me. Thank you for your help.
> > > > >
> > > > > -- david
> > > > >
> > > > > At 7:18 PM -0700 1/1/88, Ron Bonica wrote:
> > > > > >Folks,
> > > > > >
> > > > > >We believe that this problem was corrected last night at about 10PM
> > eastern
> > > > > >time. Please contact us if you
> > > > > >have any further problems.
> > > > > >
> > > > > > Ron
> > > > > >
> > > > > >> -----Original Message-----
> > > > > >> From: Kevin Thompson [mailto:kthomp@mci.net]
> > > > > >> Sent: Wednesday, June 02, 1999 9:01 PM
> > > > > >> To: David A. Guevara; mgates@dast.nlanr.net
> > > > > >> Cc: vbns-eng@mci.net; dast@nlanr.net; bill@wizard.nasa.atd.net;
> > > > > >> hammond@solarz.Colorado.EDU; sgrant@psc.edu; ncne@ncne.org
> > > > > >> Subject: Re: route from hydra-atm1.colorado.edu ->
> > > > > >> jsimpson-h.gsfc.nasa.gov
> > > > > >>
> > > > > >>
> > > > > >> David, Mark, Anne, and Shirl,
> > > > > >>
> > > > > >> We verified the routing problems you reported, and are working to
> > restore
> > > > > >> several missing ospf adjacancies correlated to our Denver switches.
> > > > > >>
> > > > > >> We'll update you in the morning with our progress.
> > > > > >>
> > > > > >> Kevin Thompson
> > > > > >> vBNS Engineering
> > > > > >> 703-715-7073
> > > > > >>
> > > > > >> At 11:02 AM 6/2/99 -0700, David A. Guevara wrote:
> > > > > >> >mark --
> > > > > >> >
> > > > > >> >the last time this connection was "working" was back in late
> > > > > >> >february, early march. vbns might be able to provide
> > > > > >> >some input with regards to the additional router in our path.
> > > > > >> >I'll cc a copy of this message to their engineering group.
> > > > > >> >
> > > > > >> >-- david
> > > > > >> >
> > > > > >> >At 10:31 AM -0700 6/2/99, Mark Gates wrote:
> > > > > >> >>Hi Anne and David,
> > > > > >> >>
> > > > > >> >>I've been investigating the route that you seem to be using. It is
> > very
> > > > > >> >>peculiar to me. Do you know what dates you started having problems
> > > > > >> >>between the machines?
> > > > > >> >>
> > > > > >> >>I would say the problem is not congestion but routing. The two
> > things
> > > > > >> >>that looked odd to me are:
> > > > > >> >>
> > > > > >> >> 1) The RTO router that is listed on both traceroutes. RTO is Los
> > > > > >> >>Angeles, and why your packets would be going through there is a
> > mystery
> > > > > >> >>to me. They should go directly between Chicago (DNG) and Denver
> > (DNJ)
> > > > > >> >>routers.
> > > > > >> >>
> > > > > >> >> 2) Looking at various round-trip-time and traffic statistics, it
> > seems
> > > > > >> >>there was a definite change in the network on May 20th, 8am UTC. The
> > > > > >> >>round-trip-time between Denver and Chicago-based institutions jumped
> > > > > >> >>from 20ms to 40ms. There doesn't appear to be any traffic going
> > directly
> > > > > >> >>from the Chicago (DNG) router to the Denver (DNJ) router.
> > > > > >> >>
> > > > > >> >>I am going to have the NLANR engineering services group look more at
> > > > > >> >>this and hopefully discover and fix whatever routing is broken in
> > the
> > > > > >> >>network. Someone from there may contact you directly if they need
> > more
> > > > > >> >>information. You may still contact me also.
> > > > > >> >>
> > > > > >> >> -mark
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>Anne Hammond wrote:
> > > > > >> >>>
> > > > > >> >>> Hi,
> > > > > >> >>>
> > > > > >> >>> This message below from David Guevara of NASA NREN
> > > > > >> >>>
> > > > > >> >>> Would you be able to help in analyzing this slowdown?
> > > > > >> >>>
> > > > > >> >>> Anne M. Hammond Anne.Hammond@Colorado.EDU
> > > > > >> >>> Network/UNIX Admin 303-492-5578 (FAX -5325)
> > > > > >> >>> JILA / University of Colorado at Boulder
> > > > > >> >>> Boulder, CO 80309-0440
> > > > > >> >>>
> > > > > >> >>> ----- Begin Included Message -----
> > > > > >> >>>
> > > > > >> >>> >From dguevara@mail.arc.nasa.gov Tue Jun 1 08:59:02 1999
> > > > > >> >>> Mime-Version: 1.0
> > > > > >> >>> Content-Type: text/plain; charset="us-ascii"
> > > > > >> >>> Date: Tue, 1 Jun 1999 07:58:54 -0700
> > > > > >> >>> To: "Anne M. Hammond" <hammond@solarz.Colorado.EDU>
> > > > > >> >>> From: "David A. Guevara" <dguevara@mail.arc.nasa.gov>
> > > > > >> >>> Subject: Re: route from hydra-atm1.colorado.edu ->
> > > > > >> jsimpson-h.gsfc.nasa.gov
> > > > > >> >>> Cc: bill@wizard.nasa.atd.net, mitchell@jila.Colorado.EDU,
> > > > > >> >>> litvanyi@spot.Colorado.EDU, hammond@hydra.colorado.edu,
> > > > > >> >>> brummell@hydra.colorado.edu, dguevara@mail.arc.nasa.gov
> > > > > >> >>> Content-Length: 1929
> > > > > >> >>>
> > > > > >> >>> anne --
> > > > > >> >>>
> > > > > >> >>> the route between hydra-atm1 and jsimpson-h is back up. there
> > seems
> > > > > >> >>> to be a congestion problem. currently our rtt is running about
> > 104ms
> > > > > >> >>> (approximately twice the value we had at the beginning of
> > > > > >> this project).
> > > > > >> >>> I think the problem lies with vBNS. See below for traceroutes
> > from
> > > > > >> >>> hydra-atm1 and jsimpson-h:
> > > > > >> >>>
> > > > > >> >>> hydra <~>traceroute jsimpson-h.gsfc.nasa.gov
> > > > > >> >>> traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30
> > > > > >> hops max, 40
> > > > > >> >>> byte pa
> > > > > >> >>> ckets
> > > > > >> >>> 1 198.11.22.25 (198.11.22.25) 1 ms 1 ms 1 ms
> > > > > >> >>> 2 204.131.62.38 (204.131.62.38) 3 ms 3 ms 3 ms
> > > > > >>
> > > > > >> >>> 3 cs-atm0-0-14.rto.vbns.net (204.147.130.117) 33 ms 33 ms 33
> > ms
> > > > > >> >>> 4 cs-atm0-0-13.dng.vbns.net (204.147.129.145) 81 ms 204 ms 81
> > ms
> > > > > >> >>> 5 204.147.128.198 (204.147.128.198) 82 ms 82 ms 83 ms
> > > > > >> >>> 6 gsfc-chi.nren.nasa.gov (198.10.64.49) 97 ms 97 ms 97 ms
> > > > > >> >>> 7 192.12.123.42 (192.12.123.42) 98 ms 98 ms 98 ms
> > > > > >> >>> 8 rtr-900-giga-a.gsfc.nasa.gov (169.154.32.3) 98 ms 98 ms 98
> > ms
> > > > > >> >>> 9 * jsimpson-h.gsfc.nasa.gov (169.154.161.43) 100 ms 100 ms
> > > > > >> >>>
> > > > > >> >>> jsimpson% /etc/traceroute hydra-atm1.colorado.edu
> > > > > >> >>> traceroute to hydra-atm1.colorado.edu (198.11.22.12), 30 hops
> > > > > >> max, 96 byte
> > > > > >> >>> packe
> > > > > >> >>> ts
> > > > > >> >>> 1 169.154.161.1 (169.154.161.1) 6.397 ms 7.808 ms 5.467 ms
> > > > > >> (6.0/6.000)
> > > > > >> >>> 2 rtr-atm-a.gsfc.nasa.gov (169.154.1.1) 8.553 ms 8.924 ms
> > > > > >> 11.023 ms
> > > > > >> >>> (8.0/9.
> > > > > >> >>> 000)
> > > > > >> >>> 3 192.12.123.41 (192.12.123.41) 9.189 ms 5.739 ms 7.83
> > > > > >> ms (7.0/7.000)
> > > > > >> >>> 4 chi-gsfc.nren.nasa.gov (198.10.64.50) 20.339 ms 22.183
> > > > > >> ms 22.424 ms
> > > > > >> >>> (22.0
> > > > > >> >>> /21.333)
> > > > > >> >>> 5 204.147.128.197 (204.147.128.197) 21.977 ms 23.711 ms
> > 23.756 ms
> > > > > >> >>> (23.0/22.
> > > > > >> >>> 333)
> > > > > >> >>> 6 cs-atm0-0-12.rto.vbns.net (204.147.129.146) 69.767 ms 72.649
> > ms
> > > > > >> >>> 69.713 ms
> > > > > >> >>> (69.0/70.000)
> > > > > >> >>> 7 cs-atm0-0-14.dnj.vbns.net (204.147.130.118) 99.653 ms
> > 101.246 ms
> > > > > >> >>>105.729
> > > > > >> >>> ms (101.0/101.667)
> > > > > >> >>> 8 204.131.62.37 (204.131.62.37) 103.403 ms 106.348 ms 104.908
> > ms
> > > > > >> >>> (104.0/104
> > > > > >> >>> .333)
> > > > > >> >>> 9 hydra-atm1.Colorado.EDU (198.11.22.12) 103.892 ms 104.416 ms
> > > > > >> 107.273
> > > > > >> >>> ms (
> > > > > >> >>> 104.0/104.667)
> > > > > >> >>>
> > > > > >> >>> ----- End Included Message -----
> > > > > >> >>
> > > > > >> >>--
> > > > > >> >>Mark Gates 139 ACB
> > > > > >> >>NLANR applications support 217-244-7959
> > > > > >> >>http://www.ncsa.uiuc.edu/People/mgates mgates@nlanr.net
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > > >
> > > > >-- End of excerpt from David A. Guevara
> > > >
> > >
> > >-- End of excerpt from Paul Lang