Thanks! Everything is back to normal once more.
-Bill
> Fixed. See traceroutes below. My apologies, our switch in Chicago
> has been having some problems. The PVC's between GSFC and Chicago
> had to be re-entered
>
> nyxx <~>traceroute jsimpson-h.gsfc.nasa.gov
> traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30 hops max, 40
> byte packets
> 1 198.11.22.25 (198.11.22.25) 2 ms 1 ms 1 ms
> 2 204.131.62.38 (204.131.62.38) 4 ms 3 ms 3 ms
> 3 gsr-atm0-0-15.dng.vbns.net (204.147.135.178) 27 ms 27 ms 27 ms
> 4 204.147.128.198 (204.147.128.198) 29 ms 28 ms 29 ms
> 5 gsfc-chi.nren.nasa.gov (198.10.64.49) 43 ms 43 ms 43 ms
> 6 192.12.123.42 (192.12.123.42) 44 ms 43 ms 43 ms
> 7 rtr-900-giga-a.gsfc.nasa.gov (169.154.32.3) 44 ms 44 ms 43 ms
> 8 jsimpson-h.gsfc.nasa.gov (169.154.161.43) 46 ms 46 ms 46 ms
> n
>
> jsimpson% /etc/traceroute nyxx-atm.colorado.edu
> traceroute to nyxx-atm.colorado.edu (198.11.22.19), 30 hops max, 96 byte
> packets
> 1 169.154.161.1 (169.154.161.1) 5.709 ms 4.469 ms 4.61 ms (4.0/4.333)
> 2 rtr-atm-a.gsfc.nasa.gov (169.154.1.1) 4.874 ms 4.851 ms 4.444 ms
> (4.0/4.000)
> 3 192.12.123.41 (192.12.123.41) 5.137 ms 5.717 ms 5.492 ms (5.0/5.000)
> 4 chi-gsfc.nren.nasa.gov (198.10.64.50) 20.612 ms 20.795 ms 19.593 ms
> (20.0/19.667)
> 5 cs-atm0-0-2006.dng.vbns.net (204.147.128.197) 21.374 ms 21.356 ms
> 21.437 ms (21.0/21.000)
> 6 cs-atm0-0-0-24.dnj.vbns.net (204.147.135.177) 45.728 ms 45.649 ms
> 45.378 ms (45.0/45.000)
> 7 204.131.62.37 (204.131.62.37) 47.801 ms 48.173 ms 47.38 ms (47.0/47.333)
> 8 nyxx-atm.Colorado.EDU (198.11.22.19) 47.877 ms 47.459 ms 48.467 ms
> (47.0/47.333)
>
> At 11:41 AM -0700 8/24/99, Bill Fink wrote:
> >It appears that the Goddard NREN router (192.12.123.41) has a
> >static route for the 198.11.22 network, but that this static
> >route via the Chicago NGIX NREN router (198.10.64.50) is failing,
> >i.e. the Chicago router does not have a route via VBNS to
> >198.11.22. Because the route on the Goddard NREN router
> >is a static route, our ATM Firewall Router still learns
> >the 198.11.22 route via BGP from the Goddard NREN router,
> >and thus continues to pass traffic there causing it to
> >effectively be blackholed. If it was a dynamic rather than
> >static route, our router would fallback to using the NISN
> >router to get to 198.11.22, and although this would reduce
> >the effective throughput of the path, it would at least still
> >work.
> >
> > -Bill
> >
> >
> >
> >> Anne/Dave/Ken/Vishy,
> >>
> >> Sorry about the late reply. I was out all of last week.
> >>
> >> This is being CC'd to the NREN engineers as it possibly a routing
> >> problem in that part of the network.
> >>
> >> I did notice in the traceroute (in your message below) that there
> >> seems to be a routing problem possibly within the nren network.
> >> Instead of the route from 204.147.128.198 going to 198.10.64.49
> >> (gsfc-chi.nren.nasa.gov) it went to 198.10.64.34 (jpl-chi.nren.nasa.gov).
> >>
> >> This is the latest traceroute from jsimpson-h.gsfc.nasa.gov to
> >> nyxx-atm.Colorado.EDU (198.11.22.19) which stops just before the hop
> >> that should go to gsfc-chi.nren.nasa.gov (198.10.64.50).
> >>
> >> jsimpson% traceroute 198.11.22.19
> >> traceroute to 198.11.22.19 (198.11.22.19), 30 hops max, 96 byte packets
> >> 1 169.154.161.1 (169.154.161.1) 4.984 ms 4.298 ms 4.4 ms (4.0/4.000)
> >> 2 rtr-atm-a.gsfc.nasa.gov (169.154.1.1) 4.811 ms 4.708 ms 4.505 ms
> >>(4.0/4.000)
> >> 3 192.12.123.41 (192.12.123.41) 5.073 ms 5.808 ms 5.475 ms (5.0/5.000)
> >> 4 * * *
> >> 5 * * *
> >> --------------------------------------------------------------------------
> >> > From hammond@plutus.colorado.edu Wed Aug 18 13:49:23 1999
> >> > From: "Anne M. Hammond" <hammond@solarz.Colorado.EDU>
> >> > To: lang@gi-joe.gsfc.nasa.gov (Paul Lang)
> >> > Subject: Re: route nyxx-atm.colorado.edu <-> jsimpson-h.gsfc.nasa.gov
> >> > Cc: nren@plutus.colorado.edu, brummell@plutus.colorado.edu,
> >> > hammond@plutus.colorado.edu
> >> >
> >> > Hi Paul,
> >> > Our route from nyxx-atm.colorado.edu (198.11.22.19) is not functioning.
> >> > This was just reported this morning:
> >> >
> >> > nyxx 15% traceroute jsimpson-h.gsfc.nasa.gov
> >> > traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30 hops max,
> >>40 byte
> >> > packets
> >> > 1 198.11.22.25 (198.11.22.25) 96 ms 1 ms 1 ms
> >> > 2 204.131.62.38 (204.131.62.38) 4 ms 3 ms 3 ms
> >> > 3 gsr-atm0-0-15.dng.vbns.net (204.147.135.178) 27 ms 27 ms 27 ms
> >> > 4 204.147.128.198 (204.147.128.198) 28 ms 29 ms 28 ms
> >> > 5 * jpl-chi.nren.nasa.gov (198.10.64.34) 83 ms *
> >> > 6 * * *
> >> > 7 * * *
> >> > 8 * * *
> >> > 9 * * *
> >> > 10 * * *
> >> >
> >> > We can get to it fine from 128.138.188.
> >> > Anne
> >> >
> >> > On Aug 4, 5:40pm, Paul Lang wrote:
> >> > > Subject: Re: route nyxx-atm.colorado.edu <-> jsimpson-h.gsfc.nasa.gov
> >> > > Anne,
> >> > >
> >> > > jsimpson-h has been moved to a new hippi switch and the giga router
> >> > > will be moved shortly. jsimpson now has a working atm interface but
> >> > > we need to transition jsimpson-a (and a number of other systems) to
> >> > > a new network number so that we can route for them via our atm router.
> >> > > We will send you updates on the above issues as things get worked out.
> >> > >
> >> > > - Paul
> >> > >
> >>=========================================================================
> >> > > > From dguevara@mail.arc.nasa.gov Wed Aug 4 00:07:51 1999
> >> > > > To: bill@wizard.nasa.atd.net
> >> > > > From: "Anne M. Hammond" <hammond@solarz.Colorado.EDU> (by way of
> >>David A.
> >> > > > Guevara)
> >> > > > Subject: route nyxx-atm.colorado.edu <-> jsimpson-h.gsfc.nasa.gov
> >> > > > Cc: hammond@solarz.Colorado.EDU, lang@gi-joe.gsfc.nasa.gov
> >> > > >
> >> > > > Bill --
> >> > > >
> >> > > > See below. I currently can't get to jsimpson-h. I'll try again
> >> > > > in the morning. The next hop in Anne's traceroute should
> >> > > > be jsimpson. The route looks correct up to our point-to-point
> >> > > > peer between NREN and your router.
> >> > > >
> >> > > > -- david
> >> > > >
> >> > > > ===========
> >> > > >
> >> > > > Hi David,
> >> > > >
> >> > > > The route between nyxx-atm.colorado.edu (198.11.22.19) has
> >> > > > gone down in the last "few" days (perhaps over the weekend?
> >> > > > Below is a traceroute, and some history.
> >> > > >
> >> > > > hydra-atm1 is down due to a system crash. But these are
> >> > > > on the same physical network.
> >> > > >
> >> > > > Thanks for your yeoman work in the past. I haven't heard
> >> > > > any summary of the differences in data rates; although
> >> > > > several people have accounts on our machines and had been
> >> > > > looking into it.
> >> > > >
> >> > > > Thanks for taking a look at this.
> >> > > >
> >> > > > Anne Hammond
> >> > > >
> >> > > >
> >> > > >
> >> > > > nyxx 32% traceroute jsimpson-h.gsfc.nasa.gov
> >> > > > traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30 hops
> >>max, 40
> >> > byte
> >> > > > packets
> >> > > > 1 198.11.22.25 (198.11.22.25) 2 ms 1 ms 1 ms
> >> > > > 2 204.131.62.38 (204.131.62.38) 4 ms 12 ms 3 ms
> >> > > > 3 cs-atm0-0-14.dng.vbns.net (204.147.129.185) 147 ms 27 ms 28 ms
> >> > > > 4 204.147.128.198 (204.147.128.198) 29 ms 29 ms 29 ms
> >> > > > 5 gsfc-chi.nren.nasa.gov (198.10.64.49) 44 ms 44 ms 44 ms
> >> > > > 6 192.12.123.42 (192.12.123.42) 44 ms 44 ms 45 ms
> >> > > > 7 rtr-900-giga-a.gsfc.nasa.gov (169.154.32.3) 44 ms 44 ms 45 ms
> >> > > > 8 * * *
> >> > > > 9 * * *
> >> > > > 10 * * *
> >> > > > 11 * * *
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Jun 3, 6:59am, David A. Guevara wrote:
> >> > > > > Subject: RE: route from hydra-atm1.colorado.edu ->
> >> > jsimpson-h.gsfc.nasa.go
> >> > > > > Ron --
> >> > > > >
> >> > > > > Looks good to me. Thank you for your help.
> >> > > > >
> >> > > > > -- david
> >> > > > >
> >> > > > > At 7:18 PM -0700 1/1/88, Ron Bonica wrote:
> >> > > > > >Folks,
> >> > > > > >
> >> > > > > >We believe that this problem was corrected last night at about 10PM
> >> > eastern
> >> > > > > >time. Please contact us if you
> >> > > > > >have any further problems.
> >> > > > > >
> >> > > > > > Ron
> >> > > > > >
> >> > > > > >> -----Original Message-----
> >> > > > > >> From: Kevin Thompson [mailto:kthomp@mci.net]
> >> > > > > >> Sent: Wednesday, June 02, 1999 9:01 PM
> >> > > > > >> To: David A. Guevara; mgates@dast.nlanr.net
> >> > > > > >> Cc: vbns-eng@mci.net; dast@nlanr.net; bill@wizard.nasa.atd.net;
> >> > > > > >> hammond@solarz.Colorado.EDU; sgrant@psc.edu; ncne@ncne.org
> >> > > > > >> Subject: Re: route from hydra-atm1.colorado.edu ->
> >> > > > > >> jsimpson-h.gsfc.nasa.gov
> >> > > > > >>
> >> > > > > >>
> >> > > > > >> David, Mark, Anne, and Shirl,
> >> > > > > >>
> >> > > > > >> We verified the routing problems you reported, and are working to
> >> > restore
> >> > > > > >> several missing ospf adjacancies correlated to our Denver
> >>switches.
> >> > > > > >>
> >> > > > > >> We'll update you in the morning with our progress.
> >> > > > > >>
> >> > > > > >> Kevin Thompson
> >> > > > > >> vBNS Engineering
> >> > > > > >> 703-715-7073
> >> > > > > >>
> >> > > > > >> At 11:02 AM 6/2/99 -0700, David A. Guevara wrote:
> >> > > > > >> >mark --
> >> > > > > >> >
> >> > > > > >> >the last time this connection was "working" was back in late
> >> > > > > >> >february, early march. vbns might be able to provide
> >> > > > > >> >some input with regards to the additional router in our path.
> >> > > > > >> >I'll cc a copy of this message to their engineering group.
> >> > > > > >> >
> >> > > > > >> >-- david
> >> > > > > >> >
> >> > > > > >> >At 10:31 AM -0700 6/2/99, Mark Gates wrote:
> >> > > > > >> >>Hi Anne and David,
> >> > > > > >> >>
> >> > > > > >> >>I've been investigating the route that you seem to be
> >>using. It is
> >> > very
> >> > > > > >> >>peculiar to me. Do you know what dates you started having
> >>problems
> >> > > > > >> >>between the machines?
> >> > > > > >> >>
> >> > > > > >> >>I would say the problem is not congestion but routing. The two
> >> > things
> >> > > > > >> >>that looked odd to me are:
> >> > > > > >> >>
> >> > > > > >> >> 1) The RTO router that is listed on both traceroutes. RTO
> >>is Los
> >> > > > > >> >>Angeles, and why your packets would be going through there is a
> >> > mystery
> >> > > > > >> >>to me. They should go directly between Chicago (DNG) and Denver
> >> > (DNJ)
> >> > > > > >> >>routers.
> >> > > > > >> >>
> >> > > > > >> >> 2) Looking at various round-trip-time and traffic
> >>statistics, it
> >> > seems
> >> > > > > >> >>there was a definite change in the network on May 20th, 8am
> >>UTC. The
> >> > > > > >> >>round-trip-time between Denver and Chicago-based
> >>institutions jumped
> >> > > > > >> >>from 20ms to 40ms. There doesn't appear to be any traffic going
> >> > directly
> >> > > > > >> >>from the Chicago (DNG) router to the Denver (DNJ) router.
> >> > > > > >> >>
> >> > > > > >> >>I am going to have the NLANR engineering services group
> >>look more at
> >> > > > > >> >>this and hopefully discover and fix whatever routing is
> >>broken in
> >> > the
> >> > > > > >> >>network. Someone from there may contact you directly if
> >>they need
> >> > more
> >> > > > > >> >>information. You may still contact me also.
> >> > > > > >> >>
> >> > > > > >> >> -mark
> >> > > > > >> >>
> >> > > > > >> >>
> >> > > > > >> >>Anne Hammond wrote:
> >> > > > > >> >>>
> >> > > > > >> >>> Hi,
> >> > > > > >> >>>
> >> > > > > >> >>> This message below from David Guevara of NASA NREN
> >> > > > > >> >>>
> >> > > > > >> >>> Would you be able to help in analyzing this slowdown?
> >> > > > > >> >>>
> >> > > > > >> >>> Anne M. Hammond Anne.Hammond@Colorado.EDU
> >> > > > > >> >>> Network/UNIX Admin 303-492-5578 (FAX -5325)
> >> > > > > >> >>> JILA / University of Colorado at Boulder
> >> > > > > >> >>> Boulder, CO 80309-0440
> >> > > > > >> >>>
> >> > > > > >> >>> ----- Begin Included Message -----
> >> > > > > >> >>>
> >> > > > > >> >>> >From dguevara@mail.arc.nasa.gov Tue Jun 1 08:59:02 1999
> >> > > > > >> >>> Mime-Version: 1.0
> >> > > > > >> >>> Content-Type: text/plain; charset="us-ascii"
> >> > > > > >> >>> Date: Tue, 1 Jun 1999 07:58:54 -0700
> >> > > > > >> >>> To: "Anne M. Hammond" <hammond@solarz.Colorado.EDU>
> >> > > > > >> >>> From: "David A. Guevara" <dguevara@mail.arc.nasa.gov>
> >> > > > > >> >>> Subject: Re: route from hydra-atm1.colorado.edu ->
> >> > > > > >> jsimpson-h.gsfc.nasa.gov
> >> > > > > >> >>> Cc: bill@wizard.nasa.atd.net, mitchell@jila.Colorado.EDU,
> >> > > > > >> >>> litvanyi@spot.Colorado.EDU,
> >>hammond@hydra.colorado.edu,
> >> > > > > >> >>> brummell@hydra.colorado.edu,
> >>dguevara@mail.arc.nasa.gov
> >> > > > > >> >>> Content-Length: 1929
> >> > > > > >> >>>
> >> > > > > >> >>> anne --
> >> > > > > >> >>>
> >> > > > > >> >>> the route between hydra-atm1 and jsimpson-h is back up.
> >>there
> >> > seems
> >> > > > > >> >>> to be a congestion problem. currently our rtt is running
> >>about
> >> > 104ms
> >> > > > > >> >>> (approximately twice the value we had at the beginning of
> >> > > > > >> this project).
> >> > > > > >> >>> I think the problem lies with vBNS. See below for
> >>traceroutes
> >> > from
> >> > > > > >> >>> hydra-atm1 and jsimpson-h:
> >> > > > > >> >>>
> >> > > > > >> >>> hydra <~>traceroute jsimpson-h.gsfc.nasa.gov
> >> > > > > >> >>> traceroute to jsimpson-h.gsfc.nasa.gov (169.154.161.43), 30
> >> > > > > >> hops max, 40
> >> > > > > >> >>> byte pa
> >> > > > > >> >>> ckets
> >> > > > > >> >>> 1 198.11.22.25 (198.11.22.25) 1 ms 1 ms 1 ms
> >> > > > > >> >>> 2 204.131.62.38 (204.131.62.38) 3 ms 3 ms 3 ms
> >> > > > > >>
> >> > > > > >> >>> 3 cs-atm0-0-14.rto.vbns.net (204.147.130.117) 33 ms
> >>33 ms 33
> >> > ms
> >> > > > > >> >>> 4 cs-atm0-0-13.dng.vbns.net (204.147.129.145) 81 ms
> >>204 ms 81
> >> > ms
> >> > > > > >> >>> 5 204.147.128.198 (204.147.128.198) 82 ms 82 ms 83 ms
> >> > > > > >> >>> 6 gsfc-chi.nren.nasa.gov (198.10.64.49) 97 ms 97 ms
> >>97 ms
> >> > > > > >> >>> 7 192.12.123.42 (192.12.123.42) 98 ms 98 ms 98 ms
> >> > > > > >> >>> 8 rtr-900-giga-a.gsfc.nasa.gov (169.154.32.3) 98 ms
> >>98 ms 98
> >> > ms
> >> > > > > >> >>> 9 * jsimpson-h.gsfc.nasa.gov (169.154.161.43) 100 ms
> >>100 ms
> >> > > > > >> >>>
> >> > > > > >> >>> jsimpson% /etc/traceroute hydra-atm1.colorado.edu
> >> > > > > >> >>> traceroute to hydra-atm1.colorado.edu (198.11.22.12), 30 hops
> >> > > > > >> max, 96 byte
> >> > > > > >> >>> packe
> >> > > > > >> >>> ts
> >> > > > > >> >>> 1 169.154.161.1 (169.154.161.1) 6.397 ms 7.808 ms
> >>5.467 ms
> >> > > > > >> (6.0/6.000)
> >> > > > > >> >>> 2 rtr-atm-a.gsfc.nasa.gov (169.154.1.1) 8.553 ms 8.924 ms
> >> > > > > >> 11.023 ms
> >> > > > > >> >>> (8.0/9.
> >> > > > > >> >>> 000)
> >> > > > > >> >>> 3 192.12.123.41 (192.12.123.41) 9.189 ms 5.739 ms 7.83
> >> > > > > >> ms (7.0/7.000)
> >> > > > > >> >>> 4 chi-gsfc.nren.nasa.gov (198.10.64.50) 20.339 ms 22.183
> >> > > > > >> ms 22.424 ms
> >> > > > > >> >>> (22.0
> >> > > > > >> >>> /21.333)
> >> > > > > >> >>> 5 204.147.128.197 (204.147.128.197) 21.977 ms 23.711 ms
> >> > 23.756 ms
> >> > > > > >> >>> (23.0/22.
> >> > > > > >> >>> 333)
> >> > > > > >> >>> 6 cs-atm0-0-12.rto.vbns.net (204.147.129.146) 69.767
> >>ms 72.649
> >> > ms
> >> > > > > >> >>> 69.713 ms
> >> > > > > >> >>> (69.0/70.000)
> >> > > > > >> >>> 7 cs-atm0-0-14.dnj.vbns.net (204.147.130.118) 99.653 ms
> >> > 101.246 ms
> >> > > > > >> >>>105.729
> >> > > > > >> >>> ms (101.0/101.667)
> >> > > > > >> >>> 8 204.131.62.37 (204.131.62.37) 103.403 ms 106.348 ms
> >>104.908
> >> > ms
> >> > > > > >> >>> (104.0/104
> >> > > > > >> >>> .333)
> >> > > > > >> >>> 9 hydra-atm1.Colorado.EDU (198.11.22.12) 103.892 ms
> >>104.416 ms
> >> > > > > >> 107.273
> >> > > > > >> >>> ms (
> >> > > > > >> >>> 104.0/104.667)
> >> > > > > >> >>>
> >> > > > > >> >>> ----- End Included Message -----
> >> > > > > >> >>
> >> > > > > >> >>--
> >> > > > > >> >>Mark Gates 139 ACB
> >> > > > > >> >>NLANR applications support 217-244-7959
> >> > > > > >> >>http://www.ncsa.uiuc.edu/People/mgates mgates@nlanr.net
> >> > > > > >> >
> >> > > > > >>
> >> > > > > >>
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >-- End of excerpt from David A. Guevara
> >> > > >
> >> > >
> >> > >-- End of excerpt from Paul Lang