Update:
On 2008-07-23 14:56 Henrik Levkowetz said the following:
www3.tools.ietf.org is currently very heavily loaded, and it's not possible
to ssh in to it. I've removed the A and AAAA records for it from the zone
file for tools.ietf.org, so any long delays you may have experienced reaching
the wg status pages, RFCs and drafts on tools.ietf.org should be eliminated.
The wiki pages and trackers are normally only available through www3, though,
unless I switch over to another machine as master. I'm going to try to get
www3 back to normal a bit longer, before I take that step, though. Sorry for
the temporary unavailability of those services.
www3.tools.ietf.org is up and running again. What triggered the incident is
that during a short period of time yesterday the server was hit by about 14000
requests from a single client. This by itself should have made it sluggish,
but shouldn't have made it unresponsive. However, the configuration of the
apache server permitted a too high number of server processes, which led to
the machine running out of available RAM first, and later running out of
swap memory. In the end, a power-down/power-up was necessary to bring it
back.
I've now changed the apache configuration so that the maximum number of server
processes will be a number I've seen the server survive at its earlier highest
load level, which should prevent this particular situation from arising again.
However, over the weeks after the IETF I'm going to add some more fail-over
mechanisms to ensure that services (like Wikis and Issue Trackers) which take
input from users, and thus can't easily be served simultaneously from multiple
servers, will be be automatically migrated to another server if the master
server for those services fails. Hopefully that will mean that the next time
a tools server has problems (and of course that will happen again, at some
point) nobody will notice it except the administrator ;-)
Best,
Henrik
_______________________________________________
Ietf mailing list
Ietf(_at_)ietf(_dot_)org
https://www.ietf.org/mailman/listinfo/ietf