In a followup to The Commandments of Systems Administration Newsforge has posted the latest installment; The Fourth Commandment of Systems Administration: Thou Shalt Keep Server Logs on Everything.
The role of system administrator is a role of details. Heavily used and updated servers are filled with details, from new tables in a database to root password changes. These details need to be documented. When you are managing three servers, these details can be easy enough to remember. However, when you have 30 or 50 or 100 servers, the details become impossible to keep track of without documenting them. When it matters, you don't want to think that the IP address of that old accounting server is 192.168.10.55, you want to know it.
All I can say about this is that: at a minimum have a central syslogd server on your network, and throw syslog (or syslog-ng)at it from everything that can throw syslog at it. Routers, switches, servers, services, print server boxes, everything. Lock that syslogd box down and run alerting tools against those logs, such as Epylog, or log2mail - configure those tools to alert you when root login failures and the like happen. Rotate those logs to keep them managable (logrotate or similar. Keep those logs forever, burn them to disc.