Saturday, January 1, 2011

Hidden Backup

Ran into a server with high network io in primetime. My first thought was it was a backup, but after checking the historical usage over the last 40 days figured that this wasn't the case. The particular server is at the hub a the Customer Billing environment so raised a severity ticket to bring teams together to figure out what was going on.

By the time we looked again the issue was gone yet nobody knew the cause. Investigation revealed that the  cause was a Full backup and that it has been cancelled for the last few months based on a previous request to get more CPU cycles to catchup on some billing runs. And it had been cancelled again while I was raising the severity ticket.

More digging revealed a new media server had been implemented to off-host this backup 6 months ago, but if was forgotten.

Step 1 get a full backup run as rebuilding from 3 months of incremental backups isn't recommended and step 2 get the Project Office to complete the original media server implementation.

Driving these types of issues isn't in the normal remit of a Capacity Planner, but sometimes when we discover something we have to drive it until a firm resolution is in place.  Make sure you partner with your Backup Team and identify any other backups that aren't getting completed as these could be capacity and performance issues.

Regards.

Scott Wardley