In the latter part of my career, it became increasingly obvious that many data centre operations were dependent on the reliability of modern disk technology. They had a "backup strategy", of course, but little in the way of a "restore strategy". The first example I remember was in about 2005, where the tape storage facility was next to the disk storage units. And, there's was no way to manually extract tapes from this facility. A fire in that case could have destroyed everything, including all backups.
Restoring from tape became increasingly rare, with clever disk-based solutions like snapshots meaning that tape had become a last line of defence. Increasingly, the backups were completely automated, to the point where no one seemed to think very much about testing the restoration of data. It was assumed, more or less, that if the backups were running, then the restoration of data from them would be possible.
Generally, there was always a difficulty in organising a full restore from backup. If the test went wrong, then in principle the data was gone! Often a full restore was tested before a system went live - but once the system was live, it could take quite a bit of ingenuity to test a restore properly. In the old days, when a system required a single Unix server with a tape backup every night, it was relatively easy to restore onto a similar server somewhere else and test the restored system. It might take one techie less than a day to do the whole off-site restoration, using tapes that were stored off-site, where nothing from the live site was needed. In the 1990's we did this sort of thing regularly - although we always kept our fingers crossed when a restore from tape was involved.
But, as systems became increasingly complex and interdependent - with an environment of perhaps several hundred virtual servers - creating a test installation was a project in itself. In one of the last projects I worked on, it took about six weeks to configure the server infrastructure. If an additional user-test environment was required, then it took about six weeks for the various Windows, Linux, Oracle and other techies to do their thing. Then, all the application software had to be loaded. (I think it cost about £1 million per environment!). By contrast, in the old days, creating a new test environment could take a couple of hours - copy the executables, copy an existing database, do some configuration, update the backup schedule(!) and that was it. And, it came at no additional cost - as long as the server had enough memory for another Oracle database. In one case, we had a single Unix server with nine test enviornments on it.
I worked on a project in about 2012 where we did a major upgrade by going live on what had been the "disaster recovery" (DR) site and swapping the roles of live and DR data centres. That all went well, but it was a controlled project over several months and not a DR test as such.
In general, there seemed to be a strong reluctance to do a DR test, even where the facilities were available and an annual test was part of the contract. People always seemed to have better things to do! Also, there was a new generation of system support techies who had a different outlook on things. I was telling anyone who would listen that we were essentially dependent on the reliability of the technology. By 2014, when I retired, I strongly believed that a fire at a data centre would have been a real disaster, with little chance of full systems and data recovery.
