Category Archives: Backup and Disaster Recovery

Restoring NetApp LUNs

Leave a reply

The other day, I was tasked to restore a NetApp LUN from several years ago. Since the backup was so long ago, I have to restore it from an archived tape. After restoring the file, it did not show up as a LUN. It turned out that there are few rules to follow in order to restore LUN from tape and that it shows up as LUN.

There are also a couple of requirements when backing up LUNs to tape. Thankfully when we backed up the LUNs, we followed these rules:

1. The data residing on the LUN should be in a quiesced state prior to backup so the file system caches are committed to disk.

2. The LUN must be backed up using a NDMP compliant backup application in order for it to retain the properties of a LUN. Symantec Netbackup is an NDMP compliant backup application.

When restoring, I learned that:

1. It must be restored to the root of a volume or qtree. If it is restored anywhere else, it will simply show up as a file and lose the metadata allowing it to be recognized as a LUN.

2. The backup software should not add an additional directory above the LUN when it is restored.

For instance, on Symantec Netbackup application, when restoring the LUN, you should select the option to “Restore everything to its original location.” If this is not possible, you can select the second option which is “Restore everyting to a different location, maintaining existing structure.” This means that you can restore it on a different volume.

For example, if the LUN resides in /vol/vol1/qtree1/lun1 and we chose to restore to /vol/vol2, the location where the LUN would be restored is /vol/vol2/qtree1/lun1 because it maintains the existing structure.

Do not select the third option which is “Restore individual directories and files to different locations” because the Netbackup software will add an extra directory beyond the qtree and the LUN will not show up as a LUN.

When restore is complete, a “lun show -v” output on the NetApp CLI will show the restored LUN on /vol/vol2 volume.

Deduplication and Disk-based Backup

Leave a reply

Disk-based backup has been around for a long time now.  I remember when I first deployed a Data Domain DD560 appliance eight years ago. I was impressed by its deduplication technology.  Data domain appliances are disk storage arrays used primarily for backup.  We used it as a disk backup destination for Symantec Netbackup via NFS.  Data Domain sets itself apart because of its deduplication technology.  For instance, in our environment, we are getting a total compression (reduction) of 20x; which means that our 124TB data only uses 6TB of space – a very large space savings indeed.

In addition, disk-based backup makes it very fast and easy to restore data. The need to find the tapes, retrieve them, and mount them have been eliminated.

In fact, the need for tapes can be totally eliminated.  A lot of companies are still using tapes to store them off site for disaster recovery.  If you have a remote office site, a Data Domain appliance can be replicated to your remote site (or disaster recovery site).  What is nice about Data Domain replication is that only the deduped data is sent over the wire.  This means less bandwidth consumption on your WAN (Wide Area Network).

Deduplication technology is getting better. Storage vendors and backup software vendors are offering it in different flavors.  With the cost of disk going down, there is really no need for tapes. Even long term retention and archiving of data can now be stored on a low cost, deduped disk storage array.

The Importance of Disaster Recovery (DR) Testing

Leave a reply

Recently, we conducted disaster recovery (DR) testing on one of our crucial applications. The server was running Windows 2008 on an HP physical box. We performed bare metal restore (BMR) using Symantec Netbackup 7.1. However, after Symantec BMR completed the restore, the server will not boot up. We troubleshoot the problem and tried several configurations. It took a couple of days before we figured out the issue. The issue, by the way, was that the boot sector got misaligned after the restore and we have to use Windows installation disk to repair it.

What if it was a real server disaster? The business cannot wait for a couple of days to restore the server. We defined an RTO (Recovery Time Objective) for that server to be 8 hours. And we did not meet it during our testing. This is the reason why DR testing is very important.

During DR testing, we have to test the restore technology and the restore procedures. In addition, we need to test if we can restore it on time (RTO) and if we can restore the data at a point in time (or RPO – Recovery Point Objective) (e.g. from a day before, or from a week ago).

With a lot of companies outsourcing their DR to third parties or to the cloud, DR testing becomes even more important. How do you know if the restore works? How do you know if their DR solution meets your RPO and RTO? Companies assume that because backups are being done, then restore will automatically work.

We perform DR testing once a year. But for crucial applications and data, I recommend DR testing twice a year. Also, perform a test every time you make significant changes on your backup infrastructure, such as software updates.

Backing Up NetApp Filer on Backup Exec 2012

Leave a reply

The popularity of deduped disk-based backup, coupled with snapshots and other technologies, may render tape backup obsolete. For instance, if you have a NetApp Filer, you can use snapshot technology for backup, and snapmirror technology for disaster recovery. However, there may be some requirements such as regulatory requirements to keep files for several years, or infrastructure limitations such as low bandwidth to remote DR (disaster recovery) site that inhibits nightly replication. In these instances, using tape backup is still the best option.

The proper way to backup a NetApp Filer to tape on Backup Exec 2012 is via NDMP. You can backup your Filer on the network, using remote NDMP. If you can directly connect a tape device to the NetApp Filer, that would even be better, because backup will not go through the network anymore, thus backup jobs will be faster.

However, using NDMP requires a license on Backup Exec. The alternative way to backup the Filer without buying the NDMP license is via the CIFS share. Configuring the Backup Exec 2012 via CIFS shares though can be a little tricky. These are the things you need to do to make it work:

1. Disable NDMP service on the NetApp Filer. This is done by issuing the command “ndmpd off” at the command line.
2. Change the default NDMP port number on the Backup Exec 2012 server. The default port number is 10000. You may use port 9000. This is done by editing the “services” file located at C:\Windows\system32\drivers\etc and adding the line “ndmp 9000/tcp” Reboot server after editing the file.
3. Make sure you have at least one Remote Agent for Windows license installed on your Backup Exec server.
4. Make sure that the “Enable selection of user shares” is checked in the “Configuration and Settings -> Backup Exec Settings -> Network and Security” settings.
5. When defining the backup job, select “File Server” at the type of server to backup.
6. When entering the NetApp Filer name, use IP address or the fully qualified domain name (FQDN).

The backup status for backing up NetApp Filer this way will always be “Completed with Exceptions,” since Backup Exec still looks for remote agent on the client. But this is fine, as long as all files are being backed up.

Upgrading Netbackup from Version 6.5 to 7.1

I recently upgraded a Netbackup infrastructure from version 6.5 to version 7.1. Here are some of my observations and advice:

1. Preparation took longer than the actual upgrade of the server. Pre-installation tasks included understanding the architecture of the backup infrastructure including the master and media server, disk-based backup, and ndmp; checking the hardware (processor, memory, disk space) and operating system version were compatible and up to snuff; checking the general health of the running Netbackup software including the devices and policies; backing up the catalog database; obtaining updated Netbackup licenses from Symantec; downloading the base Netbackup software and the patches; joining, unzipping, and untarring software and patches; and other related tasks. Planning and preparation are really the key for a successful upgrade. These activities will save a lot of trouble during the upgrade process.

2. The upgrade process was seamless. On Solaris server, I ran the “install” command to start the upgrade. The process asked for several questions. Some packages were already integrated in the base package such as ndmp, so the program asked for the existing Netbackup ndmp package to be uninstalled. The part that took longer was the catalog database upgrade.

3. Upgrade of client agents was also easy. Upgrading UNIX and Linux clients was completed using the push tool “update_clients.” Windows clients were upgraded using the Netbackup Windows installation program. One good thing though was that no reboot was necessary. Also, I found out that Windows 2000 and Solaris 8 clients were not supported on 7.1, although it will still backup using the old 6.5 agent.

4. For bmr (bare metal restore), there was no need for a separate boot server. All client install included the boot server assistant software.

5. The GUI administration interface is almost the same, except for some new features such as vmware support.

6. The java administration console is so much better, in terms of responsiveness.

Backup Infrastructure

Leave a reply

I have been designing, installing , and operating backup systems for the past several years.  I have mostly implemented and managed Symantec Netbackup (used to be Veritas Netbackup) for larger infrastructures and Symantec Backup Exec for smaller ones.

These software worked very well although some features are not very robust.  I’m very impressed for instance of the NDMP implementation in Netbackup.  Backing up terabytes of NetApp data via NDMP works very well.  However, I do not like the admin user interface of Netbackup since its not very intuitive. Their bare metal restore (BMR) implementation also is a pain.  Some of the bugs took years to fix.  Maybe because there are not too many companies using BMR.

Backup Exec works very well with small to medium systems. It has very intuitive interface, it is relatively easy to setup, and it has very good troubleshooting tools.  Lately though, Symantec has been playing catch up in their support for newer technologies such as VMware. It is so much easier to use Veeam to manage backup and restore of virtual machines.  In addition, Backup Exec has been breaking lately. Recent Microsoft patches have caused backup of System_State to hang.

But I think the biggest threat to these backup software are online backup providers. Crashplan, for instance, was initially developed for desktop backup, but it will not take long before companies will use it to back up their servers. When security concerns are addressed properly by these providers, companies will be more compelled to backup their data online. It’s just cheaper and easier to backup online.