Joe Glessner
This user hasn't shared any biographical information
Posts by Joe Glessner
How to lose your data
Jan 5th
See that picture? When it happens to you, it may not look quite that bad (or be quite that obvious), but data loss sucks. And it does happen. I’ve been working with computers for 10+ years, and I’ve had it happen a couple times myself. Did I mention how much it sucks?
I’m not going to spend a couple pages telling you why you should backup, I’m just going to be straight about it, unless you really couldn’t care less if that happened to your computer, you are flat out stupid if you are not backing up your data on a regular basis.
Instead of telling you why to backup, I’m going to tell you how to ensure that you are not going to get your data back, even if you think you are backing it up.
Method 1: I’ll just back the data up to CD/DVD.
Well sure, this will work for a bit, but:
- Ever try to save 20GB to CD? Or 250GB to DVD? Ugh.
- How long do you think that optical desk is going to be readable?
Going this route, you can quickly end up trapped behind a small mountain of plastic. Or lets say you manage to somehow keep the optical disks to a manageable quantity, will the marker you labeled it with make the disk unreadable in a year, or is the dye layer unstable, rendering your disk unreadable in six months, or will the glue on the label you made for the disk make it worthless in a year or two? These are just a couple of examples of why optical media should not be considered an archive grade solution.
Method 2: ok then, I’ll just copy the data to a USB hard drive.
Sure it’s better than nothing, but single HDD solutions are not going to keep your data safe. Hard drives fail. In fact it will happen to every single hard disk you will ever come across. The only question is; when? It’s not a matter of if, or of MTBF (mean time between failure), it is more a matter of “you never know, it could fail in ten years, or in ten seconds”.
Don’t get me wrong, if this is the only way you can back the data up, then it is your only choice, and it’s better than nothing. Just be aware, as soon as you copy the data to that USB HDD, the “Clock of Death” is ticking.
Much better would be to copy the data over to a machine with a RAID storage system (preferably RAID5).
Method 3: I bought actual Backup Software (or use a vetted Open Source solution), and run Incremental Backups (to tape!) every single day!
Ok, so you spent some money on a tape backup solution, spent hours reading the manual and configuring your backup. Congratulations, I bet you think your data is safe! Until you find out how Incremental Backups really work (this usually happens after a disaster, and the tapes is all you have left of your pr0n, illegal mp3’s downloaded movies warez mission critical data).
Let’s pretend for a minute that your backup tapes look something like this:
Full_backup_tape (tape 1 – doesn’t matter what you tell it to be, the first backup is always and without exception, a full backup)
Incremental_backup_1 (tape 2)
Incremental_backup_2 (tape 3)
Incremental_backup_3 (tape 4)
Incremental_backup_4 (tape 5)
Incremental_backup_5 (tape 6)
And then you have a catastrophic failure. So you’re sitting there at 2am merrily running the restore, and you hit a snag: tape 2 won’t read. Doesn’t matter why, the tape could be bad, maybe you left it out of the tape safe overnight, and the radio station next door managed to erase it with the magnetic waves they transmit (this actually happened), the data is gone. So is all data after it. See Incremental backups require that all tapes since the last full backup be present and working. So tapes 3-6 may as well be empty, because you are never getting the data off of them. Ever.
If you can’t run full backups every day, use Differential backups instead of Incrementals. Let’s say that in the scenario the user had been running differentials rather than incrementals. They could then restore to current using just the original full backup, and the last differential.
Method 4: Now I’m running differential backups to tape every single day!
But you fail to check the backup logs every day, and the backup job you though had been running for the last year actually failed 273 days ago, and has been requesting the “correct” tape since then. I’ve seen this one a lot (in fact, I think this would be the most popular reason for data loss if you have backup software running).
You’ve got to check your backup logs. It sucks, and it’s boring, but it’s one of those things you just have to do.
Method 5: Alright, I’m running differentials to tape, and have been checking my logs for the last 2 years every single day!
But you’ve never run a test restore. If you haven’t restored data from the tape successfully, there is no data on the tape. The tape was bad, the backup software failed (silently of course), the gremlins ate it.
Method 6: Ok, now I spend two hours reading the log and then randomly restoring files from my backups (before putting the tapes in the tape safe) every single day!
And then your server room catches fire. All machines, and the safe holding the backup tapes are destroyed. You never took any offsite, because you have a tape safe. It happens. It’s unfortunate.
Method 7: Enough, I give up on tape! Now I run a full backup to a RAID5 NAS every single day!
But you ordered your NAS with the drives form the manufacturer, and they used 4 HDD’s from the same batch, and two failed. This is the one that always gets them! The strength of RAID5 is that more than one drive has to fail before the RAID is unrecoverable. The weakness is that hard drives from the same batch tend to fail at the same time (or thereabouts).
To strengthen your RAID system, always make sure that you have drives from different batches, if not from different manufacturers (this is not always the best idea, but that is an argument for another time). For instance: to take care of my backup needs at home, I bought a Buffalo Terastation. Unfortunately, Buffalo sent me a Terastation with 4 drives from the same batch (you can usually tell if they all have the same date on them, sometimes there will be a batch code on the drive). I bought 3 more of the same model drive from 3 different manufacturers, and now have the most healthy RAID I can.
These are not the only ways to lose data, but they are by far the most common. How would I know? I was the Worldwide Manager of Technical Support for a backup software company for several years. And I always got to be the one to explain to the customers why their data is gone.
So what do I do?
There are as many answers to that question as there are IT shops with backup systems. Here is how I protect data at my office:
I backup all data every day (full backup) to a NAS configured in RAID5, with a hot spare. I check the health of the RAID every day (it takes about two minutes). Once a week I backup the entire RAID to LTO3 tape, and take the tapes offsite (currently I am taking them home, where they go into a DATA rated fire safe (there is a difference, do your homework), and then into my large safe where I keep all my other valuables. My ideal would be to have them delivered to a bank safety deposit box, but that costs money.
At home, I back up all my data to the aforementioned Terastation. Once per month, I copy all the data off to a USB HDD (actually two of them), and take one to work where it goes into the tape safe.
Is it perfect? No. Does it stand a much better chance of keeping that data alive through a catastrophic event? Absolutely. You don’t have to go to these lengths to protect your data, but you should be aware of the risks.
Mysterious Server 2003 disk space consumption
Oct 25th
So the System drive of my (primary) domain controller has been running low on disk space (it’s a 20GB partition running with about 4GB or so free). This has been a nagging issue that I’ve had off and on for a while now, and I haven’t really had the time to delve into it.
I decided to start my investigation by running WinDirStat and looking for any oddly large files. The largest portion of the System disk is consumed by the Program Files directory (no big surprise there), and aside from a couple slightly disturbing large files from my backup software there is only one group of large files on the drive – hovering in at about 12GB for the 8 or so files. And they all have the same path and are similarly named: C:\System Volume Information\{914b4760-84b2-11dd-bca9-000e0cb2b564}{3808876b-c176-4e28-b7ae-04046e6cc752}
Hmmm, interesting. A quick Google search turns up some results linking this directory (more specifically files with CSLID names in this directory) to two things: System Restore points, and virus files.
Well I’m pretty sure it’s not virus files (no other odd behavior or weird network activity), and if I’m not mistaken to enable System Restore on WS2003 you have to manually copy over some files from an XP CD (which is a pretty cool hack, but not something I’ve done on any corporate network I’ve ever worked on).
At this point I start hearing dramatic music in the back of my mind, I’ve got a bonafied mystery! Or at least initial facts would indicate so.
Well a bit more in depth investigation turns up what some of you already knew at this point, the culprit is VSS. But I never configured VSS! (queue swelling of dramatic music in the background)
Ok so this is something of a mystery after all. So I go digging around in the event logs for the last 3 years looking for the initial VSS snapshot message. It sounds like a lot of work, but Microsoft Log Parser actually makes things like this pretty trivial.
Turns out that the VSS snapshots started on the same day that I installed our current Backup software (Yosemite Backup 8.5 sp2) which cooincidentally has the ability to make use of VSS snapshots!
Now this is not a huge issue, as VSS will delete old snapshots when space is needed, however I tend to take exception to software doing things like this without my permission.
Well luckily for me, I used to be a manager at the company that makes our backup software, so I fire up my trusty IM client, and start poking at the engineering department.
Twenty minutes later I have my trusty pipe and smoking jacket firmly in place, as I am feeling quite like Sherlock Holmes. It seems that in fact it was the backup software which enabled VSS for all volumes on my server, and (because it uses the defaults when enabling VSS) had set VSS to not limit the space consumed by snapshots!
A simple trip into Disk Management, and a quick change to the drive’s Property page, and VSS is now limited to 4GB for the system partition (which is far more than I’ll ever need). Interestingly enough had I disabled the VSS service on this machine before installing the backup software, it would not have enabled VSS. I’ve asked that they include a note about VSS being automatically configured to the Yosemite Backup installer (it may exist now, I’m not sure as I haven’t actually read any of the installer screens in years), but who knows when that will make it into the software.
As a side note, I’ve spoken to the Tech Support Manager at Yosemite Technologies (they make Yosemite Backup), and they are currently writing a knowledge base article about this, and how to change the VSS settings from the defaults that Yosemite Backup enables.
Admins Arsenal: DBAN
Oct 23rd
Ever buy a used Hard Drive on Ebay or Craigslist? Ever look to see if there was any data on it? I have, and let me tell you, it is downright scary what people will leave on HDD’s when they sell them as used. I recently purchased 12 used 250 GB SATA HDD’s from Ebay for a NAS project I was working on, and of those 12, 9 of them had not been so much as formatted.
Of those 9, all but one had data that would have been usable for identity theft: files with credit card information, copies of bills, saved email that had account information, not to mention the astounding number of ummm, not safe for work pictures taken by (or of) the former owners. In all cases I could have contacted the former owner, as on all 9 drives I found current addresses and phone numbers for the former owners.
This is 2009, I would have thought that by now better than 25% of people selling used hard drives would know better.
As an IT Professional, I get used computers all the time (for some reason everyone I work with thinks I need every used machine I can get my hands on, especially if it has a “Designed for Windows 95!” sticker on it!). The ratio of these HDD’s that I get with data still on them is more like 95%. It’s easy to format a HDD, until you get one that just won’t format. Mostly the ones I’ve seen this on were disks that I installed an obscure linux distro on, and then for whatever reason decided to install Windows on. The Microsoft format tools are not always capable of handling partitions like this, which is where DBAN comes in.
Darik’s Boot And Nuke (or DBAN) is a self-contained boot disk that securely wipes the hard disks of most computers. DBAN will automatically and completely delete the contents of any hard disk that it can detect, which makes it an appropriate utility for bulk or emergency data destruction.
One of the really great things about DBAN is that it can run from a floppy, or be burned to a bootable CD, which means it can be used on almost any computer. Better than that, DBAN has many options for how it wipes the disk, ranging from the single pass “autonuke”, to the 35 pass random data Gutmann method, and of course DBAN also offers users a method for definable number of passes.
In short, DBAN excels at destroying all data on a Hard Disk. You can find more information on DBAN here.
Using Notepad++ for Version Control
Jun 29th
I have what you would call a love/hate relationship with Version Control Systems. On the one hand, it is oh so nice to be able to revert back to a working version of your document when you accidentally make way too many mistakes, and it no longer works. But on the other hand, it is a total pain in the ass.
I am duty bound to tell you that if you are doing much of anything involving code, you need to be using some sort of Version Control, however I am not going to ram Subversion or Tortoise SVN down your throat. I understand why programmers use them, and I have used them myself in the past. Since I try to live the whole “Least Amount Of Administrative Effort” thing, I just really don’t like them. Let me explain.
Ok, setting up full on Version Control makes sense if any of the following are true:
- You spend a large portion of your day writing code (like your job title is “programmer” or something similar)
- You spend more time fixing things you break while writing code, than writing code
- Anyone else is going to suffer consequences if you totally wreck some code
- Someone else is going to maintain and administer the Version Control System
- You work on files that multiple other people also edit
However it does not make sense if any of the following is true:
- You write code for your own use only
- You could care less if it gets broken or accidentally deleted
- You don’t have time to administer or maintain a Version Control System
Even if all of the last bit are true for you, it is still a good idea to use Version Control. Wait, what?
Version Control is a process, and if done correctly it works very well. If done poorly it gives you a false sense of security. Version Control is not necessarily a software package or commercial system.
If working with end users for all these years has taught me anything it is this; if a system is easy to use (especially if you don’t have to actually do anything) it will get used, if it is complicated (or time consuming) it will not get used. This also holds true (even more so in most cases) for the IT professionals I know.
Since don’t write enough code to make Visual Studio a wise investment, I use Notepad++, and unlike Visual Studio, I can take my IDE with me on a USB drive (Notepad++ Portable!). In the same vein, I don’t really write enough code to really justify setting up and maintaining a Version Control System, so I do it with Notepad++. Least Amount Of Administrative Effort.
So here is my solution:
I use the Backup settings built into Notepad++ to take care of my Version Control. To set this up:
- Install notepad++
- Open Notepad++
- Select Settings > Preferences
- On the Backup/Auto-completion tab, change the backup setting from None to Verbose
If you change nothing else there, now when you save your file, Notepad++ will automatically create a copy of the file without the changes since the last time you saved it. The copy will also be saved in a subfolder (named nppBackup) in the same folder where the document is saved. The copy will be named FileName.Extension.Date_Timestamp.bak by default. So a file named “test.vbs”, saved today would have a name similar to “test.vbs.2009-06-29_151927.bak”.
Cumbersome to be sure, but I am guaranteed to use it every time I work with a document.
Please don’t mistake this for something it is not. This is not a Version Control System. You do not check files in or out, and it doesn’t differentiate who made what changes, all it does is save every iteration of your document. It is Version Control in the truest sense of the term.
Admin’s Arsenal: Process Explorer
Jun 18th
Once in a while I will get a call from one of my users describing a problem, and immediately I think to myself “impossible”. Like “it’s just not possible that your computer is deleting your email all by itself”.
Then there are the times where I find myself five minutes into the conversation going “uhhhh, yeah that’s not good, I wonder what could cause that?” (believe it or not, us IT people don’t in fact know immediately exactly what is wrong with your computer, and we’re even wrong once in a third Tuesday of the week).
When I need to get a crystal clear picture of what is happening on a system, I turn to Process Explorer from Sysinternals (now brought to you by Microsoft!). Process Explorer is everything that Windows’ Task Manager wishes it was:
Overview
Process Explorer is an advanced process management utility that picks up where Task Manager leaves off. It will show you detailed information about a process including its icon, command-line, full image path, memory statistics, user account, security attributes, and more. When you zoom in on a particular process you can list the DLLs it has loaded or the operating system resource handles it has open. A search capability enables you to track down a process that has a resource opened, such as a file, directory or Registry key, or to view the list of processes that have a DLL loaded.
The Process Explorer display consists of two sub-windows. The top always shows a list of the currently active processes, including the names of their owning accounts, whereas the information displayed in the bottom window, which you can close, depends on the mode that Process Explorer is in: if it is in handle mode you will see the handles that the process selected in the top window has opened; if Process Explorer is in DLL mode you will see the DLLs and memory-mapped files that the process has loaded.
Process Explorer also has a powerful search capability that will quickly show you which processes have particular handles opened or DLLs loaded. The unique capabilities of Process Explorer make it useful for tracking down DLL-version problems or handle leaks, and provide insight into the way Windows and applications work.
You can obtain equivalent command-line tools, Handle and ListDLLs, at the Sysinternals Web site.
Process Explorer does not require administrative privileges to run and works on Windows 9x/Me, Windows NT 4.0, Windows 2000, Windows XP, Server 2003, Windows Vista, Windows Server 2008 and on the x64 version of 64-bit Windows XP, Windows Vista, Windows Server 2003, and Windows Server 2008.
So, why use this rather than any of the dozens of other Task Manager replacements you can find on the internet? Well for starters, Process Explorer was written by Mark Russinovich. Mr. Russinovich is acknowledged as one of the foremost experts on Microsoft Windows in general, and the NTFS file system in particular. The man is incredibly knowledgeable about the internal workings of Microsoft Operating Systems, and has authored several books on Microsoft Technologies.
Beyond that, the sheer depth of functionality in this product makes it a hands down winner in my book. Oh and did I mention that you can run it from a USB drive?
You can get more information on Process Explorer (and download it) here.
Admin’s Arsenal: Evernote 3.1
Jun 18th
Initially I had some trepidation about doing an AA on Evernote, because as of v3, it is web centric, and some of the great features of the old v2.2 have been removed. But man is it useful!
ok, so what is Evernote? It’s difficult to explain correctly. Basically it is a note taking application on the order of Microsoft’s OneNote (which I like, but can’t justify buying a copy of for every system I work on, so it’s kind of limited in its use for me), but taken to a whole different level, oh, and it’s free (sort of).
Here is the description from the Evernote site:
Remember everything.
Evernote allows you to easily capture information in any environment using whatever device or platform you find most convenient, and makes this information accessible and searchable at any time, from anywhere. Did we mention that it’s free?
Yeah, that’s about accurate. Currently there is a free Evernote app for Windows, and Mac OS X (Leopard), a bookmarklet that will work on just about any browser, a Firefox extension, and apps for the iPhone, Palm Pre, Windows mobile phones, and limited Blackberry models .
It requires Blackberry OS4.6 or above so at the moment it is limited to the Bold (and since I just got one, this is a good thing), Curve (the new one), and Storm, however I have no doubt that RIM will update all others to 4.6 soon.
The thing that makes this a killer app for me is that Evernote can make text in images searchable. So if I take a picture with my Blackberry of say, a BSoD error message, and save it to Evernote, I can then access it on my desktop (or another machine connected to the internet), and figure out what that error message is telling me.
All in all, the ability to clip portions of websites, entire web pages, text from documents, and email from Outlook (yeah it integrates with Outlook), makes this a really really handy tool for IT work (not to mention blogging).
The free accounts are limited to 40MB of Monthly upload data (text, images, audio, and .pdf files only), whereas the premium version gets you 500MB per moth, removes the small advertising window in the desktop app, and can sync any kind of file.
While 40MB may not seem like much, it is roughly 20,000 text notes, or 400 mobile snapshots, or 270 web clips, or 40 audio notes, or 11 high resolution photos. 500MB is roughly 12.5 times that amount of data.
The cost of the premium account is $5/mo. or $45/year.
If you’d like to try Evernote out, you can sign up and download it here: http://www.evernote.com/
