Joe Glessner
This user hasn't shared any biographical information
Posts by Joe Glessner
Guerrilla Event Log archiving: why and how.
Jun 8th
I am quite positive that there are as many solutions (both paid and unpaid) for handling Win32 Syslogs as there are SysAdmins out there. On my *NIX machines syslogs are a simple thing, configure Syslog-ng and move on. My Windows Syslogs are a whole different story.
First off, shame on you Microsoft for not providing built in syslogd integration capabilities. With the volume of BSD code in Windows there is just no acceptable reason for this.
But that doesn’t help me. The long term goal is of course to get a central Syslog server set up that will handle and archive log entries from all of my machines (*NIX and Win32), but that is going to take two things:
- Time I don’t have.
- Money I don’t have.
I need a solution for archiving my Windows event logs right now, in a central location, until I can get the central Syslog server set up. As I mentioned, most of the solutions for doing this on Windows machines (the ones I feel comfortable entrusting my event logs to anyway) cost somewhere in the neighborhood of an arm, a leg, and most of an ear, so those are not viable options. Now what do you do?
Well if you’re me, you roll your own solution. I’ve got several WS2003 servers that I need to log the event data from, because, well to be quite honest, because this network was built by someone that is more of a *NIX SysAdmin, and didn’t set up the Windows side correctly, so there are quite a few odd bugs in this network that will take quite a while to work out.
Now I could go through and manually export the event logs to a file once a month, but that is way too much work. I decided to script the solution to this problem using VBScript (as it is available on all of the Servers I need event log info from).
I give you logArchive.vbs:
More >
IT staff to Users ratio?
Apr 29th
This has been a rough month for me at my day job. We are in the midst of rolling our existing company over to an ESOP (basically employee owned, buying out the current owners over a period of years), which has basically doubled my workload in getting the back end ready for the change over.
In addition to this we had a senior accounting executive fired, which led me to discover that our entire AR system is being handled by a series of highly customized Excel spreadsheets. When I say highly customized I mean that not only will they only work on one computer, but they will only work under one single user’s profile (which happens to be the previously mentioned, and recently departed, senior accounting executive).
This in turn led me (with the help of the accounting department) to the inevitable conclusion that our entire ERP product needs to be completely replaced. Why? Because it is not capable of handling the volume of of a specific AR transaction we have on a daily basis without quickly locking up the server (which is how the current “Franken-Excel” system came to be in the first place).
Ok great! I’ve never liked the ERP solution we use, and have already done the research on what I would like to replace it with. After getting the current owner to sign off on it, I’ve now effectively quadrupled my workload. Oh did I mention the go live date for the new ERP system can be no later than October 1st?
Which led me to the conversation with one of the executives in our company about the fact that I need more IT staff. Anyone who has experience with an IT degree education would be great. Don’t get me wrong, I don’t mind working the occasional 14-18 hour day, but when it becomes 10-12 hours every day (and at least a couple hours over the weekend) there comes a point where I would rather just not come in.
More to the point it’s a bad situation for the company. At this point my IT staff is worn kind of thin, and we are basically paralyzed. We can continue what we are doing, but doing anything more (like when the main printer goes down because you accidentally tried to print the entire Internet, or when the pr0n fiend on the third floor infests the network with spyware, and the servers start to lag because of it) is going to force one of us to stop doing something else.
In theory this is a undesirable situation, but in reality what happens is this; we have a bunch of unhappy users, and the IT staff is making decisions about what to get done based on what is least likely to get them fired if it doesn’t get done.
So keeping that in mind, the question I am asked by this executive is; what is the ideal ratio of IT Staff to Users?
Wait, what? It just doesn’t work like that (though I sincerely and truly do wish it were that easy).
Unfortunately IT is not a commodity like so many executives seem to think it is. IT work is not like filing, or data entry, unfortunately it is just not something that your average person can do (if it were I would not have a job).
And beyond that, it can’t be quantified as simple man hours either. An issue that may be a trivial five minute fix for me, may take another admin two hours to fix (or vice versa).
Which led me to the real question, how do I quantify the number of IT staff that is required for this company? Because when you cut right to the source of the issue, each and every company is going to have a different level of IT requirements based on a multitude of factors:
- What does the company do? (a software company is going to require more IT involvement than say a restaurant)
- How screwed up is the current IT system?
- How screwed up are the company’s business processes?
- What level of technical competence do the users have?
These are by no means the only factors, but they are the ones that have the biggest impact on what any given company’s ideal “IT staff to Users ratio” should be.
So what’s the answer? If I knew that I would be making Millions telling everyone how to get it right. All I can offer is this: the people making staffing decisions need to listen when their IT people tell them they need help, because the unfortunate fact of life is this; in the modern workplace (and perhaps more than any other business unit) when IT fails, the business fails.
I could list a plethora of statistics to back that up, but I just don’t have the time.
Admin’s Arsenal: PSTools
Apr 22nd
The PSTools suite is one of those things that you’ll find new uses for every time you play with it. The PSTools suite was developed by Mark Russinovich who worked on the original NTFS file system, and hence has a rather unique insight into the inner workings of Windows systems.
The PSTools suite is comprised of the following utilities:
- PsExec – execute processes remotely
- PsFile – shows files opened remotely
- PsGetSid – display the SID of a computer or a user
- PsInfo – list information about a system
- PsKill – kill processes by name or process ID
- PsList – list detailed information about processes
- PsLoggedOn – see who’s logged on locally and via resource sharing (full source is included)
- PsLogList – dump event log records
- PsPasswd – changes account passwords
- PsService – view and control services
- PsShutdown – shuts down and optionally reboots a computer
- PsSuspend – suspends processes
- PsUptime – shows you how long a system has been running since its last reboot (PsUptime’s functionality has been incorporated into PsInfo)
While these tools work locally (and in most cases work better than the native Windows utilities, or provide functionality that is not available natively), they really shine when it comes to working with remote machines. If I had nothing else but a fresh (default) Windows install, I could probably continue to administer my network using the PSTools.
Notice I said nothing but a default windows install. Microsoft has done something rather unique with the PSTools suite (in fact with the entire Sysinternals utilities collection), and made them usable from a “live” website (to get an overview of what is available, just type live.sysinternals.com\tools into your browser’s address bar).
Now all these tools are stand alone executables (no need to install), so they can be run from a USB drive (SWEET!!!), however being able to run them without even having the executables on the machine is just awesome!
Network documentation and inventory made easy with free tools.
Feb 23rd
For reasons I cannot fathom, the words “network documentation” are like kryptonite for IT people. I’ve seen hardened SysAdmins cringe in fear at the mere mention of network documentation, and it always makes me shake my head and wonder why.
Network documentation is your friend. Unlike tech support for your backup software vendor (who universally seem to only be available 8-5 Monday to Friday), network documentation will be there to hold your hand at 3am after you’ve been trying to restore a downed server for 10 hours, and your $10,000 tape library has just gone up in flames like it was a Dell laptop battery (taking all of your current backup tapes with it).
Everyone seems to have this misconception that network documentation is this incredibly long and difficult process. I would put forth that it is neither. Let me show you just how easy some simple network documentation can be. All of the tools mentioned here are FREE (some may be ad supported, but as is the case with Spiceworks, for a small annual fee the ads can be disabled).
First we start with the Servers. These are likely the most valuable asset in your network, and should be treated as such. The first tool we will look at is the SYDI project.
SYDI is essentially a set of scripts (extensively for Windows, some Linux as well), which will gather TONS of information about the machine they are run on, and optionally create a nice Word document from that information (you must have Microsoft Word installed on the machine you run SYDI from for this feature to work).
SYDI was created by Patrick Ogenstad, and I have found it to be incredibly useful. I currently use SYDI to generate basic documentation for all of the servers on my network (currently all of my Linux servers are Gentoo, and I was able to get the Linux script for SYDI working on them with minimal effort). I have these scripts set to run on the first day of every month, and output the resulting files to a network share.
There is going to be some information (which you will definitely want) that SYDI does not gather. I’ve found that there are a great many tools that can help you get this data automatically, and rather than attempt to cover everything available, I am going to focus on the tools that I use.
I am a big fan of finding ways to do things with the Least Amount Of Administrative Effort (or LAOAE, often pronounced “lay-away”), so when available I like to use tools that can fit multiple needs. The next tool I am going to cover is Spiceworks Desktop, and it fills several of my needs.
Spiceworks is agentless, can be installed on either a server or a desktop machine, and can be accessed from any computer on the network using a web browser (don’t worry, it has extensive security capabilities, including Active Directory integration). One caveat for spiceworks is that you should not try to install it on a machine who’s name contains any “illegal” characters (-,!,?, etc. – this causes weird issues with device discovery). Spiceworks does many things, but the one we are really interested in here is inventory.
The inventory feature of Spiceworks will discover and attempt to identify every device on your network. Once discovery is complete you can view and categorize the devices it finds. The really useful feature for me is the ability to attach notes and files to the devices it finds. Also at this point you can fill in the information that SYDI omitted in your Server Documentation.
Among other things, I use Spiceworks as a repository for my network documentation and as an archive for system event logs. This provides a way for me to keep everyone in the IT department on the same page, and makes it really easy to keep a running changelog for every machine on our network.
No, how do we get those event logs? If you are running Windows machines, I have a script that I’ve written that I use to gather the event logs from my servers to a network share. The code for logArchive.vbs can be found here.
Now then, see how easy it is to generate basic network documentation?
Of course you will also want to eventually create a network map, an IT Service catalog, and… well let’s just say that you can follow this particular rabbit hole as far down as you’d like to go.
Can you hear me now? Know when your email got to their BlackBerry.
Nov 26th
I just love hearing “oh sorry, I didn’t get your email” as a response when I ask someone for a response for the third time. Especially when I know that person has a BlackBerry. When it’s from users on my BlackBerry Enterprise Server (BES) I usually just create a help desk ticket from their “oh I didn’t get your email” response, and then attach a screen shot of the BES log showing that it was in fact delivered to their BlackBerry.
Then they forget that I can do this, and in a few weeks I have to repeat the whole thing. But what do you do when the person you’re sending email to doesn’t have a BES, or is not on your network?
Apparently the good folks at RIM are one step ahead of me, as they have a solution to this nonsense built in. Keeping in mind that this will only work with actual BlackBerry devices (I’ve confirmed that it works with a BES server, and using the BlackBerry Redirector for peeps without a BES), send a email to the address that gets delivered to the BlackBerry with <confirm> as the subject, and in a few moments you should get a reply that looks something like this:
As you’ll see in the screen shot, you can use this functionality with an actual subject, or by sending just <confirm> as the subject (just make sure that <confirm> is the first thing on the subject line). The really awesome part of this is that unless the recipient knows what the <confirm> tag in the subject line does, they have no idea that you now know that the email was delivered to their device. Take note smarmy sales weasels: I see what you did there.
Admin’s Arsenal: BareTail
Nov 11th
As someone that works with Linux systems on a daily basis I have sort of a love/hate relationship with tail.exe (it can be frustrating trying to get it to display exactly what I want, though piping out to grep usually fixes that straight away).
One thing I have always wondered though is why it is not a native Windows utility. It is freaking useful!
Well all is not lost, as there are a couple of solutions available. The “just like on Linux!” alternative for Windows is Tail for Win32. This is a port for Windows systems of the “tail -f” command on *NIX systems.
My preferred solution is BareTail.exe, a more full featured offering. This is basically a GUI version of Tail which allows you to define syntax highlighting. This utility can also be run from the command line.
This utility is chock full of functionality, and is natively portable (meaning you can run it from a USB drive, yes!). BareTail offers many other useful features, including:
- Optimized real-time viewing engine
- View files of any size (> 2GB)
- Scroll to any point in the whole file instantly
- View files over a network
- Configurable line wrapping
- Configurable TAB expansion
- Configurable font, including spacing and offset to maximize use of screen space
- View the end of a growing file in real time, no matter how fast the file grows
- Like “tail -f” on Unix systems, but with many more features
- Simultaneously monitor multiple files for changes using tabs
- Visual indication on each tab of file status and changes
- Tabs may be positioned on any side of the window and oriented horizontally or vertically
- Lines containing particular strings can be highlighted to help you notice important text
- Highlight colors are fully customizable
- Windows / DOS text files (lines end in CR/LF pairs)
- Unix text files (lines end in LF)
- Microsoft IIS logfiles (and other files terminated with a string of nulls)
BareTail comes in two license formats: Free, and Registered ($25 at the time this was written). The Registered version offers all of the features of the Free version plus:
- Regular expression text search (including line numbering for search results)
- Searching while you type, to find results quicker
- Feedback on regex syntax errors while you type, to build regexes quicker
- Filter tail mode (include or exclude lines)
- Frequently used text search patterns may be saved, named and edited
- Export/copy of search and filter results in many formats
At only $25 for a license this is a worthwhile upgrade, but the free version is imminently usable as well.
You can download BareTail here. You can also read Karl’s full review of it here (via MakeUseOf.com)
