Diary of a Network Geek

Hey guess where I'm writing this from? Yep, I'm back in Baton Rouge! Yahoo! Only this time, I flew. No more of that driving across the swamp at night. Blech!
Things were going fairly well Monday. I got to start correcting the NDS (that's Novell Directory Services, for you neophytes) synchronization issues that we're still having thanks to Mother, our primary NDS Replica server, going South on me. Of course, just about every change I want to make takes about two hours to replicate out. Now, I *know* that I've explained that to the Supreme Commander, but he still doesn't seem to "get" it. He's a really Big Picture sort of guy and doesn't understand *any* of the details of what I do. The joys of working for a former programmer. That's all they truly understand or think matters. The rest is just magic that should happen instantly. *sigh* Of course, the fact that our building lost power didn't help much. Thankfully, I've got my Palm IIIc! I just whipped that out and started playing Space Trader. Ah, what fun.
Then, on Tuesday, I started on that again. Until the Baton Rouge server dropped off the network. It still was up, but it stopped communicating with the rest of the network. The Cisco router was fine, so I knew it wasn't the WAN connection, but it just wouldn't see the network. So we rebooted it. Oh, boy, was that a mistake. It never came up right. Eventually, we got to a point where all we could get was an "Invalid Partition Table" error. So, it's off to the airport for me. Leaving my poor sick wife to deal with getting our daughter from school and taking care of our leaking pond. Oh, joy.
Well, I got there and started fumbling around with stuff, but didn't get very far. Then, I noticed that we had been trying to use Windows 95 utilities on a DOS 6.22 formatted partition! Damn, no wonder we weren't getting anywhere! So, that started us root around for a DOS bootable disk. We finally found one and I was able to FDISK and FORMAT the old Primary drive and copy everything from the DOS partition of the mirror. Then, while eating my oyster po'boy, we rebooted and *shazam*! The server starts to come up. The next thing we know, it's already started running the backup. So, since all looked well, we called it a night. Now I'm waiting for everyone to get caught up so I can clean up the hardware and get the server back up, hopefully for good. Or at least until they move, when I hope they'll upgrade to real hardware. You know, something brand-name, like Compaq or Dell or even HP. I hate having to figure out clones. Then, I'll try to get an earlier flight back to Houston and just enjoy a night at home. Maybe even snuggle with the wife on the couch while we watch Star Trek. Voyager's still in reruns! ; }

Well, I finally got my cable run. Of course, I stepped through my ceiling first. *sigh*
Everything was going okay Saturday with my cable runs. I got the pull string through and got cable feeding up into the closet in the library/server room, but then it got hung up. So, it was up into the attic I went. Well, I found the first snag and got that cleared. Then, I queued up the cables (one phone line and two network cables) to go down the other side of the topmost attic. I climbed down and start to pull it, but it got snagged again. *sigh* So, back up into the attic. The heat was so oppressive that I could hardly breathe. I was pouring sweat faster than I could drink water. I was tired from my little trip to Baton Rouge. And, I slipped. I was stepping carefully from one rafter to another and then to the one-by-six that spanned that side of the attic and my foot slipped off and through the ceiling. *sigh* Well, at least I didn't fall *all* the way through! And, the hole did make a convenient breathing hole when I got too hot.
Anyway, I finished running those two network lines and the phone line without a hitch. Then, I used the pull string to get the next two network connections with minimal problems. I also managed to pull two network lines down to the master bedroom, though I had to finish that on Sunday because the drill needed a recharge. I used the same pull string for the two connections to our daughter's room, but when I went to put it down the wall, I found an AC conduit blocking my way. I also had to contend with a 110v line that was a little too close to where I wanted my connection. So, I ended up drilling the hole in the top-plate on the wrong side of the stud from where I cut the hole in the drywall. *sigh* Looks like another patch job. At least this one's not as big as the ceiling!
And, with that, I decided to stop running cables for now. I've got connections everywhere I need them. Truthfully, I doubt that I'll actually use the ones in our bedroom, but it's the idea of it. So, now I just have to punch them all down and hook up the jacks. Then, test them to make sure they work. Yahoo. After that, I'll do up a whole section of "what I learned while running cable" for the poor saps who still think they want to do this. My advice? *Pay* someone else to do it!

*sigh* Here I sit, tired to exhaustion, waiting for the weather to clear in Baton Rouge, Louisiana. How did I end up here? *sigh* Oh, my what a story. I'll keep it short, though. Once again, I'm paying for someone else's short-sightedness. The beat-up, old hardware in our Baton Rouge office curled up its toes and died yesterday afternoon. Or, at least part of it did. And, since we couldn't quite seem to get it up and running remotely, I was "volunteered" to go fix it.
Okay, that's not quite true. After the last mess I dealt with due to crappy hardware, I felt that if I didn't go, it might cost me my job. Also, as much as I like my employees, the last time I sent someone else to do this kind of work, I was blamed because they couldn't make the old, crappy hardware work. So, I decided that if I were going to take the blame anyway, I might as well do the work, too.
What that meant, however, was loading up the car with server equipment and driving over 300 miles to Baton Rouge at 9:00pm at night. That's roughly a four to five hour trip to a place I've never been at night when I was already tired. I only had one near miss. And that wasn't my fault. That was the two drunk kids in the Saturn outside of Beaumont that came off an entrance ramp and across three lanes of traffic without looking. I don't think I actually was on two tires at any point, but only because I had the extra weight of the server equipment.
Anyway, I did manage to get a couple hours of sleep before I came in to rescue the server. After I blew the dust off the SCSI ID settings on the external Artecon drive arrays, I spotted the trouble pretty quickly. So, I swapped out the bad tape drive that started all this and got them up and running. That was about 9:45am this morning. My next task is to get a set of mirror drives put together for them and get them into place. That's why I'm waiting for the weather to clear, so I can get out to my car to get the hardware I need. Then maybe my captors, er... fellow employees, will let me sleep. Probably not. *sigh* More updates later.... Or maybe tomorrow after I drive back.

ilovemyjobilovemyjobilovemyjob......

Well, I ran into a little snag running cable. Literally. I managed to get through the top plate and into the wall at the right spot, but when I actually tried to pull the cable, it snagged on the roof before it even got out of the server room closet! I'll have to get up into the nasty part of the attic to fish it around the snag. Though, I may try to push it up through with the fiberglass pull rod and see if it gets past the obstruction. Either way, I didn't feel like getting all nasty in the attic anymore, so I was done for the night.

Well, Day 1 of cable running didn't go quite as well as I'd hoped. For one thing, I found out just how wimpy my cordless drills are. But, let me start from the begining....
Saturday, after going in for some tests and passing my kidney stone, I stopped at Lowe's for some tools and supplies. Mainly, I got drill bits and wireties and that kind of junk. But, I also got a *good* stud finder and gloves and breathing masks. Very important as I was going to be crawling around in a very dusty attic. But, by the time we got through with the rest of our day, I wasn't up for running cable. So, Sunday afternoon, I got set up.
First, I changed into the crappiest clothes I could find. Jeans and an old, long-sleeved t-shirt. Then I got my hat on, scrunched my mask down, and pulled my gloves on. Then, I crawled up into the attic with my fiberglass pull-rods and started fishing around for an opening. No luck. So, I crawled out of the attic behind the "server room" (ie. closet) and headed up to the main attic with the other half of my pullrod. I fished it down and when I got back into the attic behind the server room, I was in luck. I pulled the rod back up some and tied the nylon cord that I planned to pull cable with to the end. Then, after thinking about it for a minute, I tied a second line to that one to setup my second long cable run. After that it was back up into the main attic to fish the pullrod up and over. I managed that and, as I bellycrawled back out, I pulled the second line with me. That should, I hope, miminize the number of trips to the dusties part of the attic. That was the last easy thing that happend,though.
At that point, I got into the attic over where our office will be and tried to fish the pullrod down. After a little manuvering, it seemed to go just fine. However, when I got down stairs and cut my first hole for the wall plate, there was no pullrod. So, I went back up into the attic and tried again. Still no luck. Well, this time I decide to cut a hole for the second wall plate that I planned on putting in, but still no luck. So, it was back up into the attic to see what was what. It was then that I discovered that I had fished the pullrod down the *outside* wall behind the brick facade. Damn. At that point I figured out that I needed to drill through the top plate of the wall so I could fish my line. That's when I discovered that my cordless drills were just way wimpy. *sigh* And, in fact, that's where I finally gave up for the day. I'll go back to it tonight and see if I can finish drilling that hole. If I can, then the rest should be downhill. And, I console myself with the thought that this is the hardes of the cable runs. The work of a network geek is never done!

Okay, so I *still* haven't been all that great about keeping this up to date, but I have an excuse. My wife and I got the house. In fact, we just finished moving in this last week. Now, we're surrounded by boxes. But, somehow, I don't mind. After all, we have *all* our stuff, no more storage facility, and we *still* have more room than we did in that stinking little apartment!
Now, let's see, I still haven't passed that damn kidney stone, so tomorrow, Saturday, I'll be going in bright and early to drink a radioactive milkshake and have intestinal pictures taken. Of course, the real fun will be the castor oil that I have to drink the night before. My lovely wife, who's very knowledgeable about health and medical stuff, says that it's to clean me out. I guess so that nothing interferes with the pictures. Fun.
Oh, and I bought all my supplies for running my cat 5 network cable this week, too. With any luck at all, I'll be able to start wiring up the house this weekend. I've looked all over the Internet for people who have chronicled their home cabling experience, but I came up pretty empty. There are a few folks that have talked about it, but not too many really. So, of course, that means that I'll be doing my best to log all the stuff I do. I'm even going to try and give a little technical advice, but I'm afraid that it will mostly be out of date by the time it hits the web. In part because I'm using at least *some* "found" materials. Namely, I saved myself a couple of bucks (roughly $150) by salvaging a category 5 punchdown panel. It only has 24 connections, but how many does a home network need?!? Anyway, I'll do my best to make a usable reference of everything I learn along the way. Stay tuned!

Well, it sure has been a long time since I did any updates on this page! In fact, it's been literally months!! I can explain...

First off, I've been pretty damn busy at work. After my little adventure in the New Orleans airport, I hardly had a chance to sit down. Not only have I had to roll out two sets of patches, or "Service Packs" as they're called now, on our Novell servers, but I've had to set up a Novell BorderManager Proxy server. Man, the troubles I've had with that! The initial setup was okay, but getting the VPN (Virtual Private Network) working has really bedeviled me. It's been so bad that my boss, the CIO, had to raise some ruckus with our sales reps at Novell. He's finally gotten me a connection with our local Support Engineer, who's promised to come take a look at our configuration for me. Hopefully, he can find what a CNE (Certified Novell Engineer) with more than eight years experience has missed.
Second of all, I finally got my gorgeous girlfriend/fiancé to marry me. On Saturday, March 10th, 2001, Anne and I got married in the Canterbury Wedding Chapel at the Excalibur Hotel in Las Vegas, Nevada. Allie, Anne's daughter, was our ring-bearer/flower-girl. The service was nice, short and attended by the photographer. We thought it was the perfect way to get married. No one hassling us or telling us what to do and how to do it. No meddling mothers or aunts or sisters to make us miserable. No seating charts or receptions to worry about planning. Just a cool, quick wedding and a great steak dinner afterward. After that, we spent most of the week running around Las Vegas doing tourist stuff. We went to see Sigfried & Roy, Tournament of Kings and The Blue Man Group. We also went to see Star Trek:The Experience, where I bought "genuine" Romulan Ale and Klingon Blood Wine. But, of course, work intervened and I had to come back a day early to "fix" something that had gone wrong. What would they do if I ever quit?
Finally, now that we've done all that, we've been looking for houses. It looks like we found a nice four-bedroom, two-bath, two-story house on a lot that's almost double-deep! And, even with all the flooding that we've had here in Houston, it stayed dry. What could be better?!? Two koi-ponds, that's what. That's right, the property already has two koi ponds on it, complete with koi. There's a fireplace with a Ben Franklin stove in it and a partially finished garage. There's a covered car port that's connected to the house by a covered walkway. It's really great and I hope the sale goes through okay. We're scheduled to close this Friday, 6/29/2001!
And, of course, I'd be remiss if I didn't mention my kidney stone. That's right, as if all the work I've been doing isn't enough, I was doubled over in pain by a kidney stone last week Monday, 6/22/2001. It got so bad, that I had Anne take me to the emergency room. We ran into Charles, our paramedic friend, there who told the nurses to take good care of me, which they did. I got moved through as fast as I've ever been. And a nice, young doctor gave me some great pain medication. I still haven't passed the damn thing, but I'm off to see the friendly urologist tomorrow. I plan on demanding that he "do something" about it because I'm tired of peeing through a strainer!
Until next time....

So, you'd think with all that this past week, things would be getting better, right? Well, think again. Apparently, the network has more issues than even *I* was aware. NDS (Netware Directory Services) and GroupWise (the e-mail program) were way out of synch. The bad part about this is that someone, who works for me, kept resubmitting an update. In short, we tried too many updates without out allowing these two systems to synch up and now they're broken. Mind, it didn't help much that we didn't find it until things were really bad. I think that was due to my being absent from the office. I feel like I'm bridging a gap of some kind in the department communications. There's a guy there who's very knowledgeable about networks, if not NDS in particular, but has trouble communicating with his co-workers. To give you an idea of what this means, someone congratulated me for being able to work with him the other day.
Anyway, that means that this problem was probably known, but no one did anything about it, because they were afraid to try and work with this guy. The sad thing is, he probably could have fixed this problem if he'd known about it soon enough. Ah, well, what's done is done. The damaged GroupWise database has been sent off to Novell, and we should hear something back on Monday. It probably means that I'll be working late all week again. I'm not sure how it is that this always happens, but we're going to see if we can distribute the "fun" around the department. I'll let you know how it goes.

On a *really* positive note, the nightshift police took a real interest in my Palm and my folding keyboard. They were totally amazed at my typing into such a tiny device. Hey, at least they kept me company for a few minutes. I think I'll ask them about vending machines and then go back and play some Area 51 in the video arcade. At least there's that to do while I wait. Afterward, maybe I'll take a nap. (As if that could happen with the noisy cleaning crew!) I wonder what they serve for breakfast in the airport?

Here I sit, at 12:25am, in the lobby of the New Orleans airport waiting until 6:00am so I can go home. "Why?", you ask. Well, I'll tell you. If for no other reason than to look busy so that the nice police officers stop looking at me like I'm some kind of vagrant. It all started a couple of weeks ago, when I noticed that the chuckle-head who was the Network Admin before me had not gotten all the servers onto the latest set of service packs. When I asked my boss about this, he was somewhat surprised, since he'd been told that everything was up to date. Not hardly. In fact, I had servers that went back at least two revisions! So, two weeks ago I prepared to get all the servers updated. Behold, my true troubles begin! I try to get the patches copied to all the servers and discover that it takes literally *days* to copy the files! That's right, the patches are so big, and our bandwidth is so small, that it takes almost five days to copy the Novell patches to all the servers. But, wait! They didn't really copy to all the servers!! Yikes! So, the re-copying begins. Now, this is when it starts to get really fun. You see, in most of the cases, the reason the Novell Service Pack didn't copy, is because there just wasn't enough room on the main volume. (That's SYS: for all you Novell CNEs out there. Not that there's many of us left. Certified Netware Engineers are a dying breed, I'm afraid.) Well, I try the obvious solution and recopy everything to one of the other volumes (that's a "disk drive" to you non-network engineer geeks.) At first, that seems to work well, but, on closer examination, I discover that one of the servers has still refused to copy everything. It turns out that this server lost it's wide area network connection. *sigh*
Okay, so I try copying the patches yet again, but this time I send them in a compressed format. So far, so good. While all that copying was going on, I started installing the patches on the other servers. Again, most of them worked just fine, but those few with teeny, tiny SYS: volumes don't all work. So, at this point, I've ended up with four that don't quite have enough room on the main volume to install patches and one that finally has the patches *copied* , but not installed. This is where the problems that actually led me to the New Orleans airport began. We recently purchased a nifty utility called ServerMagic from PowerQuest. This darn program is the best thing since sliced bread! Among other things, it lets you resize Netware volumes without having to destroy them first! Totally cool, and totally impossible before this tool. So, never having used this before, I installed it on one of the remote servers. I ran it and it asks to reboot the server. (GULP!) I took a deep breath and tell the program, "okay". Bang! The server goes away. Poof! It never comes back up. *sigh* Well, that was just the beginning. I let my boss know what was going on, and then we sat back to wait for Monday morning when that office discovered their crashed server. In the meantime, of course, I installed the patches on the rest of the servers. Or, at least, I *tried*. Some of these servers **still** didn't have enough room to install all the patches!! Well, I figured that I'd eventually get the space problem worked out, and just left it alone. Oh, boy, was that a mistake! Monday, the first office called to get the server up and running. We got the first signs of life about 9:30am, with some errors, of course. About then, that office's answering service lets them know that I know about the server and that they should call me. *sigh* So, okay, we get this pretty well squared away, when the development department tells me about this new, totally redesigned piece of misson-critical software that *I* need to get rolled out. Being the kind of guy I am, I start getting this all configured to run, from scratch. Now, we've survived until Tuesday morning without a tragedy.
Well, about the time that I start to work on finishing the big, new, shiny software rollout, another site calls to tell me that their server is down. "How did that happen?", I wonder. No time to worry about that, though, because we've got two whole companies down while this one fileserver is off-line. So, like I always do, I started trying to figure out what happened so that I can undo it. It turns out that they took it upon themselves to reboot the server because there were users having problems logging in. Aha! The new patches had partially loaded, because they were partially installed, and some of the new files didn't like some of the old files. Blamo! One crashed fileserver. Well, I banged away at it for most of the afternoon, one way or another. I tried loading files from backup directories, and copying files from the backup directories, neither of which worked. Then I tried to use ServerMagic to resize volumes so that I could come up with the extra space I needed to install the patches. Well, that *almost* worked. Apparently, the old disk drive just couldn't quite handle the new utility and just decided to stop working.
By this time, it's after 4:30pm and I've already figured out that I'm flying out to the remote site. Shortly after I suppose I'm going, my boss confirms it for me. We agree to try a couple more things, but not past 5:30pm. So, now, it's do-or-die time. Let's just say that I didn't "do". My boss helps me get the travel arranged, and we decide that a day trip should be enough. I gather my tools together and scare up a 4Gig external drive that I can add on to the ailing server. Then, I got the brilliant idea to burn a CD-ROM that has the patches on it. After all, since I have to add diskspace anyway, I might as well install the patches while I'm there! That was easier said than done. *sigh* I finally downloaded the required files to my PC and used my own CDR to make what I needed. It took over 6 hours to get the complete download. I got out of the house with just enough time to make the first flight from IAH to OLY. That's at 6:50am, just in case you're interested. It gets into New Orleans at about 7:00am. I then spent almost 45 minutes getting from the airport into my rental car. After that, I was ready to drive more than an hour to get to our site in Houma, which I finally did after a few wrong turns.
Finally, I was ready to start working on the server at about 10:30am. It took all of about 15 minutes to get the extra drive installed, configured and running. The fileserver itself took a little longer. I got enough of the new drive allocated to the SYS: volume and then I got the new patches installed from my CD. So far, so good. In fact, at this point was planning to try and catch an earlier flight home than the 5:20pm flight I was scheduled to fly back on. Oh, well, maybe next time. I rebooted the server, so that the new patches were activated, but started getting errors right away. Apparently, the USR volume, where most of the actual data existed, was damaged. Damn! So, I try every repair utility that I can think of, without any improvement. Eventually, we had to delete most of the data because it was too badly damaged to actually use. Unfortunately, that volume also stored the GroupWise e-mail system data. Whoops! So, now, it was about 2:00pm and I was wolfing down an oyster poboy while trying to get the USR volume back online, when it hit me to try and use ServerMagic to fix the problem. Well, that *almost* worked, but it recovered the drive at the expense of the data it contained. Double damn! We don't back up the GroupWise data for legal reasons, so now we can't restore it. But, wait! The local remote admin ran a backup last month that accidentally included the e-mail directories. Hooray!! Well, to shorten a *very* long story, we managed to recover and rebuild the mail databases so at least they'll have user accounts and mail going back a couple of weeks. Not ideal, but better than nothing.
But, how did I end up missing my flight? Well, to prove that we give the best customer service, I stayed while the Arcserve restores ran. (Okay, it was more like they ran, then got totally screwed up, then we deleted them and recreated the jobs and *then* they ran.) And, the next thing you know, it's 9:30pm and I'm running to try and make a 10:30pm flight. Obviously, I missed it. *sigh* On a more positive note, I do have a flight booked for 6:00am. Of course, that means that I'll have just enough time to shower and change before I head back into work so that I can get the super-duper software rollout working. Blech! I plan to get the most out of the damn company picnic on Sunday!