Diary of a Network Geek

The trials and tribulations of a Certified Novell Engineer who's been stranded in Houston, Texas.


DNS Redirect Attack

Filed under: Geek Work,News and Current Events,Rotten Apples,The Dark Side — Posted by the Network Geek during the Hour of the Horse which is around lunchtime or 12:34 pm for you boring, normal people.
The moon is Waning Crescent

I’m seeing traffic about this, so I thought I’d write up what I found.

I tweeted about a strange DNS-based network/malware attack that I saw on Friday, but, at the time, I didn’t see any interest, so I didn’t go into any real details.  Besides, I may be a hardcore geek, but I do have a life and was going out.  But, now, I’m seeing search engine traffic hitting my blog apparently looking for details, so I thought I’d describe the attack, as I saw it.

First of all, let me mention that I’ve seen a higher-than-usual occurrence of malware infections the past couple of weeks.  I mean, it’s a hazard of my business that, sooner or later, people are going to get infected, either through bad behavior or by accident, but the past three weeks or so I’ve seen way more problems like that than is even remotely normal.  So, bearing that in mind, I’ve been on a kind of high-alert status looking for any malware problems, but this was something new.

It started with someone from another location, who’s on a totally, physically separate network which uses a different internet service provider to connect to the Internet, calling me with a problem.  It was, apparently, a recurrence of a virus he had previously that we cleaned.  He described being taken to a webpage that featured a maroon graphic background with a white icon of a policeman holding up his hand to indicate “stop”.  The text on the page gave a message that said the user’s browser was not the correct version to access the page and that an upgrade was required.  Helpfully, it provided a button to press to receive the “upgrade”.  Obviously, the “upgrade” was an infection.  (You can see an example of the graphic here.)  Thankfully, I trained my users well enough to be suspicious of these kinds of things and no one who reported this actually clicked on it.

About the same time this happened, I noticed that my iPhone wasn’t connecting to the wifi hotspot I have setup in my office.  I checked the configuration and noticed that the DNS servers listed were wrong.  In fact, they’d all been replaced with a single DNS server;  Obviously, that seemed suspicious to me, so I opened a command prompt on my PC and did a tracert to see if I could figure out where this server was and, from that, why it had become the default DNS server on part of my network, despite my having very carefully configured totally different DNS servers that I knew were safe.  It looked like the tracert results showed me a network path that led out of the country somewhere, which was, to me, very suspicious.

Before I could really pursue that, though, I got another call from a user at my location reporting the exact same error message and graphic, but going to a totally different website! I went to his computer and checked the IP configuration and found that his DNS servers had been replaced by the rogue server as well.  I refreshed his network config, several times actually, and the DNS servers reset, but, when I thought to check some other people in the same area of the building, his configuration set itself back to the rogue DNS server!  So, I reset the local network equipment to clear the DNS cache, and whatever other caches may have gotten poisoned by this attack, and the problem seemed to go away.  Unfortunately, whatever had caused the compromise was still active and seemed to poison the DNS cache and the DNS configuration again.  It did seem sporadic, though, as if the ISP was trying to correct the issue at their end.

As far as I can tell, the attack actually seemed to be network-based in some way.
At least, I couldn’t find any computer on my network that was infected with anything that AVG, Norton Anti-virus, or Malware Bytes could find.  It is, I suppose, possible, that this attack was so new that no of those programs had an updated detection pattern for it, but, based on the lack of detection, and the fact that it happened on two physically separate networks almost simultaneously, leads me to believe that this was a network-based attack.  I suspect that an ARP cache or DNS cache or something similar was attacked and compromised on a major network router somewhere.  Possibly one of the edge routers at a trans-continental connection somewhere.  From the tracert results I had, it looked like it was the East Coast somewhere, leading to Europe via London to France, though I could be wrong.  It’s possible that was a blind alley meant to throw researchers off the trail in some way.
Also, as of this writing the rogue DNS server seems to be out of commission, though that might change, too.

The Internet is a wild and wooly place, ladies and gents, and you can’t always count on your friendly, neighborhood Network Geek to watch over you and keep you safe!  So, be careful out there!
(And, if you’re a fellow professional who’s seen this, too, leave me comments and tell me what you found!)
UPDATE: Looks like the server is still active, but my ISP has blocked DNS traffic to it, to fix the problem.
Also?  I hate the bastards that do these things.  I hate every last one of the little rat bastards!

UPDATE/FOLLOW-UP: So, it seems like a lot of people have been effected by this problem!
Check the comments for what other folks did and tools they might suggest to help with the problem.  Frankly, I wish I’d had known about those tools when I started my day!  Yes, I was *totally* wrong when I said it looked like it was coming in from outside the routers.  It was, in fact, *several* PCs that were infected with whatever it was.  I found it, much like at least one commenter, by checking the results of “ipconfig /all” in a command prompt.  I noticed that the DHCP server listed in the config was NOT my actual DHCP server!  So, as I went from machine to machine, I saw several PCs that kept coming up as DHCP servers.  I used Malware Bytes to scan the infected PCs and it seemed to clean them off.  At least, for now.  I’m not sure what I’ll find in the morning.
Apparently, Friday, when it looked like the problem was getting cleaned up, it was really just people shutting their workstations down early for the long weekend.
In any case, as at least one commenter has mentioned, it looks like updates for the various scanners should be coming out this week, so keep updating your antivirus and antispyware programs and scan your networks!  Well, scan them more completely and carefully than you already have.
And, as always, if you have any new information or suggestions for tools to clear up the issue, please, leave them in the comments!


What’s on your monitor?

Filed under: Fun Work,Geek Work,Linux — Posted by the Network Geek during the Hour of the Monkey which is in the late afternoon or 5:59 pm for you boring, normal people.
The moon is Waxing Gibbous

What do you run on your monitor server?

Do you think you’re too small to run a monitoring server? Well, I have two local servers, a remote web server and a remote e-mail server that I’m in charge of worrying about and I run a monitoring server. It’s not much of a server, really, just an old workstation to which I added a bunch of spare memory and a large, clean hard drive. Naturally, I run Linux on my monitoring server, which, ironically enough, I named Monitor. Specifically, Monitor runs Red Hat Fedora.

Monitor runs Nagios, which I’ve mentioned before. With Nagios, I monitor both my main file server and my accounting SQL server. I also watch the off-site web server and the SMTP and POP3 e-mail services on the managed e-mail server we have through our ISP, just to make sure they’re up and running. (It’s a long story on why we have that, but, rather than run my own, to reduce hassle, headache and potential disaster, I let someone else worry about it.) Nagios tells me the status of drive space, the memory usage, the CPU usage and uptime on both servers. On the accounting SQL server, it verifies that the SQL service is available and that users can log into it. On the file server, it tells me the status of the Backupexec modules. Unfortunately, I haven’t figured out a way to get Nagios to tell me more than the running status of Backupexec, but, in my spare time, I still try to find a way to have it report the status of the last backup or restore job run. No joy yet, but I keep trying.

I also have a browser window open to the SolarWinds installation at our ISP. They monitor inbound and outbound traffic over the Internet connection we have. Usually, I keep a window open on the standard “interface details” reports which update regularly. Most of the time, I also open a window to the weekly history report on the min/max/average packets in and out. I have to update that manually, but it lets me quickly compare today’s traffic to network traffic for the past week. It’s nice to see those trends!
Lately, I’ve been keeping a browser window open to the national weather forecast, by hour, for our local area. In hurricane country, keeping track of the weather can be vitally important! But, if you live in snow country, the same thing would probably be true, too. I don’t recall heavy snow causing an outage during my time up North, but it’s not out of the realm of possibility.

Finally, I almost always have Wireshark running a packet capture, too. If I see a sudden spike in traffic, having a packet capture already running could make a big difference. I have that capture set to save files locally, too, just in case. I’ve been setting the capture to rotate nine files and to keep the files at seven megabytes each. That should give me a pretty good spread of captured network data if I ever need to go back and diagnose a traffic problem. And, since the machine is actually kind of stinky hardware and crashes on occasion, when I restart the packet capture, I rename the base file using the current date. That way, I can tell at a glance when the capture was started.

One day, I’d like to move this all to another machine that’s more stable, faster and has more drive space, but, until then, this works. It’s only on the private network, so I can’t look at it directly from the Internet, but, it still does enough for me. One of these days, I’ll look into some of Nagios’ data presentation modules and teach this old dog a few new tricks, like automated uptime reports and that kind of thing.

Hopefully, that hasn’t bored too many of my non-geek readers. And, I hope it’s given my geek readers something to think about. So, tell me in the comments, if you have a monitoring server/station/whatever, what does it run? If you don’t have one, why not?


Review: Nagios

Filed under: Career Archive,Fun Work,Geek Work,Linux,MicroSoft,Novell,Ooo, shiny...,Review — Posted by the Network Geek during the Hour of the Tiger which is terribly early in the morning or 5:27 am for you boring, normal people.
The moon is Waxing Crescent

I don’t think I’ve mentioned this before.

Some time ago, I was having problems with traffic on my network. Something, somewhere was apparently causing some issues with bandwidth on our Internet connection. Or, at least, that’s what our ISP kept telling us. It was, I think, the excuse they were using to avoid dealing with an e-mail problem.
Regardless, I had to find a tool to monitor our network traffic. I ended up using Wireshark for that, but along the way, I discovered a number of OpenSource monitoring tools for various purposes. The one that impressed me the most was Nagios.

Nagios is, according to the opening paragraph on their website, “an Open Source host, service and network monitoring program.” While I never did configure anything to monitor the network, per se, I did configure this to watch both local servers and third-party web and mail servers.
First of all, it’s important to know that Nagios runs on Linux. So, to install the software, you first have to have an available Linux server on which to install it. I’m using an old workstation that I installed the latest version of Fedora, the free version of RedHat. Getting the initial install done wasn’t very hard at all. In fact, there were RPMs available, so all I had to do was use RedHat’s package manager to get the base install loaded on the machine.

After the initial software load, I mainly followed the Quick Install instructions that they link to on the first page. Then, since I was mainly monitoring Windows servers and workstations, I found the cleverly titled help page, “Monitoring Windows Machines“, and followed that. This page ran me through the basics of installing the NSClient++ on a Windows machine and configuring Nagios to connect to and monitor that client. One thing that I had to find out the hard way was that the entries for the monitored systems have to be duplicated for each host. In other words, there is no way to just list all the Windows systems you want to monitor. You have to created entries describing each host individually. That’s not a big deal, honestly, since you can open the configuration files in a text editor and just copy, paste and edit the required entries.
I did have a few false starts here, until I figured out the correct syntax and the fact that every host has to be part of a previously defined group. But, other than that little glitch, configuration was fairly simple.

It took a little more digging, but I later found instructions for passively monitoring services running on servers without a client. I now use my private installation of Nagios to monitor our company webserver, both POP3 and SMTP on our hosted e-mail server, as well as my two Windows 2003 servers. I can even check on the Microsoft SQL database, thanks to information I got from this post on the OSdir mailarchive. And, did I mention that all this software was free? Yeah, the documentation wasn’t the best and it took me a little while to figure out the install and config, but it was far easier than the other monitoring software I played with and I can let anyone who has the username and password check these stats from their own workstation via a web browser. How cool is that? Oh, and did I mention that this can be used to monitor Linux/Unix systems, Windows systems and even Netware systems? Nagios pretty well covers it all!
(Oh, and as a side note, if you’re messing around with the configuration and want to reset the statistics, just stop the service and delete /usr/local/nagios/var/status.dat, then restart the service. All your counts will zero and all the checks will start fresh.)
In short, if you’re looking for a low-cost but versatile monitoring system and aren’t afraid to read the documentation, I highly recommend investigating Nagios.


AOL Mail Problems

Filed under: Criticism, Marginalia, and Notes,News and Current Events,Personal,The Network Geek at Home — Posted by the Network Geek during the Hour of the Horse which is around lunchtime or 1:15 pm for you boring, normal people.
The moon is Waxing Gibbous

Not the best way to communicate this, but…

For those of you who contact me via e-mail with an AOL address, I’ve had trouble getting a reply through to you.  Apparently, my ISP, SBC/AT&T/Whatever they call themselves this week, is having some kind of communictation problem with AOL’s servers.
So, I got your e-mails, but I can’t get a reply back to you.

In essence, though, everyone just wrote “Um, hello?  Still there?  Couldn’t write a bit more to let us know you’re still alive?”.  So, let me go on record that there’s more coming, as soon as I have a bit of time to think the posts through and edit them and all that.

In short, more soon.

Tags: , ,


DIY Wireless ISP

Filed under: Fun,Fun Work,Geek Work,Linux,The Network Geek at Home,Things to Read — Posted by the Network Geek during the Hour of the Tiger which is terribly early in the morning or 5:43 am for you boring, normal people.
The moon is a Full Moon

Hmm, this could be fun…
Lately, IBM has been really supportive of Linux. In fact, they have a whole section on their site dedicated to cool things you can accomplish with Linux. The latest of these is an article about how to use Linux to set up a wireless ISP. They suggest setting this up for a neighborhood or office, but I’m sure there are applications far beyond that. They take you through the basics, but, after skimming the article, I sure wouldn’t reccomend this for the Linux neophyte. The article does cover, in brief, all the aspects of this project, including hardware choices, but it focuses on a series of bash scripts written by the author to help you manage your WISP. That’s all well and good, but, of course, limited in scope.
So, why not add all that functionality to a backpack and make yourself a walking “hotspot”. Yeah, that’s not a joke. A guy actually took a backpack with solar panels built into it, added some wireless hardware, and made himself into a roving hotspot.

Now, that’s entertainment! And perfect for a fun, freaky Friday link.


ISPs and Updates and Such

Filed under: Personal — Posted by the Network Geek during the Hour of the Snake which is mid-morning or 10:49 am for you boring, normal people.
The moon is a Third Quarter Moon

Wow, it’s been a long time since I updated my blog!

For one thing, I’ve been wrestling with my web host/ISP for one of my other domain names. I wanted to get a simple upgrade done and two weeks later, I didn’t even have a status report, much less the upgrade. So, I decided to move that domain, which is Fantasist.net, BTW, to a new ISP. So far, so good. They seem to be better than the last folks and they sure are less expensive!

As for other updates, I’m still working at Loomis Fargo. In fact, they’ve started to make noises about bringing me on board full-time. So, I guess I’m still doing okay. It’s been a really easy job for me so far. Nothing more challenging than checking backups and dealing with printer problems, really. Of course, this weekend, I’ll be doing an in-place upgrade on a Novell NetWare server. Got to get up to version 5.1, or we won’t get Novell support! And, at the same time, I’ll update it to the newest NetWare Directory Services revision, too. Two birds with one stone and all that.

Well, that’s enough of an update for now. More later this week. Maybe how I survived Thanksgiving at my in-laws.

Powered by WordPress
Any links to sites selling any reviewed item, including but not limited to Amazon, may be affiliate links which will pay me some tiny bit of money if used to purchase the item, but this site does no paid reviews and all opinions are my own.