Wednesday, December 27, 2006

Candy Security Opens It's Virtual Doors

Back in 1997 or so I started building FreeBSD firewalls and proxy servers for friends and business partners. To my surprise, many of those systems ran until their hardware fell apart(literally). As those systems died nearly all of them were replaced with off-the-shelf commercial products. I was no longer in the business of building firewalls and I could not easily compete with the cost advantages of the commercial options. For a small business, your average wireless router does quite well for NAT and DHCP. At the time, the cost of $100 or so was quite reasonable. For the larger businesses - there are larger needs that include SPAM filtering, Antivirus, Intrusion Detection/Prevention, and more. The answer for them would be a Cisco device or a competitive product (such as a SonicWALL).

After kicking around a SonicWALL for a few months for my home business, I just had the feeling that I could do better with hardware that I had laying around. I was looking to replace not only the NAT functionality of the hardware, but the other services as well. It didn't take me long to discover IPCop. IPCop is an open source firewall solution with a Linux core. It's easily installed, and even easier to configure through web based management. For the hardcore enthusiast there is a growing collection of compiled add-ons for the product which include ClamAV (Gateway Antivirus), Squid (for proxying), Squidguard (for content filtering), and even Snort (for Intrusion Protection). Mix all those together, and you have yourself a very affordable unified threat management system (a term I do not throw around loosely).

It wasn't long before I started deploying my IPCop firewalls to friends, family, and small business owners (many out of charity). The hardware, while never fantastic, has always been adequate for their use and has all been purely recycled components. In some cases, I have taken a companies retired desktop PC's and turned them into managed network security solutions for them. Aside from saving a ton of money on the cost of hardware and security services, we have kept the environment from having to digest a pile of toxic components.

There was only one thing missing. I had never officially given the company a title. While working on the website, that seemed necessary. While some of you may still refer to it as "Steve's firewall business", we will be putting something more pleasing on our business cards. The official title, as the subject implies, is Candy Security. I have been tossing the name around for nearly a year and I even registered the domain name some time ago in preparation. I have most recently built a website, and today it went live!

You will have to excuse the advertisements glued to the top. I opted to get "free hosting" until I had something to put up. Now that it's up, I have to have my account converted from a free account to a paid hosting account. The last time I did this with GoDaddy, it required deleting and re-creating my account.

Hope you all like the new site. For present customers (or other interests) I created an area that you can use to contact me from the website. Obviously, if you are having problems and need immediate assistance you can still call us.

http://www.candysecurity.com

-Steve Ballantyne

Bad Attitudes Have No Business In the IT Business

I thought that I would post something that's non-technical today. I have had a couple of dealings with customers these past few days that were less than pleasant. One of which I knew from days past. With that, I have thought I would write up a few words on bad attitudes in IT.

It seems that throughout the nineties there were a lot of companies that had administrators and IT personnel that nobody wanted to talk to. In some cases (that I myself have witnessed) there were overpaid administrators who literally sat in a dark room most of the day and would throw a fit if anyone bothered them for assistance. In my days as an IT instructor I would often cross paths with some of these folks and I wondered why anyone would employ them.

At some point in the history of information technology it seems that we accepted that for someone to be smart and helpful in the area of computing, it should be expected that they would come with personality flaws (or even hygiene problems). Into the late nineties and early millennium many of these administrators found themselves unemployed. The job market for information technology shrank. Or, did it? Perhaps as the economy suffered, businesses of all sizes had to cut back and see who they could do without. Could they live without that mean guy who sits in the dark playing video games all day? They were willing to try.

While my training business in was booming in mid 2005 and I had no shortage of work, I was running into old colleagues and sometimes students who said that they had been trying for weeks or months to find work. What was it that they were missing? A good attitude for starters (and better hygiene for those others). Good attitudes will always make up for what you may be lacking in the area of experience. While many will say that it's not "what you know" but "who you know", I still contest that it's how you treat people that determines if you are going to get a job.

I have also been amazed at what a small world it is when it comes to information technology. I still run into people that I worked with ten or more years ago. Almost all of those people that I bump into are still in the same field. I am assuming that they are all glad to see me. But when I run into an old colleague that I had a hard time with, I will not exchange cards. In such a small world there are few rivers. Nothing will kill your career faster than burnt bridges. And nothing accelerates bridge fires like a bad attitude!

Remember to slow down, relax, and be nice to the people you work with. Be even nicer to the people that you help, or the people that help you. Compassion and kindness are instruments of business.

-Steve Ballantyne

Sunday, December 10, 2006

Building a site-to-site VPN tunnel between SonicWALL and IPCop

To perform this task, I used:
A SonicWALL TZ170 Running SonicOS Enhanced 3.2.0.3-54e
A PC running IPCop v1.4.11, with the built in VPN functionality

SonicWALL is becoming ever more popular as a good solution for the small to mid-sized business due to its lower TCO over products such as Cisco. In the same regard, IPCop has grown in popularity for its open source nature, and its large list of features. Therefore, I thought it would be interesting to see how difficult it might be to merge two companies using these very different products. In the end I can say, “not too difficult at all”.

Some things to remember:
* You must have two different networks, with different network ID’s. That is, if one company has a network ID of 192.168.0.0/24, than the other side cannot also be 192.168.0.0/24. This will be a routed network. You cannot route between two networks, that are actually the *same* network!

* SonicWALL is a commercial product and is backed by toll free phone support (which varies by product and status). Help is most certainly guaranteed. IPCop is open source. That means that your support comes in the form of forums, IRC, etc. Also note that the use of open source software comes with an agreement. Make sure that your company or organization is permitted to use open source software.

Step #1 – Set up the SonicWALL side of the tunnel.
Log into the SonicWALL Administration page. Click “VPN” on the left side, and ensure that you are now looking at “Settings”. Now under the “VPN Policies” click the Add button.

Leave the authentication drop-down at its default. Name your policy whatever you wish. You can use spaces here, it doesn’t matter.

For “Primary Gateway” you need to enter the IP address of the IPCop firewall. If this address was obtained via DHCP and will be changing, then you will need to set up Dynamic DNS for that box (Google for help on that). The SonicWALL *will* accept a hostname here instead of an IP address if DDNS is in use. For the “Secondary”, you can enter 0.0.0.0, or if left blank, the SonicWALL will enter that for you. This would be used if you had a “backup VPN” in place to another box in case this VPN fails.

Enter a secret password into the “Pre-Shared Key” area. You will need to enter this same password on the IPCop firewall. The longer the password is, and the more obscure the characters are (!@#$%^, etc), the better your encryption will be. So be creative.

For your “peer ID’s”, I would typically recommend using the default which is “IP Address”. And normally you would leave the fields blank. But, SonicWALL and IPCop did not handle that well at all. I suggest using something textual such as “E-Mail addresses”. The idea here is you would use a contact at the SonicWALL site for the SonicWALL side, and the address of an employee at the IPCop side for the IPCop settings. It doesn’t even have to be a real e-mail address, but whatever you enter in these boxes, must match the other side when we are done (local to remote, remote to local).

When done, it will look something like the illustration below. Click the Network Tab to continue.



Things may get tricky here depending on how complicated your network is. We will assume that you have one standard reserved network address on your local network. In our case, the SonicWALL’s network is known as “192.168.199.0” with a subnet mask of 255.255.255.0. We could create a new “object” for that address, but instead used the default object, “LAN Primary Subnet”. If you have a whole slew of networks, or you want the wireless network to be reachable through the VPN, etc – I would suggest going through this once for one network. When you have success with one network, go back and add the others. Each end of this tunnel must be aware of the other ends networks!

For the Destination Network, we will need to create an object. So click the drop-down under “Choose a network from the list”, and then select “Create new address object”. You can name this whatever you want, but for the sake of other administrators – or yourself at a later date – use a sensible name (like the network ID itself).

Zone Assignment MUST be set to “VPN”! Next, choose “Network” as the type, and enter in the network ID information for the IPCop side of things.



Click “OK” when done, and you should have something like this …



Now, click the Proposals tab. We will have to make a few changes here to find a middle ground between our two different ends of this tunnel.

For Exchange, choose Main Mode. For DH Group, choose “Group 2”. Note that we also want Group 2 down below, but we can’t change that (and don’t need to) unless “Perfect Forward Secrecy” is enabled. For Encryption, go with “3DES” (aka Triple DES). Authentication, “MD5”. Could we go with better encryption? Probably. But SonicWALL tends to have a harder time working with other devices when using stronger protocols. If you want this to work well, stick to this path. For lifetime, go with the default. We will match this on the other side. For those keeping track, 28800 seconds translates to 8 hours.

When you are all done here, it should look like this …



On the Advanced tab, there is not much that will need changed here. I like to enable a “keep alive” on one end of the VPN tunnel. This will keep the tunnel up, even when no traffic is being passed through it. In most cases, you want that kind of functionality. The alternative is a tunnel that is built “on the fly” when users start trying to send traffic through it.

You may also want to enable NetBIOS. I often do for my customers. But if this was my network, I wouldn’t want Windows broadcasts going through my tunnel. This should be a routed network! Theres a good reason that broadcasts are not routable. Additionally, if you enable that here, you may be in trouble with how the IPCop handles it at the other end.

Leave this other options alone. You’re done here. Click “OK”.



The SonicWALL will go ahead and enable this policy for you. Since only one end is complete, you should disable it. Find the little checkbox under “Enable” and uncheck it.

Now, you are halfway there.

Step #2 Configure the IPCop side of the tunnel.
Connect to your IPCop’s web administration. This can probably only be done from the LAN side, unless you have enabled remote administration. Remote administration for an IPCop box is tricky, as it entails setting up SSH – and configuring an SSH tunnel at the remote end. Again, use Google or the Ipcops.com forums for help on that.

From the admin interface, click “VPN’s” and “VPN’s” as the only choice. If you are an OpenVPN user as well, you should be happy that these two seem to work together without any problems.

On the “Global Settings” screen, enter the IP address or hostname of this IPCop firewall if it’s not all read there. For the MTU, you will need to enter “1500”, or you will have serious problems. The SonicWALL defaults to 1500. Do not enable the VPN yet. Click “Save” toward the lower right of these settings. Smile.

Now scroll down a bit to “Connection status and control” and click the “Add” button lingering around the lower center.

We will be building a “Net to Net”, so choose that second option and then click “Add”.

For name, choose anything you want. But it must be one big word without spaces or special characters. For the “IPCop side” leave the default setting of “left” (this is an obscure reference to Open SWAN settings).

In the Remote Host box, enter the IP address of the SonicWALL. This is the “public address” which should be reachable from the Internet.

For “Local Subnet” enter the ID of this IPCop’s network. All ready in there? Good! Leave it alone. For the Remote Subnet, enter the SonicWALL’s network ID in the same fashion. Network ID first, then a forward slash, then the subnet mask (for us it was 192.168.199.0/255.255.255.0).

Leave “Dead peer detection” at its default, “restart”. If the VPN goes to hell in a hand-basket, it will drop and re-establish itself.

For “Options”, we will need to enter Local and Remote ID’s. As I mentioned earlier, these are textual. Normally, the IP address is used. But I had problems with that. My advice, use email addresses. You will need to put an “@” symbol in front of them.

So for Local ID, put in the address used as the remote “Peer ID” of the SonicWALL. Then for “Remote”, use the local “Peer ID” that you used on the SonicWALL. Be sure to put them in with @ symbols leading the address, but of course still plant the @ symbol in the address where it belongs (such as “@steve.ballantyne@gmail.com”).

The “Remark” is just a comment. Leave it blank.

For Authentication, we need to enter our Pre-Shared Key (the secret password). This should match what we used on our SonicWALL to a tee, or this won’t get off the runway. Before clicking “Save” at the bottom, scroll up and uncheck the “Enable” at the top. We aren’t ready to bring this up just yet! Now, click Save.

No errors? So far, so good. Scroll down to your new VPN under “Connection status and control” and find the edit button. It looks like a little pencil. Double check your settings for consistency.

Now scroll all the way to the bottom and click “Advanced”.

NOTE: In some areas, there will be two options set (such as Grouptype). That will only confuse the SonicWALL and make life miserable for you. De-select all but what we specify (if it’s darkened, or highlighted, it’s selected).

IKE Encryption: 3DES
IKE Integrity: MD5
IKE Grouptype: MODP-1024 (this equates to “Group 2”)
IKE Lifetime: 8 hours (you will need to change this)

ESP Encryption: 3DES
ESP Integrity: MD5
ESP Groutype: MODP-1024
ESP Keylife: 8 hours

Uncheck all other options at the bottom, including “PFS”.

It looks like the picture below … Click Save.



Before bringing up your tunnel, you should probably be ready to debug it. So SSH into the IPCop firewall, and run “tail –f /var/log/messages”. You will be watching your log running by. If it’s moving too fast, that indicates a busy network … which means your timing probably isn’t good for making network changes. ;-)

If possible, get the SonicWALL Administration web page open too and head into the Log area. Now we can be ready for any error messages that hit.

Enable the IPCop side first by checking “Enabled” and “VPN on Green” and clicking the Save button. You should see some stuff roll into your log. Let it settle down.

Now go to the SonicWALL, click VPN > Settings. Enable the VPN we created earlier. Watch the IPCop log now as it fills up with interesting stuff. This is not so easily read but should indicate a Phase 1, Phase 2 success followed by some confirmation messages.

Now, on the SonicWALL side, refresh your browser window. You are looking to see a “green light” on the VPN connection, as well as an active connection status displayed at the bottom. A green light does *not* indicate success. I have had many green lights that were actually crippled non-working tunnels.

On the SonicWALL, go into the Log and see what you have. You should see several messages, the last of which will be “SA Added” indicating success. No such luck? Start debugging.

The good news is that if it didn’t work, it’s probably just a mis-match in settings. The bad news is that there are a lot of things to look over and the error messages generally are not all that helpful. So check, double check, and re-check the settings at both ends. One small typo will blow the whole thing up.

To test your VPN you can (depending on your access rules) try pinging hosts. From the SonicWALL you can click System > Diagnostic and use the Ping utility. Do not try to ping the public addresses of the firewalls, and do not ping the private addresses of the firewalls themselves. The SonicWALL will not allow you to, and the IPCop will probably lie about where the reply is actually coming from. Rather, find a host on either end that will allow ICMP traffic and ping back and forth – and ping those.

Does it all work? Good. If you are have checked everything, and can’t seem to get things to work, feel free to contact me. Just know that you understand your own networks far better than I do, so a resolution may be difficult coming from someone outside.

Enjoy,

-Steve Ballantyne

Saturday, December 02, 2006

SonicWALL's Zapchast Virus Issues

Friday I got support ticket from one of our larger customers, Ohio State University. They were attempting to download and update one of their Symantec antivirus products but the SonicWALL was preventing the file transfer saying that it was infected with the "Zapchast virus". The support folks have access to their SonicWALL (while most of our customers do not). They knew that this was a false alarm and a mistake. So they went into the SonicWALL and disabled the antivirus from that zone. That ought to do it, right?

Wrong.

Here is one of my many complaints about SonicWALL products. When you identify something that isn't working correctly you should be able to easily turn it off, or make an exclusion. While SonicWALL's do give you the ability to exclude certain IP addresses from the antivirus feature, it doesn't work. That's right. You can add those addresses to the exclusion list and it will exclude them. Bit if you have one of these hum-dinger mistake signatures that feature simply "does not work". Next, I would normally recommend removing the antivirus feature from the network zone. That's Network > Zones, from the administration pages. Yet, remove it from the zone all you like. That won't work either.

The only sure fire way to stop the SonicWALL from blocking something like this is to disable the entire antivirus, anti-spyware, and intrusion prevention engines and then reboot the box. Obviously, that is not a viable solution for a University who is supporting thousands of simultaneous connections. Not to mention that this seems pretty unnecessary just to download a single file that you need.

I have had similar problems with intrusion prevention signatures. SonicWALL will write a new signature and put it out there, only to find that it's causing horrible problems and stopping all sorts of legitimate traffic. They will eventually get around to fixing or removing the signature but until an update is available you are in trouble. The difference is that you can disable an intrusion prevention signature. You can't disable a virus signature. You have to understand the logic in that. Intrusion detection is bound to have false positives. Someone may be trying to 'break in' to your network, but that could be a legitimate user trying to run some sort of cryptic report (which looks like an attack). Virus's on the other hand - are always virus's. It's not as if it "might be a virus, might not be".

I suppose the problem here, is that SonicWALL needs to admit that they make mistakes (often) and give us the ability to work around them. Until then we will continue to tell customers that "they are working on it" ... and we will hope that they actually ARE working on it.

Until next time,
-Steve Ballantyne

Tuesday, November 28, 2006

What's In a Name?

In the realm of Microsoft, it seems that "name resolution" is always the answer to the question, "why doesn't this !@#&*^ thing work?". Crossing over into a career in Network Security I have found that I will never escape this problem.

The past two days it seems that about sixty percent of my problems have been with customers who are expecting name resolution to "just work". Take for example one of my SonicWALL owners who has recently built a VPN tunnel between two offices. I am always quick to tell them that "if you cannot access a resource by name, try it by IP address". For some, accessing resources by IP address does not propose any problems. For others, this is quite an obstacle. It's getting to be that most of my calls revolve around name resolution problems of some sort. Therefore, today I would like to cover ...

Why Name Resolution Doesn't Work over my SonicWALL VPN Tunnel

Did you enable NetBIOS on the Security Association? - Remember that NetBIOS is a broadcast mechanism. It has no business being sent through a routed network. A VPN tunnel is a routed network. You should edit the VPN Policy, head into the "Advanced" tab and look for "NetBIOS Broadcasts". You need to check that option at *both* ends of the tunnel when possible.



Have you tried the IP Helper? - This is a goofy fix. At this point, I would question what you are trying to accomplish here. Remember that Microsoft has made several attempts over the years to banish "broadcast name resolution" from networks in favor of DNS. I don't want to get up on my soapbox and start yelling at you though. So you might try this - it fixes quite a bit.


Go into Network > IP Helper and first enable the helper, while selecting NetBIOS below it. You will need to "Apply" for that to be effective. Next, you will need to create TWO policies. One for each directional flow of traffic. That is, if you were doing this for a remote user VPN (WAN GroupVPN) make one that says "source: VPN DHCP Clients, dest: Firewalled Subnets", and then reverse it. Like this ...



Use Your Own DNS Server - If you have a DNS server at one end of this tunnel, consider using it for clients at the other end. What do I mean? Imagine "network A" has a DNS server which Active Directory uses. In fact, the Active Directory server is almost always also the DNS server for smaller businesses. That server should resolve the names of *ALL* your internal clients. It should also have "forwarders" configured so that when you ask it who http://www.google.com/ is, it will send that request to a DNS server on the Internet for an answer.

That being said, when users on the remote end of the VPN tunnel ("network B") are connecting, they should *also* use that DNS server. What you will find though, is that they are probably configured to use a public (Internet) DNS server when they connect. That's because you configured DNS on the SonicWALL for the WAN Interface. It then uses those DNS servers as the primary and secondary servers for it's DHCP scope. So, if this is a "site to site" tunnel, you will need to change the DHCP scope of that local SonicWALL having the name resolution issues. If you are using a remote user VPN (WAN GroupVPN) you need to change the DHCP scope of the SonicWALL that they are connecting to. You should *not* change the WAN Interfaces DNS server addresses. You don't need to!

You would do this: Click Network > DHCP Server. Then click the configure button next to the scope. Next click the DNS/WINS tab. You will notice that the SonicWALL defaults to "inherit DNS Settings". You will change it to "Specify Manually" and ensure that the first address is the DNS server on "network A" (the one with all the answers!). You might want to configure a second address of an Internet DNS server. One that will answer queries for Internet domains. That is more of a fail-safe in case the primary DNS server stops responding. It should *not* be used as an attempt to answer *all* Internet queries. Remember, your primary should be set up with DNS forwarders for that!



Use Your Domain Name - Quick, look at that illustration above. Is your domain name configured as part of your DHCP scope? It ought to be. Most folks think this is optional. But if you want your DNS server to complete your queries, this needs to be set to your name. That is, I might ask your DNS server who "SERVER1" is, and it doesn't know. But if I ask it who "SERVER1.mydomainname.com" is, I get an answer. Add your domain name to your scope, and you will fix a lot of complaints from users. If SonicWALL is not performing DHCP, and you cannot fix this on your own - consider fixing this problem on your local network adapter (or virtual VPN adapter). Basically, you drill down to IP Properties, then "Advanced", then DNS, then add a "domain name suffix" in that box near the bottom. Case closed.

Still Using WIN's? - You ought to be ashamed! But if you are using WIN's, you probably can solve your name resolution problems by adding WIN's servers to your DHCP scope. If SonicWALL is performing DHCP for you, refer to the previous illustration to see how that would be done. If you are not using WIN's, I congratulate you. WIN's was designed as a way to manage(?) NetBIOS name resolution. Most of us use DNS now (or Dynamic DNS) and we are much better for it.

If none of that worked, and you are still having name resolution problems on your SonicWALL VPN, you should send me a comment or an email. Yes, I will help you. I'm that kind of guy.

-Steve Ballantyne

Saturday, November 25, 2006

Downloaded Files Disappear

My brother in law asked me a few weeks ago about a problem he was having. He would download files from the Internet - and they would "vanish". No sooner did the download finish, than the window would disappear and there would be no file to show for it. What really bothered me is that I have had that problem in the past and I couldn't for the life of me remember what I did to fix it.

Today I was there for Thanksgiving, and I noticed a few things. First, this was a "kids" computer. Kids do strange things with PC's that adults would never think of. Second, it had a lot of anti-spyware tools (maybe a few too many). Lastly, the copy of Symantec Antivirus that it had was a tad out of date.

So I went to work first removing as much as I could. First I tried to empty out the temporary Internet files ... which oddly enough seemed to hang and never complete. Could this be a problem from installing a new browser (Internet Explorer 7)? My brother in law says "this problem started before then, and I hoped it would go away with the upgrade". I agreed.

For my next trick I headed into Add/Remove Programs in Control Panel, and started removing. Anything that he and I couldn't identify, was removed. Then I took out Microsoft Defender, Spybot Search and Destroy, and Ad-Aware. All of them are good programs, but none of them were up to date (thanks to the inability to download anything). Lastly, I updated his Symantec Antivirus with the latest stuff - which actually took several reboots to accomplish.

After all the fiddling and removing of stuff - his browser worked once again. What did we remove that fixed it? My guess is that this was caused by Spybot Search and Destroy's Tea-Timer plug-in which is designed to protect your browser from outside threats. It's never been considered all that stable, and may cause more problems than it solves. But the idea of a browser protector seemed pretty good when it debuted. At any rate, it was likely protecting the browser in a way that made downloading or installing anything through the browser relatively impossible.

And some would probably say "why didn't you just install Firefox?". Let's say that your sink backs up and water starts shooting out onto the kitchen floor. Are you going to start washing dishes in the bathroom? Internet Explorer is a Windows component these days. If it's having a breakdown of some sort, it should be fixed. While it did take a good couple hours of off and on attention, it was good to see that I fixed it. The alternative was to of course reinstall Windows - and hundreds of kids games.

I'd say we made out.

-Steve Ballantyne

The Vow of Vina

Have you ever heard of a Vina eLink device? I sure hadn't. We had this new customer who had been using one of these little devices for several years with his T1 (fiber) connection through Nuvox. Of course, this was a Time Warner customer ... which kind of made me wonder why I had never heard of one of these little things.

Heck, I had to go out Googling to find anything. What I found was a lot of forums where people were asking each other where to find documentation (a bad sign) and a couple redirectors to websites that no longer exist. This is when http://web.archive.org/ comes in very handy. I pull up this Vina companies web site from a year or so ago (when it still was "live") and it seems that they have been acquired. Good news, actually. Next I go out looking for the company that bought them. Sadly, they too have closed their doors forever and accepted financial defeat.

The best information I could find on this device ended up being from eBay, where kind folks were trying to unload these little devices, and make them sound appealing. I was at least able to get a photo of this device that I would be supporting from afar.

It looks like a very cheap cable modem ... or something. At any rate, this company had one and wanted to keep using it. Normally, I would tell them to get that peice of junk out of my way so that we could put our SonicWALL where it belongs. After talking to the Time Warner sales contact for this account, I learned that they were going to be a Nuvox customer a little longer than intended. In other words, Time Warner's installation was delayed for a few months while they cut through all the red tape trying to lay fiber conduits. But the SonicWALL had to be installed to support the VPN this guy wanted.

One might wonder why I volunteered to work on "Black Friday", the biggest shopping day of the year. There were a couple of perks. One being, that I got to choose any other day I wanted and have that day off in trade. The second being that it was a holiday for most of the free world ... and I didn't expect many phone calls. I was half right. The entire day I got maybe five phone calls. It just happens that one of those calls was to make this stupid little router work.

The customer called, and we bagan to play with various settings. I would ask him what he saw in the way of physical connections and telnet dialogue with the device, and he would try to relay that information back to me. We got nowhere fast. After an hour or so, he asked "would you like to connect and try some things?". My first response was "I have zero knowledge in configuring these devices", and my second response was "what's the password?". So in I went.

It kind of ran like a Cisco router. That is, the CLI was quite similar. That is, you had all sorts of settings, within settings, withing settings ... you get the idea. The organization was "hierarchical" and horrible to trip through. I also found that the engineers made an attempt to shorten all of the settings to one or two hyphenated words. A setting of "ip" made you wonder just what were you setting the IP to, exactly?

I also found that the customer knew far more about this device, and how he had configured it. What are these "groups" I see here? He answers, "grouped settings". I ask, "you mean like VLANS ... or trunks?". He answers "yes", indicating that he clearly has no idea himself. From the appearance, it looked like he had a combination of addresses that were being NAT'd from public addresses to private ones ... but also some addresses that were being passed clean through. Which led me to believe that his mail server might be publically accessible, without any firewall whatsoever. I knew at this point, that I would never be able to configure this device without some sort of assistance. "Let me call you back", I said.

My next phone call was to Nuvox. If you have not dealt with Nuvox, let me say a few polite words about them. Whenever I have asked them for documentation about a customer such as "what IP block do they have, and what would their gateway be" I am always given a very complete list of everything I need to know - and it's always been e-mailed quickly. When I have had to call them, I always get a live person relatively easily ... and they have never been unhelpful "screen readers". I prayed for the best and dialed the number.

I got a guy named "Jimmy" who was about to make my day. Jimmy had full knowledge of these devices, and he was happy to share that with me. It probably also made Jimmy feel a little better when I assured him that while I knew nothing about this little device, I wasn't a complete moron. He asked very simply "what is it that you want this device to do for you?". My response was "can we just pass everything through it, and put it in a transparent mode?". His answer was a resounding "absolutely". Now here is what sets the support apart from most people you are familiar with having to deal with. When Jimmy asked whose device this was, I had to tell the truth. "They bought it man ... but if it helps, they bought it from you". I went on to tell him that I know this is not his problem, but that didn't seem to matter to him. "I can't configure the device for you - but I can tell you exactly what you need to do". Thank you Jimmy.

Here is what we did. Now, pay attention - because what we are going to do here is to put a Vina router in "passthrough mode". That means that it's no longer going to perform NAT to private addresses any more. Rather, my SonicWALL which will be plugged into it - is going to have a public address. Now, on with the show.

We start by logging in, which meant a telnet session to it's public IP address ...


telnet x.x.x.x


Vina Technologies eLink 216 (5.1.1 build 12)
Built: Jun 3 2002, 18:25:46
NVRAM version 0117
PB $Id: romInit.s,v 1.7 1997/09/26 23:43:22 bob Exp $

Copyright 1996 - 2002 VINA Technologies, Inc.
Please enter name: your name here


Enter password: your password here


After putting in the credentials, you are at the "top" of a horrible hierarchy. Much like you would with a Cisco, you are going to configure this guy. So type "config" and [enter].

Next, we are at the "config" prompt, and we want to move into the Ethernet settings. So type "ether" for short, then [enter].

Once at the Ether prompt, you will see that I did a "?" [enter] to get a view of what was in the config now. Jimmy told me that I should probably do this (if for no other reason) so that I could save this session and use it to undo any actions that might screw things up. Good advice Jimmy. I also decided it would make good Blog material.


logged on as Carrier


>config

(config)>eth

(config:Ethernet)>?
IP-address : 192.168.100.1 (IP-address)
netmask : 255.255.255.0 (IP-address)
secondary-ip : 192.168.100.3 (IP-address)
sec-netmask : 255.255.255.0 (IP-address)
RIP : Disable (EnableRxOnlyTxOnly[Disable])
version-RIP : 1 (12)
Link-integrity-test : on (off [on])
Help
!

See that? This thing is running with a public IP address. That's because it's performing NAT for us know. But that will become my SonicWALL's job. Therefore, we are going to give it the public IP address that I have set aside for it. This router may also have a "secondary" IP address. I'm not sure why I would want that, or how it would even be helpful. But we are going to wipe it out, by replacing it with an address of "0.0.0.0". Note that the last two octets of the public IP address have been blocked out (with x's) to "protect the innocent". ;-)

(config:Ethernet)>ip-address 64.119.x.x
(config:Ethernet)>netmask 255.255.255.248
(config:Ethernet)>??
IP-address : 64.19.x.x (IP-address)
netmask : 255.255.255.248 (IP-address)
secondary-ip : 192.168.100.3 (IP-address)
sec-netmask : 255.255.255.0 (IP-address)
RIP : Disable (EnableRxOnlyTxOnly[Disable])
version-RIP : 1 (12)
Link-integrity-test : on (off [on])
Help
!


(config:Ethernet)>
(config:Ethernet)>secondary-ip 0.0.0.0

(config:Ethernet)>?
IP-address : 64.19.x.x (IP-address)
netmask : 255.255.255.248 (IP-address)
secondary-ip : 0.0.0.0 (IP-address)
sec-netmask : 255.255.255.0 (IP-address)
RIP : Disable (EnableRxOnlyTxOnly[Disable])
version-RIP : 1 (12)
Link-integrity-test : on (off [on])
Help
!

That last "?" was just to see it, get a good look, and to double check my work. Next, we need to go kill off NAT. We don't want it doing that any more. So first, we will type "exit" and [enter] once to get back to the main prompt. Then we will enter the "nat" config with the command "nat". Finally, we will disable it with a rather cryptic "enable off" command.

(config:Ethernet)>exit
(config)>nat

(config:NAT)>?
enable : on ([off] on)Dynamic ...
Static ...
PassThru ...
Show-sessions alludptcpicmpconfig
Clear-sessions alludptcpicmp
Help
!

(config:NAT)>enable off


(config:NAT)>exit

If at any time you might worry that we are going to disconnect ourselves, you may rest assured that these changes aren't going to go into effect until we apply them at the very end. If at any time we get cold feet, we could exit the config or power cycle this device and nothing would be changed.

Now, we also need to disable DHCP. It wouldn't make much sense for this device to continue handing out private addresses, when it will no longer perform NAT to those addresses.

(config)>dhcp

(config:DHCP)>?

enable : off ([off] on)
start-ip : 192.168.0.2 (IP-address)
end-ip : 192.168.0.254 (IP-address)
private-network : on (off [on])
lease-time : 600 (seconds([600]:600...7200))
dns-server : 192.168.0.1 (IP-address)
domain : (name-string)
Help
!

(config:DHCP)>enable off

(config:DHCP)>exit

Now we are almost done. We have one last step, and that's to put this thing in it's "transparent" mode. In doing so it will become a passive device. Yet, here is where I fouled up something that I later had to go back and fix. If you had enabled "passthru" on this device for any particular "group" that you configured, you will need to go disable that now. I didn't record this part of the configuration, as most people would have never attempted that. In fact, I am fairly sure that the customer hadn't meant to configure it that way, and Jimmy at Nuvox didn't catch it.

And now, the exciting conclusion ...

(config)>syn

(config:Synchronous-interface)>mod

(config:Frame-relay)>pvc 1

(config:FrameRelay:PVC1)>show

DLCI : 100 (dlci (16...991))
IP-address : 64.19.x.x (IP-addressEnet[Disable])
netmask : 255.255.255.248 (IP-address)
RIP : Disable (EnableRxOnlyTxOnly[Disable])
version-RIP : 1 ([1]2)
ENAT : OUT (PASSIN[OUT])
Help
!

(config:FrameRelay:PVC1)>enat pass

(config:FrameRelay:PVC1)>exit

(config:Frame-Relay)>exit

Brace yourself.

You are about to embark on a mission that you may not return from. You are about to save this configuration. I will warn you, there is no turning back (dum-dum-duuum!). I also should warn you that it's going to say that it will "interrupt your Data traffic" and then *hang*. And when I say it hangs, I mean that you wille get no response for a good minute or two. Do not panic. It will come back to you. Jimmy also assured me that he has never lost communication at this point, even when he was sure he had screwed things up.


(config)>save
Verifying system configuration...
Do you really want to update the system NVRAM configuration?
This may cause a temporary interruption of Data traffic [n]y: y
Updating Flash NVRAM,... wait
Updating Protected Boot Flash NVRAM... wait
DONE

That was it. The device rebooted itself, I thanked Jimmy for all of his help, and I assured him that if in fact we broke this configuration, I would never blame him for it. I also wished him a happy holiday, and leant him a bit of pity that I shared - as we had to be the only suckers that came into work on this would be day off.

After a few moments of silence, and prayer ... the device was back up. I had been running a continous ping to the device at it's newly assigned public IP address.


ping -t 64.19.x.x

At first I was getting "TTL expired in transit" messages, which indicated an upstream router was dumping off the packets because it couldn't find an appropriate place to send them. Once those turned into "Reply" messages, I was overjoyed.

Yet, I was still unable to ping my SonicWALL. I know I had configured it with the correct address. But my ping response was coming back "no response". It looked as if my packets were getting to the SonicWALL, but there was nothing coming back. Ah! The gateway. In my haste to get the device configured, I had never changed the SonicWALL's WAN interface gateway.

Note to self: next time, change the SonicWALL first - before losing connection to it forever.

Now I had to bite the bullet and call the customer. "We are so close to having this work", I assured, "but I am going to need your help". While the customer was not all that technically "with it" (he was the first to say so), he was pretty good at following instructions. With that, I walked him through connecting to the SonicWALL and changing the WAN interface gateway IP. What is the SonicWALL's WAN gateway? Why, it's that *same* address that we gave to the Vina eLink.

Now, to see if the SonicWALL was even connected to the Vina and "speaking to it", I ran another telnet session back into the device to check it's ARP cache.



telnet 64.19.x.x


Vina Technologies eLink 216 (5.1.1 build 12)
Built: Jun 3 2002, 18:25:46
NVRAM version 0117
PB $Id: romInit.s,v 1.7 1997/09/26 23:43:22 bob Exp $

Copyright 1996 - 2002 VINA Technologies, Inc.
Please enter name: your name here
Enter password: your password here

logged on as Carrier

(config)>arp

(config:ARP)>show
64.19.x.x at 0:6:b1:x:x:x


(config:ARP)>


(config)>exit



Good, good. What that told me, is that the SonicWALL is communicating with the Vina, and the customer must have plugged it in correctly (he actually hadn't plugged it in right, but I am omitting the 30 minutes or so that it took me to figure that out).

Now, I could once and for all connect to my SonicWALL and configure the device the rest of the way. One server at a time, I created the needed NAT Policies and Access Rules. Everything from here on was clockwork.

Lessons learned?

1) Don't believe the customer when he says he plugged it in right. Even after he says he has double checked it too. I have found that it helps to ask the customer where things go, and make them tell you what is connected to what devices. In a polite way, suggest what you all ready know - "well it just seems that the blue cable is connected to the red port ... is that possible?".

2) Don't assume that the ISP's telephone support will be worthless. While most companies send those calls overseas to a foreign "screen reader", this company took me straight to the resolution without tripping through a maze of prompts. Had I called them to begin with, I probably would have cut a few hours out of this lengthy process.

3) Remember to make your changes to devices, that you will ultimately loose access to - while you have the chance. Leading the customer through changing things for you is tedious, frustrating, and time consuming.

I might add a final step here, if you are someone in the sales Field. The motto is not "over-promise and under-deliver". You had better not agree to hooking up your equipment in an environment that you are not familiar with yourself. Not all devices are created equal. What works and is supported now, may not be once everything is connected. In this particular situation, the customer hadn't lost interest in his new service thanks to the effort we put forth. I should add here, that we started trying to get everything to work nearly one month earlier.

There is a another good story that follows this one. I will make the long story short.

The customer returned home, and connected his cable modem, and then his SonicWALL behind it which had both been sitting in a box for a couple of weeks. Then he called me. He was anxious to get the VPN tunnel built between work and home so that he could access everything at the office, from his home office. I walked him through hooking everything up, and I made my first attempts to connect to the SonicWALL by it's registered Dynamic DNS host name. The SonicWALL, as planned, had made contact and updated it's DNS record with it's new public address. What was that public address? 192.168.0.13. Yes, it had been loaned a private address from none other than the cable modem (router) in front of it. There was no possible way that I was going to be able to fix this one. Time Warner doesn't allow us access to their cable modems, and we needed to do the same thing to that modem that we worked on all day with his Vina.

I let him down easy. "I have a technician to murder Monday morning ... in the mean time this is not going to work". He was content with that knowing that I had just put four hours or more into this process and I was just as disappointed that we had hit a stone wall as he was. Again, he thanked me for the work, I thanked him for his patience, and we ended the day.

Happy Thanksgiving,
-Steve Ballantyne

Friday, November 24, 2006

Professional Blogging

For several years now I have run a Blog. Now, you may say "but Steve, where is this Blog that you speak of?” It's out there in cyberspace somewhere, publicly hidden. How do you hide something publicly? That's easy. You post under an alter ego. For many years I have entertained, impressed, and sometimes depressed my peers with the daily happenings of my life. Meanwhile, my professional career has never clearly been documented.

There have been countless occasions where I need to remember how I solved a particular issue - only to find that I never scrawled it down. Never more!

Additionally, this Blog may just be an interesting account of the stuff that Steve wastes his time on while under willful employment.

Enjoy.

-Steve Ballantyne