Saturday, November 25, 2006

The Vow of Vina

Have you ever heard of a Vina eLink device? I sure hadn't. We had this new customer who had been using one of these little devices for several years with his T1 (fiber) connection through Nuvox. Of course, this was a Time Warner customer ... which kind of made me wonder why I had never heard of one of these little things.

Heck, I had to go out Googling to find anything. What I found was a lot of forums where people were asking each other where to find documentation (a bad sign) and a couple redirectors to websites that no longer exist. This is when http://web.archive.org/ comes in very handy. I pull up this Vina companies web site from a year or so ago (when it still was "live") and it seems that they have been acquired. Good news, actually. Next I go out looking for the company that bought them. Sadly, they too have closed their doors forever and accepted financial defeat.

The best information I could find on this device ended up being from eBay, where kind folks were trying to unload these little devices, and make them sound appealing. I was at least able to get a photo of this device that I would be supporting from afar.

It looks like a very cheap cable modem ... or something. At any rate, this company had one and wanted to keep using it. Normally, I would tell them to get that peice of junk out of my way so that we could put our SonicWALL where it belongs. After talking to the Time Warner sales contact for this account, I learned that they were going to be a Nuvox customer a little longer than intended. In other words, Time Warner's installation was delayed for a few months while they cut through all the red tape trying to lay fiber conduits. But the SonicWALL had to be installed to support the VPN this guy wanted.

One might wonder why I volunteered to work on "Black Friday", the biggest shopping day of the year. There were a couple of perks. One being, that I got to choose any other day I wanted and have that day off in trade. The second being that it was a holiday for most of the free world ... and I didn't expect many phone calls. I was half right. The entire day I got maybe five phone calls. It just happens that one of those calls was to make this stupid little router work.

The customer called, and we bagan to play with various settings. I would ask him what he saw in the way of physical connections and telnet dialogue with the device, and he would try to relay that information back to me. We got nowhere fast. After an hour or so, he asked "would you like to connect and try some things?". My first response was "I have zero knowledge in configuring these devices", and my second response was "what's the password?". So in I went.

It kind of ran like a Cisco router. That is, the CLI was quite similar. That is, you had all sorts of settings, within settings, withing settings ... you get the idea. The organization was "hierarchical" and horrible to trip through. I also found that the engineers made an attempt to shorten all of the settings to one or two hyphenated words. A setting of "ip" made you wonder just what were you setting the IP to, exactly?

I also found that the customer knew far more about this device, and how he had configured it. What are these "groups" I see here? He answers, "grouped settings". I ask, "you mean like VLANS ... or trunks?". He answers "yes", indicating that he clearly has no idea himself. From the appearance, it looked like he had a combination of addresses that were being NAT'd from public addresses to private ones ... but also some addresses that were being passed clean through. Which led me to believe that his mail server might be publically accessible, without any firewall whatsoever. I knew at this point, that I would never be able to configure this device without some sort of assistance. "Let me call you back", I said.

My next phone call was to Nuvox. If you have not dealt with Nuvox, let me say a few polite words about them. Whenever I have asked them for documentation about a customer such as "what IP block do they have, and what would their gateway be" I am always given a very complete list of everything I need to know - and it's always been e-mailed quickly. When I have had to call them, I always get a live person relatively easily ... and they have never been unhelpful "screen readers". I prayed for the best and dialed the number.

I got a guy named "Jimmy" who was about to make my day. Jimmy had full knowledge of these devices, and he was happy to share that with me. It probably also made Jimmy feel a little better when I assured him that while I knew nothing about this little device, I wasn't a complete moron. He asked very simply "what is it that you want this device to do for you?". My response was "can we just pass everything through it, and put it in a transparent mode?". His answer was a resounding "absolutely". Now here is what sets the support apart from most people you are familiar with having to deal with. When Jimmy asked whose device this was, I had to tell the truth. "They bought it man ... but if it helps, they bought it from you". I went on to tell him that I know this is not his problem, but that didn't seem to matter to him. "I can't configure the device for you - but I can tell you exactly what you need to do". Thank you Jimmy.

Here is what we did. Now, pay attention - because what we are going to do here is to put a Vina router in "passthrough mode". That means that it's no longer going to perform NAT to private addresses any more. Rather, my SonicWALL which will be plugged into it - is going to have a public address. Now, on with the show.

We start by logging in, which meant a telnet session to it's public IP address ...


telnet x.x.x.x


Vina Technologies eLink 216 (5.1.1 build 12)
Built: Jun 3 2002, 18:25:46
NVRAM version 0117
PB $Id: romInit.s,v 1.7 1997/09/26 23:43:22 bob Exp $

Copyright 1996 - 2002 VINA Technologies, Inc.
Please enter name: your name here


Enter password: your password here


After putting in the credentials, you are at the "top" of a horrible hierarchy. Much like you would with a Cisco, you are going to configure this guy. So type "config" and [enter].

Next, we are at the "config" prompt, and we want to move into the Ethernet settings. So type "ether" for short, then [enter].

Once at the Ether prompt, you will see that I did a "?" [enter] to get a view of what was in the config now. Jimmy told me that I should probably do this (if for no other reason) so that I could save this session and use it to undo any actions that might screw things up. Good advice Jimmy. I also decided it would make good Blog material.


logged on as Carrier


>config

(config)>eth

(config:Ethernet)>?
IP-address : 192.168.100.1 (IP-address)
netmask : 255.255.255.0 (IP-address)
secondary-ip : 192.168.100.3 (IP-address)
sec-netmask : 255.255.255.0 (IP-address)
RIP : Disable (EnableRxOnlyTxOnly[Disable])
version-RIP : 1 (12)
Link-integrity-test : on (off [on])
Help
!

See that? This thing is running with a public IP address. That's because it's performing NAT for us know. But that will become my SonicWALL's job. Therefore, we are going to give it the public IP address that I have set aside for it. This router may also have a "secondary" IP address. I'm not sure why I would want that, or how it would even be helpful. But we are going to wipe it out, by replacing it with an address of "0.0.0.0". Note that the last two octets of the public IP address have been blocked out (with x's) to "protect the innocent". ;-)

(config:Ethernet)>ip-address 64.119.x.x
(config:Ethernet)>netmask 255.255.255.248
(config:Ethernet)>??
IP-address : 64.19.x.x (IP-address)
netmask : 255.255.255.248 (IP-address)
secondary-ip : 192.168.100.3 (IP-address)
sec-netmask : 255.255.255.0 (IP-address)
RIP : Disable (EnableRxOnlyTxOnly[Disable])
version-RIP : 1 (12)
Link-integrity-test : on (off [on])
Help
!


(config:Ethernet)>
(config:Ethernet)>secondary-ip 0.0.0.0

(config:Ethernet)>?
IP-address : 64.19.x.x (IP-address)
netmask : 255.255.255.248 (IP-address)
secondary-ip : 0.0.0.0 (IP-address)
sec-netmask : 255.255.255.0 (IP-address)
RIP : Disable (EnableRxOnlyTxOnly[Disable])
version-RIP : 1 (12)
Link-integrity-test : on (off [on])
Help
!

That last "?" was just to see it, get a good look, and to double check my work. Next, we need to go kill off NAT. We don't want it doing that any more. So first, we will type "exit" and [enter] once to get back to the main prompt. Then we will enter the "nat" config with the command "nat". Finally, we will disable it with a rather cryptic "enable off" command.

(config:Ethernet)>exit
(config)>nat

(config:NAT)>?
enable : on ([off] on)Dynamic ...
Static ...
PassThru ...
Show-sessions alludptcpicmpconfig
Clear-sessions alludptcpicmp
Help
!

(config:NAT)>enable off


(config:NAT)>exit

If at any time you might worry that we are going to disconnect ourselves, you may rest assured that these changes aren't going to go into effect until we apply them at the very end. If at any time we get cold feet, we could exit the config or power cycle this device and nothing would be changed.

Now, we also need to disable DHCP. It wouldn't make much sense for this device to continue handing out private addresses, when it will no longer perform NAT to those addresses.

(config)>dhcp

(config:DHCP)>?

enable : off ([off] on)
start-ip : 192.168.0.2 (IP-address)
end-ip : 192.168.0.254 (IP-address)
private-network : on (off [on])
lease-time : 600 (seconds([600]:600...7200))
dns-server : 192.168.0.1 (IP-address)
domain : (name-string)
Help
!

(config:DHCP)>enable off

(config:DHCP)>exit

Now we are almost done. We have one last step, and that's to put this thing in it's "transparent" mode. In doing so it will become a passive device. Yet, here is where I fouled up something that I later had to go back and fix. If you had enabled "passthru" on this device for any particular "group" that you configured, you will need to go disable that now. I didn't record this part of the configuration, as most people would have never attempted that. In fact, I am fairly sure that the customer hadn't meant to configure it that way, and Jimmy at Nuvox didn't catch it.

And now, the exciting conclusion ...

(config)>syn

(config:Synchronous-interface)>mod

(config:Frame-relay)>pvc 1

(config:FrameRelay:PVC1)>show

DLCI : 100 (dlci (16...991))
IP-address : 64.19.x.x (IP-addressEnet[Disable])
netmask : 255.255.255.248 (IP-address)
RIP : Disable (EnableRxOnlyTxOnly[Disable])
version-RIP : 1 ([1]2)
ENAT : OUT (PASSIN[OUT])
Help
!

(config:FrameRelay:PVC1)>enat pass

(config:FrameRelay:PVC1)>exit

(config:Frame-Relay)>exit

Brace yourself.

You are about to embark on a mission that you may not return from. You are about to save this configuration. I will warn you, there is no turning back (dum-dum-duuum!). I also should warn you that it's going to say that it will "interrupt your Data traffic" and then *hang*. And when I say it hangs, I mean that you wille get no response for a good minute or two. Do not panic. It will come back to you. Jimmy also assured me that he has never lost communication at this point, even when he was sure he had screwed things up.


(config)>save
Verifying system configuration...
Do you really want to update the system NVRAM configuration?
This may cause a temporary interruption of Data traffic [n]y: y
Updating Flash NVRAM,... wait
Updating Protected Boot Flash NVRAM... wait
DONE

That was it. The device rebooted itself, I thanked Jimmy for all of his help, and I assured him that if in fact we broke this configuration, I would never blame him for it. I also wished him a happy holiday, and leant him a bit of pity that I shared - as we had to be the only suckers that came into work on this would be day off.

After a few moments of silence, and prayer ... the device was back up. I had been running a continous ping to the device at it's newly assigned public IP address.


ping -t 64.19.x.x

At first I was getting "TTL expired in transit" messages, which indicated an upstream router was dumping off the packets because it couldn't find an appropriate place to send them. Once those turned into "Reply" messages, I was overjoyed.

Yet, I was still unable to ping my SonicWALL. I know I had configured it with the correct address. But my ping response was coming back "no response". It looked as if my packets were getting to the SonicWALL, but there was nothing coming back. Ah! The gateway. In my haste to get the device configured, I had never changed the SonicWALL's WAN interface gateway.

Note to self: next time, change the SonicWALL first - before losing connection to it forever.

Now I had to bite the bullet and call the customer. "We are so close to having this work", I assured, "but I am going to need your help". While the customer was not all that technically "with it" (he was the first to say so), he was pretty good at following instructions. With that, I walked him through connecting to the SonicWALL and changing the WAN interface gateway IP. What is the SonicWALL's WAN gateway? Why, it's that *same* address that we gave to the Vina eLink.

Now, to see if the SonicWALL was even connected to the Vina and "speaking to it", I ran another telnet session back into the device to check it's ARP cache.



telnet 64.19.x.x


Vina Technologies eLink 216 (5.1.1 build 12)
Built: Jun 3 2002, 18:25:46
NVRAM version 0117
PB $Id: romInit.s,v 1.7 1997/09/26 23:43:22 bob Exp $

Copyright 1996 - 2002 VINA Technologies, Inc.
Please enter name: your name here
Enter password: your password here

logged on as Carrier

(config)>arp

(config:ARP)>show
64.19.x.x at 0:6:b1:x:x:x


(config:ARP)>


(config)>exit



Good, good. What that told me, is that the SonicWALL is communicating with the Vina, and the customer must have plugged it in correctly (he actually hadn't plugged it in right, but I am omitting the 30 minutes or so that it took me to figure that out).

Now, I could once and for all connect to my SonicWALL and configure the device the rest of the way. One server at a time, I created the needed NAT Policies and Access Rules. Everything from here on was clockwork.

Lessons learned?

1) Don't believe the customer when he says he plugged it in right. Even after he says he has double checked it too. I have found that it helps to ask the customer where things go, and make them tell you what is connected to what devices. In a polite way, suggest what you all ready know - "well it just seems that the blue cable is connected to the red port ... is that possible?".

2) Don't assume that the ISP's telephone support will be worthless. While most companies send those calls overseas to a foreign "screen reader", this company took me straight to the resolution without tripping through a maze of prompts. Had I called them to begin with, I probably would have cut a few hours out of this lengthy process.

3) Remember to make your changes to devices, that you will ultimately loose access to - while you have the chance. Leading the customer through changing things for you is tedious, frustrating, and time consuming.

I might add a final step here, if you are someone in the sales Field. The motto is not "over-promise and under-deliver". You had better not agree to hooking up your equipment in an environment that you are not familiar with yourself. Not all devices are created equal. What works and is supported now, may not be once everything is connected. In this particular situation, the customer hadn't lost interest in his new service thanks to the effort we put forth. I should add here, that we started trying to get everything to work nearly one month earlier.

There is a another good story that follows this one. I will make the long story short.

The customer returned home, and connected his cable modem, and then his SonicWALL behind it which had both been sitting in a box for a couple of weeks. Then he called me. He was anxious to get the VPN tunnel built between work and home so that he could access everything at the office, from his home office. I walked him through hooking everything up, and I made my first attempts to connect to the SonicWALL by it's registered Dynamic DNS host name. The SonicWALL, as planned, had made contact and updated it's DNS record with it's new public address. What was that public address? 192.168.0.13. Yes, it had been loaned a private address from none other than the cable modem (router) in front of it. There was no possible way that I was going to be able to fix this one. Time Warner doesn't allow us access to their cable modems, and we needed to do the same thing to that modem that we worked on all day with his Vina.

I let him down easy. "I have a technician to murder Monday morning ... in the mean time this is not going to work". He was content with that knowing that I had just put four hours or more into this process and I was just as disappointed that we had hit a stone wall as he was. Again, he thanked me for the work, I thanked him for his patience, and we ended the day.

Happy Thanksgiving,
-Steve Ballantyne