A Good Old Fashioned Rant

Call comes in.  The support guy answers and the conversation goes something like this:

Tech: Hi, I'm upgrading my software today.
Support: Sure thing.  What's the problem?
Tech: Well, I need the software.
Support: ...

Call me "Old Fashioned" or perhaps years of tech support have blinded me to the realities of "Life in the Real World" but my check list would have been something like this:

Step 1: Download software

Especially if it is some old as crap stuff that hasn't been in use for ten years or so.  Why is the name of Cthulhu would you wait until the DAY OF YOUR UPGRADE to try to find the software?  So on the day of your upgrade (which happens to be a Sunday afternoon) you decide, "Well.  Might as well go dig up some software for the upgrade.  I guess once I've got that and FUBAR'd the upgrade destroying the database and server I can also read the instructions on the process to upgrade."

Honestly, this lets you know exactly how successful his upgrade will be.

The next time you sit in a queue for an hour or so, keep in mind the questions being asked.  Explaining to people that just because the feature isn't working the way they think it should work, the display doesn't look the they think it should and I don't know how to do this and I'm paying you money so why should I bother reading the documentation so just do it for me, are all likely ahead of your legitimate question.

The Art Of Troubleshooting

I've worked tech support for a long time.  Too long actually.  I've watched people come and go, new ideas get implemented and then tanked, old ideas rehashed as new ideas and then tanked.  The one thing that is constant is the methodology of troubleshooting.

I'd like to say it's an art.  It is a skill but I don't know if it can actually be taught.  I've seen a lot of people that can pass all these wonderful tests that crap themselves when a down system call comes in.

Case in point: Down system with a HIGHLY AGITATED customer.  His boss is yelling at him, the users are yelling at him.  Everyone wants the system back online but no one seems to know how to get to that state.

Phones are offline but powered up.

The server is running.

The Phone switch is running.

The IP Network is working for everyone.

Tier 1 gets the call and the customer explodes on them.  They want to troubleshoot as they have been down for over an hour and need this fixed immediately.

Troubleshooting has to begin with the flow of data.  Data is life.

Approach the problem from the most likely to the least likely point of failure.  The phones are down and are all in the same physical building.  They span across multiple phone switches, but the same phone switches that have some non-working phones have working phones.

The customer reset phones and the phone switches and things didn't fix themselves so he called TAC (which is the right thing to do when you are stuck).  The only thing I can fault him on was panicking when things went South.

Panic. Never. Helps.

Follow the Hitchhiker's Guide to the Galaxy's rule: "Don't Panic".

In this case we had to look from Most Likely to Least Likely.  It is HIGHLY unlikely that 50 phones all suddenly go bad at exactly the same time with exactly the same symptoms.  Think from the phone out.

Phones are network devices first and foremost.

Does the phone have power? Yes
Does the phone boot up? Yes
Can the phone get a DHCP address? No
Does it come up with a cached IP address? Yes
Can you ping the phone's IP address from the server? No
Can you ping the phone's IP address from the phone switch? No
Can you ping the server from the phone? No
Can you ping the phone switch from the phone? No

From these quick tests we've determined the phone is getting power from the network POE switch (so likely the network cable between the phone and network switch is OK).  The phone boots up which indicates internally the phone is OK (no hardware/firmware failure).  The phone DOES NOT get an IP address (potential network problem).  The phone does indicate it's using its cached DHCP address (so the network used to work) but you cannot ping it from the phone equipment it has to connect to in order to operate.  You also cannot ping the phone equipment from the phone sooooo... SURVEY SAYS????

*ding*           IP Network Problem!

Rebooting the network switch the phones connected to corrected the problem and the phones logged in successfully.  Total time to identify the point of failure was around 10 minutes.

I'm not bashing the customer AT ALL.  I just want to use this to give an example of how important clear thinking, identification of the problem and an understanding of data flow is to proper troubleshooting.

Laugh, Cry, Whatever

I'm on site this week.  I have a customer who is experiencing a problem with their PRI provider.  They have an existing span which fails when they move it to our equipment.  It's actually a very interesting failure as this absolutely SHOULD work.

So why am I dragging my butt out of bed, putting on my pajamas and blogging about it?

I am a low-level minion now at COMPANY NAME REDACTED so my job is to answer the phone, work the problem and move on.  I was passed over for promotion and I would normally say something like, "The other guy is really good and they had a tough choice" except for I was hired for that job.  I was supposed to be a low-level minion at COMPANY NAME REDACTED while I learned the product and then I was to be moved to a mid-level minion.  Instead, I am in support queue Hell.

Low level minions are NOT supposed to go on site.  So why am I?  As my boss put it, I am just the eyes and ears for our top-level minion.

Of course, he didn't respond to my IMs or emails all day today, so I did what I do.  I worked the problem.

I shouldn't gripe but I've been doing support for fifteen years at this point.  The way to be successful at support is to get the call, work the call, fix the problem.  Three simple steps.  Support isn't like making widgets where each widget is exactly the same and you should be able to make 30 an hour or whatever, each support case is different, different things have been tried, the setup for each is different, you must be flexible and it takes however long it takes.

With the advent of "The Great Recession" (tm) employers started hiring "Consultants" (remember kids, if you aren't part of the solution, you're a consultant) to review "Metrics".  Metrics are all about how to do more with less.  So the assembly line of support is created.

Get the call, work the call, give the customer something to do, get off the call and onto the next call.

I STRONGLY disagree with this model as I am one of the guys who gets the frustrated and now pissed of customer who has been fighting a problem all day long and feeling like he's getting the run around.  I (and some others) wind up working the call and trying to recover the customer and making sure the problem gets fixed.  This is great for CSAT, but total crap for Metrics.  I take longer to work calls and therefore pull fewer.  Although when I do send a case up, chances are fair that it is a legit problem. (Not always, I am human.)

So my stubborn insistence on doing things the right way directly impacted my moving to mid-level minion.

So, what does this mean?  Nothing really.  No time to write (beyond this), no time to rehearse music, no time to play in a band outside of church, no time for illustrations.

In short, no time for fun in life.  Makes you wonder, "Why bother?"

The Magic Of Apathy

A co-worker found a very effective way to deal with difficult cases.  He followed Roderick's Rules and performed a "Deny, Deny, Deny".

Let me expand.

I get a frustrated customer from Widgets Inc. that said my co-worker (let's call him "Fred" for this example) was supposed to have called him back last night.  I check the case and there are no notes from Fred and Fred did not take ownership of the case.  So, as far as our records showed, Fred never touched the case or talked to this customer.

Normally in tech support you follow the "Trust, but verify" model which means, the person on the phone isn't lying to you, but they might be wrong in what they are telling you.  This is a pretty effective method for everything so I tried to reach Fred on IM.  Unfortunately Fred was at lunch.

The customer was VERY SPECIFIC about what Fred did, said and told them he was going to do.  Plus they were pretty upset that they weren't called back.  So I wind up working the case with an unhappy customer trying to get them back to happy status.

It was a beautiful job on Fred's part.  He ditches the unhappy customer, the difficult case, plus his STATS look good and since he put nothing in the case, he gets no blow back.

I'm now working under a couple of different stat metrics which are screwing me.  First, the more cases you pull, the better your stats.  Balance that with the customer satisfaction survey (CSAT) and it rolls together to determine who gets a better shift at the end of the rotation.

Fool me once...

Getting Older

I've been asked a question I'm pondering how honest to be on.  The younger me would be discreet and keep my mouth shut. Maybe... I think hack to how naïve I've been in the past. Who knows.

I've been asked to provide feedback on how to improve some things.  Being the new man on the team could lead to me getting labeled as an ego maniac.  The bottom line is I've done support for fifteen years and my company is paying me too much money to work basic adds, moves and changes type stuff.  Getting through the tiered stuff is wearing on me too. What's the point in taking a case if you don't get the satisfaction of completing it?

A second thing is my first personality conflict is rearing up.  I'm trying to be lighthearted and laugh things off, but I'm having some serious issues with how a coworker is interacting with me.  I guess I'm expecting too much and I have no desire to air dirty laundry, lets just say I'm ready to change to the position I interviewed for.

I'm going to set up a one on one with my new boss.  I've decided I understand my time in T1.  All well and good to get to understand the product  (not sure why Microsoft didn't run that way) but I want to know what goals and/or timeline I can expect before moving to Tier 2.  Paying dues is one thing, I just want to understand and set expectations.  I do not want to stay in Tier 1.

So Tired

I'm counting down.  I've turned in my notice at work and I am set to start my new job Monday after next.

But I'm tired...

I've been surprised at how relentless many employers are now.  It's not a case of can you do the job, it's can you do all of these jobs with an unrealistic expectation of success.

I have a case where a customer has done the equivalent of flailing their arms and saying "Stuff doesn't work".  They don't give details, they just complain "Stuff doesn't work."

I have another customer we identified an incorrect configuration on two weeks ago.  Yesterday the guy who was sent on site to help fix things found the same setting.  I told him we'd told them two weeks ago to change the setting.  I was told they were going to change it last night.

This morning they asked me what i found in the logs.  I asked what the results were of the change.

The customer is still "Testing" in their lab before rolling it into production.

My boss is asking me what is going on.  We have managers all over the place asking what is going on.  The customer is still angry that this isn't fixed.

But their own people refuse to change the setting we KNOW is causing problems.

My main complaint about my current position is I am the fall guy.  I am tasked with fixing whatever is broken, yet I have no authority to change anything or force anyone to do anything.  So this customer continues to sit on their hands and I keep looking at the same logs.

One more week...

I Is The Dumb And Dumberer Today

Sheesh...

I hate it when I do dumb things and this one was a doozy.

I've been fighting an ongoing case for several weeks now.  Things have been extremely busy at work and we've all been going at it like gangbusters to get our work done.  Things are so hectic we have to throw some ideas out, read logs or network trace and then scramble to the next case.

It's a young man's game.  I'm not young anymore.

Regardless, the case I have been fighting with involved application sharing and slooooooooooow processing.  I tried network traces, perfmon traces, client traces... Everything I could think of.

Finally I had the customer break the clients down to a simple switch with only those two machines on it, with them in the same subnet and one connection back to the server.  It still was slooooooooooow.

What. The. Fishsticks.

So, I decided to beat my head against the wall this evening and try their antivirus program as invasive as I could make it on my lab to try to make performance suck.  IT STILL WORKED!!!!

What the crap?  How can their experience be so bad?  What could possibly be making that thing suck so hard?  What could be limiting the performance?  It was like the bandwidth was throttled down to nothing...

Ahhhhhh CRAP!

You may have guessed it by now.  In Lync you can limit the bandwidth you allow between clients in certain scenarios.  This is so you don't have Fred the Janitor hogging all the bandwidth with Sue in Accounting doing video chats when Mr. Big is needing to do his conference call with the shareholders.

My customer had limited the bandwidth for his clients through a policy.  A quick look back through the logs and I confirmed it.

"What took you so long to find something so obvious?" you man ask me.  It is all in presentation.  The description of the failure was two VERY specific scenarios.  I got so wrapped up in those, I didn't see the obvious point staring me in the face.

Never, ever, take anything for granted.  *sigh*

The Phoneman Adventures - Episode 1

Years ago I worked on a team that installed a phone system for the City County Building in Knoxville, Tennessee. The City County Building is like the town courthouse of steroids as it held most of the city and county offices, courts, the city jail and also the independent public management department.

The important part of this story is the city jail. We always enjoyed visiting the jail for service calls, kind of the way we enjoyed visiting relatives who pinch your cheeks and reeks of scotch, Marlboros and cologne.

In short, we didn't.

It wasn't because we didn't like and appreciate the folks who worked in the jail. They had a tough job and did it well, we weren't particularly fond of the "Residents" of the jail. While filled with a mix of weekend drunk drivers, petty crooks and other various miscreants, the thing that bothered us most was we were there... 

With tool pouches...

You see, it's hard to do repair work without tools and although ANY knife would quickly be confiscated from a visitor to the jail, we were free to walk among the inmates with leather pouches filled with technician death swinging from our hips.

Screwdrivers, pinch tools, pliers, hooks, wire, snips etc... were our tools of trade and they all made wonderful weapons if used improperly.

The things which kept us in the proper frame of mind was our use of gallows humor to pass the days. We enjoyed razzing each other endlessly over our failures and problems. We also enjoyed telling each other the stories of our trips to the jail.

One story involved a group that ran a new phone line for the staff. When it came time to leave the guys were checking their tools to make sure everything was there. One of our guys named Curtis noticed he had a screwdriver missing. A quick look around didn't turn it up, so Curtis turned to the guard who was escorting them and said, "I'm sure it'll turn up sooner or later." The guard then replied, "Yeah, sticking out of our back."

Needless to say after a MUCH more inspired search the screwdriver was found where it was left, above a ceiling tile.

A second story involved a set of ladies who were mapping out the location and numbers of all the phones in the jail. One of the ladies realized she had forgotten to declare a small knife when processing in.

I need to clarify the term, "Knife" here. Knife paints a picture of a deadly killing utensil. This "Knife" was about an inch long when closed and contained a tiny blade, a tiny set of scissors, a tiny nail file and a tiny toothpick. We had gotten them as gifts from the company and they came on a keying. Most of us used them to clean our fingernails and occasionally use them as screwdrivers when wiring phone jacks when we were too lazy to walk back to our trucks for tools.

With that said, she tapped a guy on the shoulder and told him of her mistake. She held the teeny, tiny, itty, bitty knife in the palm of her hand to show him. (keep in mind the first story about out "Tool Pouches of DEATH" I just shared).

The man turned white, broke out in a sweat and then snatched the "Knife" from the lady's hand. He clutched the contraband weapon in his meaty paw, enveloping it in a kind of flesh container to protect it from seizure, and holding the meaty paw filled with tiny death high in the air, immediately marched out of the jail.

Yeah... drama...

Cube Squatter

AHA!

I am firmly settled into life here at SERVERCO where I do "Something" with computers.  Its amazing how some things never change.  For example: A window cube is always a coveted prize. It gives the cube-dweller a glimpse to outside and an opportunity to dream of freedom.

I had one once...

Anyway, we have a squatter in our area.  She isn't in our group, but she has a coveted "Window Cube" and she refuses to move.  My manager is a SUPER nice guy who is trying to politely "Hint" that she needs to move along so we can use the cube.

She pretends not to take the hint.

Today she LOUDLY laughed and chatted on her phone while the rest of us minions were buried Powershell and tried to drown her out with "Megadeth" on Zune...

I feel that the Cube Squatter MUST BE OVERTHROWN!!!! 

Challenge Accepted!

NerdVana

All I have to say is "Quad Core Windows machine with 24 Gig of RAM."

Nerds everywhere wish they were me.

Goodbye PHONECO

Today is my last day at PHONECO, starting Monday I will be an underling for SERVERCO.  Basically my job description will be the same. "He does 'Something' with computers."

Leading up to all this Lucie asked THE WIFE, why I was so happy to leave PHONECO.  She didn't understand why I didn't like them anymore.  THE WIFE tried to explain that I didn't believe they had a good plan for the future and she followed up with the question, "You know what SERVERCO makes don't you?"

Lucie replied, "Yes, money."

The girl is very bright.

Why Do People Panic?

First ticket of the morning is a guy freaking out because the volume to his voicemail is too low. Everything is fine except this ONE QSIG link. When calls come in that one link, the volume is too low and we need to fix our voicemail yadda yadda yadda. My response, increase the volume on the link where you are having the problem. Seriously, this is not rocket science. Second ticket of the morning a guy tells me he is getting a 404 error when he tries to perform an action. I ask for a wireshark so I can see what file is not being returned. Something he could have done from the beginning and saved us both a TON of work. 7 more days... 7 more days... 7 more days...

Wanna See What I look At All Day

This is one VoIP call.  Go to college kids... Trust me.

VoIP In Action

Gotta love Wireshark!

Total Shocker

We had to make an emergency trip back to Tennessee for my Paternal Grandmother's funeral so with the chaos and all, I have been missing my workouts.  Not to mention I have been trying to study up for another job.

I'm sorry, but my days at PHONECO are numbered.

I don't think I will get canned, but that is part of the problem.  I will be one of the last people left here to suffer through supporting failing products when we finally get our division sold off to someone else. THEN I will get canned.  Right now there are exactly TWO of us working support. (2) Seriously.  My Big Boss stated yesterday he was confused because we had 350 or so emails a week, 150 or so cases a month and no sales.

You heard that right... no sales.

Obviously this doesn't bode well for me. The hand full of installations which have been performed are eating our lunch as far as resources go.  I have read the writing on the wall, so I have been sowing my oats (and if anyone is looking for a VoIP Ninja feel free to let me know).  I am spending my days waiting for a callback to see if my BIG LEAD pays off or not.

So anyway, I digress.  My weight after my mile and a half return to walking, 318.  I'm still on target.

Thank you, God.  All praise and glory goes to you.

Rules? Who Needs Rules?

Today was interesting.  First off we had to pull Ben out of school after the school gave THE WIFE notice that he wasn't doing so well.  They felt like he isn't doing so well, blah blah blah...

Whatever, the writing is on the wall, so the boy is now home schooled for the rest of the year.

Now, in dealing with this, I got called by some managers about a site that is so angry. (Score one).  They are threatening to pull our server. (Score two).  They don't feel like they are getting good enough support. (Score three).  And several more sales depend on this getting fixed which will result in thousands of dollars in revenue. (TECH SUPPORT BINGO!!!!)

Here is the situation; our sever has a list of phones we have tested and deemed to be fully compatible.  The salesholes push the fact we are SIP based and that any SIP phone will work on our system, but tend to forget that if we haven't tested for full compatibility, there may be some things that don't work right.  This can be caused by things being implemented in different methods or a bug on one side or the other.  I admit, we aren't perfect, but we are pretty good.

The important thing is to remember if the phones have not been fully tested, we do not guarantee full compatibility.  Basically that is a big red flag which stops troubleshooting.  If it works on a phone that is fully tested and not one one that isn't, boom, we are supposed to punt the problem away.

So of course we have a HUGE site which just purchased our server with a bunch of Cisco SIP phones that are not on our fully compatible list.  Sure enough, there is a bug with the Cisco terminal.  Does our reseller call Cisco and demand they fix the bug?

No.

They call us and demand we figure out some way to make these terminals which are not on our fully compatible list and have a bug (a bug that I have proven is legitimate and Cisco supposedly fixed several loads back) because THEY SOLD THEM TO THE CUSTOMER!

What's really sad.

Because cash is being threatened, I have been asked to try to figure out some way to make it happen.

Why make rules that you won't enforce?  Why not just say, sure we'll have our minion get right on it.

Whatever; you guys pay my salary.

Just Because It's Monday

The More You Know!

If you have phone problems it is apparently best to NOT call in while you are having a problem, but rather to wait until lunch time on the following Monday and then gripe like it's the end of the world.

Effective.

Situation Normal

Not griping, just amazed.

PHONECO does mainly online training and one of my co-workers took a course on the system I support.  He made the statement that he "Passed the test but couldn't figure out how to add a phone to his system now."

This guy is no dummy, he reads ISO recommendations and understands them, got his CWNA, has taught classes in VoIP for years and has written countless documentation for PHONECO.  This is just an example of how crappy the training is.  We cannot make upper management realize you can't sit a person down in front of what is (basically) a 12 hour Power Point Project and expect them to do anything other than follow the steps a to b to c.

That would be great, but if you don't know why you HAVE to go a to b to c instead of b to a to c (which may work sometimes) and why, they will fail on installation of the new PBX.  Go over budget.  Speak poorly of it.  Cause others to hate it.

In other words FAIL.

Losing My Mind?

I am working from home today.

Not because I am sick or have an appointment.  Nothing like that.

I overslept.

What the heck?  I am 38 and haven't overslept for work since I was a teenager.  What an embarrassment.

Furthermore, I carpool into work, so I not only slept through my alarm, but I slept through my ride-share knocking on my door and calling my phone.

I am also on call, I would have slept through that if I had been called.

I don't know what the hell is going on with me.  I really seem to be losing my mind.

I Know Too Much

There are days I wish I did not have the insight that I have as to what all is going on at work.  I write here because of a semi-anonymous way to blather to people who don't really know me or what is going on.

I just saw an email talking about how we need to make sure that company "X" doesn't get upset with us as they are still pushing our PBX.  We don't want them to stop pushing it also.

Also?  So I moved over to a department where I was promised I was working on the NEXT BIG THING, to find we are pissing people off left and right?

Wow.  Great career move huh?

Oh well, as a side note I am spending my free time prepping to get some extra cash in.  Hopefully I can start pimping out my wares.  We'll see.

Worse come to worse, I can be a phone man... Oh wait! ARRRRGH! I AM!

A Moment To Complain

It has become obvious to me my time has come to move on from PHONECO so i will be looking.  I have often griped on the blog about this or that and certain things which torqued me, but I have been loyal to the company and done what was asked of me.

I cannot say that any longer.

The final straw began when I was moved into a temporary assignment and asked to cover a product I hadn't worked on for two years to help out as they were losing one of their support guys.  I immediately agreed and was enjoying myself fairly well. Until the second support guy turned in his notice to leave.  This group only had four people, so I thought it over and decided to apply for the open position.  My thinking was to get more Microsoft experience under my belt and not be so pigeon holed.

Long story short, I was given the position with a title change, but no money.  The money didn't bother me too bad as I understood things are tight, but since I have been screwed before by a title change with no money, I know that the money will not come later.  I respectfully asked for my title to remain the same.

I was denied.

Apparently management fought so hard for me to get money that the offer of a new title was all they could get and since they made such a huge fuss they don't want to go to the big boss and tell him that this underling is willing to do whatever for the exact same pay and title.  I guess I can understand that, but a snazzy title doesn't pay the bills.

Next up we were told we had to participate in mandatory vacation and take every Friday off to burn vacation hours off the books.  This is an accounting trick to free up cash as that way they don't have to show as much money on the ledger for the half.  I was moderately irritated but the family and I were taking vacation in August, so I wasn't exceptionally worried and actually was looking forward to the long weekends.

Well, apparently even though everyone was told to take off, tech support has to cover for all the days, plus after hours support.  This resulted in my vacation getting screwed so the family is now no longer taking a road trip for the vacation.  This is minor compared to two of my co workers as one has been out of the country for a month and has to take off still and the other who is going out of the country later in the year and also has to take off.  Obviously they don't have enough hours to cover this and management gives a big shrug of the shoulders to that with an "I don't know," when they are asked how they will handle that.

The last thing which set me off was me finding a write up over the weekend talking about how our sales are up 24% from last year.

I have the offer letter sitting on my desk.  I'm sure I will sign it as I will do what I have to do to take care of my kids, but I'll be damned if I will forgive them for this one.