To UTC or not UTC, that is the question.

As things get more complex with an immense amount of devices talking to each other depending and exchanging information, accurate time synchronization between devices is crucial.

When we started making terminal servers in the late 70’s, there was not even a real time clock to derive a time stamp to use when recording data.

As time progressed, logs and logging became more important, recording the information with some kind of crude time stamp was good enough. After all, it only mattered to the single unit that you were working on locally.

In later years, these files were being dumped into SYSLOG, the format of the time became more important. Then devices started dumping huge amounts of information into programs like Splunk where the date/time, time zones and daylight savings flags matter to build a better picture of what’s happening across your entire enterprise. With thousands of devices, all at different locations, getting the date/time, time zones and daylight saving flags consistent and synchronized becomes not only important, but required.

I have asked many people on how a unit should be initially setup, Local time or UTC (not to be confused or interchanged with GMT).

The answer was a little surprising. I would have thought that UTC would be a unanimous answer, but a lot of people chose local time. The over whelming answer to “why” was predictable.  It was easier for the human at the location of the unit looking through the data to align the time zone of an event without having to do any conversion from UTC to local time. Directly clashing with letting machines compile and help you sort that immense amount of data from multiple sources.

I know that the machines can take all the different sources, from all the locations around the world and normalized them, but if a machine was not set with the correct time zone or the daylight savings profile is old, there could be skewed data ruining any collected data scattered across multiple locations.

Time may be an afterthought, but with your collection of data scattered across the globe, it may be worth taking a few minutes to solidify your event collection. Waiting until a problem occurs is a terrible time to realize any discrepancies.

With Cisco Live 2023 in Las Vegas coming up, I can’t wait to continue my research.

Put the ‘Live’ back into Cisco Live

Cisco Live

It has been a long time since Cisco Live has been Cisco “In-Person” Live. The last Cisco “In-Person” Live was Barcelona in January 2020. The timing to online only was a little disappointing personally because in the previous two years I had solved the true meaning of unlocking the full potential of Cisco Live.

I have been going to trade shows for years dating back to the original Comdex in Las Vegas in the early 80s. When I started going to Cisco Live twelve years ago, I went to the vendor booths, went to the sessions, politely smiled and then left the room continuing my show experience. I was unknowingly completely neglecting the true spirit of the show – interaction with the Cisco Live presenters.

I go to Cisco Live because our company caters their boxes to the networking crowd, but I am a code engineer, networking is not my main skill so I felt a little out of place asking questions and interacting, I was assuming I was wasting the presenter’s time.

It took Cisco DevNet at Cisco Live Orlando in 2018 to make me realize the full potential of being at the show. After someone gave their presentation, they would also hang around the show floor and you could walk up to them anytime during the day and interact. You know what, its ok if you ask a question that may seem out of place or may require some time to fully answer your question. The presenters enjoy interacting and answering your questions, it gives them valuable feedback so they can modify their presentation for the future based on the questions.

Along with everyone else I have had a crazy couple of years, I have crisscrossed the United States six times (my son goes to Penn State), my wife and I have moved from Southern California to Cape Cod on the Eastern coast.  Looking at the Cisco presenters for Cisco Live 2022 Las Vegas, a lot of them have changed, but there are just as many that are familiar names. I am looking especially forward to this particular show because it feels like coming home to a warm, familiar setting where you can rekindle old friendships and start some new ones.

Do yourself a favor, after a presenter has giving their presentation, make it a point to discuss any questions or comments you may have. If after the presentation you don’t want to wait in line to ask questions, look for them in the Cisco or DevNet area. Take that deep breath; gather your thoughts and talk to them, your show experience at Cisco Live will change forever.

It did for me.

FUD, not my cup of tea

Dear Cisco,

Please do not make your shows permanently online only.

Apple has announced that its 2020 WWDC this June will be online only due to the coronavirus, but they have suggested they might keep it online permanently going forward. I know on the surface it looks like a great idea: Save money! No travel! No hotel or airline headaches!

But online loses the whole purpose for an event: Relationship building and human contact.

A lot of engineers and developers in this world each sometimes feel like a lone tree in a large forest when they need to get verification. They’re hoping what they are doing makes sense and takes them in the right direction, but they want to reach out to verify that with someone they know. Meeting real people at the Cisco Live, Cisco Connect and DevNet events puts human faces on the Cisco online ID. They’re people you know and trust, not some untouchable individual in an ivory tower that does not have time to get to know the common person.

During the past few years, numerous Cisco employees have helped my team and I with a range of both mundane and complicated problems. That would have never happened if we hadn’t made personal connections with them first.

I would have never known that some Cisco people are avid drummers, or restore vintage video game systems, or ride the Sturgis Motorcycle Rally, or talk gloriously about video compression codecs. Or tell the a V.P. at Cisco that her new board of directors gig at a large company is in my home town (well, that conversation didn’t happen yet because of a scheduling conflict, but it will at the next show). And most of the Cisco technical employees you see online all the time are just good people and will absolutely and passionately point you in the right direction if help is needed.

Personally, if there had not been a Cisco Live or DevNet Create, I never would have ventured into the Podcast Zone and actually done a podcast with some amazing people. It took cajoling in person to make me take that leap.

Online meetings are great for keeping in touch throughout the year, but the personal face-to-face meetings and chance encounters at the shows are critical to the way we do business and keep our careers moving forward. At the conventions, you never know who you will run into between sessions or while wandering around after hours — chats that could lead to lifetime associations. I’ve lost count of the people I have met on the bus going to and from the conventions (or sponsored Cisco Events) — even at the McDonald’s in Barcelona. All these in-person moments keep our keen sense of passion within the community alive and vibrant. You learn, and your source of information learns too, creating a self-renewing circle of mutual information sharing.

Talking about creating the Pixar campus, Steve Jobs said it was designed “with collaboration in mind. Rather than separating animators, executives and editors in different buildings, I brought everyone under the same roof — with the idea that chance encounters would lead to the cross-pollination of ideas.” We should all make maintaining in-person encounters a cardinal rule.

Even after forty years of being an engineer, I still bring the same passion to solving problems, creating new ideas, and carrying other people along who don’t quite have the same forward-looking vision. That passion is a great catalyst to talk, in person, to other people that have the same passion. We should always champion in-person opportunities to champion those passions.

Now back to your regularly scheduled program.

The Phoenix Project

I was on a conference call for a company I help out that’s based in San Francisco. We have people all over the world — England, Australia, Hong Kong, Czechoslovakia, plus the West and East Coasts of the United States — and a number of them were on the call. It was just before Thanksgiving, and an English participant mentioned to the group that the average American Thanksgiving dinner costs $48. The San Francisco people went off the rails, saying (I say boasting) the $48 does not even feed one person at their Thanksgiving dinners. That made me think how different lives can be.

I had started reading “The Phoenix Project,” a book on how a fictional company becomes profitable after a crisis-driven change in its information technology management style. I love reading books, especially about the computer industry, along with fiction and nonfiction, but this one was hard to finish. It had been recommended by many people I know in Silicon Valley in the IT field as being an authentic account of their daily jobs. The reason this book was so hard to finish for me is that I kept thinking, Why do you people still continue to work at such horrible places? The protagonist of the book is promoted to a managerial position and his life starts to fall apart. The long hours he works not only burns him out, but he also spends less time at home, putting pressure on his wife (who also works) and children. I understand the point of the book is to promote the processes of IT automation, so that a real company does not fall into the trap of the fictional company in the book. Too many times you hear about these companies in real life and, every time one problem is solved at them, the management seems to fall into another company-threatening problem.

There was a story in the early ’90s when Microsoft was working on Windows NT (for you young people, that operating system is the basis of every Windows version presently produced), when the brilliant Dave Cutler and a bunch of D.E.C. computer engineers came over to Microsoft to plan and implement this ground-breaking OS. The group checked in code on Christmas Day. At the time this was unthinkable, working on that holiday; however, present day, stories like that are rampant.

In Silicon Valley, you have both members of a couple working long hours, dropping their kids off at daycare in the morning and picking them up in the evening. On the weekend, they spend time getting their scheduling projects and readying them for work in the coming week. Occasionally, they might break away to be a family, and have a few hours of family time.

My message is this to the Silicon Valley people trying to get rich. It’s not worth it. Work hard at what you love without sacrificing those that you love and who love you. You can have a comfortable life and not be consumed by work. You can relocate someplace where the small starter houses don’t start at $1.2 million and come with a two-hour commute each way to work. I know you think the rest of the country (and world) looks on at you with envy, but it’s actually the opposite: We can’t imagine throwing away everything and wasting decades at jobs that are no better than a remote chance to win the lottery.

By the way, here in Southern California our Thanksgiving dinner cost about $60. And I spent all day and night with my family.

The Persian Rug Theory

When I first started out in software engineering during the ’80s, I had a talk with an older McDonnell Douglas engineer. Among his various amusing anecdotes from the Apollo moonshot era was his recounting of the Persian Rug Theory. Being wide-eyed and curious, I asked him what the heck the Persian Rug Theory was and how it pertained to software engineering.

Here’s his story: Hundreds of years ago, when Persian rugs were au courant with the social set, rug buyers were careful negotiators and would look for any reason — say, the use of two different wools– to undercut the price on a rug they were considering.

What the rug makers might learn to do — say, when making other such rugs, elegantly weaving together two different sets of raw wool whose coloring wasn’t a perfect match — would be to put in a tiny defect. It’d be off to the side and wouldn’t detract at all from the gorgeousness of the rug, but an expert eye would be sure to notice it. As the rug maker expected, the rug buyer would become fixated on this unimportant imperfection in the effort to get the best price and would thus miss completely that the coloring across the rug was just a tad off. Both maker and merchant would be happy with the final selling price, and the eventual owner at home would be happy with their Persian rug.

The Douglas gent told me that in software, when you are passionate and really believe in what you are doing, there can be times when you know the right things to do, but the powers-that-be might not have the vision. When you want to get functionality included that you know is crucial but management doesn’t understand, it’s much easier to get the functionality approved if you purposely put in a small but obvious problem. It might be a spelling error, a wrong copyright date, or an incorrect address.

Does the Persian Rug Theory actually work? Well, all I’ll say is that the theory is part of software lore, and what I build always includes all the necessary functionality. The rest I leave to you.

Kobayashi Maru

The Kobayashi Maru simulation is from the original Star Trek television series, and later showed up in the movies in Star Trek II: The Wrath of Khan (which is still the best Star Trek film ever).  The Kobayashi Maru is a no-win training scenario used at Starfleet Academy to test a starship trainee’s resolve and examine their decision-making process. Do you save the people on a doomed ship or risk intergalactic war and maybe the death of every one of your crew? Either choice you make is a doomed choice, but the choice reveals your character. The only person to ever beat the Kobayashi Maru simulation was James T. Kirk, who went on to command the starship Enterprise. To beat this test, he changed the rules; he hacked into the main program and changed its parameters. The logic was that he didn’t believe in a no-win situation.

When I talk to other programmers and engineers at computer shows, I realize that I am in a unique position of working for a hardware manufacturer and also making software that talks to the same devices. If something comes up and doesn’t quite work within the rules, I can change the rules within the firmware of the box. It gets a little frustrating when I talk to people about interfacing our devices to open source or commercial projects and they dismiss us because we don’t have a certain feature. I do take comfort that most people don’t realize that I can change our boxes’ firmware, the actual API in the box AND the software that talks to it. There is no such thing as a no-win scenario when you have these tools at your disposal.

I first became aware of this power many years ago when the company I worked for made serial/parallel-based printer-sharing boxes. You could change the target printer by sending a string of characters (I believe it was [:]P<cr><lf> for parallel and [:]S<cr><lf> for the serial printer). We made a DOS TSR program to help issue these commands and also a Windows-based program. In the Windows-based program, we used the “generic text” printer driver, which sent exactly the characters you wanted to send without any of the printer’s native page markup language commands. A problem arose when Microsoft went from Windows 2.11 to Windows 3.0; the generic printer driver now ended all output with a <formfeed> character instead of <cr><lf>, so when a customer tried to change printer output on our device an extra sheet of paper came out of the printer. Instead of trying to contact Microsoft and complaining about this change (which we knew would go nowhere), we just changed the rules. We modified the firmware in our printer-sharing box to accept both <cr><lf> AND <ff> characters at the end of any command.

More recently with our “Out of Band” serial console and “Power Distribution” units, controlling them with automation packages was daunting at best—you could use a package’s generic shell control modules to issue cryptic commands to our devices. So, again, we changed the rules: We have added a growing set of APIs to all of our devices, which interacted with our modules we created and have been adding to the Ansible project.

My long-drawn-out point: Try to change how you look at a problem when you encounter a roadblock. Think outside the box. If there is an open source project or a type of a device that doesn’t do what you want it to do, get involved and add the functionality to make it work. For open source, download the source from GitHub, play with it and add the things you need to make your life easier. If a device you are using would be perfect if only there were one more feature, contact the manufacturer; most of the time they will be thrilled to get a phone call from customers who want new features (especially if you reach the engineer who helped make the device).

When you change your way of thinking from a programmer to an engineer, that opens up unlimited combinations of possibilities. Be a Kirk.

Sticker Shock

I am a software engineer; I have been since my first computer, a VIC-20. 1980.

Being a software engineer, I have a few compulsive disorders. Comes with the territory.

One of them is clean equipment. Nowadays, that means a clean laptop and a clean tablet, with no stickers, no dirt, no fingerprints. Clean.

But, if there is one thing I’ve learned over these four decades, it’s that the world moves on, whether I like it or not. If you don’t get with the program, you will get left behind. First, you’re being put last on the invite list to meetings. Then, worse, you’re just plain ignored.

Which brings me back to stickers. It started a little over a year ago, my first sticker on my laptop; it was a DevNet robot design from Cisco Live Orlando. Putting it on, I thought how this will be the end of life on earth as I’ve known it—and, what a waste of time this stickering is.

I was wrong.

At Comic-Con of the same year, I picked up another few stickers and put them on my devices. Then another few from food joints and some Box Lacrosse teams I like.

Now when I travel, the stickers prove not wastes of time, but maximizers of time. People see the stickers and, bang, a conversation begins. It’s amazing how many people will stop what they’d been doing and start engaging with me on a topic that they connect with when they see a sticker that speaks to them.

At Cisco Live 2019 in San Diego, I saw a sticker about drummers on a laptop of someone furiously typing an email. Somehow that made sense. And it turned out to be a good acquaintance, John McDonough from Cisco. Conversation ensued.

The point of this rather long note is, however, not about stickers; it’s about not just dismissing new things out of hand but giving them a try. For instance, consider the new DevNet Certification that was announced at Cisco Live San Diego. Many network engineers look at it as a dilution of their skills/certification that they have, while software engineers may look at it like they are lowering themselves. It’s really neither. It’s a way for the two groups to meet in the middle and make the process of automating the network easier and more reliable. I think you’ll find that this dual program is beneficial to everyone, not only within your organization, but to our community at large.

Plus, you’ll get a sticker.

Made in the U.S.A

What does Made in the U.S.A mean to you?

You may see products stamped with carefully crafted terms such as “Engineered in the USA”, “Designed in the U.S.A.” or the “Inspected in the U.S.A”.

Personally I don’t see it as a marketing spin or even a nationalistic moniker. When I buy something critical for my infrastructure, Made in the U.S.A. means that Engineering, Manufacturing, Service and yes, even Sales are all in close communication with each other. When the operation is in multiple countries, across four time-zones separated by six thousand miles, things don’t get done quickly or efficiently. The first person a customer talks to should not be an apathetic person who is just trying to fill the quota mandated on them by upper management who they have never met and never will because of the tremendous turnover rate.

When I put something on my network and a problem arises or a vulnerability shows up, I want the piece of mind that it wont be months before i get a resolution to my concern.

In the company I work at, Western Telematic Inc., we make Out of Band console and Power Distribution Units and is located in Southern California, in fact we have been located in Southern California for fifty years. When I say WE, I mean every piece of the process, Engineering, Manufacturing, Service and Sales.

When you call in for service you will either get Mario or Andy, if they cant solve your issue, they will walk down the hall to Engineering, if something needs to be checked with production, we go downstairs to the manufacturing floor. Right across the hall from them is sales so you can be contacted right away.

The Engineering group has been together for a long time, long enough so that if we look at a piece of code you know who wrote it by the style of the code written.

There is a reason WTI has a five year warranty on our devices, along with free firmware /software upgrades.

Building a Better Network

In any organization, speed and efficiency is critical when responding to network outages and interruptions. But as a company grows, and the networks expand, quick responses become more of a challenge.

Network mangers and engineers at large data centers prepare and review network failure scenarios. Failsafes and contingencies are built in, and they generally have the staff, equipment and access to respond quickly.

But when outages and interruptions occur at campus and branch facilities, fewer resources and qualified personnel are available locally to restore network service, and deploying branch-level technicians can take hours. Meanwhile, productivity at these remote locations is at an unacceptable standstill.

WTI’s Console, PDU and hybrid devices are the solution. With more than a quarter-million installations at some of the largest companies worldwide, our Out-of-Bandwidth (OoB) products have become an indispensable tool for remotely identifying and resolving network problems, and most importantly, they can keep network traffic moving.

WTI’s IP Passthrough™ technology allow branch or campus offices to stay connected through cellular modems available on WTI Console server and hybrid devices. IP Passthrough technology is a great failsafe to keep network traffic moving until normal connectivity is restored.

Is Your Organization’s Communication Infrastructure Reliable?

WTI’s devices should be a part of any organization’s network failsafe/contingency strategy, ensuring full-time remote power control, access and connectivity regardless of whether you company’s communications infrastructure is analog (POTS) or digital (cellular).

Whichever technology you use, as your organization and networks expand, it’s important to consider that POTS (Plain Old Telephone Service) and cellular are two communications technologies that are moving in two very different directions.

POTS vs Cellular

POTS

POTS is and older technology and is in-use at many organizations around the world. In fact, in some of the more remote locations around the world, POTS is all there is, but that’s changing quickly.

Despite it still being a viable technology, increasingly, carriers are putting less of a priority on POTS installation, maintenance and upgrades. In fact, as of 2018, carriers like AT&T and Verizon are no longer required to install or maintain POTS lines at all. Third-party contractors will begin picking up the slack, and the increased costs will be passed along to the consumer.

POTS copper cable and wiring is cumbersome, expensive and aging rapidly. It’s served us pretty well during the 20th century, but its reliability has always been an issue. Bad weather, accidents, and even unwelcome critters have been known to bring POTS service to its knees.

Cellular

Cellular offers significantly more advantages for both carriers and users. For carriers, cellular is much easier and economical to install, maintain and upgrade, which is why we see carriers moving away from POTS and focusing much more of their attention on cellular.

Cellular users enjoy the same simplified deployment and cost benefits as carriers, but the advantages of improved performance, functionality and scalability options moving forward can’t be overstated.

Unlike POTS, the IP-based functionality of cellular service provides simultaneous access to network devices, which is absolutely necessary to allow telemetry and network automated maintenance systems to collect and monitor network data.

The network automation options cellular offers reduces the number of time-consuming, error-prone and expensive tasks and allows network professionals to manage their time and resources more efficiently.  

Those of us who have watched cellular technology evolve know that speed, performance and reliability are rising exponentially, and costs are dropping rapidly.

Final Thoughts…

POTS technology isn’t going away completely any time soon; too many people and organizations still depend on it. But its expense and performance compared to the rapid rise of cellular technology is certainly something to consider moving forward.

Whatever choices your organization makes, rest assured that WTI products will continue to be a dependable part of your network failsafe strategy.

wtinetworkgear