- Rebecca Johnson

In 1996, I recognized a problem in the way IT services were bought and sold. The sales process wasn’t set up to solve a customer’s problems. Instead, it was set up to close a deal. Unfortunately for IT buyers, there really weren’t any better alternatives.

And the process wasn’t much better for the sellers either. Vendors were burdened with trying to provide excess requirements for the least cost, regardless of market value or actual need. In the middle, a massive amount of cash was being lost as the two tried to come together. Where did that cash go? Mostly into sales and marketing, which is ultimately about bringing two sides together to do a deal?

I first noticed this issue in the mid-1990s, when I was coaxed out of early “retirement” to work for K.B. Chandrasekhar and B.V. Jagadeesh, whose Internet services company had grown into a powerhouse called Exodus Communications.  Barry James Folsom and Mark Bonham hired me to help grow Exodus even more. Three years after starting Exodus, Chandra and B.V. took it public. 

“If we could sell our experience for what it cost us, we’d all be millionaires.”

- Abigail Van Buren

I will never forget Chandra passionately telling me I was hired to “put a face on the place.” I am almost sure he meant to create a brand that was known by all. Regardless, by the time I left, Exodus had a $37 billion valuation. There are so many stories to share about the transformation that made the Internet what is it today, from our stoic CEO Ellen Hancock, who fought off the collapse of our business model in a terse phone negotiation I set up with BBN Planet’s Vint Cerf, to our first managed-services deal with something perplexingly called an “enterprise account.”

I never had a formal title working for our marketing vice president, the great thought leader Mark Bonham, or Sam Mohammad, the vice president of sales. I was “the marketing guy.” Whatever was needed to help the company grow, from global data-center launch strategy to managing Keynote’s dumb test of our network, to advertising and marketing, to schmoozing clients, there wasn’t anything I didn’t poke my nose into.

Over time, I noticed that Exodus had a corps of salespeople busy signing all the deals my work was helping support. The salespeople were paid a ton of money (6 percent of every deal’s value) as soon as the deal closed. And they were paid long before Exodus got paid. That was interesting, I thought.

“Everyone lives by selling something.”

- Robert Louis Stevenson

As Exodus grew, it added another layer, in “strategic” sales. The strategic guys got paid another 6 to 10 percent of the deal.  And then Exodus added another layer, the channel salespeople.  They got paid a lot, too. Plus, we paid our channel partners another 10 percent of revenue.  So, at least 22 percent of every deal went to sales, rather than client services. Some sales people were making $1 million a year.

Were they worth it? I’d say yes, absolutely, given the system the company had to work in. I just never thought that system made sense. So why was that?

Because after every deal, I saw the same scenario:  the sales guys and the client’s executives toasted the deal. Meanwhile, off to one side, the project managers from Exodus and the client were saying, “Okay, how are we going to make this thing work?” That was followed by, “Let me tell you what I really need.”

That was when the real work started, and the real headaches too.

“The only thing you got in this world is what you can sell. And the funny thing is that you’re a salesman, and you don’t know that.”

- Arthur Miller

Though the dot.com meltdown put Exodus and many other companies in Chapter 11, it didn’t end the structural problems that complicated making deals.

Exodus’ successors still needed an expensive sales operation to drive the deals that sustained their companies. And their customers still needed reliable, reasonably priced IT services. Yet, between the two sides, they kept signing disastrous deals that hobbled everyone. I kept thinking there had to be a better way.

So in 2000, I created RampRate Sourcing Advisors, to build a more fluid, rational marketplace that simplified the brain-melting process of connecting the customer’s exact needs of customers with the vendor’s exact capabilities and pricing.

“Money demands that you sell, not your weakness to men’s stupidity, but your talent to their reason.”

- Ayn Rand

Over the years since then, the RampRate business model has gone through a few tweaks, but it basically works this way:

1. Clients and RampRate create a detailed set of IT requirements and preferences.
2. We run that information through RampRate’s SPY Index to determine the best vendor. The SPY Index includes the services, prices and other details of more than 350 vendors.
3. Then we create a detailed contract with service-level agreements, graduated non-performance penalties and more. 
4. We save clients time and money, and ensure they are happy with their decision. And we save the vendor time and money since they don’t chase deals that are a bad fit. Finally, we help them quickly negotiate a deal at an appropriate rate. 

“Life’s too short to sell things you don’t believe in.”

- Patrick Dixon

 So, it’s a decade later, and RampRate has saved our clients a ton of money. And 98% of the deals we’ve created actually fit both the customer and the vendor so well that they sustain the relationship through the life of the contract.  They become like a good pair of athletic shoes you hate to throw away. These are deals for the long haul.

 But what’s next? We still haven’t created the perfect fluid, dynamic and efficient IT marketplace for everyone. Entire sectors still do IT deals the old-fashioned way, with dubious sales pitches, unenforceable contracts and unsustainable expectations. We can all do better.

In the coming weeks, let’s discuss 15 myths that fuel this mess, and keep us from a better IT marketplace. Then we can work together to make it better.  

“Sell cheap and tell the truth.”

- Rose Blumkin

“Buying is a profound pleasure.”

- Simone de Beauvoir

But I’ve also been using PC-Windows and Apple Macintoshes since the early 1980s, so I think my views are informed by a pretty wide set of data sources. When I build a desktop PC today, I cherry pick some of my favorite components- including quiet cases, Asus motherboards, Nvdia video cards, Samsung monitors, Logitech mice, etc. I can depend 100% that I can install Windows 7 and everything will not only work, but work in a pretty optimal state. From there, I can do some tuning and get even better performance. For me, relative to the evolution I’ve seen in home computers since the late 1970s, that’s pretty cool. An OS that will work with just about any hardware you throw at it. Sure- the operating system is built of binaries of which I don’t have access to source code.  But even though I’m a power user, I don’t need it.

When Microsoft talks about being “all in” in terms of cloud computing, I get curious to see what they mean. Bob Muglia, head of Microsoft’s server group, recently gave a talk about Microsoft’s cloud computing efforts.

Cloud computing means something different to me than just providing a remotely-hosted operating system with the drivers to support just about any hardware. And it is very difficult to talk about cloud computing without talking about UNIX flavored operating systems- in this case Linux. (I will use Linux as a proxy for all open source variants of UNIX flavored operating systems for the purposes of this article.)

I define cloud computing as a shared hosting service that allows for self and automatic provisioning of resources, total virtualization of the operating system environment, and billing based on per unit consumption.

All of these attributes have been mostly inherent in open source UNIX variants since the beginning of their existence. When I logged onto the ucscb.ucsc.edu server while at college in 1988, the BSD 4.3 environment I saw was, more or less, a complete copy of the environment every user saw when they logged in. I could do word processing, chat, play games, send email, and develop software- as could everyone else. At the time, Microsoft didn’t have anything close to this- something that Muglia talks about in his speech:

 ”One thing I will tell you is that, I’ve been working with providing software to hosters for well over ten years, and we have not done as a company as good a job of providing you software that was designed to run in your environment, and thus you’ve had to do a lot of make work to adapt it to work.”

UNIX variants were built from the beginning to be used for hosting- hosting students, researchers, developers, and multi-user corporate environments. It is natural that Microsoft would need to catch up, and Muglia describes this as well. Muglia went on to say,

“Linux was actually quite a bit better than we were. You know, we recognized this was a deficiency… part of the reason for some of [our] these deficiencies has been that our focus has always been towards enterprise customers, and small business, not so much delivering things as a service, and particularly not so much towards the hosting community.”

So by understanding where Linux has strengths, Microsoft now clearly is in a position to compete. And it is welcome competition.  Microsoft server environments take away the “code maintenance” and “hacking” necessary to run a strong Linux based environment. A good amount of engineers at  companies who run massive Linux server farms such as Google and Akamai focus their time on “tuning the Linux kernel”, something not possible with Windows. The kernel is pre-compiled, and you tune the environment to your needs. This, among many other elements of a Microsoft server environment, is a big time saver. And Microsoft put a lot of effort into tuning their server operating systems for performance.

But performance is just the start. To be a true cloud environment, each user must have a complete virtual instance of the server at their disposal. Again this is new technology for Microsoft, but Muglia absolutely addresses its arrival as the “aspect of the evolution of the datacenter towards a compute  an overall shared compute environment… this as an opportunity to dramatically accelerate the speed of building solutions and delivering solutions into the marketplace.”

Here is where the excitement begins. It would be great to be able to roll out cloud infrastructure without having to be a kernel tuner, hardware hacker, or server architecture guru. This is where Microsoft can add tremendous value to the industry. I can even envision it for the home- to have a single disk-based device accessed by terminals throughout the house, eliminating the need for heavy machines everywhere… yes that means a return to client-server in many ways, except in this case it’s not a server, but a virtual instance of a fully powered client machine.

And in terms of business scaling out, Muglia says, “they want to allow that software to run on behalf of another organization…there’s a whole different way that we think about running this pooled set of resources”

Finally the concept of what we called for years “sub accounts” comes into play. This is what is missing from so much software out there:  the ability to set up sub accounts for business to create sub accounts for their clients to create sub accounts for their users etc… This works great on Linux- and would be great to come to the world of Microsoft.

Overall, I’m excited that Microsoft is throwing its resources behind shared server hosting technology and creating strong competition to the Linux world. Of course I am also well aware that this will create a more complex field of vendors, offerings, and service levels- and require work to decyper and differentiate between potential contracts.

But what excited me most from his talk was some of that classic Microsoft humor, as Muglia used the term “dog fooding” to mean that Microsoft will offer up its own services based on its own software.

The story is a long one, but if you leaf back a bit before Chapter 11, you’ll find misplaced trust highlighted and triple-underlined.  The UAW and big 3 trusted in their eternal entitlement to our pocketbooks and banded together to fight challenges to their complacency such as tougher mileage standards and free trade. Consumers trusted that American ingenuity will find a way as long as we kept buying domestically. Voters cheerfully approved import quotas to mask the stench of stagnation and trusted that if what was good for GM wasn’t good for the country, it was at least good for Detroit. And everyone trusted that the worst thing was for this failure to be exposed and corrected, hence the subsidies, protections and bailouts that continue to this day.

The question is, will our IT world – at least the B-to-B side of it – see the same fate before we see a truly impartial market for it? I hope not, but if we don’t want to see Silicon Valley turn into Detroit, we need to sharpen our instincts and stop trusting IT industry myths. As with the auto industry, these myths sprout on all sides:

• Sellers trust that three layers of schmoozers and regular injections of FUD will keep their cash cow accounts profitable indefinitely, so they can put innovation on the back burner

• CFOs and CEOs trust IT to build a competitive advantage but put sourcing of IT services on par with procurement of paper clips

• Buyers trust in strategic partners, magic bullets, and mystical quadrants despite years of disappointment

And just as with the car industry, the biggest fear is not failure, but being exposed – exposed as having spent too much or hitched your trailer to the wrong vendor or having fallen behind someone who was too small or too foreign to be a real threat. But although some people have trouble remembering it, being fooled once is ok. So in that spirit, I’ll cop to my mistakes first. I entered the IT business in co-location in ’96 with Exodus. Starting from 25 people, we became the darling of industry and analysts once we hit $600 mm and 3,500 people in less than 3 years, to be followed shortly by a spectacular collapse into Chapter 11. Granted, we changed the game and the assets became some of the most important data centers in IT today. I then took another roller coaster ride on the buy side, helping build a forerunner to Hulu and YouTube called the Digital Entertainment Network, which blew $72M in just over a year before imploding. Right concept, wrong time, wrong combination of people. In both places, I saw layers upon layers of inefficiency, waste, and hype centered around how IT was bought, sold, and provisioned.

Why America Loses the King of IT Throne If We Don’t Change

I thought we could do better, which is why I started RampRate, which has lasted longer than the last 2 gigs combined.  So here I am 10 years later with the same pursuit of a fluid trustworthy market. I think we can get our heads on straight and avoid the auto industry’s fate, but we still need to slay some dragons, which is why I’m compiling the master list of IT myths that we’ll need to overcome. If you want to add to the pile, or learn from mistakes of others instead of your own, or, for that matter, tell me why the good times will never end, come on down.

This post was also carried in Huffington Post.

Today’s big format war is a bit more subtle. It’s not so much about “this one or that one” but about “our ecosystem, or nobody’s ecosystem.” The Apple iPhone and iPad do not support web sites designed with Adobe Flash. I have tested extensively on both, and the results are pretty clear- blank pages or broken links.

Why would Apple want to exclude support for a large portion of internet content? It’s very simple: sites that are dependent on Flash are dependent on Flash – which is a technology that is not open source and not under Apple’s control.

Apple attempted to do the same thing with Quicktime back in the early days of computer based video and audio playback. Quicktime was not compatible with the dominant media formats: Windows and Real Media.  Quicktime never became widely adopted, but it did become the underpinnings of the iTunes client.

What is Apple’s goal here? Their goal is that the use of their devices is an “experience.” They want the experience to come from content displayed using open standards and not using technology of which Apple has no influence.

Apple fans don’t seem to mind… for now. But I wonder if the Adobe formats will continue to proliferate and Apple will give in and add support, or maybe if some clever companies will find some way to create an app that transparently encapsulates and runs Flash applications and video on the Apple devices, or if open standards such as HTML5 will take over and Adobe will create a decline.

So in a way, the format war is Apple vs Adobe. This is very interesting because back in the day, those two companies were considered to be Silicon Valley “good guys” who both fought against dark and menacing “Evil Empires” from other states.

Most don’t remember, but the Sony Betamax tape format became, after losing the consumer market, a professional standard that lasted up until relatively recently. I wonder if one Silicon Valley company will wind up catering only to professionals as well.

As a seller of IT managed services, you may spend hundreds of hours on a quote only to find that you really had no chance because the game was rigged. It can be obvious and heavy handed:

• The RFP ignores your strengths, but itemizes your competitor’s advantages;
• You’re given a week whereas others are given a month to respond;
• Any small error is grounds for disqualification rather than a clarification request.

Buy-side bid rigging can also be more subtle and behind the scenes. Sometimes you won’t even know it’s there. Ultimately it harms both clients and vendors, so it’s one of the things we’re training each of our client engagement staff to identify and prevent.

Although we’ve seen this practice in all domains, including data centers, bandwidth, and CDN, it is most frequent and intense in managed services and desktop support because of the importance of cultural fit and the difficulty of measuring it in an RFP. It’s also exacerbated by the prejudices that some IT managers feel with regard to offshore providers, which constitute a good portion of quotes in modern RIM or managed desktop project.

Rigged RFP Perpetrators and Victims

There are three audiences we’ve encountered in this situation:

• As a seller, if you suspect that you’re not given a fair shake, it’s hard to escalate or circumvent the process because you’re viewed as biased. So the most common response is to just sandbag the quote process or even no-bid
• As a CFO or CIO, you may suspect your staff is avoiding change that may materially benefit the company but have no way of verifying it
• As a perpetrator of bid rigging, you might want an external authority to rubberstamp the practice for one of two reasons:
  • It can seem perfectly ethical and legitimate to rig a bid if you believe the “powers that be” are uninformed and misguided.
  • Less appropriate is being part of what economists call the “agency problem” and we informally call “hockey ticket procurement” — using personal favors rather than benefit to the company as the selection criteria. 

Although I’m happy to recommend a DYI solution in most cases, for the first two situations, it’s critical to get an unbiased 3rd party framework that has sufficient domain fit and credibility.  This makes a sourcing advisor useful to both sellers and buyers — providing a neutral ground for evaluation.  When it comes to bid rigging that is fundamentally rooted in trust and ability to explain strategic and cultural value, we can also build a model that will demonstrate to upper management the folly of being penny-wise and pound foolish. 

For the last case, well, you probably shouldn’t call us. The SPY Index doesn’t have a field for sports event tickets and junkets, and objective evaluation is our reputation.

Uptime Metaphor

One of the biggest mistakes that buyers make when outsourcing for the first time is assuming that their internal performance standards (usually tuned to “net” server uptime) are what the provider is promising, whereas most of the time what they’re getting is a “gross” site uptime that excludes pre-scheduled maintenance and force majeure. We covered force majeurein a previous post, but a reasonable estimate of maintenance time and putting boundaries around it is a good part of due diligence for a true 24×7 operation with strong uptime monitoring. Failure to understand this tidbit from the vendor playbook can result in loss of 1%-2% of project revenue due to server unavailability of online ordering or other mission critical services — which can amount to 7 or 8 figure losses for a top-30 online retailer.  It can also affect standard business hours operations, but in a much less severe / quantifiable way.

Maintenance is a frequently overlooked footnote in most hosting contracts.  For a business-hours operation, the biggest problem usually is not too much maintenance, but an overly restrictive schedule — for example only providing a window that requires the payment of overtime to internal IT staff or a time that interferes with critical processes.  These are not typically deal-breakers, but something that works better when mediated up front in the contract negotiation process.

For a 24×7 operation, however, regular, required, and enforced maintenance downtime can be an unpleasant surprise.  As the Uptime Institute notes, Tier III and IV data centers are supposed to be concurrently maintainable.  You will also notice that there are surprisingly few data centers certified as Tier III or IV, but most providers we deal with do actually fall into Tier III or close enough to make no difference to most customers, so the data center itself should not need maintenance that takes equipment offline.  But that doesn’t mean that your vendor won’t find it more convenient to take your site down — and with a blank check written into the contract, it’s much more likely to do so.  Similar considerations apply to patches, network equipment upgrades, etc. — you can theoretically maintain equipment that has some redundancy without taking sites offline, but it’s more cumbersome and resource-intensive.

So if you can question the exclusion of maintenance from your uptime metrics(or other SLAs), you can often save yourself a lot of headaches and potential revenue losses. Better yet, in a round-the-clock operating scenario you should consider multisourcing, with both geographic and vendor diversity that can limit impact of maintenance that can’t be negotiated away.

Justin’s video, “Baby”, on Vevo has 58,012,064 views as of my writing this article.

That’s right. 58 million. And he is just starting out at age 16.

58 million is a big number. To give you a sense of scale, the #1 selling album of all time in the USA is Michael Jackson’s “Thriller” at 29 million copies. This isn’t quite comparable because an album represents a unit of sale of about $10 or so, and a video view is free to a user- but the scale still is pretty amazing.

The internet is well known for being a place of terrifying variable costs for content owners. Looking at this 58 million number, I was reminded how radically things have changed. Let’s look at the numbers.

The “Baby” video is 27.4MB in size. This means, for about 58.012M views, a total of 1,589,531GB of data has been transmitted. This assumes each view is complete and wasn’t cancelled during download.

In today’s pricing environment or $.01/GB for top customers, the cost of 1,589,531GB is a mere $15,895.

That’s right, a mere $15,895.

In the year 2002, at “buck a gig” pricing of $1/GB, this video usage would have cost the content owner $1,589,531. Ah how times have changed.

The video host of “Baby” is Vevo, the YouTube sponsored music video site. One of Vevo’s business model elements is based on advertising.

Video CPMs are approximately $11 for an in-video banner. There are 58,012 “Ms” (thousands of views) of “Baby” so, if fully monetized (which is really an academic view only- who knows the reality of the ad deals and their success?) “Baby” could earn a video host like Vevo $638,133.

That revenue is of course also split with many potential entities such as Google, the rights owners (music companies) etc etc etc… But the point remains the same.

Cost $15,895. Potential revenue $638,133.

Finally the bits make for a business model!

But only if you have a young good looking massively popular artist at your side. Reduce any of these numbers, or increase costs, and it won’t make sense anymore.

The lesson? The internet can indeed work as a business model for content owners… but large scale delivery, popularity, and fully monetized advertising campaigns are needed. Three legs of a barstool that will fall over and toss the singer off the stage if even slightly misaligned.

Force Majeure (less secularly known as “Act of God”) clauses are standard contract language and should be included in your data center checklist. Having your data center wiped out by a meteor strike means you’re not getting service credits and there’s not much you can do about it (there are concessions that can soothe that pain, but that’s a separate topic). However, although the laundry list of what is and isn’t force majeure is almost standard, a frequent trick play is layering in some non-standard exclusions that take away data center risk that the vendor should bear. As with other failures to measure non-performance, an overabundance of exclusions can mean the risk of losses anywhere from $2K in service credits to accepting a 7-figure early exit penalty when you’d much rather terminate for cause… and for free.

As juicy as they sometimes are, we can’t quote the data center contracts we negotiate, but here’s a relatively standard force majeure clause from a public source documenting a relationship between Fusion Telecom and Terremark that we weren’t party to in any way:

Neither Party shall be liable for default if nonperformance is caused by an occurrence beyond Party’s reasonable control (whether, in whole or in part) including, without limitation, fire, flood, acts of God or the public enemy, epidemics, quarantine restrictions, strikes, unusually severe weather, and delays of common carriers or other circumstances or conditions that render it hazardous for a Party’s personnel to travel to or enter onto the affected site; strikes, lockouts and other labor disturbances; acts or omissions of governmental authorities, other telecommunications operators (not due to the fault of the claiming party under this provision) administrators or other competent authorities; military operations; riots and industry wide government codes, ordinances, laws, rules, regulations or restrictions that render performance under this Agreement impossible.

Context-free, this is actually a decent provision. Knowledge of both of these providers makes it somewhat interesting. For example Terremark built its business on having a bunker in Miami specifically designed to withstand “unusually severe weather” and if your data center is in a hurricane-prone area, you might want to think about carving that particular act of God out of the contract. On the other hand, Fusion is a carrier, and, as every other mid-size network, relies heavily on upstream providers rather than its own fiber. If these providers ever drop service to Fusion either accidentally or in a peering dispute, the clients are out of luck. But these are comparatively minor nitpicks.

For a more egregious abuse of the force majeure leniency, we can turn to an example from Yale. It’s a perfectly generic sentence that would give me all kinds of uneasiness if left in a data center hosting contract

<liability doesn’t apply for reasons> including, without limitation, failure of suppliers, subcontractors, and carriers, or party to substantially meet its performance obligations.

So, say you just signed a major data center contract / lease with a real estate company and you’re trying to hit a committed install deadline. But turns out the “supplier” of the UPS equipment was unable to expedite. Oops, too bad, we’re not responsible for the blown deadline. And if the operating company (a “subcontractor”) turns out to be unable to operate the equipment properly and you experience outages, well, they’re a subcontractor, so that’s an exclusion. And then if you bought some network access via your landlord, you’re locked out of carrier screw-ups as well. The items on your data center checklist are growing.

All of the above examples are more errors of ignorance than intentional wrongdoing — some terms may be ok in a generic contract but underestimate the responsibility an IT service provider bears for managing upstream resources such as power and network. We have, however, also seen an addition that considers the connection of every single piece of equipment not explicitly approved by the vendor in writing to serve as an exclusion for all future SLAs on the entirety of the infrastructure. In other words, plug in a new server, and you have zero SLA leverage. That’s something that’s much more likely to be an intentional trick play designed to get separation between the vendor and its liability for failures, and is something your data center checklist should cover.