Socrata

Most of us, especially in the software industry, are data driven by nature. Given that propensity, I’m surprised by how few applications are purposely instrumented to provide real, hard data about how they are functioning.

This is one area where Microsoft does a very good job. Most Microsoft applications generate “Watson” data – information captured whenever an application has a problem. Subject to your personal privacy settings, that information is sent over the web to Microsoft so it can be analyzed, which hopefully results in the application being improved.

At blist, we’re doing similar things. We instrument the application to tell us things that we could potentially just ask you voice to voice. But we can produce better, more comprehensive and accurate answers simply by measuring it from within the application.

For example, we measure whenever anyone starts to import a CSV file but abandons the process before completion. We now know with certainty that we have more work to do in this area because we’re seeing that an unacceptable and unhealthy percentage of CSV imports are abandoned. Is the process too slow? Are we failing to provide enough feedback as the import progresses? Are we choking on big files? Are we having problems with the data in some kinds of files? These are the kinds of questions that can best be answered by instrumenting the application to tell us with certainty. And that’s exactly what we are doing. Look for the CSV import process to improve over the upcoming weeks as we analyze the operational data our application generates.

12 or 15 years ago a boss of mine gave me some investing advice. He suggested that as I was an early evaluator of technology, the things that I was choosing to buy for my company were likely the things that other people in similar roles at other companies would eventually choose to buy. If I chose a product that was produced by a publicly traded company whose stock price didn’t yet reflect these anticipated sales, it might be a good idea to buy the company’s stock. I heeded this advice successfully over the years.

Last night my wife and I were at a charity auction held to benefit my kindergartener’s elementary school. I think I observed an interesting social indicator. At these events, there’s both a silent auction and a live auction. For the silent auction, an item is placed on a table for viewing and if you’re interested, you fill in your bidder number into the next available bid slot. The bid sheet discloses the real value of the product (as you can only declare a charitable deduction for the amount in excess of the value of the item). Typically the minimum bid is somewhere around 25% of the value and often the winning bidder pays well above the real value (this is, after all, a charity event to help the school raise funds for playground equipment, field trips and teaching specialists). Most people don’t mind overpaying for a good cause, not to mention there’s plenty of wine flowing to get people into the spirit and loosen up their checkbooks.

The interesting phenomena I observed is that two items had no bidders – tickets to a Seattle Sonics game and a retail copy of Windows Vista. Hmm. What does that say? Nobody wants these items even at 25% of their real value! For the Sonics tickets, I think most of us have written the Sonics off. We’ve resigned to the idea that they’re moving to Oklahoma. Nobody bidding on Vista actually surprised me. The retail value was either $499 or $599 and not one person had bid at the minimum level of $50.

Here we are in Seattle, in Microsoft’s own stomping grounds, and no one wants Vista. Ouch. I think it’s time to sell my Microsoft stock.

I’m one of those folks who likes to check references for prospective employees. Yesterday was a perfect example of the ideal situation. On a scale of 1 to 10, I was at about an 8.5 on the enthusiasm scale when considering extending an offer to one candidate. Then I called his references. Wow. Talk about off the charts raving fans. These weren’t the “Yeah, Pete’s a great guy” kind of comments. I interview references. I ask questions that require specificity. After 3 reference calls my enthusiasm for hiring the candidate was at 9.5 or 10. That’s what great references can do.

Earlier in the week I had just the opposite experience. I was at about an 8 in enthusiasm for an engineering candidate. I called two references. One was pretty good, but there were a lot of caveats in his testimony. Then I called another reference and he provided quite a bit of detail about how this software engineer had struggled within their team. We decided to pass on extending an offer.

The reality is that even in a controlled interview loop with four members of our team, it’s impossible to know for certain. References can provide more color and insight than an interview alone can provide. Hiring good people is such an important element of success, I’m surprised how many people opt not to invest in checking references.

When you go to a fairly academic conference, it’s frowned upon to award a best in show. Yesterday I attended the Hadoop Summit and expected to hear all the cool stuff Yahoo and Powerset were doing with it. By far, however, the runaway winner for “best use of Hadoop” in my book goes to Facebook. Joydeep Sen Sarma and Asish Thusoo gave a talk on a project called Hive that helps the analysts and engineers at Facebook grok their clickstream and logfile data. Good geeks are, well, geeks. I know many of them. What really impressed me about these two gentlemen and the Hive project was just how business driven it is.

Joydeep started his talk by saying “We asked our current BI [business intelligence] users what tools they could and couldn’t use and they told us they know how to use SQL.” So often technologists forget about their audience. Hive was developed iteratively by a 2 or 3 person team (I think Jeff Hammerbacher was also involved) making it easy for business analysts to ask ad hoc questions of terabytes worth of logfile data by abstracting MapReduce into a SQL like dialect. Think of it as a data warehouse sitting on top of thousands of servers’ logfiles. Beneath the surface Hive leverages Hadoop and translates SQL-like imperatives into MapReduce jobs. It’s really a great use of technology. My highest compliments to the Facebook team for their work in this area.

I’d also like to commend IBM Research for their work on JAQL. It’s essentially a query interface into a JSON data store. It’s really intriguing. Conceptually I love JAQL and think it could be extremely useful. I have concerns about it coming from IBM Research and how open its open source license will be once it gets through IBM legal.

The Hadoop Summit was a great day long event attended by about 400 folks interested in Internet scale computing. It was a pleasant surprise to learn that folks outside of those whom I expected are doing really interesting and innovative work.

I have a passion for the operation of systems at Internet scale. Part of that passion is academic. Part is pragmatic. Facebook, Google, Yahoo and Amazon each have differing approaches to building services that scale to support tens of millions of users and billions of objects. I look forward to working at that scale again with blist.

Today I’m out of the office, in the field so to speak. I’m attending the Hadoop Summit in Santa Clara with about 225 other folks who are interested enough in distributed computing to give up a day from their normal routines. It should be a great day to learn how other people are using and/or evaluating Hadoop. As much as I’ll enjoy and benefit from that, I’m as eager to meet other folks interested in solving challenges at Internet scale.

If you’re at the Hadoop Summit, track me down and say hello.

Happy Easter everyone.

We all think of Easter eggs as the wonderfully decorated eggs and candy or prize filled plastic eggs hunted by kids of all ages on Easter.

Easter eggs have been a part of the software industry for a long time too. Like the annual pastime of hunting in the yard or park for the hidden orbs, software Easter eggs are hidden within applications by developers. In the 80’s software was dry and almost all business. Shrink wrapped and tightly following rigid specifications, there was little room for artistic creativity. Software engineers exercised some creativity by embedding Easter eggs within applications. Often the business folks are unaware of the hidden elements.

So what is an Easter egg in the context of a software application? It varies widely but often it’s some kind of attribution to the development team. Remember how there was always a likeness of Alfred Hitchcock in his movies? Often that’s the spirit of an Easter egg. Sometimes it’s an embedded picture of the development team. Sometimes it’s an animated movie credit scrolling by. Sometimes it was much more – like an entire game hidden within a business application. In the Office 97 suite, Excel had a hidden flight simulator; Word had a hidden pinball game; and Access had a hidden Magic 8 ball game. Excel 95 had a hidden game called “The Hall of Tortured Souls” which was a clone of Doom.

Finding the hidden eggs was deliberately obfuscated. For example, to launch the Doom clone in Excel 95 you needed to open a new, empty worksheet, select row 95, press the[tab] key, then select Help –> About from the menu, then press and hold [ctrl] and [shift] while clicking on Tech Support button.

Thursday we presented at Under the Radar in Mountain View. It was a lot of fun. I didn’t see all of the company presentations due to my own preparation and there being two separate, parallel tracks.

The two startup company presentations I liked the most were SlideRocket and SlideShare. I liked them for different reasons. Here’s my take on each.

SlideRocket is a really impressive presentation creation tool. SlideRocket is to PowerPoint as blist is to Access. That is to say that SlideRocket is a web-based, beautifully designed, updated version of PowerPoint. Like blist, SlideRocket is a rich Internet application (RIA) built on Adobe Flex. As much as I like what’s there already in the private beta version, I love the vision founder/CEO Mitch Grasso has for where SlideRocket is going. Leveraging its position “in the cloud,” two things really set SlideRocket apart: 1) it has a pluggable architecture that allows the content on a slide to be dynamic. Imagine a table created in blist being live and dynamic on a slide inside a SlideRocket presentation. 2) SlideRocket understands that presentations are often used for marketing and has built in analytics. You can easily determine how many people have viewed the presentation, where people are abandoning, how long they’re spending on each slide, etc. It’s really useful data. SlideRocket recently closed their first funding round, from Hummer Winblad. It’s a really promising application. Congratulations to Mitch and team.

SlideShare, while sounding similar to SlideRocket, takes a very different approach. Where SlideRocket’s emphasis is on content creation, SlideShare focuses on presentation sharing and distribution and a community around presentations. SlideShare users create presentations in PowerPoint or other presentation creation tools that allow you to save as PDF (for example, you could upload Visio docs if you want). SlideShare is a great place to share, discover and discuss presentations that matter. I was unbelievably impressed by SlideShare CEO Rashmi Sinha. She’s smart, articulate and full of passion and energy. SlideShare has demonstrated strong traction since launching in late 2006. One of their investor/advisers is Dave McClure, one of the smartest Internet marketers around (warning: Dave could exercise better constraint over his language. Sometimes its tough to reconcile his intelligence with his potty mouth ).

To show you how cool SlideShare is, here’s the blist Under the Radar presentation:

| View | Upload your own

Keep an eye on both of these promising startups. I look forward to integration between blist and SlideRocket and also look forward to getting to know and sharing experiences with Ramshi at SlideShare. You should check each of them out.

I’ll be on stage at Under the Radar this afternoon between 3:00 and 3:30 at the conference center at Microsoft’s Mountain View campus. I’ve got another 6 minutes to show people what blist is all about. I did the rehearsal yesterday. The format of Under the Radar is a little like American Idol. I get up and sing for 6 minutes. Then a 3 or 4 person panel tells me how I did in front of a conference hall full of attendees.

Besides continuing to convey our message that blist is the world’s easiest database and the first online social database, I have a personal goal for my presentation today. After our launch at DEMO one commenter who saw the video of my presentation commented on ReadWriteWeb’s write-up of blist “That is one stiff fella.” So today, I’ll try to loosen up a bit and have some fun.

And after I’m done, I’m looking forward to Werner Vogels‘ fireside chat on the future of apps in the cloud. Werner’s an informative, engaging and entertaining speaker. I doubt anyone’s ever called him one stiff fella.

From time to time I’ll meet someone working for one of the big, established software companies. Let’s just call them BigCo. The individual will tell me how they’re not *really* working for BigCo, but rather a startup inside. They go on to tell me how it’s really an entrepreneurial team and that it’s just like working at a startup. The ones who usually say that have never worked at a startup before. What they mean is that their team is working on a new project and maybe at the outset has decided to try to be less bureaucratic than other teams at BigCo.

Like many readers of this blog, I’ve worked in both environments. Literally, I’ve been employee #1 at a startup and employee #277251 at BigCo. So I thought I’d take a moment to share some of the differences between working at a startup and working on a new project at BigCo:

*) At a startup, you know your first few dozen customers by name. They usually know your cell phone number too.

*) At a startup, you build what customers want, not what your boss wants.

*) At a startup, your customers invite you to their weddings.

*) At a startup, you know which of your teammates like mayonnaise on their hamburger. And which don’t.

*) At a startup, you personally feel the joy of each happy customer.

*) At a startup, you know that capitalization is about money, not punctuation.

*) At a startup, you become addicted to the sound of the bell being rung when a new customer is landed.

*) At a startup, you decide who joins the team.

*) At a startup, you decide the tools being used.

*) At a startup, you decide if you want a Mac or a PC.

*) At a startup, when you call IT the line is always busy. Finally you realize you’re dialing your own desk.

*) At a startup, there are no reorgs.

*) At a startup, you man the booth at the tradeshow.

*) At a startup, you do the demo during a sales call.

*) At a startup, the crappy legacy code you have to fix is your own.

*) At a startup, a spec is a discussion. And maybe a sketch on the whiteboard.

*) At a startup, the outcome of a meeting is a decision.

*) At a startup, you find any reason to celebrate.

*) At a startup, there’s no sag wagon. You patch your own flats.

*) If you’re lucky, at the end of the road at a startup, you feel like you accomplished something truly remarkable. Then you’re ruined for life and you can’t work for BigCo ever again.

We all know how important is to have a good reputation. When it comes to engaging a service provider, finding a doctor or hiring an employee, there’s no substitute for another person’s recommendation. Of course the recommendations we receive are influenced by the reputations of the recommender as well.

All this leads me to opinion. Have one. Make it honest. Make it count. Your reputation stems in part from the quality and accuracy of the opinions you offer about the reputations of others.

There’s a startup here in Seattle which once flew high but now flies much lower. They raised a lot of money, spent it, retooled a number of times and now are trying to find their way again. Coincidentally I know two men who previously worked at this startup. From time to time I’ll run into a candidate, now interested in blist, who also worked at the startup. I’ll do what I normally do and I’ll see who I know who knows the candidate. I’ll ask these two their opinions of the candidate. One is frank, sincere, genuine and supports his opinion with supporting anecdotes and information. The other offers the same bland position for each candidate. “Oh, he was a great guy. You’d love him. Fun in the office. Great guy.”

The candid opinions shared by the one man has boosted his reputation in my mind. I trust his recommendations. The other has diluted his own reputation by offering only vague opinions.

When someone asks for your opinion, consider the impact on your own reputation by providing as thoughtful an opinion as you can.