Submitted by markbirbeck on Thu, 11/27/2008 - 21:29.
When considering how to test the Hubbub beta, we faced a not uncommon dilemma.
On the one hand we were tempted to keep Hubbub a secret, and just let a few close friends and colleagues play with it; that way we felt it might be possible to control whether the world said nice or nasty things about our baby. This is what is usually called a private beta.
But on the other hand, we felt that we had something interesting to show, and we were pretty certain that there would be people out there who agreed. They would likely be happy to tell us about the bugs they had found, and perhaps more importantly, give us suggestions for the kinds of features that an open, social-networking framework for the desktop, should support. This approach would usually be called a public beta.
Backwards and forwards we went, one minute fearful of the barbs of bloggers, the next looking forward to a flood of constructive criticism that would send us on our way to the next round of features.
After much deliberation over whether to have a public beta or a private one, we decided on a public private beta.
Of course that sounds like a contradiction, so I'll explain.
Public private beta
Usually a private beta is where you ask people to test software that isn't ready for prime-time, and which in fact may change a lot (due to users' suggestions). But the people that you ask to join such a beta are almost certainly going to be people that you already know, because you hope that they are going to give their criticism to you, rather then sending it out in waves across the blogoshpere.
Of course we wanted to have such a beta, since we wanted that friendly feedback.
But we also had a strange moment of enlightenment, when we realised that it really made little difference whether we actually knew the people who were involved in such a beta. After all, the key thing was whether those involved in the beta were interested enough in what we were doing to spare some time, and give us their suggestions.
And as we thought about it more, we realised that in the age of Twitter, this was exactly the right approach to take. After all, with services like FriendFeed and Twitter, you might have hundreds of people following you, whom you have never met, but who would still gladly help you to improve your products, or to make progress in some project.
To put it a different way; with a traditional beta you would rely on the goodwill of your friends and colleagues to test your software. But nowadays you will almost certainly have many sources of goodwill coming at you.
Once we'd reached this point, it seemed the only approach to take was to say that if someone wanted to join the beta, then we'd be happy to have them.
Of course, it was difficult to shake those feelings that this might be a high risk strategy, and that people would treat Hubbub as if it had just launched, and call open season.
But we decided to trust the audience, and it turns out we need never have worried.
Crashes and spam
Within hours of the Hubbub beta being announced on TechCrunch we were alerted to the fact that a crucial file that we had assumed was present on all Windows environments actually wasn't (so causing a crash for some people).
A new installer was promptly issued.
Also, we kept receiving puzzling reports that users with Hotmail or Yahoo! email addresses weren't receiving their sign-up messages. After much investigation it transpired that there is no way to be sure that email coming directly from an Amazon EC2 instance (which we use for all of our sites), is not rejected as spam by some services.
This problem is a little trickier to solve, and has required a number of steps, but the last one should be in place soon. (UPDATE: This is now fixed.)
No barbs?
Of course we don't imagine that we'll get a completely smooth ride, and there is still much to do.
But at the end of the first day of our public private beta, with two serious problems spotted (and nearly resolved), it feels like we made the right choice to trust in the goodwill of the community.
