Running a beta and getting feedback

[Mike Morris]

Hi all...I'm going to be releasing a beta version of a new database related product fairly soon and wanted to get some feedback as to what tools/mechanisms people use to make it very easy for beta users to submit bug reports and/or feedback in general. From my viewpoint, it would need to support the following:

1) Ability to wire it into the code.... trap unhandled exceptions and present user with form to submit via web or email (with stack trace and other info automatically included)

2) Allow users to effortlessly submit general comments

3) Be fairly easy to implement and not cost prohibitive

Thanks!

Mike

[Mark Dalgarno]

I'd give the beta testers some guidance on the type of feedback you're looking for. If this is your first attempt at beta testing then I'd recommend keeping it as simple and as easy for them to use as possible and for you to manage. You should be guided by how much and what type of feedback you can actually use. Are you testing with 1, 10s or 1000s of users? What parts of the product do you want them to focus on and from what aspects?

If I'm a beta tester and not that familiar with you or your company I might not be that keen to have auto-generated web or email transactions created as in criteria 1. In this case just giving the user some text to paste into an email would be a good low-tech approach and would meet criteria 2 and 3, it would probably also be easier for you to implement => quicker time-to-market :-)

[Mike Morris]

Thanks for the feedback Mark. I would anticipate it would be in the 10s range. My main focus will be to gather information around bugs/exceptions as opposed to feature requests. Your point around making it simpler and quicker time-to-market is a very good one.

[Gustavo Orrillo]

I am in the process of starting a beta test program and our approach is to start small. We are helping the beta testers with documentation, tutorials and (many times) a list of tasks we expect them to perform in order to test the site. We are starting with 10 testers and growing 10 testers each week. We don't have the resources to manage hundreds or thousands of testers today. We also reach to them one week after they started testing the tool via e-mail or phone asking them about their experience with the tool.

Hope this information helps, if you have further questions please let me know and if you feel like testing a code-generator tool you will be more than welcome

Gustavo

[Rachel Clements]

Hi Mike,

We've written code into both beta and retail versions of our software to catch unhandled exceptions and allow the user to email them to us. Aware of customer privacy, we give them the option of adding in their email address. It's a great way of knowing about those bugs you never find in your test environment :)

We also give beta testers a short document to complete (we try not to overwhelm them and take up too much time or the process becomes a time-consuming chore for them), with a mixed style of questions: some easy ones rating 1 to 5 of how easy was the product to install, how do you rate the UI etc - questions where a simple grading is all that is necessary. And then a few questions asking them if they used the help file, certain features and if they have any suggestions of general feedback.

Rachel :)

[Mike Morris]

Gustavo & Rachel,

Thanks for the great feedback. These are all things that I'm going to really consider as I approach the beta...! Now I just need to drive the final beta bits home and get it out there. Thanks again.

Mike

[Sam Ng]

We trialled Fog Bugz, and I have to say I'm now a convert. It's much better than anything else we had earlier. It will help with points 2 and 3 but I'm not sure it handles point 1 (wiring into code).

[Atle Iversen]

We're in the middle of a public beta ourselves, and this is our (simple) approach:

- we catch unhandled exceptions, and generate an e-mail with details about the error (and details about our product, like version number); the user may then review the information, and choose if they want to send the e-mail or not

- we want reports via e-mail, as this is easy for both us and the testers.

The method you choose depends on the feedback you want. We've already been through an extensive private beta to remove most 'obvious' bugs, and therefore use a simple method for our public beta as we don't expect too many problems (hopefully :-).

We don't want TOO many testers, but new quality testers are always welcome :-)