In most web-based projects I am currently involved in, one of the main challenges is the translation between user requirements/perception and software. One of the reasons is probably the fact that we are mainly in the business of social software, which is rather new and not well-defined. Every different project has different expectations, and uses a different concept.
As a matter of fact, we are working on completely different communities at the same moment, using the same software. The way users are looking at the community, is very different though. The translation between a specific user requirement and a generic feature in our software is extremely important to us.
I recently noticed the website www.utest.com , which pays Internet users to find bugs in software. At first sight, this looked like a perfect concept. Developers are not end-users, and outsourcing parts of the testing would help developers focus on the software. However, I see 2 reasons why we cannot use this concept internally at this moment. First of all, we have our internal non-interactive tests, that keep at least the algorithmic part of the code under control. These tests should somehow be connected to the external users tests. Apart from that, and more important, is the fact that it is not just the feedback from "a" user that is important. We need the feedback from the target group.
Different social networks attract different people. What some people consider an "annoying bug" in one network, might be considered a "security feature" in another network.
Nevertheless, I think that using (and paying) the Internet population as a test-group is a very good idea for many software products.
Johan's blog 