Imagine how much less would get done overall and how many fewer people would participate if we did not let people work on what they wanted to work on.

The only choice left would be to contribute or not and more people would choose not to contribute (probably the choice you have made).