I've taken a very tiny first step towards something like this for computer graphics and vision: trying to make user studies as easy as possible: http://www.imcompadre.com It's not a ready product by any means, but two paper submissions have been made with it so far.
It's a difficult problem to solve, because these pesky researchers are always trying out new things that you didn't anticipate - who would've thought! But still, for the mundane things that can be automated, something like this is definitely the way to go. Of course, as other people here point out, figuring out what to actually test is always the hardest part.
It's a difficult problem to solve, because these pesky researchers are always trying out new things that you didn't anticipate - who would've thought! But still, for the mundane things that can be automated, something like this is definitely the way to go. Of course, as other people here point out, figuring out what to actually test is always the hardest part.