I think it is going to be complementary. Where the Prediction API drops off, then it looks like you could crowd source better predictions and analysis through Kaggle. MLComp might help those trying to use Google’s service evaluate and chose the best performing learning technique. It could potentially be used on the same data to make that judgment not just on Google’s algorithms but others as well.
I also had a thought about how it might compare to Wolfram Alpha. Alpha may have superior learning techniques but its data sets are closed. Wolfram might do well to open up submission of data and be open enough to allow comparison through something like MLComp. That is assuming Wolfram himself is justified in his views of how much more advanced his technology is than anyone else’s.