> OpenAI would naturally optimize for the tests published by Marcus as a critique of GPT-2
It would be difficult for them to do so since Marcus's GPT2 critique came out after they collected the dataset for GPT3.
Marcus's article: Jan 2020
GPT-3 dataset: "Table 2.2 shows the final mixture of datasets that we used in training. The CommonCrawl data was downloaded from 41 shards of monthly CommonCrawl covering 2016 to 2019"
It would be difficult for them to do so since Marcus's GPT2 critique came out after they collected the dataset for GPT3.
Marcus's article: Jan 2020
GPT-3 dataset: "Table 2.2 shows the final mixture of datasets that we used in training. The CommonCrawl data was downloaded from 41 shards of monthly CommonCrawl covering 2016 to 2019"