nlp中What is Oracle experiment?

迈不过友情╰ 2022-05-15 04:34 147阅读 0赞

转：[https://stackoverflow.com/questions/10542937/what-is-oracle-experiment][https_stackoverflow.com_questions_10542937_what-is-oracle-experiment]

**Question:**

I have read a paper about machine learning and it contains an Oracle experiment to compare between his study and another study? But it does not seem to be so clear what is Oracle experiment?

**Answer：**

An "oracle" is an imaginary entity that always gives the right answer. An oracle experiment is used to compare your actual system to how your system would behave if some component of it always did the right thing.

For example, in the NLP domain, let's assume you built a parser that takes part-of-speech (POS) tagged sentences as input. In the real world, you would have to run real sentences through an actual POS tagger. This tagger would probably produce results with accuracy above 90%, but less than 100%. Since the accuracy of your parser depends on the accuracy of the incoming tags, your parser's performance will be negatively affected by this loss.

In order to see how well your parser would perform *if the POS tagger was perfect*, you could run an experiment with an oracle tagger. In this experiment, you would replace the real POS tagger with a program that knows the actual POS tags for the sentences, thus always returning tag results with 100% accuracy.

So, if your parser gets 85% accuracy in an experiment with a real tagger, and 90% in an experiment with an oracle tagger, then you know that 5% of your performance loss is directly due to the mistakes of the tagger.

**总结(不一句一句翻译)：**

如果有系统依赖别的组件，为了方便评估，需要提供依赖神组件（oracle），能100%给出正确答案。这样提供依赖组件100%正确的情况和非100%正确真实组件的情况比较，从而对系统本身做出评估。

[https_stackoverflow.com_questions_10542937_what-is-oracle-experiment]: https://stackoverflow.com/questions/10542937/what-is-oracle-experiment