Data Portal @ linkeddatafragments.org

Matches in ESWC 2020 for { ?s ?p ++++ COMMENTS AFTER REVIEW ++++ While I still hold some reserves, the authors' response has been clear and convincing on some of the points raised by me. Additionally, the required changes do not require a full revision of the work but only clarifications. I thus revise my evaluation to "weak accept". ++++ INITIAL REVIEW ++++ The paper reports on the latest evolutions and improvements in the evaluation campaign OAEI, introducing in particular the latest gold standards for the recently added Knowledge Graph track and discussing the results of a hidden task – an evaluation of a run of matchers over datasets with no-overlapping domains - the that they performed over the last competing systems of the initiative. The paper is well and clearly written, it introduces the initiative for those who are new to it, briefly updates the reader on the latest additions to the evaluation and discusses the benchmarks, the way they have been built and the challenges characterizing the task as much as its evaluation I have a doubt about the trustworthiness of the Gold Standards. If the GS 2019 revealed so many matches, then there are a lot of them missing from GS 2018. In particular, why, if the models have been always matched by experts (i.e., even in 2019, as claimed by the authors), the two dataset pairs that are present in both GSs (i.e. memoryalpha-memorybeta and memoryalpha-stexpanded) have so different results? Have they simply been improved in 2019? However, if GS 2019 shows 4 and 7 trivial matches (respectively, for the two pairs, i.e. 14 total – 10 = 4 and 13 – 6 = 7) for them, why these slipped out of the attention of the crowdworkers on such a small number of elements? Additionally, if the number of matched instances with the link method is decently reliable (at least as an order of magnitude), how can the poor numbers of negative matches from 2018 be of any support in the evaluation? (as in section 3.1 it is said that only them have been adopted for the precision). I’m guessing they are mostly negative trivial matches, thus avoiding a phenomenon which might be smaller; however I’m not so sure of that and I don’t think it is understandable from this data. As these corpora of matches are used for the evaluation, noting such large differences across years raises some doubts about how much these can be called “gold standards” Later on, in the golden hammer bias section, in order to assess how much a 50-sample is reliable for the golden hammer, a statistical test of confidence should have been carried on, repeating experiments with different 50-matches samples on the same systems and checking their variation. The observations on the golden hammer bias are also interesting, but it should be, again, statistically assessed how much those numbers in the domain-overlapping case are not a result of those overfitting matchers, yet a purely proportional result of a matcher that “throws out some results” in an ocean populated with many positive matches versus one where there are not. I’m ambivalent, on the one side I value the importance of publicly disseminating on the results of such a renowned initiative such as OAEI, on the other side we must critically address flaws in the initiative itself or on its related dissemination, where the results might be questionable and the way to analyze them not as complete as it could be. I’d really suggest, for future investigation, to collaborate with some expert in statistics, to understand the statistical significance of the results. Numbers are important, but their meaning (as we all know working in the field of semantics) is even more. TYPOs. * the reference to table 3.2 in pag. 6 is actually table 4 MINOR REMARKS: The concept of trivial match (which can – rightly, as the reviewer is aware of it – be guessed from the end of the paragraph where “same names” are mentioned) should be clarified to the reader, as they might not be aware of it.". }

• Paper.62_Review.0 hasContent "++++ COMMENTS AFTER REVIEW ++++ While I still hold some reserves, the authors' response has been clear and convincing on some of the points raised by me. Additionally, the required changes do not require a full revision of the work but only clarifications. I thus revise my evaluation to "weak accept". ++++ INITIAL REVIEW ++++ The paper reports on the latest evolutions and improvements in the evaluation campaign OAEI, introducing in particular the latest gold standards for the recently added Knowledge Graph track and discussing the results of a hidden task – an evaluation of a run of matchers over datasets with no-overlapping domains - the that they performed over the last competing systems of the initiative. The paper is well and clearly written, it introduces the initiative for those who are new to it, briefly updates the reader on the latest additions to the evaluation and discusses the benchmarks, the way they have been built and the challenges characterizing the task as much as its evaluation I have a doubt about the trustworthiness of the Gold Standards. If the GS 2019 revealed so many matches, then there are a lot of them missing from GS 2018. In particular, why, if the models have been always matched by experts (i.e., even in 2019, as claimed by the authors), the two dataset pairs that are present in both GSs (i.e. memoryalpha-memorybeta and memoryalpha-stexpanded) have so different results? Have they simply been improved in 2019? However, if GS 2019 shows 4 and 7 trivial matches (respectively, for the two pairs, i.e. 14 total – 10 = 4 and 13 – 6 = 7) for them, why these slipped out of the attention of the crowdworkers on such a small number of elements? Additionally, if the number of matched instances with the link method is decently reliable (at least as an order of magnitude), how can the poor numbers of negative matches from 2018 be of any support in the evaluation? (as in section 3.1 it is said that only them have been adopted for the precision). I’m guessing they are mostly negative trivial matches, thus avoiding a phenomenon which might be smaller; however I’m not so sure of that and I don’t think it is understandable from this data. As these corpora of matches are used for the evaluation, noting such large differences across years raises some doubts about how much these can be called “gold standards” Later on, in the golden hammer bias section, in order to assess how much a 50-sample is reliable for the golden hammer, a statistical test of confidence should have been carried on, repeating experiments with different 50-matches samples on the same systems and checking their variation. The observations on the golden hammer bias are also interesting, but it should be, again, statistically assessed how much those numbers in the domain-overlapping case are not a result of those overfitting matchers, yet a purely proportional result of a matcher that “throws out some results” in an ocean populated with many positive matches versus one where there are not. I’m ambivalent, on the one side I value the importance of publicly disseminating on the results of such a renowned initiative such as OAEI, on the other side we must critically address flaws in the initiative itself or on its related dissemination, where the results might be questionable and the way to analyze them not as complete as it could be. I’d really suggest, for future investigation, to collaborate with some expert in statistics, to understand the statistical significance of the results. Numbers are important, but their meaning (as we all know working in the field of semantics) is even more. TYPOs. * the reference to table 3.2 in pag. 6 is actually table 4 MINOR REMARKS: The concept of trivial match (which can – rightly, as the reviewer is aware of it – be guessed from the end of the paragraph where “same names” are mentioned) should be clarified to the reader, as they might not be aware of it."".