Matches in ESWC 2020 for { ?s ?p I acknowledge the rebuttal answers by the authors, though I am not fully happy about those. I am still not fully convinced about the relevance of this paper for thee track. I also think that evaluation results are not fully convincing and mature. These considerations preserve my initial score of weak accept. ------------------- This paper presents the results of an evaluation of the CoModIDE tool for graphical modular ontology engineering in an experiment with 21 subjects carrying out some typical (simple) modeling tasks. Having noted that, I believe that the paper does not fully fit into the scope of Ontology and Reasoning track of ESWC. Overall, the paper is well written and structured. The methodology used in the presented research is mature and well presented. the data collected in the exaluation exersize is presented and discussed in fully sufficient detail. I cannot however say that all the results, as interpreted and presented in the discussion, including some generalizations are well supported and convincing. My major concerns are: - Chosen evaluation measures and the size of the subjects pool: The paper states that the tools were compared based on the task completion time (with the threshold of 20 min) and the correctness of the subjects' outputs. while reading Section 5.2, I wondered why completion time has at all been selected as a measure. It was not a quick chess tournament, I believed. A reasonably convincing argument needs to be given in support of this experiment design choice. Also, why 20 minutes point has been chosen as a threshold? I got the answer to my completion time question only two pages later - in Fig. 5. Still, it has not been made clear, why a 20 minute time slot to completion. As for Completeness, I feared that this approach is too simplistic and coarse, as presented. Furthermore, I felt that there was a high risk that the results would have been under-representative having quite a small pool of subjects (21 people). Generalization: In the conclusion to Section 6.2, the authors state that they "... believe that our results are generalizable, due to the strength of the statistical signicance". I think that this statement is pre-mature. A larger-scale experiment, e.g. via croud-sourcing, is needed before you may state any generalization of a statistical sort. In my opinion, the pool of subjects is too small to be considered statistically representative. Some comments to free text responses and future work: Graph layout: This is so common for all diagram editors, that you might have taken it for granted before asking. The reason for the use of the force-directed graph layout in VOWL is, most probably, the way to build better layouts. This is not perfect - as you described in thee related work section. However, having nothing for improving the layout is not good. Future work: The presentation in Section 7.1 is the future work on the tool, but not on its evaluation. However, the paper focuses on the evaluation. So,iss this (evaluation) line of research fully accomplished? There are several imperfections in the text and figures in terms if readability, reference numbering, and English. Examples: - References need to have consecutive numbering in the order of their citation appearance in the text. Hence, [17] cannot be the first citation. This is one of the indicataor of a quick reshuffling of a previously written manuscript, btw. - P.2: "we have formulated for the following hypotheses" - remove "for" - P.4: "... provides an a graphical overview of an ontology's structure ..." - to be: a graphical overview of an ontology structure - P.4: "We also invite the reader to download and install CoModIDE themselves ..." - I would remove "themselves" - Fig. 1: text in the figure is not readable - Acronyms: acronyms have to be given in full at the place of their first appearance. Please check OPLa in p.5. Does OPLa stand for an Office of the Principal Legal Advisor? - P.7: "...the time taken to complete each tasks" - each task - P.8: "... we provided a 10 minute tutorial ..." - I would rather call it a briefing. You do not teach, but put people into the context of the activity and instruct. - P.9: "... quick and dirty ..." - please avoid emotionals and colloquials - Fig. 5: Please make H1 and H2 in one tense. To conclude, I believe that the paper presents ongoing research and better fits into a relevant workshop as a work-in-progress report.". }
Showing items 1 to 1 of
1
with 100 items per page.
- Paper.111_Review.0 hasContent "I acknowledge the rebuttal answers by the authors, though I am not fully happy about those. I am still not fully convinced about the relevance of this paper for thee track. I also think that evaluation results are not fully convincing and mature. These considerations preserve my initial score of weak accept. ------------------- This paper presents the results of an evaluation of the CoModIDE tool for graphical modular ontology engineering in an experiment with 21 subjects carrying out some typical (simple) modeling tasks. Having noted that, I believe that the paper does not fully fit into the scope of Ontology and Reasoning track of ESWC. Overall, the paper is well written and structured. The methodology used in the presented research is mature and well presented. the data collected in the exaluation exersize is presented and discussed in fully sufficient detail. I cannot however say that all the results, as interpreted and presented in the discussion, including some generalizations are well supported and convincing. My major concerns are: - Chosen evaluation measures and the size of the subjects pool: The paper states that the tools were compared based on the task completion time (with the threshold of 20 min) and the correctness of the subjects' outputs. while reading Section 5.2, I wondered why completion time has at all been selected as a measure. It was not a quick chess tournament, I believed. A reasonably convincing argument needs to be given in support of this experiment design choice. Also, why 20 minutes point has been chosen as a threshold? I got the answer to my completion time question only two pages later - in Fig. 5. Still, it has not been made clear, why a 20 minute time slot to completion. As for Completeness, I feared that this approach is too simplistic and coarse, as presented. Furthermore, I felt that there was a high risk that the results would have been under-representative having quite a small pool of subjects (21 people). Generalization: In the conclusion to Section 6.2, the authors state that they "... believe that our results are generalizable, due to the strength of the statistical signicance". I think that this statement is pre-mature. A larger-scale experiment, e.g. via croud-sourcing, is needed before you may state any generalization of a statistical sort. In my opinion, the pool of subjects is too small to be considered statistically representative. Some comments to free text responses and future work: Graph layout: This is so common for all diagram editors, that you might have taken it for granted before asking. The reason for the use of the force-directed graph layout in VOWL is, most probably, the way to build better layouts. This is not perfect - as you described in thee related work section. However, having nothing for improving the layout is not good. Future work: The presentation in Section 7.1 is the future work on the tool, but not on its evaluation. However, the paper focuses on the evaluation. So,iss this (evaluation) line of research fully accomplished? There are several imperfections in the text and figures in terms if readability, reference numbering, and English. Examples: - References need to have consecutive numbering in the order of their citation appearance in the text. Hence, [17] cannot be the first citation. This is one of the indicataor of a quick reshuffling of a previously written manuscript, btw. - P.2: "we have formulated for the following hypotheses" - remove "for" - P.4: "... provides an a graphical overview of an ontology's structure ..." - to be: a graphical overview of an ontology structure - P.4: "We also invite the reader to download and install CoModIDE themselves ..." - I would remove "themselves" - Fig. 1: text in the figure is not readable - Acronyms: acronyms have to be given in full at the place of their first appearance. Please check OPLa in p.5. Does OPLa stand for an Office of the Principal Legal Advisor? - P.7: "...the time taken to complete each tasks" - each task - P.8: "... we provided a 10 minute tutorial ..." - I would rather call it a briefing. You do not teach, but put people into the context of the activity and instruct. - P.9: "... quick and dirty ..." - please avoid emotionals and colloquials - Fig. 5: Please make H1 and H2 in one tense. To conclude, I believe that the paper presents ongoing research and better fits into a relevant workshop as a work-in-progress report."".