Opinion Marks: A Human-Based Computation Approach to Instill Structure into Unstructured Text on the Web

Abstract

Despite recent improvements in various computational approaches such as machine learning, natural language processing, and computational linguistics, making a computer understand unstructured, human-generated text still remains a difficult problem to solve. To alleviate the challenges, we propose an approach called "Opinion Marks," which enables writers to mark positive and negative aspects of a topic on their own text. In addition, Opinion Marks incorporates an automatic marking suggestion algorithm to offload a user's marking effort. The phrases marked with Opinion Marks can be further used to clarify sentiments of other text in the similar context. We implemented Opinion Marks on a question answering website http://caniask.net. To test the efficacy of Opinion Marks, we conducted a crowdsourcing-based experiment with 144 participants in a between-subject design under the three different conditions: 1) human marking only; 2) machine marking only (automatic marking suggestion); and 3) human-machine collaboration (Opinion Marks). This study revealed that Opinion Marks significantly improves the quality of marked phrases and usability of the system.

Publication
KDD Workshop on Interactive Data Exploration and Analytics (IDEA)
Date