A Cyclic Cascaded CRFs Model for Opinion Targets Identification Based on Rules and Statistics

. Opinion sentences on e-commerce platform, microblog and forum contain lots of emotional information. And opinion targets identification plays an import role in huge potential commercial value mining, especially in sales decision making and development trend forecasting. Traditional CRFs-based method has achieved a pretty good result to a certain extent. However, its discovery ability of out-of-vocabulary words and optimization of the mining model are both insufficient. We propose a novel cyclic cascaded CRFs model for opinion targets identification which incorporates rule-based and statistic-based methods. The approach acquires candidate opinion targets through part-of-speech, syntactic and semantic rules, and integrates them in a cyclic cascaded CRFs model for the accurate opinion targets identification. Experimental results on COAE2014 dataset show the outperformance of this method.


Introduction
With the development of the Internet, social platform has gradually integrated into people's lives, resulting in the increasing expansion of mass information. More and more opinion sentences on the Internet are generating. For the government, business or individual, the study of these opinion words is of great significance. Compared with regular grammar and news text, opinion sentences on social plat-form are more colloquial, interactive, and also contain a large number of advertisements and junk information. These bring new challenge to opinion targets identification, and how to effectively extract the useful information has become more and more important. Sentiment analysis, also called opinion mining, is to process, induce and infer the subjective texts [1]. Sentimental elements extraction is the basis of sentiment analysis. Sentimental elements extraction is to extract the opinion elements in the sentence, including opinion words (such as "好"), opinion targets (such as "三星手机"), opinion holder (such as "张三" in the sentence "张三认为……"). In this paper, we mainly study opinion targets identification.
Traditional CRFs-based method has achieved a pretty good result to a certain extent.
However, its discovery ability of out-of-vocabulary words and optimization of the mining model are both insufficient. We propose a novel cyclic cascaded CRFs model for opinion targets identification which incorporates rule-based and statistic-based methods. The approach acquires candidate opinion targets through part-of-speech, syntactic and semantic rules, and integrates them in a cyclic cascaded CRFs model for the accurate opinion targets identification. Existing opinion targets identification methods cannot comprehensively discovery the out-of-vocabulary words, and do not optimize the mining model. To address these shortcomings, it is intuitive to consider the combination of rule-based and statisticbased methods, and at the same time take special features of opinion sentences on social platform into consideration. In this paper, we propose a novel cyclic cascaded CRFs model for opinion targets identification which incorporates rule-based and statistic-based methods. The approach acquires candidate opinion targets through partof-speech, syntactic and semantic rules, and integrates them in a cyclic cascaded CRFs model for the accurate opinion targets identification. In experiments on the COAE 2014 dataset we find that our method can substantially extract opinion targets more effectively under different evaluation metrics.

Related Work
The methods of opinion targets extraction are mainly divided into two categories: unsupervised and supervised methods. In the unsupervised methods, Hu and Liu [2] used association rules to excavate opinion targets and regarded the top-frequency words as opinion targets. Li and Zhou [3]extracted tuples like <emotional words, opinion targets> based on emotional and topic-related lexicons. Popescu and Nguyen [4] extracted properties of products with mutual information. Yao [5] used domain ontology to extract the topics and their attributes from a sentence, and summed up the subject-predicate structure rules based on syntactic analysis for opinion targets identification. Liu [6] used syntactic analysis to obtain the candidates , and then combined PMI with noun pruning algorithm to decide the final opinion targets. Besides, in the supervised methods, Zhuang [7] proposed a multi-knowledge-based approach which integrated WordNet, statistical analysis and movie knowledge. Jakob [8] modelled the task as a sequence labelling question and employed CRFs for opinion targets extraction. Wang [9] proposed a method of opinion targets identification based on CRFs, and selected morphology, dependency, relative position and semantics as features.
However, existing opinion targets extraction methods only took lexical-related features into account. Consequently, considering the specific features of Chinese microblog, we propose a new method for opinion targets extraction towards microblog using syntax and semantics in which we adopt a new approach of PDSP for domain lexicon construction and select groups of features for CRFs.

Candidate Opinion Targets Identification Based on Rules
The task of candidate opinion targets identification is automatically extracting the opinion targets using rule-based methods. Considering the importance of syntax and semantics in opinion targets identification, we propose a method of candidate opinion targets identification which incorporates POS, dependency structure and semantic role. Opinion targets are usually nouns or noun phrases. Through statistics on corpus, we design six templates based on Part-of-Speech which are shown in Table 1 where n, adj, adv, aw,cmp and OTrepresents for noun,adjective, degree adverb, advocating word, comparative word and opinion target. Here we get adv and aw from Hownet , and acquire cmp from [10].  Table 1.

Part-of-Speech sequence templates
As we all know, when we express opinions towards a product, we need some opinion words which usually have strong semantic relation with the opinion targets. Therefore, we collect opinion words from Hownet and NTUSD 1 andperform HIT-LTP 2 for dependency parsingto discuss the relation "ATT" and "SBV" between opinion words and opinion targets，relation "COO" between already known opinion targets and unknown opinion targets.
As a necessary part of shallow semantic parsing, sematic role [11] occupies an important position in lexical and semantic analysis. People usually express opinions through opinion words in opinion sentences. And adjective andverb are two main forms of opinion words. Through investigation, we find when the opinion word are adjective, A0(agent) is opinion target. Furthermore, when the opinion word are verb, A1(patient) is opinion target.

CCCRFs: A Cyclic Cascaded CRFs Model for Opinion Targets Identification
In this paper, we propose a novel cyclic cascaded CRFs model for opinion targets identification which incorporates rule-based and statistic-based methods. The approach first acquires candidate opinion targets through method in 3. And then, this model adopts two-layer cascaded CRFs. In the first layer, we select opinion words and manual features for opinion words identification. And in the second layer, the outputs of the first layer are added as input and we select opinion words, candidate opinion targets and manual features for opinion targets identification. In each iteration, we choose sentences whose confidence value is larger than C as training data. And the remaining sentences are regarded as testing data.

Cascaded CRFs Model
CRFs(Conditional Random Fields, CRFs) is proposed by Lafferty [12] in 2001. Its chain structure is shown in Figure 1. Given a set of input random observed variables, this conditional probability distribution model can generate another set of implicit output random variables by training the model. CRFs are often used for sequence labelling tasks, such as part-ofspeech tagging, named entity recognition and so on.
However, in the actual labelling process, we find it exists nesting phenomenon. For example, in Chinese named entity recognition, other named entities elements, such as names, places will be included in organization name. And this situation has led to incorrect identification. To solve this problem, you need to refine these tasks, step by step and gradually completed. In this paper, we adopt CCRFs(Cascaded Conditional Random Fields) as shown in Figure 2 to solve the above problems.
CCRFs reduced the coupling relationship between different layers of the model. Each layer of the model can be built independently, and each sub task can be done independently without interfering with each other. The complexity has linear relationship with the number of the model layers. Before the high-level model, some of the necessary pre-treatment can be carried out in the output of the underlying model and filter some errors. Consequently, CCRFs can avoid error propagation and diffusion and further improve the performance.

Cyclic Model
The selection of training data is always a focus of machine learning methods. In order to improve the recall rate of this method, we add cyclic method under CCRFs through screening each experimental results. Figure 3 is the flowchart of Cyclic Cascaded CRFs Model. If the confidence C is greater than threshold, we add them into training corpus circularly. If not, we treat them as testing data. Then, this model loops as this until the iteration number reaching a certain value N. In cyclic model, for every opinion targets identification results, its confidence degrees C, can be calculated as follows.
1 is the confidence value of first layer in CCRF, and 2 is the confidence value of second layer in CCRFs. These can be acquired from CRFs tool.
We selected sentences whose confidence value larger than M into training set, and the remainder continued as the testing data. Then we re-trained the model, and extract the results for N times iteration to get the final results.

Feature Selection
In feature selection, we refer to the features which are employed by Jakob [8] and Lu [13] in English and meanwhile put forward some new features based on the specific grammar of Chinese. Generally, we think opinion targets extraction is primarily related with four kinds of features which are named as lexical features, dependency features, relative position features and semantic features. As words with the same Part-of-Speech usually appear around the opinion targets, we select the current word itself and the POS of current word as lexical features. Dependency parsing reflects the semantic dependency relations between core word and its subsidiaries words [14]. Consequently, we select whether the dependency between current word and core word exists, the dependency type, parent word and the POS of parent word as the dependency features. As we all know, since words which appear around emotional words are more likely to be opinion targets, we determine the boolean value by judging whether the distance between current word and emotional word is less than 5. Considering there is a strong relationship between the sematic roles and POS of emotional words, we select the sematic role name of current word and POS of emotional word in this sentence for CRFs.

Experiments and Analysis
In experiments, we firstly obtained candidate opinion targets through method in section 3, and then we employed CCCRFs with the candidate opinion targets and features in section 4 together to extract opinion targets.

5.1.Dataset
Through filtration, we finally obtain 5,000 normalized sentences with opinion orientation from COAE2014. And we perform segmentation, part-of-speech, syntactic parsing and semantic role labelling through LTP [15].In this paper, we conduct experiments on such a dataset and assess it with traditional Precision, Recall and Fmeasure under strict and lenient evaluations which respectively represents the extraction result is exactly the same or overlapped with the labelled one.

Parameter selection
In this section, we make comparing experiment on parameter selection of cyclic cascaded CRFs model. And the experiment results between loop number N and confidence value C in cyclic cascaded CRFs model are in Figure 4.

Fig.4.Comparing experiments between loop number and confidence value in cyclic cascaded
CRFs model From this figure we find that, it performs best when the confidence value C=0.9 and loop number N=4. We also find the curve is slowly tend to be stable under four confidence value C with the increase of loop number N. As the confidence value C increases, opinion sentences have been added to the training corpus completely and the training corpus is no longer increased, so the model tends to a stable state. Consequently, this experiment not only demonstrated the effectiveness of cyclic cascaded CRFs model, but also revealed the importance of parameter selection to opinion targets identification.

Comparing results with different methods of opinion targets identification
In this section, we compare different methods of opinion targets identification, rulebased method, CRFs-based method, cascaded CRFs-based method and cyclic cascaded CRFs-based method. To verify the effectiveness of cyclic cascaded CRFs model for opinion targets identification, we conduct experiments under different methods of opinion targets identification and obtain the experimental results as shown in Table 2. And the rule, CRFs, CCRFs and CCCRFs respectively represent for rule-based method, CRFs-based method, cascaded CRFs-based method and cyclic cascaded CRFs-based method.

Method
Strict It can be seen that the effect of opinion targets identification is highly improved after adopting cyclic cascaded CRFs model, which is mainly because this method not only uses candidate opinion targets identification in section 3to obtain candidate opinion targets, but also adopts machine learning method of CCCRFs to make up for the defect of rule-based method and so as to reach a higher precision, recall and F-measure. So this experiment strongly demonstrates the effectiveness and applicability of cyclic cascaded CRFs model.

Conclusions and Future Work
In this paper we propose a cyclic cascaded CRFs model for opinion targets identification which takes rules and statistics into consideration. We combine the candidate opinion targets extraction into a cyclic cascaded CRFs model to get the final opinion targets. The experimental results show that it performs better than other baseline approaches.
In the future work, we will take the following points into consideration: -Considering the various expressions of Chinese microblog, we should excavate more rules and extract the kernel sentence for opinion targets extraction.
-In this paper, we perform opinion targets extraction on sentence level. We will investigate the effect of opinion targets extraction on corpus level.