Iterative Data Augmentation with Large Language Models for Aspect-based Sentiment Analysis

Starting from an unsupervised corpus $U$, we first attempt to extract the aspects relevant to a specific domain. Specifically, we carefully design an aspect extraction (denoted as “EX”) prompt¹¹1Due to the space limitations, we present the detailed prompts of IterD in Appendix A.4. to enforce the LLM to automatically extract domain-related aspects for each sentence $S_{i}\in U$. After doing that, we deduplicate the aspects and obtain the initial aspect set $A$. Considering that aspects are generally nouns and their variants, we perform the part-of-speech processing with a Python library Textblob²²2https://pypi.org/project/textblob/ on all candidate aspects of $A$ to remove those that are difficult to accurately generate the samples. Then, to further improve the diversity of extracted aspects, we introduce an aspect extension module to expand $A$. In particular, for the Noun aspects in $A$, we enforce the LLM to expand them with their homonyms and synonyms by an aspect extension (denoted as “ET”) prompt. Lastly, the extend aspect set is merged into $A$. Moreover, for better generating the sentiment-aware data, we split the $A$ into three sub-sets with different sentiment polarities, i.e., positive aspects $A_{pos}$, negative aspects $A_{neg}$, neutral aspects $A_{neu}$, by performing a word sentiment analysis on each aspect.

After obtaining the domain-related aspects, we then generate the pseudo labeled data, i.e., triplet $\{S_{i},T_{i},y_{i}\}$. Specifically, for each aspect sub-set, we append the aspects with their corresponding sentiment polarities to construct the aspect-sentiment set. For instance, for the aspect in $A_{pos}$, we append it with the positive polarity. Consequently, we can basically design a prompt to guide the data generation of LLMs based on the aspect-sentiment set. However, during the preliminary experiments, we found that as the generation of LLMs continued, LLMs suffer from the problem of repetitive generation, i.e., the generated samples tend to be similar and low-diversity. Hence, we propose a more powerful Iteration Teaching Analysis Prompt (denoted as “ITAT”), which randomly selects samples from each round of generating samples as feedback to guide the next-round generation. By doing so, ITAT can prompt the LLMs to generate more richer and diverse pseudo triplet data.

Inspired by prior studies Wang et al. (2022), we recognize that multi-aspect data, i.e., data with multiple aspects in a sentence, is greatly beneficial to the training of ABSA models. To this end, in addition to the vanilla single-aspect pseudo data generation, we further utilize a mix-aspect pseudo data generation branch to obtain the more complex yet effective multi-aspect data. To have a close look, we provide the illustrations of single-aspect/mix-aspect pseudo data generation in Figure 2.

Despite the powerful capability of LLMs, they might generate unexpectedly low-quality data, hindering the performance of DA. Thus, it is critical to evaluate the quality of generated data and filter the lower-quality one. To achieve this goal, we introduce a new discriminator, as illustrated in Figure 3, containing a judgment module and an auto-scoring mechanism. Specifically, in the judgment module, we employ the popular LLM-as-a-Judge method to enforce the LLM to determine the domain relevance and sentiment relevance of generated data. That is, LLM is utilized to verify whether the generated data is relevant to the given domain and sentiment. After filtering the data with lower domain relevance and sentiment relevance, we further use the auto-scoring mechanism to quantitatively measure the data quality, in terms of Syntactic Structure, Lexical Richness, and Real Scenario Conformity. The scoring mechanism takes a sample judgment on a scale of 1-10, where larger scores mean higher data quality. For filtering the low-quality data, we set a filtering threshold³³3The analysis of $\mathcal{T}$ can be found in §4.3$\mathcal{T}$. The data exceeding the threshold is used as final training data, while the others are discarded. Notably, for promoting the aforementioned ITAT strategies, we use the high-quality generated data as the feedback.

We conduct the main experiments on 4 widely-used ABSA benchmarks, i.e., Laptop14, Restaurant14 Pontiki et al. (2014), Restaurant15 Pontiki et al. (2015) and Restaurant16 Pontiki et al. (2016). Following Tang et al. (2019), we remove a few instances with conflicting sentiment polarity. For the evaluation of aspects extracted by our IterD, we use “Precision” (P), “Recall” (R) and “Macro-F1” (F₁) as the metrics, while the “Accuracy” (Acc) and F₁ score are used to evaluate the final ABSA models. The details of all used datasets can be found in Appendix A.1.

For simulating the real-world scenarios, we use the unlabeled sentences in training sets of the above ABSA benchmarks (i.e., ignoring the aspect and polarity information) as the initial unsupervised corpus for our IterD. The aspects of the original training sets are used as gold labels to evaluate our extracted aspects. After obtaining the augmented ABSA data, we train the models with these data and evaluate them on the test sets of the above benchmarks. Specifically, we use the powerful GPT-3.5-turbo ⁴⁴4https://platform.openai.com/docs/models/gpt-3-5-turbo as the LLM in our IterD. For each benchmark, we enforce IterD to generate the ABSA data, the number of which is similar to that of the original training set.

To investigate the effectiveness of our IterD, we mainly apply it to improve 5 representative baseline ABSA models, i.e., ATAE-LSTM Wang et al. (2016), ASGCN Zhang et al. (2019), and BERT-SPC Song et al. (2019), R-GAT Wang et al. (2020) and KGAN Zhong et al. (2022). For each model, we utilize the BERT-base-uncased ⁵⁵5https://huggingface.co/google-bert/bert-base-uncased as the backbone and train it following the default settings in the original papers. Due to the space limitations, we present the details of all baseline models in Appendix A.2.

We conduct the main results in 3 different settings, i.e., 1) “Original data”: training the ABSA models with the original labeled ABSA data, 2) “Generated data”: training with only the synthetic data generated by our IterD and 3) “Mixed data”: training with the mix of original data and our generated data. We additionally compare IterD with several cutting-edge DA methods, including Back-Translation (BT) Sennrich et al. (2016), EDA Wei and Zou (2019), CBERT Wu et al. (2019) and C3DA Wang et al. (2022). The detailed descriptions of these compared DA methods can be found in Appendix A.3.

As mentioned in §3, we expand the aspect set to improve its diversity. Here, to verify its effectiveness, we compare IterD with a simple alternative, “-w/o Extension”, i.e., removing the aspect extension module. The contrastive results are shown in Table 4. It can be seen that removing the aspect extension causes clear performance degradation, indicating the effectiveness of aspect extension.

In the sample generation phase of IterD, we use two different strategies, i.e., single-aspect and mix-aspect generation. Specifically, the latter strategy is to simulate the multi-aspect problem Wang et al. (2022) in ABSA. Notably, for a fair comparison, we generate the same number of training data for both strategies and present the compared results in Table 5. As seen, by generating more multi-aspect data, IterD brings consistent and significant performance gains against the vanilla single-aspect data. This is similar to the findings of Wang et al. (2022), as training on multi-aspect data can encourage the models to extract more fine-grained aspect-specific information, thus leading to better performance.

Moreover, in the preliminary experiments, we empirically found that IterD falls short in generating single-aspect data with neutral sentiment polarity. However, IterD can effectively generate the correct data of multiple aspects with the same neutral polarity during multi-aspect data generation. One possible reason is that LLMs struggle to distinguish neutral emotions, which is also found by prior empirical studies Zhong et al. (2023). In Table, we further report the results of adding the corrected neutral multi-aspect data, denoted as “Multi_Neu”. Obviously, these neural training data can further boost the ABSA performance effectively.

In our IterD, we introduce a discriminator to filter the low-quality generated data. Here, we verify its effectiveness and report the contrastive results in Table 6. Compared to vanilla IterD, i.e., directly using the generated data without filtering, IterD with the discriminator achieves much better performance. This highlights the importance of filtering the low-quality data, and indicates that data quality is more important than the data quantity for the field of ABSA.

The $\mathcal{T}$, which is used to control the threshold for filtering data, is an important hyper-parameter in IterD. Here, we analyze its influence by evaluating the performance with different $\mathcal{T}$, spanning {0, 2, 4, 6, 8}. Figure 4 illustrates the contrastive results of R-GAT on Laptop14. With the increasing of $\mathcal{T}$ in a certain range (i.e., 0 to 6), IterD continues achieving better performance. This indicates that filtering low-quality data is beneficial. Conversely, too large $\mathcal{T}$ values (e.g., 8) lead to performance degradation, as filtering too much data might lead to limited available data for training. More specifically, $\mathcal{T}=6$ performs best, thus leaving as the default setting.

Intuitively, based on more accurate aspects, IterD can generate more relevant training data and bring more performance gains. To verify it, we use the gold aspects in the original training sets as the upper bound to guide the generation of IterD. The contrastive results are illustrated in Figure 5, from which we find that IterD with gold aspects indeed achieves much better results. This indicates that the performance of IterD relies on the accuracy of extracted aspects and more accurate aspects can result in better performance.

In the iterative generation module of IterD, we use a few-shot prompt to guide the sample generation of LLMs. Here, we compare it with a zero-shot prompt, i.e., removing the labeled examples in the prompt, and show the results in Figure 6. As seen, comparing the zero-shot prompt, IterD with the few-shot prompt achieves better and more stable performance, indicating that adding some examples in the generation prompt is beneficial to generate more high-quality data.

Here, we investigate the number of training data generated by IterD. Specifically, let $R$ be the number ratio of generated data relative to that of original training data, and we evaluate the performance of IterD with different $R$ ranging from 50% to 250%. Figure 7 illustrates the contrastive results of R-GAT on Laptop14 and Restaurant14 benchmarks. It can be found that the performance on both datasets shows a rising, falling, and then rising trend. With the increase in the amount of generated data, there will inevitably be more noisy samples in the generated data, which leads to performance degradation. However, with the generation of more reliable and stable quality samples, IterD brings performance improvements again. In general, these results show that more generated data does not always lead to better performance, i.e., data quality is more important than quantity.