Financial Knowledge Large Language Model

Artificial intelligence (AI), particularly Natural Language Processing (NLP) technologies, has significantly transformed the finance industry. These technologies streamline complex tasks, automate customer services, and enhance decision-making processes. Large language models (LLMs), such as GPT-4, represent a cutting edge in this evolution, offering extensive capabilities that can analyze and interpret vast amounts of unstructured financial data quickly and effectively.

Despite their potential, the practical application of LLMs in finance faces substantial challenges. One of the primary concerns is their tendency for "hallucination," where the model might generate plausible but factually incorrect information. This is particularly problematic in finance, where accuracy and reliability are paramount. Additionally, the high cost of continuously updating these models to keep pace with the rapidly changing financial landscape poses another significant hurdle.

Moreover, there is an ongoing challenge in enhancing LLMs with factual knowledge. In finance, the demand for precise and up-to-date information is critical, and the static nature of trained models often clashes with the dynamic financial environment. To address these issues, the development of hybrid models that combine the generative power of LLMs with real-time, verified data sources could be a potential solution. This integration would aim to leverage the strengths of AI in understanding and processing language while ensuring the accuracy and reliability required in financial applications.

In this thesis, we aim at building a trustworthy LLM in finance area, which is enhanced by factual knowledge.

In Chapter 3, we introduce IDEA-FinBench, a novel benchmark for assessing financial knowledge in LLMs by leveraging questions from two internationally recognized and esteemed financial professional examinations. These questions are presented in both Chinese and English, employ four distinct question formats, and cover sixteen financial disciplines. This comprehensive array allows for an in-depth evaluation of LLMs’ proficiency in directly responding to finance-related examination questions. Furthermore, IDEA-FinBench incorporates a modular evaluation suite that supports the integration of external datasets, offering flexibility in customizing evaluation methods and interfacing with various LLMs. This feature enhances the adaptability and scalability of the evaluation framework.

In Chapter 4, we present IDEA-FinKER, a Financial Knowledge Enhancement Framework, aimed at facilitating the swift adaptation of general LLMs to specialized financial contexts at a reduced cost, eliminating the need for extensive external pre-training. IDEA-FinKER is supported by a meticulously curated and extensive database of Chinese financial examination questions and features an embedding similarity retrieval system. It underpins the development of a retrieval-based few-shot learning approach, termed the soft-injecting paradigm, for real-time contextual knowledge enhancement. Moreover, IDEA-FinKER introduces a structured set of financial knowledge instructions for fine-tuning general LLMs, described as the hard-injecting paradigm. Empirical results indicate that IDEA-FinKER markedly improves the expert-level capabilities of LLMs in the financial domain, significantly boosting their performance on IDEA-FinBench, particularly concerning Chinese financial examination questions such as the CPA.

In Chapter 5, we introduce IDEA-FinQA, a dynamic financial question-answering system powered by LLMs. IDEA-FinQA operates under a real-time knowledge injection and factual enhancement paradigm, utilizing external knowledge bases. The system consists of three primary modules: the data collector, which is tasked with gathering and amalgamating data from the financial domain through both online and offline methods and data storage solutions; the data querying module, which implements search functionalities using both traditional text-based and contemporary embedding-based indexing systems for various recall and ranking stages; and four specialized LLM-based agents comprising a query rewriter, intention detector, extractor and refiner, and a response generator. Each agent is designed to perform specific tasks within different prompts and contexts, thereby driving the effectiveness of the IDEA-FinQA system.

The main contributions of of this thesis are:

1. 1.

   A benchmarking tool that evaluates the financial knowledge of LLMs using questions from prestigious global financial exams in both Chinese and English across sixteen disciplines, featuring a modular suite for flexible customization and scalability.

2. 2.

   A framework that enhances the rapid adaptation of LLMs to the financial sector through a comprehensive database of financial exam questions, supporting both soft and hard knowledge injection paradigms, leading to significant performance improvements in domain-specific applications.

3. 3.

   A financial question-answering system driven by LLMs, utilizing real-time knowledge injection and supporting various data collection and querying methodologies, structured around four specialized LLM-based agents for optimized task-specific responses.

Natural Language Generation (NLG) tasks, such as machine translation, text summarization, question answering, text completion, and others, which not only demand a comprehensive understanding of natural languages, but also an additional downstream module is necessary to handle text generation. Sequence-to-sequence (seq2seq) [2] learning has emerged as an exceedingly popular solution, defining the Encoder-Decoder as its foundational architecture. Here, the encoder part takes a sequence as input and maps it to a latent space to obtain a continuous representation, serving as the initial state for the decoder module, which generates words sequentially. As pioneering efforts, Recurrent Neural Networks (RNN) and their variations, such as LSTM (Long Short-Term Memory) [3] and GRU (Gated Recurrent Unit) [4], have been demonstrated to achieve tremendous success in NLG tasks, especially in the realm of machine translations. The attention mechanism was also initially introduced to seq2seq, designed to guide each word to adjust the weights of its attention across different words in the sequence through a scoring module [5]. However, due to difficulties in parallelizing the model and limitations imposed by the hidden state that cap the network’s ability to process long sequences of information and uncontrollable gradient variations during training, the Transformer model [6] made a striking entrance. Its unique self-attention mechanism not only introduces an exceptionally performant parallelism to the training and inference processes but also makes capturing long-range dependencies in sequences more feasible through positional encoding. Figure 2.1 shows an architecture of Transformer.

As Transformer models continue to unlock their seemingly boundless potential for performance, the traditional paradigm of fully supervised learning is increasingly challenged by the pre-training and fine-tuning approach. Pre-trained Language Models (PLMs) adjust their vast number of parameters through unsupervised training on large-scale corpora, thereby incorporating the patterns and knowledge embedded within vast amounts of natural language, which are then applied to downstream tasks. By deconstructing the transformer model and selecting different pre-training tasks, PLMs are not confined to a singular Encoder-Decoder architecture.

Alignment in the context of LLMs refers to the process and the goal of making these models understand and follow human intentions, ethics, and instructions as closely as possible [13, 14, 15]. The concept of alignment is centered around ensuring that the outputs of LLMs are not only relevant and informative but also adhere to the moral and social norms that govern human discourse. This involves training models to avoid generating harmful, biased, or misleading content while enhancing their ability to produce responses that are truthful, appropriate, and aligned with the explicit instructions or goals set by their users [16]. Some pioneering work [12] has already attempted to enhance the alignment of language models with human intentions by utilizing a two-step process that combines Supervised Fine-Tuning (SFT) with Reinforcement Learning from Human Feedback (RLHF) [17], leveraging human-written prompts and feedback to finely adjust its responses, which is shown in Figure 2.4.

The excellent performance of PLMs in knowledge-intensive tasks such as open-domain question answering and fact verification [54, 55] suggests their significant potential in supplanting traditional knowledge bases, and in offering high-quality logical reasoning as a complement to structured queries [56, 57, 58]. Current state-of-the-art models like GPT-4 [28] have demonstrated prowess beyond the general public and even human experts in popular benchmarks for common sense question answering and reasoning [46, 59]. However, tasks involving factual knowledge often cast doubt on the reliability of these language models, especially due to their propensity for producing hallucinations [60, 61] or inaccuracies in time-sensitive [62, 63] or domain-specific contexts, such as the medical, law, and the financial fields we have discussed in Section 2.4.

In summary, we initially introduced the Transformer Model, the currently most prevalent deep neural network architecture. Particularly, we discussed PLMs based on the Transformer’s encoder and decoder, covering some pioneering pre-training methodologies. As the scale of parameters expands and computational power increases, the emergence of intelligent language models capable of responding to prompts necessitates significant attention towards alignment. Returning to the core focus of our thesis, the LLMs in the financial sector, we reviewed open-source contributions and benchmarks. Another focal point is the trustworthiness of LLMs, especially those evaluated around factual knowledge and evaluation suites. In the next chapter, we will concentrate on constructing trustworthy LLMs within the financial sector to ensure fact-based questioning and reasoning.

Currently, generative pre-trained language models (GPLMs) have achieved exceptional results on a variety of standard benchmarks for natural language understanding and generation, reaching performances that rival the average human level [28]. This success is attributed to their extensive knowledge base and robust logical reasoning capabilities. Such evaluations often focus on skills traditionally believed to require advanced cognitive abilities and thought to be exclusive to humans, including logic, mathematics, coding, physics, and causal reasoning [46, 48, 47]. These are considered prime choices for objectively assessing the potential intelligence level of large language models, which are increasingly approximating the capabilities of General Artificial Intelligence (AGI). However, in the financial sector, there remains a gap and a lack of comprehensive benchmarks. For instance, assessments of financial LLMs are still limited to traditional NLU tasks, such as entity recognition, sentiment analysis, and event extraction [50, 26, 83, 39]. These measures are inadequate for evaluating the models’ abilities to perform high-level office tasks within the financial industry. Moreover, while some studies have recognized the potential for language models to perform intelligent tasks in finance, they have been hindered by a narrow focus on specific scenarios, causing a lack of generalizability, such as in numerical reasoning [51, 52] and investment decision-making [34].

Wall Street, the epitome of top-tier financial firms, shows a strong preference for professionals holding the CFA certification, highlighting the importance of specialized training and a deep reserve of financial knowledge and problem-solving skills, which is shown in Figure 3.1. In light of this, our benchmark evaluations aim to include the CFA certification to measure LLMs’ capabilities in the financial sector. Despite the advanced abilities of current models like GPT-4, previous tests indicate they fall short of the CFA certification requirements [85]. Therefore, integrating these prestigious financial certification exams into our benchmarks is crucial to thoroughly assess LLMs’ proficiency in finance.

We introduce IDEA-FinBench ¹¹1The IDEA-FinBench data and evaluation code are available at https://github.com/IDEA-FinAI/IDEAFinBench., a benchmark that incorporates authoritative financial professional certification exams, including the Chinese Certified Public Accountant (CPA) and the international CFA exams. This benchmark spans both Chinese and English languages, four types of question formats, and sixteen financial subjects, ensuring a comprehensive assessment of LLMs’ abilities to apply financial knowledge and solve real-world financial industry problems. Building upon the foundation of prior work on C-Eval [49], IDEA-FinBench offers the community a modular evaluation suite for assessing LLMs on multiple-choice question datasets. Our suite focuses on adaptability and scalability, supporting customization of reasoning modes and interface coupling for various LLMs. It includes a parallel mechanism for accelerated evaluation, a complete logging system, and cross-linguistic prompt settings.

We undertake a comprehensive integration of the various LLMs currently prevalent in the field, which encompasses both those that have open-sourced their model parameters and those that merely offer access to their model interfaces. By establishing a standardized and equitable inferencing framework, we have been able to derive a unified, quantifiable score. This score serves as a direct benchmark for assessing the LLMs’ proficiency in grasping financial knowledge and their capacity to apply this understanding in reasoning and problem-solving contexts.

Our contributions can be summarized below:

1. 1.

   Establishment of a Benchmark for Financial Knowledge Assessment in LLMs: We introduced IDEA-FinBench, a fair benchmarking system for evaluating the intelligence level of LLMs regarding financial knowledge. Moreover, we have publicly disclosed the scores of popular LLMs on this benchmark, providing a transparent and standardized metric for comparison.

2. 2.

   Development of an Adaptable and Scalable Evaluation Suite: Our evaluation suite not only includes customization of reasoning modes and interface coupling for various LLMs, but also supports a parallel mechanism for accelerated evaluation, a comprehensive logging system, and cross-linguistic prompt settings.

In the following chapters, we will introduce the experimental settings of IDEA-FinBench, covering the evaluation setup, the model list, and the experimental results.

In Table 3.3, we present the comprehensive performance of 21 LLMs on IDEA-FinBench, evaluated by averaging the accuracy of each model across four categories: CPA with single answer (CPA-SA), CPA with multiple answers (CPA-MA), CFA Level 1 (CFA-L1), and CFA Level 2 (CFA-L2). In Table 3.3, "Random" serves as the baseline, representing the probability of randomly selecting an answer from the options available for each question. It is important to note that CPA questions have four options (A, B, C, D), resulting in a random accuracy rate of 25.00% for CPA-SA and 10% for CPA-MA (when randomly selecting a combination of answers). In contrast, CFA questions have three options (ABC), leading to a random accuracy rate of 33.33%.

GPT-4 demonstrated remarkable leadership across almost all categories and subjects, particularly in the English-medium CFA exams. LLaMA showed weaker performance in Chinese questions but ranked closer to the middle in English questions. Chinese-Alpaca improved in CPA questions due to its domain expansion in Chinese corpora, but this came at the expense of its capabilities in the English domain. Observations based on the evaluation results of ChatGLM, Baichuan, and Qwen reveal that, generally, base models exhibit a slight advantage over their corresponding chat models, but the loss incurred due to instruction-tuning is not significant. Furthermore, expanding the parameter size of the same model architecture leads to a stable increase in accuracy, enhancing problem-solving abilities. However, simply increasing the parameter size is insufficient to overcome performance variations caused by different model selections and pre-training corpus construction strategies among organizations. For instance, the Baichuan2-13B series lagged behind or matched the Qwen-7B series across various categories, while the Yi-6B series, with the smallest model size in IDEA-FinBench, demonstrated exceptional performance, even surpassing GPT-4 in CPA questions, which dominated the CFA categories.

Finally, we discuss observations on Financial LLMs that underwent secondary training on financial corpora based on base models. Surprisingly, these vertical-specific models did not achieve the anticipated improvements on IDEA-FinBench, despite being more domain-adapted compared to general-purpose models. Further experiments are needed for a deeper investigation.

In this chapter, we present IDEA-FinBench, an innovative benchmark designed to assess financial knowledge in LLMs by utilizing questions from two globally recognized and authoritative financial professional exams. The benchmark encompasses questions in both Chinese and English, four types of question formats, and spans sixteen financial disciplines, thereby providing a comprehensive evaluation of LLMs’ ability to directly address exam questions pertinent to the finance sector. Moreover, we introduce a modular evaluation suite that can integrate external datasets, allowing for flexible customization of evaluation modes and interfaces with various LLMs. This feature enhances the adaptability and scalability of the evaluation framework, making it a versatile tool for assessing financial knowledge in LLMs.

In this study, our experimental results demonstrate that GPT-4 exhibited exceptional performance across nearly most of categories and subjects, particularly in the English-medium CFA exams. According to our observations, generally, base models have a slight advantage over their corresponding chat models, with the loss incurred due to instruction-tuning being insignificant. Additionally, increasing the parameter size of the same model architecture leads to a stable increase in accuracy, enhancing problem-solving abilities. However, merely increasing the parameter size is insufficient to overcome performance variations caused by different model selections and pre-training corpus construction strategies among organizations. Finally, observations on Financial LLMs that underwent secondary training on financial corpora based on base models revealed that these vertical-specific models did not achieve the anticipated improvements on IDEA-FinBench, despite being more domain-adapted compared to general-purpose models.

This observation can be made by examining the training processes of some Chinese community open-access LLMs specific to the financial domain. The dataset used for fine-tuning DISC-FinLLM [39] includes a large number of instruction-following tasks, encouraging the model to enhance its instruction-following ability for language understanding tasks such as sentiment analysis, intent recognition, entity extraction, and retrieval-based question answering. However, the lack of financial knowledge injection results in a decline in performance on IDEAFinBench and FinEval after full-parameter fine-tuning. Similarly, the supervised fine-tuning dataset for CFGPT [40] also heavily favors tasks like text summarization, sentiment classification, and entity recognition. In contrast, Tongyi-Finance [41], despite using high-quality financial corpora for incremental pre-training and instruction fine-tuning, potentially causes significant damage to the original parameter structure determined by pre-training. These projects might be overly focused on training LLMs to be text assistants that follow a fixed work paradigm, rather than learning and mastering financial knowledge as experts.

Our contributions can be summarized as follows:

1. 1.

   A Novel Framework for Financial Domain Adaptation: IDEA-FinKER represents a unique approach to adapting general LLMs to the financial domain. Unlike traditional methods that rely heavily on extensive pre-training with financial corpus, IDEA-FinKER facilitates rapid adaptation without incurring high costs, making it a cost-effective solution for enhancing the financial expertise of LLMs.

2. 2.

   Investigation on Knowledge Injection Paradigms: We explore two distinct paradigms for knowledge injection in LLMs: the soft-injecting paradigm, which employs retrieval-based few-shot learning for real-time context-level knowledge injection, and the hard-injecting paradigm, which involves fine-tuning LLMs with a set of high-quality financial knowledge instructions.

3. 3.

   Analysis on IDEA-FinKER’s Performance on Knowledge Injection: Our empirical evaluation demonstrates that IDEA-FinKER significantly improves the performance of LLMs on financial tasks. It’s observed that IDEA-FinKER achieves the best performance when integrate both the two paradigms together on different base models.

In this chapter, we will introduce the methodology of IDEA-FinKER. First, we collected and organized FinCorpus from the internet. Our framework includes two paradigms of knowledge injection: the soft-injecting paradigm and the hard-injecting paradigm. To adapt to different paradigms, our FinCorpus needs to be processed accordingly and constructed into different formats for context insertion or fine-tuning.

The first approach is the soft-injecting paradigm for financial knowledge infusion. Due to the large scale of the financial question bank used for dynamic context knowledge injection, we selected forty thousand questions as the candidate set. We utilize ChromaDB [95] as the vector database and employ the popular Chinese text embedding method bge-large-zh-v1.5 from BAAI [96] to calculate and store the text embeddings of all the questions in the candidate set. Next, we traverse the test set and use cosine similarity as the metric to retrieve the five most similar candidate questions as example questions for knowledge injection, thereby enhancing the performance of few-shot learning. The second approach is the Hard-Injecting Paradigm for financial knowledge injection, for which we need to use the LLaMA-Factory framework.

LLaMA-Factory [97] is a unified framework integrating various cutting-edge techniques for efficiently fine-tuning LLMs. It supports the fine-tuning of popular open-source LLMs such as LLaMA [35], Mistral [98], Qwen [44], Yi [88], among others. Utilizing a scalable modular design, LLaMA-Factory facilitates additional training and alignment of base models, including incremental pre-training, instruction-supervised fine-tuning, reward model training, and RLHF. Moreover, LLaMA-Factory allows for the selective configuration of specific fine-tuning methods, such as LoRA [89] and QLoRA [99], and supports training acceleration algorithms like Flash-Attention [100]. As part of the LLM training suite, real-time monitoring and model deployment are also included.

In our experimental setup for hard-injecting paradigm, the fine-tuning process is executed on four GPUs. We use LoRA as the fine-tuning method, targeting some specific layers with a rank of 64 and an alpha value of 128. The training batch size per device is set to 4, with an equivalent size for evaluation batches. Gradient accumulation is applied every four steps to effectively increase the batch size. The pre-processing is handled by eight workers, and a cosine learning rate scheduler is used. Model checkpoints are saved and evaluated every 1000 steps, with the validation set comprising 0.01 of the training data. The evaluation strategy is based on fixed steps, and the best model is loaded at the end of training. The initial learning rate is set to 5e-5. The training is carried out for three epochs, with the option to plot the loss curve. The model is trained with bfloat16 precision for enhanced efficiency, and flash-attention is enabled for accelerated training. Finally, we will integrate both the soft and hard injecting paradigms to enhance the same model.

From Table 4.2, it can be observed that in the majority of cases, models processed through IDEA-FinKER exhibit improvements compared to their vanilla counterparts. Here, the term ’vanilla model’ refers to native models that directly employ zero-shot learning to complete tasks.

In this chapter, we introduce the IDEA-FinKER framework, a novel approach designed to enhance the financial knowledge of LLMs. Distinct from traditional methods that primarily depend on extensive pre-training with financial corpora, IDEA-FinKER facilitates rapid adaptation without incurring substantial costs, thereby presenting a cost-effective solution for augmenting the financial expertise of LLMs. This framework is particularly notable for its exploration of two distinct knowledge injection paradigms: the soft-injecting paradigm and the hard-injecting paradigm. The soft-injecting paradigm utilizes retrieval-based few-shot learning for real-time, context-level knowledge injection, while the hard-injecting paradigm involves fine-tuning LLMs with a curated set of high-quality financial instructions. Additionally, this chapter provides an analysis of IDEA-FinKER’s effectiveness in knowledge injection. Empirical evaluations demonstrate that IDEA-FinKER significantly enhances the performance of LLMs on financial tasks, particularly when integrating both injection paradigms across various base models.

As observed in Chapter 4, effectively enhancing the cognitive abilities of LLMs in the domain of finance—specifically their recognition, understanding, and mastery of authoritative financial knowledge—can be achieved through the injection of knowledge across multiple paradigms. This enhancement not only improves performance on financial examinations but also helps overcome illusions and heightens the model’s factual awareness. However, factual knowledge in financial scenarios is not limited to key points from books, encyclopedias, and textbooks, such as those concerning "leveraged buyouts" or "return on investment." It also extends to temporally characterized facts in the real world [62].

There is no shortage of work modifying model parameters to inject dense factual knowledge. For instance, some studies, such as [101, 102, 103], target specific knowledge triplets by detecting activation sites of the key vector, encoding the fact relations in the value vector, and ultimately updating the weight matrix of the multi-layer perceptron (MLP) by adjusting the projection layer. Similarly, methods like [104] leverage external knowledge bases to fact-check and score the quality of model responses, combining generated confidence with preference scores as rewards in a direct preference optimization (DPO) [105] algorithm to enhance the factuality of the model’s output. Another approach [106] focuses on augmenting the knowledge perception of LLMs, integrating explicit knowledge triplet extraction and implicit multidimensional knowledge preference scoring to structurally update model parameters and strengthen the model’s knowledge perception strategies.

However, the method of adjusting model parameters to inject knowledge still does not alter the black box nature of LLMs’ parameters, thus, the factuality of the content they generate cannot be guaranteed. Retrieval Augmented Generation (RAG) has effectively improved the performance of language models on knowledge-intensive tasks by utilizing external knowledge sources [64]. By designing specific interfaces and constructing relevant queries, LLMs gain the ability to tap into worldwide knowledge. This access helps reduce the occurrence of generated inaccuracies and the outdatedness of the knowledge encoded in their parameters [67, 68, 69, 70].To ensure that LLMs reference trustworthy sources when generating responses, the RAG technology is a worthwhile method to explore.

There has been a considerable amount of research on factual knowledge previously. These efforts extend the design principles of traditional PLM tasks, such as fact-checking or fact verification [80, 81, 82], but exclusively retain the claim part as the input for LLMs. In the knowledge-intensive domains, financial scenarios pose a more stringent test and challenge to the timeliness of world facts compared to fields like mathematics, medicine, and law.

Therefore, we propose FinFact, the first Chinese financial domain factual knowledge verification dataset. We use authoritative Chinese news media as information sources, such as Xinhua Net (http://www.news.cn/), China Youth Online (https://www.youth.cn/index.htm), and China Economic Net (http://www.ce.cn/), collecting over 200 high-quality news texts to construct over 1,500 factual question-answer pairs, ultimately forming a fact verification dataset for the financial sector.

We also introduce IDEA-FinQA, a financial question-answering system driven by LLMs. IDEA-FinQA adheres to a scheme of real-time knowledge injection and factual enhancement using external knowledge for LLMs. The system comprises three main modules: the data collector is responsible for collecting and integrating financial domain data, including data storage solutions, online and offline collection; the data querying module offers data search methods based on two types of search engines, traditional text-based indexing and popular embedding-based indexing, for multiple stages of recall and ranking; the driving force of IDEA-FinQA is four LLM-based agents, performing corresponding tasks given different prompts and contexts, including a query rewriter, intention detector, extractor and refiner, and a response generator.

Our contributions can be summarized as follows:

1. 1.

   We build the first Chinese financial domain factual knowledge verification dataset, utilizing authoritative Chinese news media as sources.

2. 2.

   We construct an advanced financial question-answering system driven by LLMs. IDEA-FinQA integrates real-time financial information retrieval, text embedding search engines, and a sophisticated financial question-answering agent.

3. 3.

   By leveraging trustworthy sources and sophisticated data retrieval and processing techniques, the IDEA-FinQA system ensures that the financial advice and information it provides are not only timely but also factually accurate.

The creation of factual datasets, as exemplified by sources like [80, 81, 82], is a meticulous and deliberate endeavor. The development of a Chinese financial factual dataset represents a pioneering effort in this domain. It is crucial to ensure the authenticity and reliability of the information sources. To this end, we utilize reputable Chinese news outlets such as Xinhua Net (http://www.news.cn/), China Youth Online (https://www.youth.cn/index.htm), and China Economic Net (http://www.ce.cn/) as foundational resources.

Our strategy aims to encompass a broad range of topics rather than restricting the dataset to a singular domain. Consequently, we prefer to compile annual summary-type news articles rather than searching for news based on specific thematic keywords. An illustrative URL is (https://www.sohu.com/a/747923588_118392), which links to a selection of the top ten domestic economic news stories of 2023 by China’s Economic Daily. These news cover diverse themes such as macroeconomic policies, agricultural economics, real estate, China’s A-shares, and industrialization, ensuring a rich content variety. Ultimately, we amassed a total of 120 financial news articles to serve as the dataset’s source material. Additionally, we gathered a smaller set of 90 articles from international, technology, and sports news to further support generalization across different domains.

Regarding the methodology for generating inputs for LLMs, unlike the multiple-choice question format used in the previous FinKBench, the FinFact dataset interacts with LLMs through a conversational format. We utilize GPT-4 for this generation process.

In constructing the FinFact dataset, we adopted two approaches. The first approach involves structured question-and-answer sessions where we require GPT-4 to generate questions based solely on the news content, focusing exclusively on specific and objective entities. These include but are not limited to names, places, organizations, dates, and numbers, while consciously avoiding subjective opinions, attitudes, and perspectives. The corresponding answers are also generated accordingly. An example of this can be seen in Figure 5.1. For instance, consider the question: "In 2023, the People’s Bank of China precisely and effectively implemented a prudent monetary policy, conducting several reserve requirement ratio cuts. How many percentage points were reduced in total?" This question explicitly references numerical entities, leading to a definitive answer: "In 2023, the People’s Bank of China implemented two cuts in the reserve requirement ratio, totaling a reduction of 0.5 percentage points."

In the construction of conversational questions, our generation strategy for GPT-4 tends to focus on questions that revolve around the exposition, attitude, viewpoint, and perspective presented in the news material concerning the event, rather than targeting specific, objective entities. Furthermore, the answers are already included within the news content itself. An example of this can be seen in Figure 5.2.

In this section, we will dismantle the functions of each module of the IDEA-FinQA system, including the data collector, the data search engine and the LLM-driven agents. An overview of IDEA-FinQA is shown in Figure 5.3

Our IDEA-FinQA system utilizes the open-source model Qwen1.5-14B-Chat [44] to power all its agents. To test the fact-based question-answering capability of IDEA-FinQA on the FinFact dataset, we selected the vanilla Qwen1.5-14B-Chat, Yi-34B-Chat [88]—one of the strongest current Chinese models, and the globally popular GPT series. Since the GPT series is only available through APIs, we chose gpt-4-turbo-preview and gpt-3.5-turbo for testing, with the model knowledge updated up to December 2023 at the time of testing. For each question in FinFact, we collect and save the complete text returned by the LLMs. Subsequently, GPT-4 is used as a judge to assess the quality of the LLM outputs. For structural questions, standard answers generated during the dataset creation phase are also input to evaluate factuality, whereas for conversational questions, original news texts are input to similarly assess factuality. To objectively evaluate the generated response quality, we also incorporate "relevant" and "informational" as additional dimensions for assessment.

IDEA-FinQA demonstrates a distinct advantage in fact-based question-answering compared to other models. Across the three dimensions—factual, relevant, and informational—IDEA-FinQA leads all other models. Particularly in the factual dimension, IDEA-FinQA outperforms other models with a winning rate of 70%. In assessments of text generation quality, including relevance and informational content, IDEA-FinQA also exhibits strong performance.

We first introduce FinFact, the first Chinese financial domain factual knowledge verification dataset. Subsequently, we present IDEA-FinQA, a financial question-answering system driven by LLMs. IDEA-FinQA adheres to a scheme of real-time knowledge injection and factual enhancement using external knowledge for LLMs. Our experiments demonstrate that IDEA-FinQA surpasses the majority of models in factual question-answering, even when the facts are from different years.