Self-Translate-Train: A Simple but Strong Baseline
for Cross-lingual Transfer of Large Language Models

Abstract

Cross-lingual transfer is a promising technique for utilizing data in a source language to improve performance in a target language. However, current techniques often require an external translation system or suffer from suboptimal performance due to over-reliance on cross-lingual generalization of multi-lingual pretrained language models. In this study, we propose a simple yet effective method called Self-Translate-Train. It leverages the translation capability of a large language model to generate synthetic training data in the target language and fine-tunes the model with its own generated data. We evaluate the proposed method on a wide range of tasks and show substantial performance gains across several non-English languages.

Self-Translate-Train: A Simple but Strong Baseline
for Cross-lingual Transfer of Large Language Models

Ryokan Ri  and Shun Kiyono  and Sho Takase SB Intuitions {ryokan.ri,shun.kiyono,sho.takase}@sbintuitions.co.jp

1 Introduction

Cross-lingual transfer is a technique to solve tasks in a target language by leveraging training data from the other languages (Pikuliak et al., 2021). This has been increasingly feasible with the rise of multilingual pre-trained models, which are trained on multilingual corpora and capture commonalities across languages (Conneau et al., 2020; Xue et al., 2021; Scao et al., 2022). Capable multilingual models can perform tasks in a target language without being trained on task-specific data in that language, which is known as zero-shot cross-lingual transfer (Artetxe and Schwenk, 2019; Chen et al., 2021). This technique is expected to reduce the disparity between high-resource and low-resource languages.

Cross-lingual transfer is also exhibited in large language models (LLMs), which refers to auto-regressive language models with billion-scale parameters that are trained with a massive amount of text data (Brown et al., 2020; Touvron et al., 2023). A common approach for zero-shot cross-lingual transfer is fine-tuning the model with supervised training data available in a source languages, mostly English, and then applying the model to the target language (Chen et al., 2024; Shaham et al., 2024). However, we argue that this approach does not fully elicit the model’s cross-lingual capability as the model has no clue the input language at the test time. To achieve better cross-lingual performance, we let the model teach itself how to solve the task in the target language. In our proposed method, Self-Translate-Train, we produce the target language’s translation of the training data leveraging the strong capability of the LLM to generate text, and train the LLM with its own generated translation.

Figure 1: An overview of Self-Translate-Train. An LLM translates training data to the target language and then fine-tuned on its own generated data.

We evaluate Self-Translate-Train with several tasks including question answering, text-pair classification, and mathematical reasoning across multiple languages. Our experiments show that Self-Translate-Train consistently improves the performance of baselines given a multilingual capability of the LLMs. Our results indicate that we can achieve better cross-lingual performance by correctly elicit the model’s translation capability, which encourages further exploration of how to better utilize the model’s cross-lingual capability.

2 Related Work

2.1 Cross-lingual Transfer Learning

There are two main approaches to transfer task knowledge across languages: data transfer and model transfer (Pikuliak et al., 2021).

Data transfer translates the source language data to the target language. In the Translate-test approach, models are trained on source language data and at inference time, the task inputs are translated into the source language (Conneau et al., 2018; Asai et al., 2018). Although the training stage is simple, it incurs additional translation costs at inference time. The Translate-train approach, on the other hand, translates the training data and the resulting model is used to predict the target language data directly (Conneau et al., 2018). Data transfer is quite effective in terms of performance (Hu et al., 2020), but one drawback is its requirement of additional translation systems.

Model transfer alleviates the need for translation systems by using multilingual pretrained models, which are trained on a large amount of data from multiple languages and capture the commonality between languages. These models can be fine-tuned on task-specific data in a single source language and generalize to solve the task in other languages (Pires et al., 2019; Mulcaire et al., 2019; Conneau et al., 2020), eliminating the need for translation systems.

Our approach, Self-Translate-Train, leverages LLMs’ translation and cross-lingual generalization capabilities. It combines the advantages of data transfer and model transfer by using explicit training signals in the target language while eliminating the need for external translation systems.

2.2 Self-Improvement of LLMs

LLMs have demonstrated remarkable text generation capabilities, which has been leveraged to generate training data for various purposes (Li et al., 2023b; Lee et al., 2024). The generated data can be used to further specialize the LLM itself for downstream applications, without requiring an extensive collection of additional data. This process can be viewed as a form of self-improvement (Bai et al., 2022; Huang et al., 2023; Sun et al., 2023; Li et al., 2023a).

Self-Translate-Train is also a self-improvement approach to specialize the LLM to a target language by translating the source language data to the target language.

3 Self-Translate-Train

Our framework focuses on fine-tuning LLMs on a small amount of data for a specific task. Let the training corpus in a source language, say English, be $\mathcal{D}_{\text{src}}=\{(\mathbf{x}_{\text{src}}^{i},\mathbf{y}_{\text{src}}^{i})\}_{i=1}^{N}$, where $\mathbf{x}$ is the input and $\mathbf{y}$ is the output. In a typical cross-lingual transfer setting, the model is fine-tuned only on $\mathcal{D}_{\text{src}}$ and expected to generalize to target languages.

Translated Synthetic Data

Given the LLM’s translation capability, we can let it translate the training corpus into a synthetic corpus in the target language $\mathcal{D}_{\text{tgt}}$. The synthetic data can be added to $\mathcal{D}_{\text{src}}$ to achieve a better generalization to the target language.

The translation can be performed in various ways depending on the model’s capabilities or available resources. In this paper, we experiment with the few-shot prompting technique (Section 4.4).

Code-switched Synthetic Data

The generated data has an interesting aspect: each synthetic instance has a corresponding instance in the original dataset with the same semantics. We can exploit this to further synthesize data by generating code-switched instances where the input and output are in different languages.

We pair the original and translated instances to construct $\mathcal{D}_{\text{cs}}=\{(\mathbf{x}_{\text{src}}^{i},\mathbf{y}_{\text{tgt}}^{i})\}_{i=1}^{N}\bigcup\{(\mathbf{x}_{\text{tgt}}^{i},\mathbf{y}_{\text{src}}^{i})\}_{i=1}^{N}$. When the task output is natural language, we manually translate the prompt “Please answer in {{ tgt }}.” into the target language, and add it to the input $\mathbf{x}$.

4 Experimental Setups

To verify the effectiveness of Self-Translate-Train, we conduct extensive experiments on multiple tasks and languages.

4.1 Task and Datasets

We present a list of datasets for experiments in Table 1. For each task, an English dataset is used for training and a multilingual dataset for evaluation. To make the computational cost feasible, we use a 10,000-sample subset of the training data for SQuAD and MultiNLI.

Task	Training	Evaluation
QA	SQuAD (Rajpurkar et al., 2016)	XQuAD (Artetxe et al., 2020)
Classification	MultiNLI (Williams et al., 2018)	XNLI (Conneau et al., 2018)
Math	GSM8k (Cobbe et al., 2021)	MGSM (Shi et al., 2023)

Table 1: List of datasets for experiments. The details are described in Section A.1.

4.2 Languages

We conducted evaluation on four languages: German (de), Russian (ru), Thai (th), and Chinese (zh). German is a Germanic language, which is phylogenetically close to English and expected to show better cross-lingual transfer, while Russian, Thai, and Chinese are from different language families. In particular, Thai is a low-resource language with a different script from English, which is expected to be more challenging for cross-lingual transfer.

4.3 Language Models

Our main experiments use Llama2-7B (Touvron et al., 2023), a public LLM. Although 90% of its pretraining corpus is English, the model has a multilingual capability (e.g., Table 2) from the remaining fraction of multilingual data.

4.4 Synthetic Data Generation

Recent LLMs are known to exhibit a translation capability without much task-specific data Briakou et al. (2023). In our experiments, we elicit the translation capability of the LLMs via few-shot in-context learning (Brown et al., 2020).

To construct few-shot translation samples, we sample eight pairs from the train or validation splits of the multilingual datasets, where instances across languages form parallel data. The translation was performed for each field individually, e.g., for GSM8k, we translated the question and answer separately. The prompt template simply alternates the source and target text prepended with the language tag (Section A.2).

An important step to ensure the quality of the synthetic data is to filter out the low-quality data (the details in Section A.3). To remove under- or over-translation (Tu et al., 2016), we filter out texts with an extreme source-target length ratio. Also, to address the repetition problem (Holtzman et al., 2020), we set the max number of tokens for generation and filter out the translation that does not end with the EOS token. With the translations from Llama2-7B, this process removes around 10% of the data for most languages and around 50% for Thai due to the model’s limited generation quality.

To provide the sense of the translation quality, we report the BLEU score (Papineni et al., 2002) measured by the parallel data constructed from questions in the MGSM test set in Table 2. Overall, the translation quality is sufficiently high except for Thai. As we will see in Section 5.1, this poses a challenge for cross-lingual transfer to Thai.

Model	de	ru	th	zh
Llama2-7B	37.1	27.2	1.9	29.4

Table 2: BLEU scores from the MGSM test set. The configuration of BLEU is described in Section A.4.

4.5 Fine-tuning

All the tasks are cast as text generation tasks, where the LLM is given the inputs as a prompt and generate the answer. Fine-tuning is conducted with causal language modeling loss, computed only for output tokens. We use LoRA (Hu et al., 2022), a parameter-efficient tuning technique, to reduce computational cost.

We use AdamW (Loshchilov and Hutter, 2019) and the cosine learning rate schedule for optimization, training with a batch size of 64 for 1,000 steps. For each setting, we conduct six runs with two learning rates (5e-5 and 3e-4) and different random seeds, reporting summarization statistics of the top four runs based on validation accuracy to remove runs with optimization failure. See Section A.5 for other hyperparameters.

5 Results

5.1 Main Results

	MGSM				XQuAD				XNLI
	de	ru	th	zh	de	ru	th	zh	de	ru	th	zh
$\mathcal{D}_{\text{src}}$	30.1 $\pm$0.4	25.0 $\pm$0.7	8.1 $\pm$0.7	21.1 $\pm$1.7	60.3 $\pm$0.8	49.3 $\pm$0.4	34.5 $\pm$1.0	66.3 $\pm$0.8	79.7 $\pm$0.4	76.9 $\pm$0.1	53.7 $\pm$0.9	74.1 $\pm$0.2
$+\mathcal{D}_{\text{tgt}}$	* 36.4 $\pm$1.3 *	* 34.0 $\pm$1.7 *	7.7 $\pm$3.4	* 27.1 $\pm$1.2 *	61.7 $\pm$0.9	* 57.8 $\pm$0.8 *	* 46.4 $\pm$1.3 *	* 77.7 $\pm$0.4 *	* 81.6 $\pm$0.7 *	* 78.5 $\pm$0.6 *	56.3 $\pm$1.5	* 78.5 $\pm$0.3 *
$+\mathcal{D}_{\text{tgt}}+\mathcal{D}_{\text{cs}}$	* 35.9 $\pm$1.3 *	* 34.5 $\pm$2.4 *	10.4 $\pm$1.6	* 28.8 $\pm$1.9 *	* 62.2 $\pm$0.9 *	* 58.0 $\pm$0.6 *	* 46.2 $\pm$1.7 *	* 77.3 $\pm$0.7 *	* 81.6 $\pm$0.5 *	78.4 $\pm$1.2	* 58.9 $\pm$1.5 *	* 77.6 $\pm$0.6 *

Table 3: Results on multilingual evaluation datasets. Scores are marked with ^∗ if its improvement is statistically significant ($p<0.05$ in Welch’s t-test) compared to the baseline $\mathcal{D}_{\text{src}}$. The significant and highest score in each column is marked in bold.

(a) de

(b) th

(c) zh

Figure 2: Accuracy in the MGSM dataset with different model sizes of Llama2.

As the baseline, we fine-tune the LLM with the source language dataset $\mathcal{D}_{\text{src}}$. To ensure a fair comparison, we augment $\mathcal{D}_{\text{src}}$ with the eight target language samples used for few-shot translation (Section 4.4). We then compare the baseline with the models fine-tuned on the data generated from Self-Translate-Train ($\mathcal{D}_{\text{src}}$ and $\mathcal{D}_{\text{cs}}$) in Table 3.

First, Self-Translate-Train is indeed an effective method; $+\mathcal{D}_{\text{tgt}}$ almost consistently outperforms the baseline $\mathcal{D}_{\text{src}}$. The only exception is Thai (th), where there is no significant improvement. This is likely due to the low translation quality of the model in Thai (Table 2).

The effectiveness of code-switching dataset is limited. When we add $\mathcal{D}_{\text{cs}}$ to $\mathcal{D}_{\text{tgt}}$, there is no significant improvement from adding $\mathcal{D}_{\text{tgt}}$ only ($p<0.05$ in Welch’s t-test). This indicates that the code-switching data does not provide additional information for the model to generalize in the task.

5.2 Does the model size matter?

The size of the language model can influence both its ability to generalize across languages and the quality of its translations, which in turn may impact the effectiveness of Self-Translate-Train. We compare the performance of Llama2 with different sizes, i.e., 7B, 13B, and 65B, on the math task in de, th, and zh (Figure 2).

The larger model size generally tends to perform better, and the improvement from Self-Translate-Train remains consistent across different model sizes. In Thai (th), we did not observe a significant improvement in the 7B model, but do in the larger models (13B and 65B), likely due to their better translation quality. The 7B model has a low Thai translation BLEU score of 1.9 (Table 2), while the 13B and 65B models have BLEU scores of 5.1 and 12.0, respectively.

The improvement in Thai (th) with the 70B model is the most significant (+19.8 average points). This implies that Self-Translate-Train is particularly effective when the model struggles with generalizing across the source and target languages but can still generate reasonable translations.

6 Conclusion

We introduced Self-Translate-Train, a method to improve cross-lingual transfer performance by generating synthetic training data in the target language. We validated its effectiveness on various tasks and languages, demonstrating substantial performance gains across several non-English languages. Self-Translate-Train is effective when the zero-shot cross-lingual transfer performance is suboptimal and the model can generate reasonable translations.

Self-Translate-Train neither requires external translation systems nor intensive additional data collection, making it a simple yet effective method for cross-lingual transfer. We encourage practitioners to try this approach as an improved baseline for cross-lingual transfer of LLMs.

Our research also shows that relying solely on the model’s generalization capability may be suboptimal, and there is a better way to elicit the cross-lingual capability of the model. We hope this work encourages further exploration of how to better utilize the model’s cross-lingual capability.

7 Limitations

Our experiments are conducted on a modern type of LLM, an autoregressive Transformer decoder, and centered around the Llama2 model families (Touvron et al., 2023). Although we further validate our method in Appendix B, the effective of our proposed method is uncertain when applied to other types of LLMs developed in the future.

Our method is based on the assumption that the model can generate reasonable translations in the target language. This may be challenging when the task inputs are long or complex. One solution is to split the input into smaller segments, as we have done with the SQuAD dataset (Section A.2).

Finally, when the task requires generating long and natural text, the quality of the generated translation matters more. If the translation quality is low, the model outputs may degrade due to translation errors or unnaturalness. The application of our method on more challenging tasks requires further investigation.

Appendix A The Details of Experimental Setups

A.1 Tasks and Datasets

We provide the details of the datasets introduced in Section 4.1.

A.1.1 Question Answering (QA)

SQuAD (Rajpurkar et al., 2016) is an English QA dataset created from Wikipedia articles as training data. Given a question and a passage, the task is to extract the answer from the passage. Evaluation is conducted with XQuAD (Artetxe et al., 2020), which consists of translation of SQuAD into multiple languages.

Context: Architecturally, the school has a Catholic character. Atop the Main Building’s gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend "Venite Ad Me Omnes". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary. Question: To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?

Saint Bernadette Soubirous

Figure 3: An input and output example of the SQuAD dataset.

A.1.2 Text-Pair Classification

We also evaluate our method on cross-lingual text-pair classification tasks. The MultiNLI dataset (Williams et al., 2018) involves determining the logical relationship between a premise sentence and a hypothesis sentence. XNLI (Conneau et al., 2018) is a multilingual NLI dataset for evaluation.

Premise: Conceptually cream skimming has two basic dimensions - product and geography. Hypothesis: Product and geography are what make cream skimming work. What is their logical relation? Entailment, Neutral or Contradition.

Neutral

Figure 4: An input and output example of the MultiNLP dataset.

A.1.3 Mathematical Reasoning

GSM8k (Cobbe et al., 2021) is an English dataset of 8.5K high-quality grade school math problems. Each problem is annotated with a solution that shows the mathematical steps required to reach the final answer. As the evaluation dataset, we use MGSM (Shi et al., 2023), a multilingual version of the GSM8k dataset.

The LLM is trained to generate the step-by-step solution to math problems. The answer is extracted from the LLM output as the final digits, and the accuracy is calculated based on the exact match of the extracted answer and the ground truth.

Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?

Natalia sold 48/2 = clips in May. Natalia sold 48+24 = 72 clips altogether in April and May. #### 72

Table 4: An input and output example of the GSM8k dataset.

A.2 Prompt Format for LLM Translation

To translate training data using a LLM (Section 4.4), we employed the following prompt template for each task. The template simply consists of the source text and target text prepended with the language tag. The text is surrounded by backticks and the LLM starts generating the target text the open backtick until the close backtick is found.

{% for sample in few_shot_samples %} en: ‘{{ sample.data_field }}‘ {{ target_language }}: ‘{{ sample.data_field }}‘ {% endfor %} en: ‘{{ data_field }}‘ {{ target_language }}: ‘

Figure 5: Prompt format for LLM translation.

The SQuAD dataset annotates answer spans in the context passages. We translate the annotations using the mark-then-translate approach (Chen et al., 2023). We mark the answer span in the context passage with the tokens “<answer>” and “</answer>”, translate the marked text, and then extract the translated answer span from the translated context. Note that in this case, the few-shot samples are also marked with the answer span.

The context passages in the SQuAD dataset are relatively long, and it is challenging for the LLM with a limited context window to fit the entire few-shot samples and the source text. To address this issue, we split the context into sentences using the spaCy library and translate them separately, i.e., the few-shot samples and the source text are sentences.

A.3 Data Filtering for Synthetic Data

We remove pairs where the target length is less than one-third or more than three times the source length. The text length is heuristically determined to account for character length differences between languages. For example, phonogram-based text (e.g., English) has much more characters than ideograph-based text (e.g., Chinese). We set normalization factors where English, German, Thai, and Russian characters count as 1, and Chinese characters as 3.

We also filter out incomplete translations which are typically produced by repetitive generation. We set the maximum number of tokens for generation (Table 5) and remove the outputs not ending with the token indicating the end of the translation, in our case, the backtick character used in the prompt format.

	Data Field	Max Number of Tokens
SQuAD (Rajpurkar et al., 2016)	context	512
	question	256
MultiNLI (Williams et al., 2018)	premise	256
	hypothesis	256
GSM8k (Cobbe et al., 2021)	question	512
	answer	512

Table 5: Maximum number of tokens set for generating translations.

A.4 Assessing the Translation Quality

To evaluate the translation quality, we use the BLEU score (Papineni et al., 2002) measured by the parallel data constructed from questions in the MGSM test set. The translation is performed in few-shot in-context learning with 8 translation samples constructed from the train set of the MGSM dataset. The BLEU score is calculated using the SacreBLEU library (Post, 2018). As the tokenizer option, we use “13a” for de and ru, “flores101” for th, and “zh” for zh.

Table 6 shows the BLEU scores from the LLMs evaluated in this paper. The result of Qwen1.5-1.8B is discussed in Appendix B, and gpt-3.5-turbo-0125 in Section C.3.

Model	de	ru	th	zh
Llama2-7B	37.1	27.2	1.9	29.4
Llama2-13B	41.3	33.4	5.1	34.3
Llama2-70B	45.6	41.9	12.0	42.4
Qwen1.5-1.8B	21.9	11.3	1.5	41.3
gpt-3.5-turbo-0125	48.0	44.6	23.1	47.2

Table 6: BLEU scores from the MGSM test set.

A.5 Hyper-parameters for Fine-tuning

We provide the hyper-parameters used for fine-tuning the LLMs in Table 7.

Hyper-parameter	Value	Hyper-parameter	Value
Batch size	64	Adam $\epsilon$	1e-8
Number of steps	1,000	Adam $\beta_{1}$	0.9
Learning rate	[5e-5, 3e-4]	Adam $\beta_{2}$	0.999
LR Scheduler	Cosine	Weight decay	0.1
Warmup ratio	0.05

Table 7: Hyper-parameters used for fine-tuning the LLMs.

Appendix B Results from Qwen1.5-1.8B

To increase the robustness of the results, we also conducted experiments with Qwen1.5-1.8B . While the model is mainly trained on Chinese and English data, it is also constructed with the multilingual use cases in mind.

	gsm8k
	de	ru	th	zh
$\mathcal{D}_{\text{src}}$	8.0 $\pm$0.3	6.5 $\pm$0.4	2.8 $\pm$0.5	23.3 $\pm$0.5
$+\mathcal{D}_{\text{tgt}}$	* 18.7 $\pm$1.8 *	* 15.0 $\pm$1.2 *	3.4 $\pm$1.8	21.4 $\pm$1.2
$+\mathcal{D}_{\text{cs}}$	* 17.1 $\pm$1.3 *	* 14.9 $\pm$0.5 *	2.7 $\pm$0.8	23.2 $\pm$0.9

Table 8: Results on the MGSM dataset with Qwen1.5-1.8B. Scores are marked with ^∗ if its improvement is statistically significant ($p<0.05$ in Welch’s t-test) compared to the baseline $\mathcal{D}_{\text{src}}$. The significant and highest score in each column is marked in bold.

We observe that the results are consistent with the main experiments: Self-Translate-Train is effective when the zero-shot cross-lingual transfer performance is suboptimal and the model can generate reasonable translations. The performance is improved by adding the target language data $+\mathcal{D}_{\text{tgt}}$. However, when the translation quality is poor as in Thai (1.5 BLEU score in Table 6), the improvement is not observed. Additionally, Qwen1.5-1.8B seems to have good cross-lingual capability between English and Chinese, as indicated by the high BLEU score (41.3 in Table 6). With this, tuning on the source language data alone is sufficient to achieve high performance.

Appendix C Frequently Asked Questions

In this section, we discuss questions that are outside the scope of the main topic of this paper but are somewhat relevant and may be of interest to readers.

C.1 Does Self-Translate-Train improve the performance in the source language?

The performance somtimes improves, given the task is challenging and the translation quality is sufficiently high.

Table 9 shows the results on the English test set with Llama2-7B. The performance improves in the MGSM dataset when adding the synthetic data from de, ru, and zh. The Thai language does not show the improvement possibly due to the low translation quality.

However, the improvement is not observed in the XQuAD and XNLI datasets. This might be because the task performance is already high with the source language data alone, and the synthetic data does not provide additional information to improve the performance.

	MGSM				XQuAD				XNLI
	de	ru	th	zh	de	ru	th	zh	de	ru	th	zh
$\mathcal{D}_{\text{src}}$	37.8 $\pm$0.8				70.2 $\pm$0.5				88.2 $\pm$1.3
$+\mathcal{D}_{\text{tgt}}$	* 42.6 $\pm$1.9 *	* 41.8 $\pm$1.3 *	40.0 $\pm$1.9	* 42.7 $\pm$1.0 *	69.1 $\pm$0.7	69.6 $\pm$0.5	70.0 $\pm$0.5	69.9 $\pm$0.4	89.0 $\pm$0.4	88.3 $\pm$0.7	88.6 $\pm$1.0	89.3 $\pm$0.3
$+\mathcal{D}_{\text{cs}}$	* 42.7 $\pm$0.7 *	* 42.9 $\pm$0.7 *	39.8 $\pm$1.6	40.8 $\pm$2.0	69.5 $\pm$0.5	69.6 $\pm$0.3	69.8 $\pm$0.5	68.7 $\pm$0.5	88.7 $\pm$0.2	88.5 $\pm$1.0	88.0 $\pm$0.7	88.7 $\pm$0.6

Table 9: Results on the Englihs test set with Llama2-7B. Scores are marked with ^∗ if its improvement is statistically significant ($p<0.05$ in Welch’s t-test) compared to the baseline $\mathcal{D}_{\text{src}}$. The significant and highest score in each column is marked in bold.

C.2 Does the synthetic data alone improve the performance in the target language?

Yes, but adding the source language data is more effective. Table 10 shows the results with Llama2-7B on the multilingual evaluation datasets. Tuning on the synthetic data alone ($\mathcal{D}_{\text{tgt}}$) improves the performance in the target language, but the improvement is not as significant as adding the synthetic data to the source language data ($+\mathcal{D}_{\text{tgt}}$). In practice, we recommend using the synthetic data in combination with the original data to achieve the best performance.

	MGSM				XQuAD				XNLI
	de	ru	th	zh	de	ru	th	zh	de	ru	th	zh
$\mathcal{D}_{\text{src}}$	30.1 $\pm$0.4	25.0 $\pm$0.7	8.1 $\pm$0.7	21.1 $\pm$1.7	60.3 $\pm$0.8	49.3 $\pm$0.4	34.5 $\pm$1.0	66.3 $\pm$0.8	79.7 $\pm$0.4	76.9 $\pm$0.1	53.7 $\pm$0.9	74.1 $\pm$0.2
$D_{\text{tgt}}$	* 32.1 $\pm$0.6 *	* 30.4 $\pm$1.8 *	8.7 $\pm$0.6	24.6 $\pm$2.5	* 62.5 $\pm$0.4 *	* 57.4 $\pm$1.1 *	* 44.3 $\pm$0.8 *	* 75.4 $\pm$1.1 *	* 81.5 $\pm$0.6 *	77.9 $\pm$0.7	53.8 $\pm$1.7	* 77.1 $\pm$0.5 *
$+\mathcal{D}_{\text{tgt}}$	* 36.4 $\pm$1.3 *	* 34.0 $\pm$1.7 *	7.7 $\pm$3.4	* 27.1 $\pm$1.2 *	61.7 $\pm$0.9	* 57.8 $\pm$0.8 *	* 46.4 $\pm$1.3 *	* 77.7 $\pm$0.4 *	* 81.6 $\pm$0.7 *	* 78.5 $\pm$0.6 *	56.3 $\pm$1.5	* 78.5 $\pm$0.3 *

Table 10: Results on the Englihs test set with Llama2-7B with the setting of tuning the synthetic data alone $\mathcal{D}_{\text{tgt}}$.

C.3 Is tuning on $\mathcal{D}_{\text{tgt}}$ generated by another model still effective?

Yes, if the other model can generate reasonable translations. Such approach can be seen as the Translate-train approach (Section 2.1) or sequence distillation from a teacher model (Kim and Rush, 2016).

As an upper-bound experiment using the math task, we fine-tune Llama2-7B on the synthetic data generated by gpt-3.5-turbo-0125 from the OpenAI API, which produces high-quality translations across the languages explored in this paper (Table 6).

	MGSM
	de	ru	th	zh
$\mathcal{D}_{\text{src}}$	30.1 $\pm$0.4	25.0 $\pm$0.7	8.1 $\pm$0.7	21.1 $\pm$1.7
$+\mathcal{D}_{\text{tgt}}$	* 37.4 $\pm$1.0 *	* 35.9 $\pm$1.2 *	* 28.5 $\pm$1.4 *	* 33.2 $\pm$1.1 *
$+\mathcal{D}_{\text{cs}}$	* 38.6 $\pm$1.5 *	* 34.4 $\pm$0.5 *	* 28.7 $\pm$1.3 *	* 34.8 $\pm$1.3 *

Table 11: Results on multilingual evaluation datasets with Llama2-7B tuned on the synthetic data generated by gpt-3.5-turbo-0125.

Adding the synthetic data generated by gpt-3.5-turbo-0125 improves performance across languages.

However, the outputs from external models are often restricted in its usage, while the method explored in this paper can be used with the model at hand without other resources. Additionally, our interest in this paper is rather to explore the cross-lingual potential of the model itself and how to better utilize it.