# Abstract

Chain-of-thought (CoT) prompting has demonstrated the capacity of large language models to perform complex reasoning through intermediate steps. While effective, current CoT methods face challenges: Zero-shot-CoT can lead to reasoning errors, and Few-shot-CoT requires labor-intensive manual demonstrations. Auto-CoT attempts to address these issues by automatically generating diverse demonstrations, but this diversity can lead to inconsistent reasoning patterns. We propose ECHO (Self-Harmonized Chain of Thought), a novel method that unifies diverse solution paths into a consistent and effective reasoning pattern. ECHO employs an iterative process to refine and harmonize automatically generated demonstrations, mitigating the limitations of existing approaches. Our comprehensive experiments across arithmetic, commonsense, and symbolic reasoning tasks demonstrate that ECHO outperforms Auto-CoT by an average of 2.8%. These findings suggest that ECHO represents a significant step towards more robust and generalizable automated reasoning in large language models.

Code available: https://github.com/Xalp/ECHO

# Introduction

Large language models (LLMs) have demonstrated remarkable capabilities in various natural language processing tasks. However, their performance on complex reasoning tasks has been a persistent challenge. A recent technique known as chain-of-thought (CoT) prompting [Wei et al., 2022] has significantly enhanced LLMs' ability to tackle such tasks by decomposing complex problems into a series of intermediate steps.

[IMAGE: Figure 1 - A comparison between ECHO and other CoT baselines. "Zero-CoT" is short for Zero-shot-CoT and "Few-CoT" is short for Few-shot-CoT. The demonstrations generated by Auto-CoT and ECHO will be applied as few-shot examples during inference.]

CoT methods typically employ two prompting paradigms: Zero-shot-CoT [Kojima et al., 2022] and Few-shot-CoT [Wei et al., 2022]. Zero-shot-CoT uses a universal prompt like "Let's think step by step" to guide models in forming reasoning chains without specific examples. Few-shot-CoT, on the other hand, involves few-shot prompting with human-crafted demonstrations, pairing each question with a detailed reasoning chain.

While Few-shot-CoT has shown promising results, it requires the creation of human-crafted examples for each specific domain, which can be time-consuming and expensive. To address this limitation, Auto-CoT [Zhang et al., 2023] was developed to automate the process of creating demonstrations by employing Zero-shot-CoT. However, Auto-CoT faces its own challenges:

- **Misleading by similarity:** Some demonstrations generated by Zero-shot-CoT may contain incorrect reasoning processes or answers, potentially misleading the model when solving similar problems.

- **Ineffective diversity:** To mitigate the misleading effect, Auto-CoT selects diverse demonstrations. However, this approach can lead to demonstrations that are too dissimilar or irrelevant to the actual question, reducing their effectiveness.

- **Inconsistent solution patterns:** Diverse demonstrations might encompass varied solution patterns, including nuanced ones, making them less representative for the model to learn from.

To address these limitations, we propose ECHO (s**E**lf-Harmonized **Ch**ain of Th**o**ught), a novel method that aims to unify diverse rationale patterns into one general pattern. Our approach is inspired by Cognitive Load Theory [Sweller, 1988], which posits that learning is most effective when the cognitive load on working memory is minimized. By unifying demonstrations, ECHO creates a more coherent set of examples, potentially reducing the cognitive load, thus facilitating more effective reasoning patterns that can be easily followed.

ECHO consists of three main steps: (1) Cluster a given dataset and select representative questions. (2) Generate initial rationales using Zero-shot-CoT. (3) Employ a dynamic prompting mechanism to iteratively improve demonstrations using each other as in-context examples.

This iterative process not only refines the quality of individual demonstrations but also promotes consistency across the entire set, potentially reducing the cognitive load required to process and apply the demonstrated reasoning patterns.

Our experiments across three different reasoning domains demonstrate that ECHO outperforms other baselines by 2.8% overall. We also conduct comprehensive ablation studies to understand the benefits of unifying diversity in improving performance. These results support the hypothesis that a more unified set of demonstrations, as suggested by Cognitive Load Theory, can indeed lead to improved reasoning capabilities in LLMs.

The main contributions of this work are:

- A novel approach (ECHO) that automatically improves the quality of demonstrations in the CoT process by unifying diversity.

- An iterative unifying prompting framework that is effective across various tasks by reducing demonstration variety.

- Extensive experiments showing competitive results on arithmetic, commonsense, and symbolic reasoning domains through diversity reduction, supporting the efficacy of our cognitively-informed approach.

[IMAGE: Figure 2 - Overview of our ECHO method. In the demonstration unification process, ECHO iteratively re-generates the rationale of one demonstration with other demonstrations as in-context examples.]

# Related Work

## Chain-of-Thought (CoT) Prompting

CoT prompting has emerged as a powerful, gradient-free method for enhancing the reasoning capabilities of Large Language Models (LLMs). Wei et al. [2022] introduced Few-shot-CoT, which utilizes manually crafted demonstrations to guide the model's reasoning process. Kojima et al. [2022] then proposed Zero-shot-CoT, extending the method to zero-shot scenarios.

Recent studies have focused on creating more complex demonstrations or employing ensemble-like strategies. For example, some prompting methods [Zhou et al., 2022; Wang et al., 2023; Yao et al., 2023] adopt a problem decomposition approach, breaking down complex problems into simpler subproblems. Self-consistency and other CoT methods [Wang et al., 2022; Asai et al., 2023; Sun et al., 2024] involve reasoning from multiple paths or iteratively. Moreover, some prompting methods can generate executable programs to aid in the computation process within the arithmetic domain [Gao et al., 2022; Chen et al., 2022; Pi et al., 2022].

## Automated Prompt Generation

Although designing increasingly complex prompts improves CoT effectiveness, it also increases the human effort involved. Auto-CoT [Zhang et al., 2023] automates the process by clustering a dataset and selecting a representative question from each cluster, followed by using Zero-shot-CoT to generate rationales for these questions.

A key consideration in Auto-CoT [Zhang et al., 2023] is the avoidance of using questions in the same cluster as demonstrations. The rationale behind this choice is rooted in the observation that if demonstrations are too similar to the target question, there is a higher risk that the model might replicate its mistakes, since Zero-shot-CoT [Kojima et al., 2022] does not guarantee correctness in its responses (i.e., the demonstrations might be wrong). However, the use of diverse demonstrations might include irrelevant demonstrations or non-representative solution patterns.

# Self-Harmonized Chain of Thought

Our method, ECHO, involves three main steps as shown in Figure 2:

1. **Question clustering:** Partition questions of a given dataset into clusters based on their similarity.

2. **Demonstration sampling:** Select a representative question from each cluster and generate its reasoning chain using Zero-shot-CoT.

3. **Demonstration unification:** Iteratively update rationales to build a coherent pattern across all demonstrations.

These steps are designed to address the limitations of previous methods by creating a unified, representative set of demonstrations. The first two steps are similar to Auto-CoT [Zhang et al., 2023] with minor differences, while the third step is unique to our approach.

## Question Clustering

We begin by applying clustering to a set of questions $Q$. Each question is transformed into a fixed-size vector representation using Sentence-BERT [Reimers and Gurevych, 2019], which provides semantically meaningful embeddings. We then employ a $k$-means clustering model to categorize these vector representations into $k$ distinct clusters. Within each cluster $i$, the questions are ordered in a list $\mathbf{q}^{(i)} = [q_1^{(i)}, q_2^{(i)}, \ldots]$, based on their distance to the centroid of the cluster.

Unlike Auto-CoT [Zhang et al., 2023], where the number of clusters equals the number of output demonstrations, our approach allows for a greater number of clusters. This expansion enables the inclusion of a wider range of demonstrations in the unification process, facilitating effective learning from more diverse patterns and ensuring that the final pattern can be applied to a wider range of demonstrations.

## Demonstration Sampling

For each cluster $i$, we sample one demonstration $d^{(i)}$ by evaluating the questions in $\mathbf{q}^{(i)} = [q_1^{(i)}, q_2^{(i)}, \ldots]$ against predefined selection criteria. For each question $q_j^{(i)}$, we generate its rationale using the prompt 'Let's think step by step' [Kojima et al., 2022], following the methodology used in Auto-CoT [Zhang et al., 2023].

Our selection criteria follow [Zhang et al., 2023], which include two key constraints: (1) The question length should not exceed 60 tokens. (2) The corresponding rationale $r^{(i)}$ should be limited to no more than 5 steps. These constraints are chosen to ensure manageable and focused demonstrations. Steps are demarcated with '\n', and we count the steps by tallying the number of these separators.

## Demonstration Unification

The core of our method lies in the demonstration unification step, where we regenerate the reasoning chain for the sampled questions to form a convergent pattern.

In each iteration, each demonstration $d^{(k)}=q^{(k)}\circ r_0^{(k)}$ in the set $\mathcal{D}$ is updated once. The process is as follows:

1. Randomly select a demonstration $d^{(i)}$.

2. Use the remaining shuffled demonstrations $\mathcal{D}\setminus d^{(i)}$ as in-context examples.

3. Regenerate the rationale $r^{(i)}$ for the selected demonstration.

4. Replace the previous rationale with the newly generated one.

This process is repeated for each demonstration in the set. Through successive iterations, this leads to convergence, resulting in a uniform pattern across all rationales. The random selection and shuffling of demonstrations ensure that each rationale learns from a diverse set of examples, promoting robustness in the final pattern.

In our method, the number of clusters $k$ typically exceeds the number of output demonstrations $m$. This oversampling approach ensures that the final unified pattern is more robust, generalizable, and suitable for a wider array of samples. It's akin to information compression, where insights from a larger set of demonstrations are distilled into the final set of $m$ demonstrations, enhancing both the representativeness and adaptability of the output.

Note that although the demonstration unification process introduces more demonstrations, we apply the same number of demonstrations during the inference stage as used in prior approaches to ensure a fair comparison.

By unifying diverse rationale patterns into a coherent set of demonstrations, ECHO addresses the limitations of previous methods, particularly the potential for irrelevant or non-representative demonstrations in diverse sets. This approach aims to create a more effective and generalizable Chain-of-Thought prompting method. We demonstrate detailed implementation of ECHO in Appendix (ECHO Algorithm section).

# Why does it Work?

In this section, we offer some insight into why the proposed approach, ECHO, can be effective by presenting a mathematical framework for understanding its iterative refinement process.

**Research Question 1: Why will regenerate rationale using others as demonstrations result in convergence?**

Few-shot-CoT demonstrates that models naturally follow the patterns shown in few-shot demonstrations, providing reasoning steps before producing the final answer. Subsequent research, such as PAL [Gao et al., 2022], extends this concept by using code rationales to generate code for solving math problems. These studies collectively show that models not only follow the general question-answering structure but also inherit the patterns from these demonstrations.

**Research Question 2: Why will regenerate rationale using others as demonstrations yield better rationale?**

Consider a set of questions $\mathcal{Q}=\{q^{(1)},q^{(2)},\dots,q^{(m)}\}$. Assume that we can use Zero-Shot-CoT to arrive at the rationales (with answers): $\mathcal{R}=\{r^{(1)},r^{(2)},\dots,r^{(m)}\}$.

Auto-CoT performs the following: First, it constructs demonstrations $\mathcal{D}=\{d^{(1)},d^{(2)},\dots,d^{(m)}\}$, where $d^{(k)}=q^{(k)}\circ r^{(k)}$. During the inference stage, such demonstrations are then used as few-shot examples for generating "refined" rationales for the $i$-th instance, based on $\mathcal{{D}}\backslash d^{(i)}$, which we denote as $r_0^{(i)}$. Completing this refinement process leads to $\mathcal{R}_0=\{r_0^{(1)},r_0^{(2)},\dots,r_0^{(m)}\}$.

Empirically, Auto-CoT was shown to yield better results than Zero-Shot-CoT. Mathematically, this leads to the following hypothesis:

$$p(\mathcal{Q},\mathcal{R}_0)\geq p(\mathcal{Q},\mathcal{R})$$

where $p(\mathcal{Q},\mathcal{R})$ returns the probability for the set of rationales $\mathcal{R}$ to be assessed as correct for the set of questions $\mathcal{Q}$. This probability can be interpreted as a measure of the overall quality and accuracy of the rationales.

This equation shows that the set of refined rationales is likely to be more accurate than the original set of rationales, which were generated without any demonstrations. This improvement can be attributed to the beneficial effect of using in-context examples during the refinement process.

In practice, we can keep refining the set of rationales by reconstructing them. Specifically, we can first construct $\mathcal{D}_0=\{d_0^{(1)},d_0^{(2)},\dots,d_0^{(m)}\}$, where $d_0^{(k)}=q^{(k)}\circ r_0^{(k)}$. Repeating the above process, we arrive at $\mathcal{R}_1=\{r_1^{(1)},r_1^{(2)},\dots,r_1^{(m)}\}$.

If the above hypothesis for Auto-CoT is true, it is not unreasonable to state the following hypothesis as its extension:

$$p(\mathcal{Q},\mathcal{R}_1)\geq p(\mathcal{Q},\mathcal{R}_0)$$

We can repeat the above process $T$ times to arrive at a chain of inequalities:

$$p(\mathcal{Q},\mathcal{R}_T)\geq \dots \geq p(\mathcal{Q},\mathcal{R}_1)\geq p(\mathcal{Q},\mathcal{R_0})$$

This chain of inequalities argues why our proposed approach works empirically. Each iteration of refinement is likely to produce a set of rationales that are at least as good as, if not better than, the previous set.

It's important to note that while the above process updates the rationales in a batch mode, in our ECHO method, we adopted an online approach. In this approach, we utilize the recently updated rationales for the next instance within the same iteration. This online approach allows for more immediate incorporation of improvements, potentially leading to faster convergence to high-quality rationales. However, this behaviour may also negatively impact model performance, as some patterns are specific to particular questions and cannot be generalized effectively to others (we demonstrate performance on combined dataset in Section 6.7).

# Experimental Setup

## Tasks and Datasets

Following prior work on CoT, we evaluate our method on 10 reasoning datasets, including: (1) 6 arithmetic datasets: SingleEq [Koncel-Kedziorski et al., 2015], AddSub [Hosseini et al., 2014], MultiArith [Roy and Roth, 2015], GSM8K [Cobbe et al., 2021], AQUA-RAT [Ling et al., 2017], and SVAMP [Patel et al., 2021]; (2) 2 commonsense reasoning datasets: CommonsenseQA [Talmor et al., 2019] and StrategyQA [Geva et al., 2021]; (3) 2 symbolic reasoning datasets: Last Letter and Coin Flip [Wei et al., 2022].

## Models

Following Wei et al. [2022], we used the OpenAI API for our experiments. We chose GPT-3.5-Turbo-0301 in our main experiments because it is easy to access and more affordable. We also tested on Mixtral-8x7B-Instruct in the ablation study to validate the generalizability of our method. To ensure the reproducibility of our experiments, we fixed the temperature parameter at 0.

## Configuration

For our primary experiment, we set the following parameters:

- Iteration count: $T=1$

- Number of demonstrations for unification process: $k=m$ (equal to the number of output demonstrations). We followed Wei et al. [2022] for the number of demonstrations: 4 for AQUA, Coin Flip, and Last Letters; 6 for StrategyQA; 7 for CSQA; 8 for all other datasets.

# Results

## Main Experiment Analysis

The data presented in Table 1 offers a detailed comparison of various methods across three categories: Arithmetic, Commonsense, and Symbolic. In these evaluations, ECHO consistently outperforms Auto-CoT [Zhang et al., 2023] in each domain, matching Few-Shot-CoT's overall performance. Specifically, ECHO achieves an overall performance improvement of 1.7% over Auto-CoT. The overall performance of ECHO and Few-Shot-CoT stays the same, demonstrating ECHO's ability to match human-crafted demonstrations without manual effort.

Notably, Auto-CoT falls short in performance compared to Few-Shot-CoT across all domains, with a particularly significant gap of 1.7% in Commonsense reasoning. This suggests that Auto-CoT may not fully replace human effort in Few-Shot-CoT, highlighting the value of ECHO's approach. ECHO, on the other hand, aligns closely with Few-Shot-CoT's overall performance, indicating its potential as a significant advancement towards a powerful reasoning framework without manual efforts.

Interestingly, ECHO outperformed Few-Shot-CoT's average score in symbolic tasks by 3.0%, but slightly lags in arithmetic and common sense tasks by 0.6% and 1.1% respectively. We observe that symbolic reasoning tasks tend to have a more uniform pattern, making a single iteration sufficient for effective reasoning, whereas arithmetic and common sense reasoning tasks are more diverse in question contexts and answer patterns. We hypothesize that this lag in performance is due to the insufficient number of rationales and iterations for the self-harmonization process in these more diverse domains. This suggests the potential for further improving ECHO's performance, particularly in tasks with greater pattern variety.

[IMAGE: Figure 3 - Performance for ECHO initialized by manual prompts and Zero-Shot-CoT generated prompts with 0, 1 and 3 iterations. We conclude the generated prompts perform better after the unification process, while the manual ones perform better when applied directly.]

## A comparison between ECHO-generated and Manual Prompts

To understand the difference between ECHO and manually written prompts, we applied both to our method. Our experiments employ the average score from six arithmetic reasoning benchmarks.

The rationales of ECHO are initialized with Zero-Shot-CoT [Kojima et al., 2022], while manual prompts are crafted by humans. We initialized our method with both automatically generated and human-written prompts to check if ECHO can be directly applied to existing manual prompts. The outcomes, depicted in Figure 3, reveal three key findings:

1. Initially, manual prompts surpassed those generated via Zero-Shot-CoT.

2. After one iteration of ECHO, the performance difference diminishes significantly.

3. Following three iterations, automatically generated prompts exceed manual ones, achieving peak performance.

Table 2 presents three examples from both manually written and ECHO-generated prompts. Manually written prompts start coherent and maintain uniformity across iterations. In contrast, generated rationales begin highly varied but unify towards a singular pattern through ECHO iterations. This suggests that generated rationales provide a more diverse initial set of patterns, aiding in identifying the most effective pattern.

To quantify the convergence, we compared the average divergence between manual rationales (0 iterations) and generated rationales (3 iterations). Using "roberta-large-nli-stsb-mean-tokens", "sentence-t5-large", and "sentence-t5-xxl" as encoders, we computed pairwise cosine similarities and defined average divergence as 1 - average similarity. Table 3 shows that the average divergence is significantly reduced in the generated demonstrations, confirming ECHO's ability to unify diverse initial patterns into a coherent, effective set of prompts.

## Effect of Hyperparameters

Building on our initial insights, we explored how the initial diversity impacts our method's performance. We tested ECHO in a configuration where the number of demonstrations considered, $k$, exceeds the original count, $m$, in the self-harmonized process. This adjustment does not alter the number of output demonstrations used during inference.

The rationale behind this enhancement is to increase diversity by starting with a more varied set of patterns, potentially increasing ECHO's adaptability across different reasoning tasks. To maximize diversity, we chose the largest $k$ within the model's token limit.

Results of this adjustment, shown in Table 1, reveal some improvements in the arithmetic domain but a noticeable decline in overall performance. Based on observations from Figure 3, we hypothesize that a single iteration may be insufficient when dealing with a larger number of demonstrations, suggesting the need for more iterations.

To determine the optimal number of iterations, we tested ECHO across various iteration counts (Figure 4). In all cases, our method's average overall performance remained superior to Auto-CoT [Zhang et al., 2023]. However, we observed a trend of overfitting with excessive iterations, evident from the model's performance peaking at an optimal iteration count before gradually declining.

A case study (Appendix) provides further insights:

- After a single iteration, the model adopts a consistent rationale structure, with phrases like "Sure, let's break this down" or "First, we need to find" becoming standard openings.

- After 32 iterations, the model attempts to condense multi-step reasoning into a single step, resulting in complex equations with meticulous unit notation, suggesting an overemphasis on uniformity at the cost of conciseness and clarity.

Analysis of Figure 4 indicates that an iteration count of $T=4$ offers an optimal balance, with peak overall average performance and generalizability across various domains.

[IMAGE: Figure 4 - Performance of ECHO in different domains with iteration increases exponentially. We found that T=4 results in the best overall performance.]

[IMAGE: Figure 5 - Performance of ECHO and Auto-CoT under 3 different settings of demonstrations: one from each cluster, randomly sampled and all from same cluster. ECHO with more iterations gain more performance from diverse demonstrations.]

## Does ECHO Suffer from "Misleading by Similarity"?

The Auto-CoT approach [Zhang et al., 2023] acknowledges a vulnerability to "misleading by similarity." This occurs when incorrect demonstrations from Zero-shot-CoT [Kojima et al., 2022] closely resemble the target problem, causing the LLM to replicate mistakes.

To investigate this limitation, we conducted experiments on mathematical reasoning tasks under three conditions:

1. **Diverse:** Demonstrations chosen from the centroid of each cluster, ensuring diverse representation.

2. **Random:** Randomly selected demonstrations from the dataset.

3. **Uniform:** Demonstrations strictly from the same cluster as the target question, promoting similarity.

We set the iteration count $T = 1, 2$ and the number of demonstrations $m = 8$ for all tests.

Results, illustrated in Figure 5, reveal key findings:

- Unlike Auto-CoT, ECHO improved as demonstrations became more uniform, suggesting that closer relationships between demonstrations and questions enhance utility, outweighing the risk of misleading information.

- ECHO with one iteration performed best in the random setting, indicating effective learning from all available demonstrations, regardless of direct relevance.

- ECHO with two iterations illustrated that: (1) Overly diverse demonstrations may require more than one iteration to establish a uniform solution pattern. (2) Excessively similar demonstrations risk overfitting after just one iteration.

These findings suggest that diverse demonstrations, even if not directly related, can help build a robust, generalizable problem-solving framework. We conclude that a strategic mix of demonstrations, balancing relatedness and diversity, could offer generalization without sacrificing effectiveness.

Future work could explore a hybrid selection strategy that dynamically adjusts the demonstration set based on dataset properties, potentially optimizing the balance between diversity and similarity.

## Results with Mixtral-8x7B

In addition to GPT-3.5-Turbo-0301, we tested our method on 10 benchmarks aligned with the main experiments using Mixtral-8x7B. The results, reported in Table 4, show that our approach outperforms Auto-CoT by an average margin of +2.3% with Mixtral-8x7B. This consistency with our existing findings demonstrates ECHO's generalizability to other models.

However, we note that the overall performance metrics are lower than those achieved with GPT-3.5-Turbo-0301, indicating the significant impact of the underlying model used. Notably, our method failed to outperform Few-shot-CoT on average when applied to Mixtral-8x7B. We believe there are two main reasons for this outcome.

First, the quality of the generated rationales may differ depending on the model's capability, while the quality of manual prompts remains constant. In this case, the rationales generated by Mixtral-8x7B may not be as sophisticated as those produced by GPT-3.5-Turbo-0301.

Second, we observed a significant performance drop in the "Coin Flip" dataset. Consider an example question: "A coin is heads up. Irving flips the coin. Hans flips the coin. Moses does not flip the coin. Nicole does not flip the coin. Is the coin still heads up?" We found that the rationales generated by Auto-CoT or ECHO explicitly track the coin's state when compared with GPT-3.5-Turbo-0301. In contrast, the human-written prompts applied by Few-shot-CoT employ a shortcut solution by counting overall flips to determine the outcome. As the coin always starts heads up, if the number of total flips is even, the coin will remain heads up. We observed that Mixtral-8x7B is less adept at tracking coin states, which may explain its inferior performance on this particular task.

These observations highlight the potential limitations of automated rationale generation across different language models.

## Results with 50% Demonstrations

To assess the robustness of our method, we evaluated the impact of reducing the number of demonstrations by half. The results reveal interesting patterns across different approaches.

When we decrease the number of demonstrations by half, the overall performance of ECHO decreased by only 0.8%, compared to a 1.3% decline for Few-Shot-CoT. This smaller performance drop suggests that ECHO's unified rationales are more robust to demonstration reduction. We infer that the consistency achieved through ECHO's unification process allows each demonstration to effectively capture patterns from the others, minimizing the impact of reducing their number. In essence, the remaining demonstrations retain information from those that are dropped, contributing to this enhanced robustness.

Interestingly, Auto-CoT showed a slight performance improvement of 0.1% with the reduced number of demonstrations. This counterintuitive result suggests that diversity in demonstrations can sometimes hinder performance. We hypothesize that this improvement stems from the reduced diversity of demonstrations, which may lead to more consistent reasoning patterns.

These findings highlight an important insight: increasing the number of demonstrations (or "shots") does not necessarily lead to better performance. The quality and consistency of demonstrations, rather than just their quantity, play a crucial role in the effectiveness of reasoning methods.

## A Study on Combined Dataset

Similar to Zhang et al. [2023], our work requires access to the entire dataset to construct demonstrations. We assume that each dataset exhibits a certain level of internal similarity. For example, a math dataset should consist exclusively of math questions, while a yes-or-no dataset should contain only yes-or-no questions. However, some datasets are more diverse in nature. In this section, we aim to evaluate our method in a more complex scenario: we randomly select 500 samples each from both GSM8K and StrategyQA. GSM8K focuses on arithmetic tasks, whereas StrategyQA involves yes-or-no commonsense reasoning tasks. We apply our method to this combined dataset, and the results are presented in Table 6.

Among the 8 demonstrations that are automatically selected after clustering, 3 are taken from GSM8K and 5 from StrategyQA. We observed a performance drop on both datasets when using our method. This outcome highlights the limitations of our approach, as it attempts to identify a uniform solution pattern. Clearly, no single pattern can be effectively applied across two fundamentally different domains.

## Results with GPT-4o

We experimented with our method using the up-to-date GPT-4o model. Due to cost constraints, we conducted experiments only on GSM8K. The results are summarized in Table 7.

These results demonstrate that ECHO outperforms both Few-shot-CoT and Auto-CoT under this setup. However, we also observe that Zero-shot-CoT achieves comparable performance to ECHO in this scenario. This finding suggests that a superior model like GPT-4o can achieve strong results without demonstrations. It already has the capability to answer arithmetic questions in a well-structured format, making additional demonstrations potentially redundant.

# Conclusion

In this work, we propose a novel method called ECHO for improving chain-of-thought prompting in large language models. We have shown that our method results in consistent improvements in three domains, confirming the feasibility and significance of adopting the self-harmonization mechanism in the CoT prompting process. We also carried out extensive experiments and case studies to understand the behavior of the proposed approach. We hope that these insights can help to improve future automatic reasoning frameworks.

# Limitations

Our study has identified several limitations within the ECHO method:

**(1)** The method incurs a higher inference cost due to the necessity of an additional unification process for demonstrations. This process requires extra computational resources as it involves generating multiple inferences to consolidate the demonstrations into a coherent pattern. For a benchmark of $n$ samples, while the other method requires inference for $n$ times, our method requires $n + T \cdot k$ times ($T$ is the number of iterations and $k$ is the number of samples used). For example, for GSM8K, the ECHO with 4 iterations requires 5.8% more number of inferences.

**(2)** The method is prone to overfitting, which can lead to a decrease in generalizability. Although we have introduced an equation to estimate the optimal number of iterations, this equation may not hold universally across different domains or datasets.

**(3)** The method assumes a certain level of similarity within the data from which it learns, which might not always be the case. In scenarios where the data are highly unrelated or where the relationships between data points are complex, the unification process may struggle to recognize a representative pattern. Future work may explore adaptive mechanisms that can recognize and adapt to the diversity of the data, ensuring that the unification process remains effective in various types of problem.

**(4)** Similar to Zhang et al. [2023], our method can be applied to a dataset containing multiple questions rather than a single question.

# Appendix: ECHO Algorithm

**Algorithm: Self-Harmonized CoT**

**Input:** A set of questions $\mathcal{Q}$, the number of demonstrations for unification $k$, number of iteration $T$ and number of demonstration for the output $m$

**Output:** Demonstration list $\mathcal{D} = [d^{(1)}, \ldots, d^{(m)}]$

1. Encode each $q$ in $\mathcal{Q}$ by Sentence-BERT (Question Clustering)
2. Cluster all the encoded question representations into $k$ clusters
3. For each cluster $i = 1, \ldots, k$:
   - Sort questions $\mathbf{q}^{(i)} = [q_1^{(i)}, q_2^{(i)}, \ldots]$ in the ascending order of the distance to the cluster center
4. For each cluster $i = 1, \ldots, k$ (Demonstration Sampling):
   - For each question $q_j^{(i)}$ in $\mathbf{q}^{(i)}$:
     - Generate rationale $r_j^{(i)}$ for $q_j^{(i)}$ using Zero-Shot-CoT
     - If $q_j^{(i)}, r_j^{(i)}$ satisfy selection criteria:
       - Add $d^{(k)}=q^{(k)}\circ r_0^{(k)}$ to $\mathcal{D}$
       - break
5. For each iteration $t = 1, \ldots, T$ (Demonstration Unification):
   - For each demonstration $d^{(i)}$ in $\mathcal{D}$:
     - Create prompt $P$ with demonstrations $\mathcal{D}\setminus d^{(i)}$ in random order
     - Regenerate rationale $r_{new}^{(i)}$ for question $q^{(i)}$ using Few-Shot-CoT
     - Update $d^{(i)}=q^{(i)}\circ r_{new}^{(i)}$ in $\mathcal{D}$
6. Keep the first $m$ elements and drop the remaining elements from $\mathcal{D}$
7. Return $\mathcal{D}$

# Appendix: Data Statistics

| Reasoning Type | Dataset       | Size  | Answer Type     |
| -------------- | ------------- | ----- | --------------- |
| Arithmetic     | SingleEq      | 508   | Numeral         |
| Arithmetic     | AddSub        | 395   | Numeral         |
| Arithmetic     | MultiArith    | 600   | Numeral         |
| Arithmetic     | GSM8K         | 1,319 | Numeral         |
| Arithmetic     | AQUA          | 254   | Multiple Choice |
| Arithmetic     | SVAMP         | 1,000 | Numeral         |
| Symbolic       | Coin Flip     | 1,000 | Yes or No       |
| Symbolic       | Last Letter   | 254   | String          |
| Common Sense   | StrategyQA    | 2,290 | Yes or No       |
| Common Sense   | CommonsenseQA | 1,221 | Multiple Choice |

# Appendix: Effect of Incorrect Demonstrations on Performance

Our research indicates that demonstrations with incorrect answers do not necessarily impair overall performance. This observation is particularly evident in the context of the AQUA dataset, where a notable instance occurred: among the four demonstrations generated by ECHO, two contained wrong answers, while in the manually written prompts, all demonstrations were correct. Despite this, ECHO's performance significantly surpassed that of the human-written prompts.

This outcome suggests that in the ECHO method, the collective contribution of the demonstrations to the reasoning pattern is more critical than the individual precision of each demonstration. It appears that the model can effectively extract and leverage useful patterns from the demonstrations, even if some contain errors. This ability to distill valuable reasoning patterns from imperfect data underscores the robustness of the ECHO approach. It indicates that the model's effectiveness relies more on the breadth and diversity of demonstrations rather than their individual correctness. This finding opens up possibilities for utilizing a wider range of demonstrations, including those with inaccuracies, without necessarily compromising the model's overall performance.

# Appendix: More Demonstrations Helps Better Commonsense Reasoning

The overall performance of ECHO ($k=\max, T=1$) is not as strong as that of ECHO ($k=m, T=1$). We hypothesize that the reason for this is a single iteration is not enough when handling a larger number of demonstrations. To further investigate the impact of increasing the number of demonstrations in the unification process, we conducted experiments with both 8 and the maximum possible number of demonstrations in 4 iterations. ('maximum' refers to the highest number of demonstrations that can be accommodated within the token limit, approximately 20.) The results are illustrated in Table 8.

Our results reveal that in the arithmetic domain, the performance of ECHO under both conditions is comparable, with each achieving an average score of 83.1%. The notable difference emerges in the realm of commonsense reasoning. In this area, ECHO ($k=\max, T=1$) with the maximum number of demonstrations surpasses its counterpart by a margin of 3.0% across both benchmarks. This outcome suggests that the sensitivity to the number of demonstrations varies across different domains, with commonsense reasoning being particularly influenced by the count of demonstrations.
