The predictive capabilities of an ensemble of LLMs
The predictive capabilities of an ensemble of LLMs
In recent months, the field of artificial intelligence (AI) has seen remarkable advances, particularly with the development of large language models (LLMs). These models, built on the Transformer architecture, have demonstrated an impressive ability to perform a wide range of tasks, from natural language processing to complex decision-making, often rivaling or even surpassing human capabilities in specific areas. The advent of LLMs such as GPT-4, Claude 2, and others has sparked growing interest in their potential applications across various fields, including forecasting.
Forecasting, which is the process of predicting future events, has traditionally relied on human expertise and the so-called “wisdom of crowds” effect. This phenomenon occurs when the collective judgment of a large group of individuals, each with their own knowledge and perspectives, produces predictions that are often more accurate than those made by individual experts. The effectiveness of this method has been well documented in a variety of fields, such as economics, politics, and technology.
However, the question remains whether LLMs can replicate or even surpass the prediction accuracy of human crowds in real-world forecasting scenarios. While some studies have explored the forecasting capabilities of individual LLMs, the results have been mixed, with LLMs often underperforming aggregates of human crowds. This raises the possibility that the underperformance of individual LLMs could be mitigated by using an ensemble approach—aggregating the predictions of multiple LLMs to exploit a similar “silicon crowd wisdom” effect.
The article by Schoenegger, Tuminauskaite, Park and Tetlock, “ Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy ", aims to fill this gap by examining whether an ensemble of LLMs can achieve forecasting accuracy comparable to that of human crowds.
In Study 1, the authors evaluate the predictive accuracy of a diverse set of twelve LLMs on 31 binary forecasting questions and compare it to the accuracy of a human crowd drawn from a large forecasting tournament. The authors also examine potential biases in LLM predictions, such as acquiescence bias, that could affect their accuracy.
In Study 2, the authors further explore whether the accuracy of LLM predictions can be improved by incorporating human cognitive inputs. Specifically, the authors test the impact of submitting median predictions from a human crowd to LLMs and assess whether this information improves the accuracy of their predictions.
The results of these studies have significant implications for the future use of LLMs in forecasting and other real-world applications. By demonstrating that LLM ensembles can match the accuracy of the human crowd, this research paves the way for new and effective methods to leverage AI in decision-making processes across various industries. Furthermore, it provides insights into the potential synergies between human and artificial intelligence, particularly in contexts where accurate forecasts are essential.
The potential to improve LLM forecasts with human data opens the door to collaboration between humans and artificial intelligence. By combining the strengths of human and artificial intelligence, organizations can develop more robust and accurate forecasting systems that leverage the best of both worlds. This hybrid approach could be particularly effective in scenarios where data is sparse or highly uncertain, and human judgment is crucial to interpret complex signals.
Study 1: “LLM set” approach
The main objective of Study 1 was to evaluate the prediction abilities of an ensemble of large language models (LLMs) and compare their accuracy to that of a human crowd in a real-world prediction tournament. This study sought to determine whether aggregating predictions from multiple LLMs can reproduce the “wisdom of the crowd” effect traditionally observed in human forecasting.
To construct the LLM set, the authors selected twelve diverse models representing a broad spectrum of current major language models. The set included both proprietary and open-source models, with varying architectures, sizes, and training data characteristics. The models used in this study are: GPT-4; GPT-4 with Bing; Claude 2; GPT-3.5-Turbo-Instruct; Solar-0-70B; Llama-2-70B; PaLM 2 (Chat-Bison@002) Coral; Mistral-7B; Bard (PaLM 2); Falcon-180B; and Qwen-7B.
Access to these models was via publicly available web interfaces, allowing the study to replicate a real-world application scenario. No APIs were used, and default parameters (e.g., temperature) were applied uniformly to all models. This approach was chosen to maximize the accessibility and reproducibility of the ensemble method, while maintaining heterogeneity in model design and function.
The forecasting questions used in this study were drawn from a public forecasting tournament held on the Metaculus platform between October 2023 and January 2024. This tournament attracted 925 human forecasters, who provided predictions on a wide range of binary probabilistic questions. The questions covered a variety of domains, including geopolitics, economics, technology, and science. Examples of questions included predictions about interest rates, political election outcomes, and advances in consumer technology.
For each question, the median prediction of the human crowd was used as a reference for comparison with the entire LLM. A total of 31 binary questions were selected for analysis, ensuring a diverse set of forecasting challenges. For each forecasting question, the authors queried the twelve LLMs individually, asking each model to provide a probabilistic forecast for the binary outcome. Each model was queried three times for each question to account for the variability in their responses. Predictions were recorded and aggregated by calculating the median prediction of each model across all three queries.
The final ensemble forecast was obtained by calculating the median of the twelve median forecasts of the individual LLMs. This ensemble approach was designed to take advantage of the diversity of the models, with the expectation that aggregating the forecasts would reduce biases and errors of the individual models, thereby improving the overall accuracy. To ensure a fair comparison, the human crowd forecasts were recorded at the same time as the LLM ensemble forecasts. Specifically, the human predictions were collected at the end of the day in which the LLM predictions were captured, ensuring that the human and LLM predictions were made under similar temporal conditions.
The results of Study 1 suggest that the LLM ensemble approach has the potential to replicate the “wisdom of the crowd” effect in the context of AI-driven forecasting. By aggregating predictions from multiple diverse LLMs, the ensemble was able to achieve levels of accuracy comparable to those of human forecasters, challenging the traditional notion that human crowds are the gold standard for forecasting.
The results also highlight the importance of diversity within the ensemble. The varying performance of individual models underscores the value of aggregating forecasts from multiple sources to reduce the impact of biases and errors inherent in any single model. Furthermore, the ability of the LLM ensemble to match human accuracy, despite the presence of acquiescence biases in individual models, further demonstrates the robustness of the aggregation approach.
Study 1 provides strong evidence that LLM ensembles can serve as a viable alternative to human crowds in forecasting applications. This result has significant implications for the future of AI-based decision making, suggesting that organizations could leverage LLM ensembles to generate accurate forecasts across a wide range of real-world scenarios.
Study 2: Improving Human-Directed Forecasts
Building on the results of Study 1, Study 2 explores the potential for improving the forecast accuracy of LLMs by incorporating human input. While Study 1 demonstrated that an ensemble of LLMs could match the accuracy of a human crowd, Study 2 examines whether exposing LLMs to human-generated predictions can further improve their forecasting performance.
For this study, the authors focused on two “frontier” models from the ensemble used in Study 1: GPT-4 and Claude 2. These models were selected because of their high performance and widespread use in various applications, making them ideal candidates to examine the impact of human input on LLM prediction accuracy.
As in Study 1, the forecasting questions were drawn from the same set of 31 binary probabilistic questions used in the Metaculus Forecasting Tournament. These questions cover a variety of topics, including geopolitics, economics, and technology, ensuring a robust test of the LLMs’ forecasting abilities.
For each question, GPT-4 and Claude 2 were asked to provide an initial prediction using the same methodology described in Study 1. Next, the models were exposed to the human crowd-generated median prediction for the same question. This human median comes from the collective contribution of 925 human tipsters who participated in the Metaculus tournament.
The study used an internal model, where the prediction accuracy of each LLM was assessed before and after exposure to the human median prediction. This design allowed a direct comparison of model performance with and without human input.
The process consisted of two phases for each forecast question:
1. Phase 1: Initial forecast – LLMs provided their forecasts without any external input, as in the procedure of Study 1.
2. Phase 2: Forecast Update – After receiving the human median prediction, LLMs were instructed to update their forecasts, if necessary. The models were asked to reconsider their initial forecasts, incorporating the new information while balancing the risk of over-reliance on crowd judgment and the possibility of underutilizing valuable human intuition.
The results of Study 2 provide compelling evidence that human cognitive input can significantly improve the accuracy of LLM predictions. By incorporating median predictions from a human crowd, LLMs not only improved their accuracy but also demonstrated greater confidence in their predictions, as shown by the reduction in prediction intervals detailed in the paper.
These results highlight the potential of human-AI collaboration in forecasting tasks. While LLMs bring advanced data processing capabilities and the ability to process large volumes of information, human input provides valuable cognitive insights that can refine and improve machine predictions. This symbiotic relationship between human and AI could be particularly beneficial in high-stakes decision-making scenarios where forecast accuracy is critical.
Furthermore, the strong correlation between initial forecast deviations and the magnitude of adjustments highlights the ability of LLMs to efficiently incorporate external information and adjust their forecasts accordingly. This suggests that LLMs do not simply follow a simplistic updating mechanism but engage in a more complex reasoning process that takes into account the relevance and weight of human input.
However, the study also found that while LLMs benefit from human input, their updated forecasts are less accurate than a simple average of human and machine forecasts. This suggests that while LLMs are capable of improving their predictions, there is room for further refinement in how they integrate and apply human insights.
Study 2 demonstrates the substantial benefits of combining human cognitive input with LLM forecasting capabilities. This hybrid approach could serve as a powerful tool in diverse domains, improving the accuracy and reliability of predictions in contexts ranging from economics to public policy. Future research could explore more sophisticated methods of human-AI integration, which could lead to even greater improvements in forecast accuracy.
Implications of AI for forecasting and decision making
Study 1 demonstrated that an ensemble of LLMs could reproduce the “wisdom of the crowd” effect traditionally associated with human groups. By aggregating the predictions of twelve diverse LLMs, the ensemble was able to achieve forecasting accuracy equal to that of a human crowd. This finding is important because it challenges the conventional wisdom that human crowds, due to their diversity of thought and individual cognitive abilities, are inherently superior at prediction tasks.
The success of the LLM ensemble suggests that the diversity of training data, architectures, and methodologies of different LLMs can compensate for biases and errors of individual models, resulting in more accurate and reliable predictions. This “wisdom of the silicon crowd” effect not only supports the use of LLM ensembles in practical forecasting applications, but also opens new avenues to leverage AI in decision-making processes where accurate predictions are essential.
Furthermore, the ensemble approach was found to be robust even in the presence of some biases, such as the observed acquiescence bias, where LLMs tend to predict positive outcomes more frequently than warranted. The median aggregation method used in the ensemble was found to be effective in countering this bias, demonstrating the potential of LLM ensembles to provide balanced and accurate predictions.
Study 2 built on the foundations established in Study 1 by exploring the interaction between human cognitive input and LLM forecasting abilities. The results revealed that exposing LLMs to human-generated predictions could significantly improve their accuracy. Both GPT-4 and Claude 2 improved their scores after incorporating median predictions from the human crowd, indicating that human insights can serve as valuable inputs to refine machine-generated forecasts.
This finding highlights the potential of human-AI collaboration in forecasting tasks. While LLMs excel at processing large amounts of data and identifying patterns, human forecasters bring a different type of cognitive processing that can improve the quality of forecasts. By integrating these two approaches, organizations can achieve higher levels of forecast accuracy, especially in complex or uncertain environments where human intuition and experience play a crucial role.
The narrowing of prediction intervals observed in Study 2 further supports the notion that LLMs may become more confident in their predictions when informed by human judgment. This increased confidence is indicative of the ability of models to appropriately adjust their predictions in light of new information, balancing the risks of overconfidence and underuse of human input.
However, the study also highlighted that while LLMs benefited from human input, their updates were not as accurate as a simple average of human and machine predictions. This suggests that LLMs, despite their advanced reasoning capabilities, may still struggle to fully integrate and apply human insights in a way that maximizes accuracy. Future research could explore more sophisticated methods of combining human and machine predictions, which could lead to even greater improvements in forecasting performance.
The results of this research have significant implications for the future of AI in forecasting and decision-making. The demonstrated success of LLM’s ensemble approach suggests that organizations can increasingly rely on AI-powered predictions, reducing the need for costly and time-consuming human forecasting tournaments. This could prove particularly useful in industries where fast and accurate forecasting is essential, such as finance, politics, and technology.
However, the study’s findings also caution against overreliance on AI alone. While LLMs and their ensembles have shown remarkable capabilities, they are not infallible and can benefit greatly from human supervision and input. The limitations observed in integrating human ideas in Study 2 suggest that there is still work to be done to refine how LLMs integrate external cognitive input.
The research presented in the paper opens several avenues for further experimentation. One area of interest is the continued development of methods for integrating human and machine predictions. While Study 2 showed that human input can improve LLM predictions, it also revealed that there is room for improvement in how these models process and apply this input. Future studies could explore different aggregation techniques, weighting schemes, or iterative feedback loops that could better capture the strengths of both human and AI predictions.
Another promising direction is the expansion of the LLM ensemble approach. Although this study used twelve models, future research could investigate the effects of larger and more diverse ensembles. Increasing the number and variety of LLMs in the ensemble could potentially lead to even greater accuracy and robustness, especially in very complex or novel forecast scenarios.
Finally, there is a need to explore the ethical and practical implications of deploying LLM ensembles in real-world decision-making contexts. As AI becomes integrated into critical forecasting tasks, it is essential to consider potential risks, such as overconfidence in machine predictions, transparency of AI decisions, and liability in the event of errors. Addressing these concerns will be essential to ensuring that AI-driven forecasting systems are both effective and trustworthy.
To learn more: read our article on RIG vs RAG et GraphRAG.