Foundation models
Recently, the field has experienced a paradigm shift due to the introduction of so-called foundation models – large-scale models that serve as the basis for various applications. Instead of being trained from scratch for each specific use case, they are pre-trained on vast amounts of data, learning its patterns and structure. In addition, they require relatively little domain-specific data to be customized for a specific task, allowing for fast and efficient data processing.
They have shown great potential in tasks like understanding and generating text, recognising and analysing images, and organising and processing data in graphs. Foundation models can be used to analyse images, generate artwork, help make personalized recommendations, assist in healthcare, detect fraud, model climate patterns, and more. Today, the most famous examples of foundation models are the so-called language models such as GPT, that are made to understand and generate human-like text. Trained on large amounts of text data from sources like books, websites, and other written materials, they have achieved remarkable performance across a range of language-related tasks, providing a powerful starting point for developing more specialised AI applications, including virtual assistants, translation, and content generation. [1]
How can we use foundation models?
While the most famous examples of foundation models are provided by OpenAI, there are other alternatives. They include commercial ones accessible through paid APIs, and public ones that are openly available for downloading, modifying, and deploying in your own environment. [2]
While currently large language models are expensive to train and deploy due to their size, efforts are underway to make them smaller and more efficient without sacrificing their capabilities. [3, 4]
What is next?
Datasets used for training AI models usually do not fully represent the underlying processes, but instead provide observations that can be used to find statistical patterns in the data. Thus, even the most efficient models that outperform humans at certain tasks, still face fundamental challenges at reasoning and capturing causal relationships between events. Researchers argue that one way to address this is to provide autonomous agents with the ability to actively interact with the world around them. Instead of passively learning from large amounts of data such as text and images from the internet, these agents would actively collect the necessary data, and continuously improve their understanding of the world. [4]
Furthermore, there is a fundamental difference between finding correlations in the data and identifying causes and effects of events. As humans, we are often able to predict outcomes of events we have never observed. For instance, we do not need to drop a glass on the floor to understand that it would break; in contrast, current AI models need to be exposed to such observations through training data. An interesting research direction, referred to as causal machine learning, aims to provide tools for formalizing causal relationships in the underlying processes, and a way to estimate the effects of interventions and model counterfactual scenarios. [5]
What does it mean for SEB?
Foundation models open exciting opportunities for every individual and our organisation as a whole. Help in writing and debugging code, summarising and analysing large amounts of unstructured text, detecting anomalies in text data, and drafting reports and articles are just a few examples of applications that can be useful across the bank. While many of us have already tried to use services such as ChatGPT, systematic exploration, understanding, and adoption of foundation models is essential for us to future-proof our ways of working. Certainly, we should always remember that, no matter how impressive current AI tools are, they lack human intelligence, and therefore we should not blindly rely on them when making important decisions, and instead must always verify the output. Finally, staying up to date with the advances in other promising areas of AI that are still not mature enough to be widely applied in practice, such as causal machine learning, will help us stay ahead of the progress curve.
Original article can be found here
References for further reading
[1] A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT
[2] Google "We Have No Moat, And Neither Does OpenAI"