AI: What’s in Store for '24

AI: What’s in Store for '24

Here at the start of 2024, the future of AI stands at an exciting juncture. Given the amazing developments we’ve witnessed over the last twelve months or so, it’s a safe bet that 2024 will be another mind-blowing year. But what, specifically, will the developments be? I asked some fellow members of the Tenyx team, as well as a few external friends and colleagues, to give me their expert vision of what the year holds for AI. I’ve divided them into four groups: Consciousness, Economics, Large Language Models (LLMs), and Active AI.

1. Machine consciousness next year? Of course not, but...

Let’s start with the most provocative topic: AI consciousness. Leading computational neuroscientist and author of the best-selling book Being You, Anil Seth, makes a safe prediction — but then adds a twist: “In 2024, AI will not become conscious, but increasingly sophisticated combinations of language models and generative visuals - such as deepfakes - will deliver systems that increasingly appear to be conscious." Interacting with such systems as if they were conscious may become well-nigh irresistible.

Human-AI interactions will increase not only in quality, but in quantity. My Tenyx teammate Sarath Shekkizhar predicts a surge in human interactions with AI agents, potentially surpassing day-to-day human-human interactions, at least in developed nations. Because of this, he stresses, “it is crucial that we acknowledge and proactively navigate a secure path amidst this rising influence and integration of AI into our everyday lives."

2. The Economics of AI

Lower Costs

The lion’s share of the predictions I received concerned the costs and economic effects of AI — and there was reassuring unanimity in their expectations: lower costs.

Chien-Wei Lin, also at Tenyx, bases his predictions on data from Taiwan Semiconductor Manufacturing Corporation (TSMC). In Q3, TSMC reported that although AI-driven processors presently contribute 6% to their revenue stream, projections indicate a promising trajectory, with TSMC anticipating a 50% Compound Annual Growth Rate (CAGR) in this sector over the upcoming five years (see, e.g., https://seekingalpha.com/article/4636287-tsmc-ai-winner-at-attractive-valuation). Chien-Wei Lin’s analysis: “This growth is likely to drive an increased supply of GPUs, subsequently leading to a reduction in the costs associated with GPU instances.”

AI on the Edge

This surge in demand will have knock-on effects. Chien-Wei Lin expects it to “steer chip manufacturers towards prioritizing the development of energy-efficient inference chips tailored for edge devices in the forthcoming year.” This dovetails with one of Paco Nathan’s predictions. Nathan, a respected AI Industry veteran, predicts “an increase in low-power/embedded/edge use cases for large models.” Nathan also independently predicts a continued drop in the Inference costs of large AI models, which will, he foresees, “change the landscape of use cases”.

Empowering AI Start-Ups

Sarath Shekkizhar is in broad agreement with these expectations for 2024, predicting “a substantial and exponential reduction in AI system costs, driven by advancements in both infrastructure and algorithms.” He expects the effects of this will be far-reaching: “This shift towards quicker and more efficient methodologies is poised to reshape the foundation of next-generation AI architectures and propel certain AI companies to the forefront.” One upshot will be that smaller models will be more powerful than they are at present.

Impact on Labor

Jobs centered around data creation and refinement will continue to proliferate, perhaps doubling globally, presenting a significant boost to job markets, especially in rural areas. This surge in opportunities has the potential to uplift hundreds of thousands of individuals out of poverty, marking a pivotal shift in economic landscapes worldwide.

3. Large Language Models (LLMs)

Currently, Large Language Models dominate the AI scene. What can we expect for them in 2024?

Again, Nathan and Shekkizhar are in broad agreement.  Nathan predicts that we’ll see “a large range of specialized tasks solved by deep learning in general, thanks to investments in tooling due to LLMs.”  Shekkizhar foresees a transition away from one-size-fits-all approaches, particularly in enterprises. "I believe we will see a pronounced shift towards customized LLMs, underlining the imperative for personalization beyond one large pre-trained model + prompt engineering."

Angie Howard, a Natural Language Engineering Lead at Tenyx, concurs, adding that we should expect to see more (and more powerful) Open Source LLMs: “More Llamas, Mistrals, etc. matching or exceeding GPT-4’s capabilities but optimized for on-premise use.” This will have an especially big impact in the commercial sector: “More and more companies will develop ‘in-house’ LLMs fine-tuned to their specific use-cases rather than focusing on generalizability. This will be an advantage for them, especially when they support niche domains.”

An important driver of the predicted increase in LLM power will be improvements in fine-tuning methods. Howard anticipates the development of “more ‘clever’ ways to fine tune LLMs to remember what people want, whether that be mitigating catastrophic forgetting, not hallucinating, staying within ethical bounds, etc.” She anticipates that a knock-on effect of these new fine-tuning methods will be an increase in LLM creativity: “LLMs/transformer generated tech will become increasingly more creative when coupled with these novel fine-tuning methodologies. Overfitting will be less of a problem and we will be able to generate novel content based on better priors.”

But we may see the dominance of LLMs challenged by alternatives. Damjan Kalajdzievski, another of my Tenyx colleagues, foresees a significant shift in AI models. "Linear time attention methods will increasingly supplant the original transformer architecture,” he predicts. To support this, he points to the most recent such alternative, “MAMBA” (https://arxiv.org/abs/2212.14052), which is reported by its creators to significantly outperform vanilla transformers. "These types of models have been the subject of increasing research attention, as they open up new research directions and applications." For example, Kalajdzievski points to applications that involve learning very long sequences, such as DNA data. Concerning these alternatives to vanilla LLMs and transformers, Kalajdzievski notes: “Such models have seen some popularity in the open-source LLM community with the RWKV model (https://arxiv.org/abs/2305.13048), and this popularity will only increase as linear time attention models and their advantages become further fleshed out."

4. Active AI

Currently, almost all foundational Machine Learning models learn by passively optimizing a loss function over a dataset that is provided to them, a dataset that is not generated through their interactions with the world (and which therefore tends to be very similar for all models). Anil Seth predicts that this paradigm will begin to be disrupted in 2024: "ML models will move towards learning by actively seeking data rather than mass osmosis of the internet.” The information that one gets as the result of a contextualized query is greater than the (passive) content of the result of that query. This will be a game changer.

Related to Seth’s prediction, I’ll make one of my own: I think “pollution” of the public Internet dataset as a result of unconstrained contributions to the internet on the part of generative AI models (most of which are of very poor quality) will be taken more seriously as an issue, and wide-ranging solutions will be offered (though I hold out little hope for a consensus on what, if anything, should be done).

What are your predictions for AI in 2024?

‍

Contributors:

Angie Howard is a Natural Language Engineering Lead at Tenyx.

Damjan Kalajdzievski is a Research Engineer Lead at Tenyx.

Sarath Shekkizhar is a Research Engineer Lead at Tenyx.

Chien-Wei Lin is a Production Engineer Lead at Tenyx.

Paco Nathan is Managing Partner at Derwen, Inc., author of Latent Space, and a tech industry veteran.

Anil Seth is a Professor of Cognitive and Computational Neuroscience at the University of Sussex, and is the author of Being You: A New Science of Consciousness.

Ron Chrisley is Co-Founder and Chief Scientific Advisor at Tenyx, and is Professor of Cognitive Science and Artificial Intelligence and AI at the University of Sussex.