2025-05-30 - high level view on AI

in the early 2000s AI was a bit of a niche tool. it was useful, but was applicable in certain cases, required a lot of specialized knowledge to be used… and results were often not very impressive. even where it delivered quality results it wasn't even remotely prompt. i planned to use AI in my master thesis, to control moving robot in real time. however it quickly turned out that with 0.5fps it's just not there. i had to fallback to more “classical” take on machine vision, and offload computations to my main PC to get 10x speedup, that'd be be barely there, to be able to control my robot in real time.

fast forward ~10 years and deep-learning revolution has begun, backed up by rapid GPUs advancements. it was probably 1st time where AI became the news, since winning with human in chess in 1997. with an advent of LLMs and GPUs AI is nowadays a commodity tool, that regular people use to solve their problems at hand. i'm one of those people, too – i use AI on a daily basis both for work and private / pet projects.

however very often i end up in disputes against the grain of how awesome AI is and / or how it will be able to do ~anything very soon, because the progress is so fast nowadays. LLM can be really good at providing you with spinets of knowledge on a particular data from a training set, or connect some dots from the data used there. however with problems outside of training set, it shows crucial lack of fundamental understanding of the problem… while still producing plausibly looking “solutions”.

on top of that there's autonomous vehicles (AV) revolution, that's just just the round a corner… for at least a decade. in the press i mean. in real live? it's gonna be ~40 years, soon. not many people know that first autonomous car drove parking lots in 1986 and drove on a highway in 1987 at 100km/h. in 1994 the computer fit in the trunk and was driving on a highway, as a part of a regular traffic. we're now over 30 years after that, and we're still not quite there. so called “AVs” are available in some places in the USA, but are far from the safety standards we'd expect and cannot cope well with unusual decisions.

i think there is a common, underlying pattern in both cases. modern AI is effectively a statistical model – correlation based representation of what's in the training data. as long as data is well represented in the training data – it's usually quite decent. things that require “common sense” to be applied to situations that were never in the training data – like oddball situations on the road, or a new variation of some game – it won't preform well.

i think we have fundamentally a few major flaws in the current take on AI:

1. AI does not think – it statistically correlates
2. AI does not handle well data outside of training set – it fails short on “long tail” problems
3. we're running out of training data – AI corporations literally scraped whole internet already
4. AI does not learn on the fly – it can be taught once and then only used

on top of that there's limits on what transformers can do. in particular, while its conversation memory is O(N), the query length if O(N^2).

training itself also is hitting some fundamental limits: image taken from AI winter is well on its way article. it can be seen there, how drastically more compute power is needed to train new models. this of course directly translates to money. we already use models that cost $100M to train, with current record being gemini ultra that cost almost $200M.

i think we're currently around the peak of hype cycle. the first one, to be precise.

after LLMs and AVs took world by storm, we're starting to see some real-world limitations and also find places where these model excel. i'm not so sure nowadays if we'll hit another AI winter soon, as (at least for LLM-AI) we actually do see widespread adoption of AI of today. i'd however expect hype to start showing age now.

to really be able solve general problems, and that i believe will also include AV development, we might need a few more revolutions, of LLM-scale, to solve fundamental constraints we're dealing with now. i do not believe these are solvable with incremental upgrades, that we see to AI models of today. we need something new. something that was not yet invented… and we'll see what time will bring. :)