2021-04-19 - less is more

recently in one of the Lex Fridman's podcast episodes i came across an interesting example of how less is more it in data science. imagine you have a library with all the science books ever written. now imagine you have another library with all possible books of 500 pages (i.e. first book is all “AAAA…AA”, next is “AAAA..AB”, etc…). in the former case you have far more data… but it is essentially useless. even though in the first library you have far less data, you still have more information. all thanks to having less data, but of a greater value.

the problem with data-driven AI is that it does not generalize in a “smart way”. it does find patterns, but it's all about correlations in the data – it does not determine underlying laws of the universe. eye opening example is with learning to add two numbers. if you'd feed only even numbers to a computer, it will learn that lowest bit of the response is always zero, because it was zero in all the examples. human will understand the rule behind addition and would never make such a mistake. similarly if you'll teach addition of up to 10 digits, humans will have no issue with adding 20 digits numbers, using the same algorithm. this would not work that smooth in case of an AI, though.

both of these examples come down to another interesting idea in AI – intelligence is the ability to compress. instead of trying to remember all possible ways a ball can fly in the air when throw, you can instead research physics behind it and boil it down to a single equation, that will cover all the infinite ways a ball can be thrown.

while achievements of today's AI are really impressive, there are also fundamental limitations with our current approaches and learning algorithms we use. AI still cannot learn on the fly – there are modes of learning and then using of accuired knowledge. w/o a major breakthrough in this field, i would not expect fully autonomous cars anytime soon.