Humans have fantastic brains capable of solving problems quickly and in amazing ways. Computers can solve a great number of problems more quickly than humans, but problem solving in humans and computers works in vastly different means. Richard Feynman said it best.
[A computer is] a glorified, high-class, very fast but stupid filing system.— Richard Feynman, one of the greatest men in history
Computers use brute force methods to solve problems. When that takes too much time, they can be programmed to use heuristics based on probabilty of success from particular states. When that seems too dumb, the heuristics can be trained by adjusting the probabilities gradually based on a variety of factors, and random elements can be introduced. This is the heart of machine learning through gradual mutation of heuristics. However, these heuristics as we now know them fail to capture one essential human quality, that of intuition.
Intuition is Essential to Human Understanding
I am a fan of chess. I am by no means a great chess player, but I appreciate the game. Chess is a fascinating game from the standpoint of information theory. The general problem of forming a decision on how to play a perfect game of chess is intractable. It is here how we see a fantastic example of the importance of intuition. World champion Garry Kasparov said it better than anyone.
Intuition is the defining quality of a great chess player. That’s because chess is a mathematically infinite game. The total number of possible different moves in a single game of chess is more than the number of seconds that have elapsed since the big bang created the universe.— Garry Kasaparov, Chess Champion
A human cannot consider a vast number of possible chess positions which could arise after playing a particular move, but only a few. A computer could consider more positions than a human could, or use heuristics to weight the effectiveness of a move. A human can do one different and essential thing, use their intuition to contribute to their decision making process.
Computer programming is an exercise in problem solving. Problem solving as a computer programmer consists of a cycle of thinking, writing code, and observing the behaviour of the program. Was the problem solved by your contribution? Did you fix the bug you experienced? How did the program behave before and after you modified the code? Did you write an automated test or use a formal proof to verify it does what you think it does?
Like chess, an essential trait for problem solving is intuition. If you are honest with yourself, you will admit that you honestly have no idea why you believe that a change you have made to a program solves a bug. There will come a time when you say to yourself, "I believe I have fixed the bug," but you will not be able to provide a completely rational explanation as to why you hold that belief. Often, you will hold such a belief incorrectly, and have to later correct yourself. However, an incorrect belief on occasion is not reason enough to discount all belief in lieu of a rational explanation. Ideas backed by flawed arguments can still hold true.
A Challenge for A.I.
I propose a challenge for A.I. researchers. I posit that either intuition is an essential capability which no A.I. can possibly be trained to achieve, or that intuitive reasoning can be learned by computers.
If computers can learn intuitive reasoning, then we might be able to safely establish that computers will be capable of achieving the same level of understanding any human could achieve. This would indicate that computers would be capable of exhibiting the same mental capabilities of any human, perhaps even greater intelligence.
If we discover that A.I. is completely incapable of utilising intuitive learning, then we must therefore establish that computers can never totally learn as humans learn. If computers are incapable of learning as humans learn, then we will have to accept that humans will always be an important factor in generalised problem solving.
I personally believe that computers might be able to design near perfect heuristics for a game of chess and capture your pieces swiftly, but they will never truly capture the genius of Bobby Fischer.