When you say “massive amount of energy” are you comparing the energy requirements to a single human or to the billions of years of solar and geothermal energy that went into producing the human species?
I don’t think this is an apt comparison, but I do think the amount of energy it takes to grow a human into brain maturity in adulthood is an interesting one. Brains + bodies over a 20 year development cycle is still probably much less than training even a low quality Llm.
Let’s say a human needs an average of 2000 calories a day. A calorie is roughly equivalent to 1 Watt hour, so over 20 years, it takes about 15 MWh to sustain a human.
Let’s say a single A100 has a peak power draw of 250W, and you need 100 to train an LLM. So each hour of training consumes 25,000 Wh of energy. 15 MWh / 25,000 W = 600 hours, or 25 days, which is probably pretty close to the true training time.
So the numbers are actually pretty close. But a human brain doesn’t start out as a set of random weights like an LLM. The human brain has predefined structure that’s the result of an extremely long evolutionary process.
It's probably taking the analogy too far, but perhaps the brain's predefined structure is akin to the original LLM training and our "life" is the fine tuning.
I wonder how many MWh the entirety of evolution represents.
By that token the amount of energy for neural networks will be bound to some extent by the development of the biosphere and the creators of neural networks.
Not really? The point is that most artificial neural networks are started from basically zero (random noisy weights), where as a human neural network is jump-started with an overall neural structure that has been shaped by millions of years of evolution. Sure, it's not fair to compare the overall energy required to get there, but the point is just that a biological neural network starts with a huge headstart that is frequently forgotten when talking about efficiency.
