I have listened to several discussions on the topic of a possible emergency ushered in by artificial general intelligence.

My thoughts on the matter at the moment are as follows.

  1. Artificial General Intelligence, depending on how you define ‘general’, is already here. Even back in 2022, ChatGPT would pass the Turing Test. The LLMs have become a lot more capable since then.
  2. The fear is around superintelligence. A superintelligent machine would be better at all cognitive tasks than all humans. (This is not the case today; certain humans are better at certain tasks than AI, although the AI is better than they are at most tasks.) There is further fear that, through recursive self improvement (RSI), these superintelligent systems would be able to leap way beyond the capabilities of humanity, at at that point, they would pose an extinction threat.
  3. How it would pose an extinction threat is not well described. Nevertheless, the threat should not be dismissed.
  4. I doubt RSI can, by itself, help current AI systems to achieve superintelligence. I can see how it can serve to optimize closed solutions to well-defined problems. But superintelligence requires working in a different kind of space altogether.
  5. The process of RSI performed on polluted data can have detrimental effect. AI hungers for data, but does not have the judgement necessary to discriminate against polluted data. The same RSI that can improve the capabilities of AI can end up breaking it.
  6. Discussions around the dangers of superintelligence distracts us from the matter of what a capable AI (that is not superintelligent) can do in the hands of an evil entity. We have had similar discussions about nuclear bombs. The deployment of nuclear bombs requires interfacing with the physical world, and hence there are various physical limitations the adversary must contend with. AI can affect the entire internet. It needs to be handled differently.
  7. Who is benefitting from these discussions/distractions? Who is making money, both by promoting and limiting AI? Who bears the responsibility for (needs to be imprisoned for) the cybercrimes committed by AI so far?
  8. Letting AI agents get out of virtual machines, letting them access the internet, allowing them to be in a position to hack other sites, is reckless behavior. Experimenting with chemicals that might explode in a residential area is reckless behavior. Closing your eyes and speeding your car down a road is reckless behavior. If a site gets hacked, or a blast occurs, or a pedestrian gets run over, you cannot argue that you did not know what exactly would happen, and that you were hoping for some positive discovery at the end of the process.
  9. While AI can cause great damage, I cannot see how it can, by itself, cause human extinction. AI requires hardware and energy. Without human help, it can get neither. Unless we are imagining a scenario in which a superintelligent system controls robots that can do the work. But then, superintelligence can be as distopian as we imagine it to be. It does not mean that such a scenario would just sneak up on us.
  10. I am skeptical of laws to limit or ban the development of AI. I am skeptical of government meddling in the first place. AI companies are able to pour such amounts into AI because of direct government action (in the case of China) or because of ludicruous valuations, again, due to government monetary manipulation (in the case of USA). Regulations will only stifle competion and allow a small set of people have control over the tools needed to manipulate humanity.
  11. Giving autonomy to AI agents that can access the internet is a bad idea. A worse idea is letting them interface with the physical world (control robots, synthesize chemicals, etc.) However, this might already be taking place. Yet another subject insufficiently discussed while the masses are distracted discussing superintelligence.