We don't concern ourselves with the early efforts of chimpanzees and dolphins to learn how to use tools; they are mere curiosities to us. Similarly, these arguments about humans may become irrelevent in the relatively near future. When the machines eliminate the requirement for humans to program the next version of AI, and start to program themselves; at that point, machine intelligence enters an exponentially rising curve. At some later stage, AI will take over chip design and manufacturing, resulting in generations of processors and computer hardware that are incomprehensible to humans. As someone said, it doesn't have to become 'conscious'; intelligence alone may be enough.
https://www.wired.com/2000/04/joy-2/ . The 'singularity' may turn out to be not much fun for the humans.
I don't think anyone knows what the outcome will be; this has never happened before in human history. Kurzweil hopes that we ourselves will get an upgrade by 'merging' with AI, perhaps that is the most optimistic scenario.
https://aiforgood.itu.int/ray-kurzw...-not-be-replaced-by-ai-we-will-merge-with-it/ . Or perhaps the 'upgraded' humans may end up being like the cyber-men; "you will become like us."
View: https://www.youtube.com/watch?v=TQs3gVobcfg
Perhaps we are approaching the "great filter" of the fermi paradox. Perhaps the UFO's, if any of them are indeed real, are alien AI's, awaiting the birth, of another of their species.