2. Superintelligence
Manufacturing intelligent objects to do our jobs and make our lives easier seems like a great plan. Intelligent entities are meant to set goals and achieve them and that usually works well because our goals align with theirs. However, if the goals are set against us, they can use their power to achieve disastrous ends. We have no reason to think that intelligence itself will make something behave ethically. Humans are separate from these intelligent entities because we have empathy and the understanding that what we cannot always get what we want. Software-based intelligence can very quickly go from primitive to overwhelmingly powerful. An “intelligence explosion” is said to happen when a software can successfully make better software. If a quick or powerful intelligence explosion does take place, things won’t bode well for mankind.