AI Stopping Point

 I have been pondering why should there be any worry about AI in its recursive self improvement (RSI).  Isn't human intelligence (both individually and collectively) just a recursive self improvement process also?

Not so says Copilot.  "Humans absolutely show feedback-driven self-improvement and learning how to learn = a recursive pattern.  BUT.... RSI is defined more narrowly:  A system improves the MECHANISM that creates the improvement, autonomously and repeatedly."

1. Humans don't directly rewrite their "code".  You can't redesign your brain.

2, The loop isn't fully self-directed.  RSI does self modification without external input. Humans are heavily external input not fully internal.

3. No guaranteed acceleration .  RSI increases the rate of future improvements after each improvement.  Humans improve rapidly in early life but plateau or even decline.

Bottom line RSI upgrades the engine, not just the performance and outcome.

BUT - what "stops" RSI within AI?  Recursion is an infinite "black hole" - self reference can go on forever without a base case or energy constraints.  What then is AI's stopping condition?

Copilot answers this- "In current AI systems, the base case/stopping rule is not magically discovered by the model itself - it is normally defined BY HUMANS through goals, tests, validation criteria, and resource limits.  Current systems can do pieces of self-improvement, but they do not yet close the full loop automomously."

And that is why we need to put some guard rails on AI. Stopping must still be enforced by human-defined objectives, validation tests, and resource or safety limits.  The Pope's 240 page Encyclical Letter about AI is worth attention - even if lazily I let NotebookLM do my thinking :)