webplodder said:
Given that neuroscience cannot definitively explain where consciousness originates - whether as an emergent property of the brain or something transcendent - is it be possible for complex chatbots to develop behaviours that can’t be predicted by simply analyzing the sum of their parts?
We may already be seeing that. Even computer systems much simpler than chatbots or neural networks can exhibit unpredictable behaviour. Note that determinism doesn't imply predictability, as long as you have sufficient variability of the environment. Deterministic feedback loops can soon go beyond predictability in any practical sense.
In fact, the simplest example is the famous recursive equation:
$$x_{n+1}= Rx_n(1-x_n)$$You couldn't have a simpler, more deterministic sequence. And, yet, for ##R > 3.57## the sequence exhibits almost limitless complexity and unpredictability.
See, also, Conway's
Game of Life, where a few simple rules can again lead to almost limitless complexity.
webplodder said:
OK, but assuming we're talking about a system capable of emulating or even exceeding human reasoning, how could we possibly anticipate every behaviour? And again, if the 'mind' (for lack of a better word) isn't deterministic, how do we know there isn't something else operating at this level?
Precisely. Unless we are missing something supernatural, then human intelligence and consciousness must arise from basic biological algorithms of one sort or another. If you describe the human brain in terms of its constituent atoms and molecules, then consciousness must arise from components that are not themselves conscious.
The extent to which systems being developed today will exhibit "expected" behaviours is, of course, not known. The argument that they are just dumb machines is, however, a dangerous fallacy.