HUZZAH! My kindred!!!
Everything you said is bang on the nose!
The dimensionality reduction and integration of our sensory observations is definitely a critical component of prediction (otherwise there is information overload and quick impacts on upward efficiency and scalability).
My own project began simply as an attempt to make an AI spider (since then it has exploded to encompass all forms of animal life)... As it happened, spiders turned out to be one big walking super-sensitive sensory organ (
all the sights, all the sounds, all the vibrations, etc...), which is to say they have incredibly dense and "multi-modal" sensory inputs (they have sight, hearing, vibrational sensing, noses on two or more of their legs, and sensory neurons throughout their body that informs them about things like temperature, injury, body position, and acceleration, and more). And to integrate these many senses, spiders have a relatively puny brain with which they get up to an uncountable number of interesting behaviors with an ultra complicated body (if their flavor of intelligence can be scaled up, it would amount to a powerful general intelligence). It's not just the sensory density and body-complexity, but also the fact that they actually exhibit a very wide range of behaviors. Not only can they extract and learn to encode high level features from individual senses, they can make associations and corroborate those features in and between many different complimentary sensory modalities (if it looks like a fly, sounds like a fly, and feels like a fly:
fly 100%).
I could expound the virtues of spider's all day, but the point that is worth exploring is the fact that spider's have a huge two-ended dimensionality problem (too much in, too much out), and yet their small and primitive-looking nervous systems make magic happen. When I set out to make a spider AI, I didn't have any conception of the dimensionality curse, but I very quickly discovered it... At first, my spider just writhed epileptically on the ground; worse than epileptically (without coherence at all).
I had taken the time to build a fully fleshed out spider body (a challenge unto itself) with many senses, under the assumption that agents need ample clay to make truly intelligent bricks. And when I set a reinforcement learning algorithm to learn to stand or to walk (another challenge unto itself), it failed endlessly. A month of research into spiders and machine-learning later, I managed to train the spider to ambulate...
And it wasn't pretty... All it would ever learn is a bunch of idiosyncratic nonsense.
Yes, spider, only ever flicking one leg is a possible ambulation strategy, but it's not a good one dammit! Another month of effort, and now I can successfully train the spider to run... Like a horse?
Imagine my mid-stroke grin; I finally got the spider
running (running? I wanted it to be hunting by now!). It took so much fine-tuning to make sure the body and mind/senses has no bugs or inconsistencies, and in the end it still fucking sucked
:rofl: ... A spider that gallops like a horse is no spider at all. I almost gave up...
I decided that there must be more to it... Thinking about spiders and centipedes (and endlessly observing them) convinced me that there has to be some underlying mechanism of leg coordination. After creating a crude oscillator and hooking up the spider muscles with some default settings I got instant interesting results:
It's still not too pretty, but compared to seizure spider and Hi Ho Silver, this was endlessly satisfying.
Over the next half a year or so, I have been continuing to research and develop the underlying neural circuitry of locomotion (while developing the asset and accompanying architecture). I branched out to other body-types beyond spiders, and in doing so I forced myself to create a somewhat generalized approach to setting up what amount to *controller systems* for learning agents. I'm still working toward finalizing the spider controller system (spider's are the Ferrari of control systems), but I have already made wildly good achievements with things like centipedes, fish, and even human hands!
(note: the centipede and the hand have no "brain"; they're headless)
These fish actually have a central nervous system, whereas the centipede and the hand are "headless" (im essentially sending signals down their control channels manually). They can smell and see the food balls,
and they likey!
The schooling behavior is emergent here. In theory they could get more reward by spreading out, but since they have poor vision and since smell is noisy and sometimes ambiguous, they actually are grouping together because combined they form a much more powerful nose (influencing eachother's direction with small adjustments acts like a voting mechanic for group direction changes). (I can't be sure of this, but im in a fairly good position to make that guess).
They have two eyeballs (quite low resolution though, 32x32xRGB) each, and these images are passed through a convolutional neural network that performs spatially relevant feature extraction. it basically turns things into an encoded and shortened representation that is then passed as inputs to the PPO RL training network that is acting as the CNS of the fish (the PPO back-propagation passes into the CNN, making it the "head" classifier, if you will).
This observational density aspect of the system has absorbed almost as much of my focus as the output side (where my cpg system is like a dynamic decoder for high level commands for the CNS, i need a hierarchical system that can act as
encoder to give high level and relevant reports to the CNS. Only that way can I lighten the computational load on the CNS to make room for interesting super-behaviors that are composed of more basic things like actually walking elegantly). I have flirted with auto encoding, and some of the people interested in the project are helping me flirt with representation encoding (there's actually a whole zoo of approaches to exhaust).
The most alluring approach for me is the hierarchical RL approach. Real brains, after-all, are composed of somewhat discrete ganglia (they compartmentalize problem spaces), and they do have some extant hierarchy thanks to evolution (we have lower and higher parts of the brain. Lower tending to be basic and older, evolutionary speaking, with higher areas tending to be more complexly integrated, and more recent acquisitions of nature). Sensory data comes in at the bottom, goes up through the low levels, shit happens at the top and everywhere in-between, shit flows back down (from all levels), and behavior is emerged. One evolutionary caveat of this is that before the higher parts could have evolved, the lower parts had to actually confer some kind of benefit (and be optimized). Each layer needs to both add overall utility, AND not ruin the stability and benefits of the lower ganglia (each layer must
graduate the system as a whole). The intuitive take-away from this that we can start with basic low level systems, make them good and stable, and then add layers on-top to achieve the kind of elegant complex intelligence we're truly after. For most roboticists and researchers, the progress-stalling rub has been finding an elegant and generalized low level input and output approach.
Like with pecking, it's likely to be the case that features that distinguish good to peck targets (like seeds' shapes and sizes or bugs' motion and leg movements) from bad to peck targets become heavily impactful in the learning process, as once an agent has cottoned onto a task relevant feature, they can respond quicker, with less effort, and just as accurately, as features efficiently summarise task relevant environmental differences.
Edit: in some, extremely abstract, respect, feature learning and central pattern generators address the same problem; Imagine succeeding at a task as a long journey, central pattern generators direct an agent's behavioural development down fruitful paths from the very beginning, they give an initial direction of travel, feature learning lets an agent decide how to best get closer to their destination along the way; to walk the road rather than climb the mountain — fdrake
One of the more impressive things i have been able to create is a cpg system that can be very easily and intuitively wired with reflexes and fixed action responses (whether they are completely unconscious like the patellar knee reflex, or whether they are dynamically modulated by the CNS as excitatory or inhibitory condition control). But one of the trickier things (and something that I'm uncertain about) is creating autonomic
visual triggers (humans have queer fears like
tryptophobia; possibly we're pre-wired for recognizing those patterns). I could actually train a single encoder network to just get really good at recognizing stuff that is relevant to the critter, and pass that learning down through generations, but something tells me this could be a mistake, especially if novel behavior in competition needs to be recognized as a feature).
I'm still in the thick of things (currently fleshing out metabolic effects like energy expenditure, fatigue, and a crude endocrine system that can either be usefully hard-coded or
learned (e.g: stress chemicals that alter the base tension and contraction speed of muscles)).
In the name of post-sanity I'll end it here, but I've only scratched the surface!