The document describes a temporal classifier system that uses spiking neural networks to handle tasks with continuous space and time. It uses Integrate-and-Fire neurons in the spiking networks to introduce temporal functionality. The system includes self-adaptive parameters that control mutation rates, neural constructivism for adding/removing neurons, and connection selection for pruning connections. This allows the system to autonomously control its learning and adapt the network topology based on the environment. The system is tested on continuous grid world and mountain car tasks, as well as a robotics simulation, and is able to learn optimal policies for the tasks by leveraging the temporal aspects of the spiking networks.
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
A temporal classifier system using spiking neural networks
1. A temporal classifier system using spiking neural networks Gerard David Howard, Larry Bull & Pier-Luca Lanzi {david4.howard, larry.bull} @uwe.ac.uk pierluca.lanzi @polimi.it 1
2. Contents Intro & Motivation System architecture – Spiking XCSF Constructivism (nodes and connections) Working in continuous space Comparison to MLP / Q-learner Taking time into consideration Comparison to MLP Simulated robotics 2
3. Motivation Many real-world tasks incorporate continuous space and continuous time Autonomous robotics are an unanswered question: will require some degree of knowledge “self-shaping” or control over their internal knowledge representation We introduce an LCS containing spiking networks and demonstrate the usefulness of the representation Handles continuous space and continuous time Representation structure dependent on environment 3
4. XCSF Includes computed prediction, which is calculated from input state (augmented by constant x0) and a weight vector – each classifier has a weight vector Weights are updated linearly using modified delta rule Main differences from canonical: SNN replaces condition and calculates action Self-adaptive parameters give autonomous learning control Topology of networks altered in GA cycle Generalisation from computed prediction, computed actions and network topologies 4
5. Spiking networks Spiking networks have temporal functionality We use Integrate-and-Fire (IAF) neurons Each neuron has a membrane potential (m) that varies through time When m exceeds a threshold, the neuron sends a spike to every neuron in the network that it has a forward connection to, and resets m Membrane potential is a way of implementing memory 5
6.
7. Self-adaptive parameters During a GA cycle, a parent’s µ value is copied to its offspring and altered The offspring then applies its own µ to itself (bounded [0-1]) before being inserted into the population. Similar to ES mutation alteration Mutate µ µ * e N(0,1) Insert Copy µ [A] Mutate µ µ * e N(0,1) Copy µ Insert 7
8. Constructivism Neural Constructivism - interaction with environment guides learning process by growing/pruning dendritic connectivity Constructivism can add or remove neurons from the hidden layer during a GA event Two new self-adaptive values control NC , ψ (probability of constructivism event occurring) and ω (probability of adding rather than removing a node). These are modified during a GA cycle as with µ 8 Randomly initialised weights
9.
10.
11. Continuous Grid World Two dimensional continuous grid environment Runs from 0 – 1 in both x and y axes Goal state is where (x+y>1.9) – darker regions of grid represent higher expected payoff. Reaching goal returns a reward of 1000, else 0 Agent starts randomly anywhere in the grid except the goal state, aims to reach goal (moving 0.05) in fewest possible steps (avg. opt. 18.6) 1.00 0.50 Agent 0.00 0.50 1.00 11
12. Discrete movement Agent can make a single discrete movement (N,E,S,W) N=(HIGH,HIGH), E=(HIGH,LOW) etc… Experimental parameters N=20000,γ =0.95, β=0.2, ε0=0.005, θGA=50, θDEL=50 XCSF parameters as normal. Initial prediction error in new classifiers=0.01, initial fitness=0.1 Additional trial from fixed location lets us perform t-tests. “Stability” shows first step that 50 consecutive trials reach goal state from this location. 12
16. Continuous duration actions Reward usually calculated as Reward is now calculated as Two discount factors that favour overall effectiveness and efficient state transitions respectively =0.05, ρ=0.1 tt = total steps for entire trial ti = duration of a single action Timeout=20; new steps to goal is 1.5 15
23. Spiking TCS canlearn to optimally solve this environment by extending an action set across multiple states and recalculating actions where necessary
24.
25.
26. Guide a car out of a valley, sometimes requiring non-obvious behaviour
42. Robotics 26 Steps to goal Connected hidden layer nodes Percentage enabled connections Self-adaptive parameters, μ, ψ, τ all plotted on RHS axis
43.
44. Initially seeding with 6 hidden layer nodes still let’s us use connection selection to generate behavioural variation in the networks
45. Temporal functionality of the networks is exploited so that a single action set canDrop unwanted classifiers to change it’s favoured action at specific points (e.g. just before a collision) Alter the action advocated action of a majority of classifiers in [A] for the same effect