Neocortical Principles
Research led by organizations such as Numenta focuses on reverse-engineering the structural properties of the neocortex. The objective is to identify fundamental principles of intelligence beyond static, layer-based neural networks.
Hierarchical Temporal Memory (HTM)
HTM is a theoretical framework that models the algorithmic properties of neocortical circuits. Key features include unsupervised learning, high levels of sparsity, and temporal sequence prediction.
The Thousand Brains Theory
This theory posits that the neocortex is organized into processing units called cortical columns, which learn complete models of objects through sensory-motor integration. This decentralization aims for robustness and computational efficiency.
Historical Connectionist Approaches
- Self-Organizing Maps (Kohonen): A type of unsupervised neural network that produces a low-dimensional, discretized representation of the input space, similar to topographic maps in the brain.
- Adaptive Resonance Theory (ART): A theory developed by Stephen Grossberg on how the brain processes information, focusing on the stability-plasticity dilemma in learning.
- Sparse Coding (Olshausen & Field): Foundational research demonstrating that the primary visual cortex utilizes sparse representations to encode natural images efficiently.
- Neocognitron: A precursor to CNNs that drew heavy inspiration from the hierarchical structure of the visual cortex.
Algorithmic Implementation
- Sparse Representations: The use of Sparse Distributed Representations (SDRs) provides high noise tolerance and aligns with CPU-centric sparse processing.
- Online Learning: Biological models prioritize continuous learning from data streams without catastrophic forgetting.
- NuPIC (GitHub): Implementation of HTM algorithms for research and evaluation.