Biological Concepts

Bio Transformations is built on several key concepts from neuroscience. This page provides an overview of these concepts and how they relate to our implementation in artificial neural networks.

Synaptic Diversity

In biological neural networks, synapses (the connections between neurons) exhibit a wide range of properties and behaviors. This diversity is crucial for the complex information processing capabilities of the brain.

1. Diversity in Synaptic Plasticity

Biological Concept: In the brain, not all synapses change their strength at the same rate or in the same way. Some synapses are more plastic (changeable) than others, and this plasticity can vary over time and based on different mechanisms.

Implementation: The fuzzy_learning_rates() method in our BioModule applies different learning rates to different “synapses” (weights) in the artificial neural network. These rates can be determined by various distribution strategies:

  • Uniform Distribution: Random variability around a central value

  • Normal Distribution: Bell-curve distribution of learning rates

  • Log-normal Distribution: Skewed distribution favoring smaller adjustments with occasional larger changes

  • Gamma/Beta Distributions: Distributions that model specific biological properties

  • Layer-Adaptive Distribution: Mimics layer-specific plasticity observed in cortical layers

  • Weight-Adaptive Distribution: Adjusts learning rates based on weight magnitudes

  • Temporal Distribution: Evolves learning rates over time, modeling developmental changes

  • Activity-Dependent Distribution: Adjusts learning rates based on neural activity patterns

2. Spontaneous Spine Remodeling

Biological Concept: Dendritic spines, the primary sites of excitatory synapses in the brain, are dynamic structures. They can form, change shape, and disappear over time, even in the absence of explicit learning signals.

Implementation: The rejuvenate_weights() method simulates this by selectively reinitializing weights in the network. The probability of rejuvenation is inversely proportional to weight magnitude, mimicking how smaller, weaker spines are more likely to be pruned and replaced. This allows for the “formation” of new connections and the “pruning” of others, similar to spine remodeling in the brain.

3. Multi-synaptic Connectivity

Biological Concept: In biological neural networks, multiple synaptic connections often exist between the same pair of neurons. This redundancy can enhance the reliability and flexibility of neural circuits.

Implementation: The weight splitting mechanism (controlled by weight_splitting_Gamma in the configuration) implements this concept by allowing multiple “synapses” (sub-weights) to exist for each connection in the artificial neural network. This enhances the network’s ability to learn complex patterns through redundant pathways.

Homeostatic Plasticity

Biological Concept: Neurons have mechanisms to maintain their activity levels within a functional range, preventing over-excitation or complete silencing. This is crucial for the stability of neural circuits.

Implementation: Several mechanisms in Bio Transformations implement homeostatic principles:

  • The crystallize() method stabilizes well-optimized weights, preventing further large changes

  • The scale_grad() method implements synaptic scaling by adjusting gradients

  • Activity-dependent learning rates adapt based on neuron activation patterns

Volume-Dependent Plasticity

Biological Concept: In the brain, the size (volume) of a synapse is often correlated with its strength and stability. Larger synapses tend to be more stable but less plastic, while smaller synapses are more dynamic.

Implementation: The volume_dependent_lr() method implements this concept by adjusting learning rates based on the magnitude (analogous to volume) of the weights. It applies three key principles:

  1. Larger weights receive exponentially smaller learning rates

  2. Smaller weights receive larger, more variable learning rates

  3. There is significant variability in learning rates among weights of similar sizes

This approach mimics the observation that spine size in biological systems strongly correlates with stability and plasticity characteristics.

Dale’s Principle

Biological Concept: Dale’s principle states that a neuron releases the same neurotransmitter(s) at all of its synapses. This introduces certain constraints on how neurons can influence each other - a neuron is either excitatory or inhibitory, not both.

Implementation: The enforce_dales_principle() method ensures that all outgoing weights from a given artificial “neuron” have the same sign, mimicking the constraints imposed by Dale’s principle in biological neural networks. The dalian_network_initialization() method assigns each neuron to be either excitatory (positive weights) or inhibitory (negative weights).

Synaptic Stabilization

Biological Concept: In the brain, frequently used synapses tend to become more stable over time, a process sometimes called “crystallization.” This helps preserve important learned patterns while still allowing for adaptation.

Implementation: The crystallize() method identifies weights with small gradients relative to their magnitude (indicating they are well-optimized) or unusually large gradients (indicating important learning signals) and reduces their effective learning rates. This selectively stabilizes important connections while allowing others to remain plastic.

By incorporating these biological concepts, Bio Transformations aims to create artificial neural networks that capture some of the complex dynamics observed in biological brains. This bio-inspired approach has the potential to enhance the learning, adaptability, and robustness of artificial neural networks.