Ever wondered how machines mimic human brains? With just a sprinkle of binary magic and logical gates, these simple models laid the foundation for today’s deep learning marvels.
Introduction
The basic bulding block of deep neural networks is the artificial neuron. We try to emulate human brain. Human brain has around 80-100 billion neurons. It is a network of many neurons. Different parts of our brain specializes in different inference mechanisms, for example, some of them are responsible for motor skills, some are for analytical skills, artistic skills, etc.
In deep learning, we try to model this complex information procesing system from our understanding of the humain brain.
What is a MP Neuron?
The basic unit of a neural network is a neuron. In 1943, McCulloch and Pitts (MP) proposed a threshold logic unit which is the first mathematical model for a neuron. This is called as MP neuron. This is a very simple model whose inputs and output are Boolean.
The center part is the neuron, which has a threshold logic unit. If the sum of the inputs is greater than a threshold \(\theta\), the output is 1. The neuron gets activated and sends its own signal to other neurons. If the sum of the inputs is less than \(\theta\), the neuron doesn’t send any signal, it outputs 0. This can be modelled as \(y = f(x) = 1 \left( \sum_{i=1}^n x_i \geq \theta \right)\).
The inputs \(x_1, \dots, x_n\) are boolean signals, each taking values 0 or 1. The input signals can be of excitatory or inhibitory in nature. A neuron can receive some excitatory and some inhibitory signals.
Excitatory Signals:
These signals excite the neuron toward possibly sending its own signal. When a neuron receives an excitatory signal, 1 is sent to the neuron. When a neuron does not receive an excitatory signal, 0 is sent to the neuron.
Inhibitory Signals:
These signals inhibit the neuron from sending a signal. When a neuron receives an inhibitory signal, it becomes less excited. In effect, inhibitory signals subtract from the total of the excitatory signals, making the neuron more relaxed, and moving the neuron away from its threshold.
-
When a neuron receives an inhibitory signal, -1 is sent to the neuron (this value reduces the activity of the neuron).
-
When a neuron doesn’t receive an inhibitory signal, 0 is sent to the neuron (this value reflects no inhibitory action).
We can make signals either excitatory or inhibitory, and we can change the threshold. By making different choices for these adjustments, we can make MP neurons that produce a variety of results.
Example 01:
Say we want to model \(f(x,y) = x.y'\) using a MP neuron. Note \(.\) is the logical AND operator.
\(X\) and \(Y\) are our inputs and we need to realize \(X.Y'\). Say we set \(X\) as excitatory and \(Y\) as inhibitory signals.
-
When \(x=0\) and \(y=0\), the neuron receives \(0+0=0\) and it should output 0. As the summation is 0 here and our neuron should not get activated, the threshold should be \(\geq 0\).
-
When \(x=0\) and \(y=1\), the neuron receives \(0+(-1)=-1\) and it should output 0.
-
When \(x=1\) and \(y=0\), the neuron receives \(1+0=1\) and it should output 1. As the summation is 1 here and our neuron should get activated, the threshold should be \(\leq 1\).
-
When \(x=1\) and \(y=1\), the neuron receives \(1+(-1)=0\) and it should output 0.
So the threshold can be set to 1 to realize the function \(X.Y'\).
The design principle of MP neurons can be modified to give inhibitory inputs dominant effect in the neuron’s logic. This simplifies the logic for specific decision-making tasks. In this design, if an inhibitory input is active (i.e., its value is 1), it overrides all other excitatory inputs and forces the output \(y\) to be 0. This is a logical rule added to emphasize the dominance of inhibitory signals in the neuron’s function.
In this design, an MP neuron with one excitatory and one inhibitory input with a threshold set to 0 acts as
When the inhibitory input is set, the summation and processing are not required. The output \(y\) will be 0 irresepctive of other inputs.
Example 02:
With our new design principle, say we want to model the below relationship using a MP neuron.
-
For \(x_1=0\) and \(x_2=0\), the neuron should get activated. The summation here is 0, so the threshold should be \(\leq 0\).
-
For \(x_1=1\) and \(x_2=0\), the neuron shouldn’t get activated. The summation here is 1, so the threshold should be \(\geq 1\). This condition conflicts with our previous condition. So let’s alter the nature of the input. On making the signal \(x_1\) inhibitory, the output in this case can be made 0.
-
For \(x_1=0\) and \(x_2=1\), the neuron shouldn’t get activated. As the inhibitory signal is not set, we have to process it. The summation here is 1, so the threshold should be \(\geq 1\). This condition conflicts with our first condition. On making the signal \(x_2\) inhibitory, the output in this case can be made 0.
-
For \(x_1=1\) and \(x_2=1\), the neuron shouldn’t get activated. The input signals are inhibitory, so the output is 0.
Thus we can model this relation with the neuron (on the right side) with a threshold value of 0. This is infact X NOR Y logic function.
What does one unit do?
The unit has a threshold logic unit which does \(\sum_i x_i \geq \theta\). It sets a threshold or boundary \(\sum_i x_i = \theta\). For a two variable case, this will be a line \(x_1 + x_2 - \theta = 0\). For three variables, it will be a plane. For more than three, it will be a hyperplane. It basically creates a linear separation in the input space, i.e., a linear classifier.
|
Caution
|
Here the inputs take only binary values (0 or 1), a continuous boundary line is drawn only for illustration purpose. |
There is no learning here. It is a heuristic approach, not an algorithmic. We try different inputs and outputs, and set the threshold accordingly to realize a function. \(\theta\) is the only parameter in this model. We could do hit and trial to find what threshold works only when the number of inputs is less and they take only binary values.
References
-
McCulloch-Pitts neurons. (n.d.). Retrieved January 26, 2025, from https://mind.ilstu.edu/curriculum/mcp_neurons/index.html
Why do we look at logical reasoning and model the Boolean functions first? The answer can be found in the above mentioned source with an interesting example.
