Unit 6: UNDERSTANDING NEURAL NETWORKS

Notes

Introduction to Neural Networks

• Definition:
  A neural network is a machine learning model inspired by how the human brain processes information. It consists of layers of connected nodes (neurons) that learn to recognize patterns in data.
• Biological Inspiration:
  The structure mimics biological neurons, where information flows between interconnected cells.
• Real-world Examples:
  • Email auto-replies
  • Spam filtering
  • Image tagging (e.g., Facebook)
  • Recommendation systems
  • Google’s search algorithm

Structure of a Neural Network

A neural network is composed of layers of interconnected units, known as neurons or nodes. These layers work together to process input data and make predictions or decisions. Each layer performs a specific function in the data transformation process.

🔸 Layers in a Neural Network

a) Input Layer

• This is the first layer of the neural network.
• It receives the raw input data from the user or an external source.
• Each node (neuron) in this layer corresponds to one input feature.
  For example, if you’re predicting house prices using 3 features (size, number of rooms, location), the input layer would have 3 neurons.
• No computation is performed here; it only forwards the data to the next layer.

---

b) Hidden Layer(s)

• Hidden layers lie between the input and output layers.
• There can be one or many hidden layers depending on the network’s complexity. When there are multiple hidden layers, the model is referred to as a deep neural network.
• Each neuron in a hidden layer:
  • Receives inputs from the previous layer.
  • Applies a weight to each input.
  • Adds a bias.
  • Passes the result through an activation function (like ReLU or Sigmoid).
• The output of each neuron becomes input for the next layer.

🧠 Why is it called “hidden”?

Because it is not directly exposed to either input or output — it’s “hidden” inside the model.

🔁 Interconnections

• Every neuron in one layer is typically fully connected to all neurons in the next layer (especially in feedforward networks).
• Each of these connections has an associated weight that adjusts during training.

---

c) Output Layer

• This is the final layer of the network.
• It produces the result of the neural network after all transformations are complete.
• The number of neurons in the output layer depends on the type of task:
  • Binary classification: 1 neuron with a sigmoid activation function.
  • Multi-class classification: 1 neuron per class, often with a softmax function.
  • Regression: 1 neuron with a linear activation.

---

🔸 Flow of Data (Forward Propagation)

• In a neural network, data flows forward, layer by layer, in a process called forward propagation.
• Each neuron computes a weighted sum of inputs plus a bias, then applies an activation function.
• The output of one layer serves as the input to the next layer, until the final prediction is produced in the output layer.

---

🔸 Activation and Thresholding

• If the output of a neuron exceeds a certain threshold, it is considered “activated” and passes its value to the next layer.
• If it doesn’t exceed the threshold, it outputs zero or a low value, effectively muting that path.

---

🔸 Deep vs Shallow Networks

• Shallow Neural Network: Has only one hidden layer.
• Deep Neural Network (DNN): Has two or more hidden layers.
  • The depth increases the model’s capacity to learn hierarchical and complex features.
  • Deep Learning refers to training these deep networks.

---

🔸 Visualization Analogy

Think of the network as a flowchart:

• The input layer gathers the raw materials.
• The hidden layers are like a processing plant where raw materials are refined.
• The output layer delivers the final product — the network’s prediction.

---

Components of a Neural Network

A neural network is made up of multiple interconnected components that work together to process inputs and generate accurate outputs. Understanding each component is essential to grasp how neural networks learn, make decisions, and generalize patterns from data.

---

🔸 Neurons (or Nodes)

• Definition: The fundamental processing units of a neural network.
• Each neuron:
  • Receives one or more inputs.
  • Multiplies each input by a corresponding weight.
  • Adds a bias to the result.
  • Passes the final value through an activation function.
• This output is then passed to neurons in the next layer.

🧠 Think of a neuron as a mini calculator that decides whether or not to “fire” based on the inputs it receives.

---

🔸 Weights

• Weights are numerical values that determine the strength and direction of the connection between neurons.
• They are learnable parameters — adjusted during training to reduce prediction errors.
• Larger weights indicate that a feature has more influence on the final output.
• Each connection in the network has its own weight.

🧮 Mathematical role:
In a neuron, if inputs are x1,x2,…,xnx_1, x_2, …, x_nx1​,x2​,…,xn​, and corresponding weights are w1,w2,…,wnw_1, w_2, …, w_nw1​,w2​,…,wn​, the neuron computes the weighted sum: z=w1x1+w2x2+…+wnxn+bz = w_1x_1 + w_2x_2 + … + w_nx_n + bz=w1​x1​+w2​x2​+…+wn​xn​+b

---

🔸 Activation Functions

• Activation functions determine whether a neuron should be activated (i.e., whether it should pass information forward).
• They introduce non-linearity into the model, which allows the network to learn complex relationships and patterns.

Common Types:

1. Sigmoid:
   • Outputs range between 0 and 1.
   • Ideal for binary classification.
   • Formula: σ(x)=11+e−x\sigma(x) = \frac{1}{1 + e^{-x}}σ(x)=1+e−x1​
2. Tanh (Hyperbolic Tangent):
   • Outputs range between -1 and 1.
   • Better than sigmoid for zero-centered data.
3. ReLU (Rectified Linear Unit):
   • Outputs 0 for negative inputs and linear for positive inputs.
   • Speeds up training and reduces vanishing gradient issues.

🧠 Without activation functions, a neural network would behave like a linear model and fail to capture complex patterns.

---

🔸 Bias

• A bias is an additional parameter added to the weighted sum before the activation function is applied.
• It allows the neuron to shift the activation function left or right, helping the model learn patterns not passing through the origin.
• Biases improve the model’s flexibility and learning capacity.

🧮 In formula: z=∑wixi+bz = \sum w_ix_i + bz=∑wi​xi​+b

Here, bbb is the bias.

---

🔸 Connections

• Connections are the links between neurons, transmitting data from one neuron to another.
• Each connection carries a weight and contributes to the learning process.
• In fully connected (dense) layers, every neuron in one layer is connected to every neuron in the next layer.

🧠 Each connection is like a path along which information travels, weighted to determine how much it influences the destination neuron.

---

🔸 Learning Rule

• The learning rule defines how the neural network updates its weights and biases based on the error of its predictions.
• The most common learning rule is Backpropagation:
  • It calculates the gradient of the loss function with respect to each weight.
  • Then it updates weights using optimization algorithms like Gradient Descent.

🧠 Think of backpropagation as a feedback loop where the network “learns from its mistakes.”

---

🔸 Propagation Functions

Propagation refers to how information flows through the network:

a) Forward Propagation

• The process where input data is passed through the layers to produce a prediction.
• At each neuron, a weighted sum is computed, bias is added, and the activation function is applied.
• Output is compared with the actual result to compute loss/error.

b) Backward Propagation (Backpropagation)

• The process of updating the weights and biases to reduce the error.
• The gradients of the loss with respect to weights are calculated and propagated backward.
• Gradient Descent (or its variants) is used to minimize the loss.

Working of a Neural Network

A neural network functions like a mathematical engine that learns from data by performing a series of computations. At its core, it transforms inputs into outputs through a flow of weighted calculations, activation decisions, and iterative learning. Understanding the math behind this is key to mastering how neural networks function.

---

🔸 Basic Working Mechanism

The computation in a neuron typically follows this mathematical formula: z=∑(wi⋅xi)+bz = \sum (w_i \cdot x_i) + bz=∑(wi​⋅xi​)+b Output=f(z)\text{Output} = f(z)Output=f(z)

Where:

• xix_ixi​ = input values
• wiw_iwi​ = corresponding weights
• bbb = bias
• zzz = weighted sum + bias
• f(z)f(z)f(z) = activation function output (e.g., sigmoid, ReLU)

This output is then passed to the next layer as input.

---

🔸 Step-by-Step Flow

✅ Step 1: Input Layer

• Accepts raw feature data.
• Each input is passed to all neurons in the first hidden layer, multiplied by respective weights.

✅ Step 2: Weighted Sum

• Every neuron calculates a weighted sum of its inputs:

z=w1x1+w2x2+w3x3+…+bz = w_1x_1 + w_2x_2 + w_3x_3 + … + bz=w1​x1​+w2​x2​+w3​x3​+…+b

✅ Step 3: Activation Function

• The neuron applies an activation function to the weighted sum:

a=f(z)a = f(z)a=f(z)

• This output determines whether the neuron “fires” (activates) and passes data to the next layer.

✅ Step 4: Output Layer

• After passing through all hidden layers, the final layer computes the prediction or output value.

---

🔸 Threshold Concept

Sometimes, a threshold is used as a decision boundary: If ∑wixi+b≥threshold, output = 1 (active)\text{If } \sum w_ix_i + b \geq \text{threshold, output = 1 (active)}If ∑wi​xi​+b≥threshold, output = 1 (active) Else, output = 0 (inactive)\text{Else, output = 0 (inactive)}Else, output = 0 (inactive)

This is often seen in binary classification or in simplified models like the Perceptron.

---

🔸 4.4 Numerical Example (Case Study: Simple Neuron)

Let’s take the example from the PDF:

Inputs (Features):

• x1=2, x2=3, x3=1x_1 = 2,\ x_2 = 3,\ x_3 = 1×1​=2, x2​=3, x3​=1

Weights:

• w1=0.4, w2=0.2, w3=0.6w_1 = 0.4,\ w_2 = 0.2,\ w_3 = 0.6w1​=0.4, w2​=0.2, w3​=0.6

Bias = 0.1, Threshold = 3.0

Computation:

z=(0.4⋅2)+(0.2⋅3)+(0.6⋅1)+0.1=0.8+0.6+0.6+0.1=2.1z = (0.4 \cdot 2) + (0.2 \cdot 3) + (0.6 \cdot 1) + 0.1 = 0.8 + 0.6 + 0.6 + 0.1 = 2.1z=(0.4⋅2)+(0.2⋅3)+(0.6⋅1)+0.1=0.8+0.6+0.6+0.1=2.1

Since 2.1 < 3.0, output = 0 → Neuron is inactive

---

🔸 4.5 Second Case (Different Neuron in Output Layer)

Given:

• w1=0.7, w2=0.3, bias=0.2w_1 = 0.7,\ w_2 = 0.3,\ bias = 0.2w1​=0.7, w2​=0.3, bias=0.2
• Input from previous layer = 0

z=(0.7⋅0)+(0.3⋅0)+0.2=0.2z = (0.7 \cdot 0) + (0.3 \cdot 0) + 0.2 = 0.2z=(0.7⋅0)+(0.3⋅0)+0.2=0.2

Assuming threshold = 0.1
Since 0.2 > 0.1 → Output = 1 (Neuron is active)

---

🔸 Real-Life Analogy Example (Surfing Decision)

Inputs:

• Wave quality = 1 (Yes)
• Lineup empty = 0 (No)
• Shark activity = 1 (No recent attack)

Weights:

• Waves: 5
• Crowd: 2
• Shark fear: 4
  Bias = -3

Computation: y^=(1⋅5)+(0⋅2)+(1⋅4)+(−3)=6ŷ = (1 \cdot 5) + (0 \cdot 2) + (1 \cdot 4) + (-3) = 6y^​=(1⋅5)+(0⋅2)+(1⋅4)+(−3)=6

Since 6 > 0 → Decision = 1 → Go surfing ✅

---

🔸 Feedforward Propagation (Forward Pass)

This is the first phase where:

• Inputs are passed through the network layer by layer.
• Each neuron computes its output using weights, bias, and activation.
• Final output is generated and compared to the actual value.

---

🔸 Backpropagation (Backward Pass)

Once an output is predicted:

• Error (loss) is calculated using a loss function.
• The error is propagated backward through the network.
• Gradients are computed and weights are updated using optimization algorithms like gradient descent.
• The process continues over many epochs to improve accuracy.

Types of Neural Networks

Neural networks come in different architectures, each suited for specific kinds of data and tasks. This section introduces the most common and practical types of neural networks used in modern artificial intelligence.

---

🔸 Perceptron (Standard Neural Network)

• Definition:
  The simplest type of neural network, developed by Frank Rosenblatt in 1958.
• Structure:
  • Contains only one layer of input nodes connected directly to an output layer.
  • Uses Threshold Logic Units (TLUs) for activation.
• Functionality:
  Performs binary classification using simple threshold-based decisions.
• Limitation:
  Cannot handle non-linear data patterns — only works for linearly separable problems.
• Application Examples:
  • Spam vs. non-spam email classification
  • Basic yes/no decision tasks

---

🔸 Feedforward Neural Network (FFNN)

• Definition:
  Also known as Multi-Layer Perceptron (MLP). This is the most basic and widely used type of neural network for supervised learning.
• Structure:
  • Input layer → One or more hidden layers → Output layer
  • Data flows in one direction — forward from input to output.
  • Uses activation functions like ReLU, Sigmoid, etc.
• Key Feature:
  No cycles or loops — each neuron’s output is used only once per prediction cycle.
• Application Examples:
  • Image recognition
  • Natural language processing
  • Predictive analytics (e.g., house prices)

---

🔸 Convolutional Neural Network (CNN)

• Definition:
  A specialized network for processing image data and recognizing patterns.
• Structure:
  • Includes convolutional layers that apply filters to the input image.
  • Pooling layers reduce dimensionality and highlight key features.
  • Fully connected layers at the end perform final classification or regression.
• Working:
  • Convolutional layers detect features like edges, textures, shapes.
  • Pooling layers (e.g., max pooling) downsample the feature maps.
• Application Examples:
  • Facial recognition
  • Object detection
  • Medical imaging (e.g., tumor detection)
  • Image classification (e.g., identifying cats vs. dogs)

---

🔸 Recurrent Neural Network (RNN)

• Definition:
  Designed for sequential data where the current input depends on previous inputs.
• Structure:
  • Neurons have feedback loops that allow outputs from previous steps to be reused.
  • Maintains an internal memory of prior inputs.
• Key Feature:
  Handles time-series or context-dependent data very effectively.
• Advanced Forms:
  • LSTM (Long Short-Term Memory): Helps overcome vanishing gradient problem.
  • GRU (Gated Recurrent Unit): A lighter alternative to LSTM.
• Application Examples:
  • Language translation
  • Chatbots and virtual assistants
  • Speech recognition
  • Stock price prediction
  • Sentiment analysis

---

🔸 Generative Adversarial Network (GAN)

• Definition:
  A type of neural network architecture that uses two models:
  • Generator: Creates fake (synthetic) data.
  • Discriminator: Judges whether the data is real or fake.
• How it Works:
  • The generator tries to fool the discriminator with increasingly realistic data.
  • The discriminator tries to detect which data is fake.
  • Over time, both networks improve through adversarial training.
• Key Feature:
  Used in unsupervised learning and data generation.
• Application Examples:
  • Creating realistic images of faces, landscapes
  • Deepfakes
  • Art and music generation
  • Data augmentation for training other models
  • Synthetic voice and video content