What is an Artificial Neural Network

Modeling in Artificial Neural Networks

The input layer sends input initially into the network. Given that it is a fully linked network, weights between the input layer and hidden layer are created at random. Finally, data is transferred from hidden layers into the output layer by initialising the weights at random. Since data is moving from the input layer to the output layer from left to right, this process is known as the forward pass. Gradient Descent Algorithm and Backpropagation Algorithm are then applied to optimise the weights.

Backpropagation in ANN

Back-propagation is a method of propagating the total loss back into the neural network to determine how much of the loss each node is responsible for, and then updating the weights in such a way that the loss is minimized by giving the nodes with higher error rates lower weights and vice versa.

Gradient Descent Algorithm

An iterative optimisation technique for locating the local minimum of a function is gradient descent. Before passing the inputs through the network, the weights and bias must first be initialised randomly in one of three methods.

• Batch gradient descent (every input is delivered simultaneously)
• Stochastic Gradient Descent-SGD (when each input is transmitted separately)

After passing the inputs through the network, it will give the output predictions. By considering actual values of output and predicted output we can calculate the error. Next, apply the backpropagation technique - go to each neuron and update their respective weights. Next, go back to the first step and repeat the process till the error is reduced within the specified range.

Types of Neural Networks

• Feed Forward Network

  Feed Forward Networks are the most fundamental kind of neural network. The input data in this network travels via the input layer before reaching the output layer. In a feedforward neural network, the output neuron is given the sum of the products of the inputs (features) and their weights (parameters). These Feed Forward networks are easy to create since, like the other neural networks, they don't use backpropagation. Additionally, a feedforward network might have numerous levels (including hidden layers) or only one layer (the output layer).

  

  Image of a Feed Forward Neural Network

• Multi Layer Perceptron (MLP)

  MLP has three or more layers which include the input layer, hidden layer, and output layer. In this type of neural network, there exists a full connection between the layers which means each neuron in a layer is connected to every other neuron in the other layer. These networks are used to solve the nonlinear patterns so the non-linearity is introduced into the network by adding the activation functions such as ReLu activation function. MLP uses backpropagation techniques to optimize the weights and minimize the error.

  

  Image illustrating the back propagation techniques in neural networks

• Convolutional Neural Network (CNN)

  The Multi-Layer Perceptron and CNN are nearly identical; however, CNN uses the convolutional technique to implement Convolutional layers. In addition to accepting unstructured data (image data) as input, CNN is also in charge of extracting features from the picture data. CNN often adheres to compositionality. Let's attempt to comprehend it. A few milliseconds are needed for the human brain to comprehend the visual information. When a person sees an image, their eyes record the information, which then travels to the cerebral cortex, which is directly behind their eyes and where the visual cortex is located. The visual cortex, which comprises layers, receives the images that humans view from the eyes. These layers are in charge of extracting the features from the picture data; the initial layers do this for the low-level features, the intermediate layers do this for the mid-level features, and the high-level layers do this for the high-level features. The retrieved features, also known as activation maps, are down sampled by a new sort of layer in CNN called the pooling layer. Image categorization, Object identification, Pose estimation, and many more uses for CNN are among the most common.

  

  Image representing feature extraction and classification using CNN

• Recurrent Neural Network (RNN)

  A recurrent neural network (RNN) is a type of artificial neural network which uses time series data and sequential data. This neural network exhibits a dynamic approach. RNN has internal memory which can store the representation of the input and give it as feedback for the next input. This property helps it to forecast the time series data. The common activation function used in an RNN is hyperbolic tan (tanh). Along with the time-series data it also accepts the sequential data so it is best situated for text processing. A sentence(Sequence of words) can be given as an input to the RNN and perform certain analyses such as sentiment analysis, next-word prediction, etc.

  RNN faces a challenge called the vanishing gradient problem which is caused due to longer length of sentences and this challenge can be overcome by using LSTM & GRU.

  LSTM - Long Short Term Memory is a neural network that is the same as RNN but the difference is LSTM has gates that can control how much information to remember or forget. So when there is a long sequence of words it tries to forget the unimportant words from its memory. LSTM has 3 gates, the forget gate, input gate, and output gate. Whereas GRU (Gated Recurrent Unit) has only 2 gates: Update gate(responsible for how much of the memory to retain) and Reset gate(responsible for how much of the memory to forget).

  

• Radial Basis Function Neural Network

  One of the forms of neural networks that employs RBF as an activation function is a radial basis function neural network (RBF Neural Network).

  Any point's distance from the centre is taken into account by the radial basis function. There are two layers in these neural networks. The features and radial basis function are merged in the inner layer.

  

  Image representing RBF Neural network