Artificial Neural Networks

Introduction

An artificial neural network (ANN) is an imitation of the human brain. A natural brain has the ability to learn new things, adapt to new and changing environments. The brain has the most amazing capability to analyze incomplete and unclear, fuzzy information, and make its own judgment out of it.

Artificial Neural Network (ANN) is gaining prominence in various applications like pattern recognition, weather prediction, handwriting recognition, face recognition, autopilot, robotics, etc. In electrical engineering, ANN is being extensively researched in load forecasting, processing substation alarms and predicting weather for solar radiation and wind farms. With more focus on smart grids, ANN has an important role. ANN belongs to the family of Artificial Intelligence along with Fuzzy Logic, Expert Systems, Support Vector Machines.

Brain Neuron

Brain is made of cells called neurons. Interconnection of such cells (neurons) makes up the neural network or the brain. There are about 1011 neurons in the human brain and about 10000 connections with each other. ANN is an imitation of the natural neural network where the artificial neurons are connected in a similar fashion as the brain network. A biological neuron is made up of the cell body, axon and dendrite. Dendrite receives electro-chemical signals from other neurons into the cell body. Cell body, called Soma contains the nucleus and other chemical structures required to support the cell. Axon carries the signal from the neuron to other neurons. Connection between dendrites of two neurons, or neurons to muscle cells is called synapse.

Fig 1: Brain Neuron

The neuron receives signals from other neurons through dendrites. When the strength of the signal exceeds a certain threshold, this neuron triggers its own signal to be passed on to the next neuron via the axon using synapses. The signal sent to other neurons through synapses triggers them, and this process continues. A huge number of such neurons work simultaneously. The brain has the capacity to store large amounts of data.

Artificial Neuron

An artificial neural network consists of processing units called neurons. An artificial neuron tries to replicate the structure and behavior of the natural neuron. A neuron consists of inputs (dendrites), and one output (synapse via axon). The neuron has a function that determines the activation of the neuron.

Fig 2: Model of an Artificial Neuron

x1…xn are the inputs to the neuron. A bias is also added to the neuron along with inputs. Usually the bias value is initialised to 1. W0…Wn are the weights. A weight is the connection to the signal. Product of weight and input gives the strength of the signal. A neuron receives multiple inputs from different sources, and has a single output. There are various functions used for activation. One of the most commonly used activation function is the sigmoid function, given by

Fig 3: Sigmoid Function

The other functions that are used are Step function, Linear function, Ramp function, Hyperbolic tangent function. Hyperbolic tangent (tanh) function is similar in shape to sigmoid, but its limits are from -1 to +1, unlike sigmoid which is from 0 to 1. The sum is the weighted sum of the inputs multiplied by the weights between one layer and the next. The activation function used is a sigmoid function, which is a continuous and differentiable approximation of a step function. An interconnection of such individual neurons forms the neural network. The ANN architecture comprises: a. input layer: Receives the input values b. hidden layer(s): A set of neurons between input and output layers. There can be single or multiple layers c. output layer: Usually it has one neuron, and its output ranges between 0 and 1, that is, greater than 0 and less than 1. But multiple outputs can also be present.

Fig 4: Neural Network Architecture

The processing ability is stored in inter-unit connection strengths, called weights. Input strength depends on the weight value. Weight value can be positive, negative or zero. Negative weight means that the signal is reduced or inhibited. Zero weight means that there is no connection between the two neurons. The weights are adjusted to obtain the required output. There are algorithms to adjust the weights of ANN to get the required output. This process of adjusting weights is called learning or training.

ANN Training Process

Categories of ANN are based on supervised and unsupervised learning methods. The simplest form of ANN architecture is the Perception, which consists of one neuron with two inputs and one output. The activation function used is step function or ramp function. Perceptions are used for classification of data into two separate classes. For more complex applications, multilayer perceptions (MLP) are used, which contain one input layer, one output layer, and one or more hidden layers as given in Fig 2. Back propagation algorithm is the most commonly used method in training the neural network. Here the difference in targeted output, and the output obtained, is propagated back to the layers and the weights adjusted. A back propagation neural network (BPNN) uses a supervised learning method and feed-forward architecture. It is one of the most frequently utilized neural network techniques for classification and prediction. In the BP algorithm, the outputs of hidden layers are propagated to the output layer where the output is calculated. This output is compared with the desired output for the given input. Based on this difference, the error is propagated back from the output layer to hidden layer, and from hidden layer to input layer. As the flow moves back, it changes the weights between the neurons. This cycle of going forward from input and output, and from output to input is called an epoch. A neural network is first given a set of known input data and asked to obtain a known output. This is called training the network. The network undergoes many such epochs till the error (difference between actual output and desired output) is within a certain tolerance). Now the network is said to be trained. This process of training sets the weights between all the neurons in all the layers. The weights obtained from a trained network are used in calculating the response of the network to an unknown data.

ANN Methodology

1. Feed Forward Networks

This is a subclass of acrylic networks in which a connection is allowed from a node in layer i only to nodes in layer i+1 as shown in Fig below These networks are succinctly described by a sequence of numbers indicating the number of nodes in each layer. For instance, the network shown in Fig below is a 3–2–3–2 feed forward network; it contains three nodes in the input layer (layer 0), two nodes in the first hidden layer (layer 1), three nodes in the second hidden layer (layer 2), and two nodes in the output layer (layer 3). These networks, generally with no more than four such layers, are among the most common neural nets in use, so much so that some users identify the phrase “neural networks” to mean only feed forward networks. Conceptually, nodes in successively higher layers abstract successively higher level features from preceding layers. In the literature on neural networks, the term “feed forward” has been used sometimes to refer to layered or acrylic networks.

Fig. Feed Forward Networks

1. Neural Learning

It is reasonable to conjecture that neurons in an animal’s brain are “hard wired.” It is equally obvious that animals, especially the higher order animals, learn as they grow. How does this learning occur? What are possible mathematical models of learning? In this section, we summarize some of the basic theories of biological learning and their adaptations for artificial neural networks. In artificial neural networks, learning refers to the method of modifying the weights of connections between the nodes of a specified network. Learning is the process by which the random-valued parameters (Weights and bias) of a neural network are adapted through a continuous process of simulation by the environment in which the network is embedded. Learning rate is defined as the rate at which a network gets adapted. Type of learning is determined by the manner in which parameter change takes place. Learning may be categorized as supervised learning, unsupervised learning and reinforced learning. In Supervised learning, a teacher is available to indicate whether a system is performing correctly, or to indicate a desired response, or to validate the acceptability of a system’s responses, or to indicate the amount of error in system performance. This is in contrast with unsupervised learning, where no teacher is available and learning must rely on guidance obtained heuristically by the system examining different sample data or the environment. Learning is similar to training i.e. one has to learn something which is analogous to one has to be trained. A neural network has to be configured such that the application of a set of inputs produces (either ‘direct’ or via a relaxation process) the desired set of outputs. Various methods to set the strengths of the connections exist. One way is to set the weights explicitly, using a priori knowledge. Another way is to ‘train’ the neural network by feeding it teaching patterns and letting it change its weights according to some learning rule. We can categorize the learning situations in two distinct sorts. These are

1. Supervised learning
2. Unsupervised learning
3. Reinforcement learning

C. Back Propagation Network

The back propagation algorithm (Rumelhart and McClelland, 1986) is used in layered feed-forward ANNs. This means that the artificial neurons are organized in layers, and send their signals “forward”, and then the errors are propagated backwards. The network receives inputs by neurons in the input layer, and the output of the network is given by the neurons on an output layer. There may be one or more intermediate hidden layers. The back propagation algorithm uses supervised learning, which means that we provide the algorithm with examples of the inputs and outputs we want the network to compute, and then the error (difference between actual and expected results) is calculated. The idea of the back propagation algorithm is to reduce this error, until the ANN learns the training data. The training begins with random weights, and the goal is to adjust them so that the error will be minimal. Back propagation network has gained importance due to the shortcomings of other available networks. The network is a multi-layer network (multilayer perceptron) that contains at least one hidden layer in addition to input and output layers. Number of hidden layers & numbers of neurons in each hidden layer is to be fixed based on application, the complexity of the problem and the number of inputs and outputs. Use of non-linear log-sigmoid transfer function enables the network to simulate non-linearity in practical systems. Due to these numerous advantages, the back propagation network is chosen for present work. Implementation of the back propagation model consists of two phases. First phase is known as training while the second phase is called Testing. Training in back propagation is based on gradient descent rules that tend to adjust weights and reduce system error in the network. Input layer has neurons equal in number to that of the inputs. Similarly, output layer neurons are the same in the number as number of outputs. Number of hidden layer neurons is decided by trial and error method using the experimental data.

D. ANN Development & Implementation

In this work, both ANN implementation & training is developed, using the neural network toolbox of MatLab. Different ANNs are built rather than using one large ANN including all the output variables. This strategy allowed for better adjustment of the ANN for each specific problem, including the optimization of the architecture for each output.

Strengths and Weaknesses of ANN

ANN is different from a normal computer program in many ways. Some of its features are: a. Adaptive learning: ANN replicates the human brain in the way it learns how to do tasks while learning. A normal program cannot adapt to other types of inputs b. Self organization: ANN can create its own organization while learning. A normal program is fixed for its task and will not do anything other than what it is intended to do c. Parallel operation: ANN works in parallel like a human brain. This is dissimilar to a computer program which works serially. d. Fault tolerance: One of the most interesting properties of neural networks is their ability to work even on the basis of incomplete, noisy, and fuzzy data. A normal program cannot handle incomplete, unclear data and will stop working once it encounters the smallest wrong data. e. In comparison with the human brain, ANN is quite fast, as brain processing time is slower f. In comparison with normal programs, the method in which ANN calculates output is not clear. The time taken keeps changing with different sets of inputs, though they are similar. g. ANN can be used for data classification, pattern recognition, and in applications where data is unclear h. ANN cannot be used when the nature of input and output is exactly known, and what needs to be done is clearly known.

Conclusion

As the ANN is an emerging technology it can be used for data analysis in applications such as pattern recognition, prediction, system identification & control. From above theories it can be seen that ANN is a radial basis function back propagation network. The network is capable of predicting the parameters by an experimental system. The network has parallel structure and fast learning capacity. The collected experimental data such as speed, load, & values of pressure distribution etc. are also employed as training and testing data for an artificial neural network. The neural network is a feed forward three layered network. Quick propagation algorithm is used to update the weight of the network during the training. The ANN has a superior performance to follow the desired results of the system and is employed to analyze such systems parameters in practical applications.

References

[1] Kuldeep Shiruru’s Paper on “AN INTRODUCTION TO ARTIFICIAL NEURAL NETWORK”.

[2] A.D.Dongare, R.R.Kharde, Amit D.Kachare’s Paper on “Introduction to Artificial Neural Network”.