ebook img

Neural Networks Using C# Succinctly PDF

128 Pages·2014·1.87 MB·english
Save to my drive
Quick download
Download
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Neural Networks Using C# Succinctly

Neural Networks using C# Succinctly James McCaffrey Copyright © 2014 by Syncfusion Inc. 2501 Aerial Center Parkway Suite 200 Morrisville, NC 27560 USA Contents Chapter 1 Neural Networks ............................................................................................................ 11 Introduction ................................................................................................................................. 11 Data Encoding and Normalization ................................................................................................. 13 Overall Demo Program Structure .................................................................................................. 14 Effects Encoding and Dummy Encoding ........................................................................................ 18 Min-Max Normalization ................................................................................................................. 23 Gaussian Normalization ............................................................................................................... 24 Complete Demo Program Source Code......................................................................................... 25 Chapter 2 Perceptrons ................................................................................................................... 30 Introduction ................................................................................................................................. 30 Overall Demo Program Structure .................................................................................................. 32 The Input-Process-Output Mechanism........................................................................................... 32 The Perceptron Class Definition .................................................................................................... 34 The ComputeOutput Method......................................................................................................... 35 Training the Perceptron ................................................................................................................ 36 Using the Perceptron Class .......................................................................................................... 40 Making Predictions....................................................................................................................... 42 Limitations of Perceptrons ............................................................................................................ 43 Complete Demo Program Source Code......................................................................................... 44 Chapter 3 Feed-Forward ................................................................................................................ 49 Introduction ................................................................................................................................. 49 Understanding Feed-Forward ....................................................................................................... 50 Bias Values as Special Weights .................................................................................................... 52 Overall Demo Program Structure .................................................................................................. 52 Designing the Neural Network Class ............................................................................................. 54 The Neural Network Constructor ................................................................................................... 56 Setting Neural Network Weights and Bias Values........................................................................... 57 Computing Outputs ...................................................................................................................... 58 Activation Functions ..................................................................................................................... 62 Complete Demo Program Source Code......................................................................................... 66 Chapter 4 Back-Propagation .......................................................................................................... 70 Introduction ................................................................................................................................. 70 The Basic Algorithm ..................................................................................................................... 71 Computing Gradients ................................................................................................................... 72 Computing Weight and Bias Deltas ............................................................................................... 73 Implementing the Back-Propagation Demo .................................................................................... 75 The Neural Network Class Definition ............................................................................................. 78 The Neural Network Constructor ................................................................................................... 79 Getting and Setting Weights and Biases ........................................................................................ 81 Computing Output Values............................................................................................................. 82 Implementing the FindWeights Method .......................................................................................... 84 Implementing the Back-Propagation Algorithm ............................................................................... 85 Complete Demo Program Source Code......................................................................................... 88 Chapter 5 Training ......................................................................................................................... 95 Introduction ................................................................................................................................. 95 Incremental Training .................................................................................................................... 96 Implementing the Training Demo Program ..................................................................................... 97 Creating Training and Test Data ................................................................................................... 99 The Main Program Logic ............................................................................................................ 102 Training and Error ...................................................................................................................... 105 Computing Accuracy .................................................................................................................. 109 Cross Entropy Error ................................................................................................................... 112 Binary Classification Problems .................................................................................................... 114 Complete Demo Program Source Code....................................................................................... 116 Chapter 1 Neural Networks Introduction An artificial neural network (sometimes abbreviated ANN, or shortened to just "neural network" when the context is clear) is a software system that loosely models biological neurons and synapses. Before explaining exactly how neural networks work, it is useful to understand what types of problems they can solve. The image in Figure 1-a represents a typical problem that might be solved using a neural network. Figure 1-a: A Typical Problem The goal of the problem is to predict a person's political inclination based on his or her gender, age, home location, and annual income. One hurdle for those new to neural networks is that the vocabulary varies greatly. The variables used to make a prediction can be called independent variables, predictors, attributes, features, or x-values. The variable to predict can be called the dependent variable, the y-value, or several other terms. The type of problem shown in Figure 1-a is called a classification problem because the y-value can take one of three possible class values: conservative, liberal, or moderate. It would be perfectly possible to predict any of the other four variables. For example, the data could be used to predict a person's income based on his or her gender, age, home location, and political inclination. Problems like this, where the y-value is numeric, are often called regression problems. 11 There are many other related problem scenarios that are similar to the one shown in Figure 1-a. For example, you could have several million x-values where each represents the pixel value in a photograph of a person, and a y-value that represents the class of the picture, such as "on security watch list" or "not on watch list". Such problems are sometimes called image recognition problems. Or imagine x-values that represent digitized audio signals and y-values that represent vocabulary words such as "hello" and "quit". This is speech recognition. Neural networks are not magic and require data with known y-values, called the training data. In Figure 1-a there are only four training items. In a realistic scenario you would likely have hundreds or thousands of training items. The diagram in Figure 1-b represents a neural network that predicts the political inclination of a male who is 35 years old, lives in a rural area, and has an annual income of $49,000.00. Figure 1-b: A Neural Network As you will see shortly, a neural network is essentially a complicated mathematical function that understands only numbers. So, the first step when working with a neural network is to encode non-numeric x-data, such as gender and home location, into numeric data. In Figure 1-b, "male" is encoded as -1.0 and "rural" is encoded as (1.0, 0.0). In addition to encoding non-numeric x-data, in many problems numeric x-data is normalized so that the magnitudes of the values are all roughly in the same range. In Figure 1-b, the age value of 35 is normalized to 3.5 and the income value of $49,000.00 is normalized to 4.9. The idea is that without normalization, x-variables that have values with very large magnitudes can dominate x-variables that have values with small magnitudes. The heart of a neural network is represented by the central box. A typical neural network has three levels of nodes. The input nodes hold the x-values. The hidden nodes and output nodes perform processing. In Figure 1-b, the output values are (0.23, 0.15, 0.62). These three values loosely represent the probability of conservative, liberal, and moderate respectively. Because the y-value associated with moderate is the highest, the neural network concludes that the 35- year-old male has a political inclination that is moderate. 12 The dummy neural network in Figure 1-b has 5 input nodes, 4 hidden nodes, and 3 output nodes. The number of input and output nodes are determined by the structure of the problem data. But the number of hidden nodes can vary and is typically found through trial and error. Notice the neural network has (5 * 4) + (4 * 3) = 32 lines connecting the nodes. Each of these lines represents a numeric value, for example -1.053 or 3.987, called a weight. Also, each hidden and output node (but not the input nodes) has an additional special kind of weight, shown as a red line in the diagram. These special weights are called biases. A neural network's output values are determined by the values of the inputs and the values of the weights and biases. So, the real question when using a neural network to make predictions is how to determine the values of the weights and biases. This process is called training. Put another way, training a neural network involves finding a set of values for the weights and biases so that when presented with training data, the computed outputs closely match the known, desired output values. Once the network has been trained, new data with unknown y- values can be presented and a prediction can be made. This book will show you how to create neural network systems from scratch using the C# programming language. There are existing neural network applications you can use, so why bother creating your own? There are at least four reasons. First, creating your own neural network gives you complete control over the system and allows you to customize the system to meet specific problems. Second, if you learn how to create a neural network from scratch, you gain a full understanding of how neural networks work, which allows you to use existing neural network applications more effectively. Third, many of the programming techniques you learn when creating neural networks can be used in other programming scenarios. And fourth, you might just find creating neural networks interesting and entertaining. Data Encoding and Normalization One of the essential keys to working with neural networks is understanding data encoding and normalization. Take a look at the screenshot of a demo program in Figure 1-c. The demo program begins by setting up four hypothetical training data items with x-values for people's gender, age, home location, and annual income, and y-values for political inclination (conservative, liberal, or moderate). The first line of dummy data is: Male 25 Rural 63,000.00 Conservative The demo performs encoding on the non-numeric data (gender, locale, and politics). There are two kinds of encoding used, effects encoding for non-numeric x-values and dummy encoding for non-numeric y-values. The first line of the resulting encoded data is: -1 25 1 0 63,000.00 1 0 0 After all data has been converted to numeric values, the data is stored into a matrix in memory and displayed. Next, the demo performs normalization on the numeric x-values (age and income). The first line of encoded and normalized data is: -1.00 -0.84 1.00 0.00 0.76 1.00 0.00 0.00 13 The demo uses two different types of normalization, Gaussian normalization on the age values, and min-max normalization on the income values. Values that are Gaussian normalized take on values that are typically between -10.0 and +10.0. Values that are min-max normalized usually take on values that are between 0.0 and 1.0, or between -1.0 and +1.0. The demo program uses two different types of normalization just to illustrate the two techniques. In most realistic situations you would use either Gaussian or min-max normalization for a problem, but not both. As a general rule of thumb, min-max normalization is more common than Gaussian normalization. Figure 1-c: Data Encoding and Normalization Overall Demo Program Structure To create the demo program, I opened Visual Studio, selected the C# console application project template, and named it Normalize. The demo program has no significant .NET version dependencies, so any version of Visual Studio should work. After the template code loaded in the editor, in the Solution Explorer window I renamed the Program.cs file to the slightly more descriptive NormalizeProgram.cs, and Visual Studio automatically renamed the Program class. 14 At the top of the source code I deleted all using statements except the one that references the top-level System namespace. The demo was written using a static-method approach rather than an object-oriented approach for simplicity and ease of refactoring. The overall structure of the demo program is presented in Listing 1-a. Methods GaussNormal and MinMaxNormal operate on a matrix of numeric values and normalize a single column of the matrix. Methods ShowMatrix and ShowData are just convenience helpers to keep the Main method a bit tidier. Method EncodeFile operates on a text file and performs either effects encoding or dummy encoding on a specified column of the file. Methods EffectsEncoding and DummyEncoding are helpers that are called by the method EncodeFile. The demo program has all normal error-checking code removed in order to keep the main ideas as clear as possible. using System; namespace Normalize { class NormalizeProgram { static void Main(string[] args) { Console.WriteLine("\nBegin data encoding and normalization demo\n"); // Set up raw source data. // Encode and display data. // Normalize and display data. Console.WriteLine("\nEnd data encoding and normalization demo\n"); Console.ReadLine(); } static void GaussNormal(double[][] data, int column) { . . } static void MinMaxNormal(double[][] data, int column) { . . } static void ShowMatrix(double[][] matrix, int decimals) { . . } static void ShowData(string[] rawData) { . . } static void EncodeFile(string originalFile, string encodedFile, int column, string encodingType) { . . } static string EffectsEncoding(int index, int N) { . . } static string DummyEncoding(int index, int N) { . . } } // Program class } // ns Listing 1-a: Encoding and Normalization Demo Program Structure All program control logic is contained in method Main. The method definition begins: static void Main(string[] args) { Console.WriteLine("\nBegin data encoding and normalization demo\n"); string[] sourceData = new string[] { 15

See more

The list of books you might like

Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.