Solved-Neural Network & Image Classification- Solution

Description

a The assignment should be submitted in the PDF format through Collob. If you prefer hand-writing QA parts of answers, please convert them (e.g., by scanning or using PhoneApps like o ceLens) into PDF form.

• For questions and clari cations, please post on piazza.

• Policy on collaboration:

Homework should be done individually: each student must hand in their own answers. It is acceptable, however, for students to collaborate in guring out answers and helping each other solve the problems. We will be assuming that, with the honor code, you will be taking the responsibility to make sure you personally understand the solution to any work arising from such collaboration.

d Policy on late homework: Homework is worth full credit at the midnight on the due date. Each student has three extension days to be used at his or her own discretion throughout the entire course. Your grades would be discounted by 15% per day when you use these 3 late days. You could use the 3 days in whatever combination you like. For example, all 3 days on 1 assignment (for a maximum grade of 55%) or 1 each day over 3 assignments (for a maximum grade of 85% on each). After you’ve used all 3 days, you cannot get credit for anything turned in late.

• Question: Neural Network Playground

Tensor ow Playground is an interactive tool for learning neural networks (more speci cally, multi-layer-perceptron¹ networks). A customized version of Tensor ow Playground is available at http://www.cs. virginia.edu/yanjun/HW/playground/.

First, we will get familiar with the interface of Tensor ow Playground. Let’s get to it!

You can choose from four di erent datasets on the left side of the page in the DATA panel: Circle, Exclusive Or, Gaussian, and Spiral.The actual data point locations are available on the right side under the OUTPUT label. Do not change the ratio of training to test data or click the REGENERATE button throughout this question. Batch size indicates how many samples are used in your mini-batch gradient descent. You may change this parameter if you want.

The neural network model is in the middle of DATA and OUTPUT. This model is a standard \feed-forward” neural network, where you can vary: (1) the input features (2) the number of hidden layers (3) the number of neurons at each layer. By default, it uses only the raw inputs X₁ and X₂ as features, and no hidden layers. You will need to change theses attributes later.

Several hyper-parameters are tunable at the top of the page, such as the learning rate, the activation (non-linearity) function of each neuron, the regularization norm as well as the regularization rate.

¹A perceptron model typically uses a heaviside step function as its activation. Since we use a sigmoid (or other di erentiable nonlinearity), our representation not exactly a perceptron model, but that is what we call it

There are three buttons at the top left for you to control the training of a neural network: Reset, Run/Pause and Step (which steps through each mini-batch of samples at a time).

1.1 Hand-crafted Feature Engineering

Now that we are familiar with the Playground, we will start to build models to classify samples.

First you are going to earn some experience in hand-crafted feature engineering with a simple perceptron model with no hidden layers, sigmoid activations, no regularization. In other words, don’t change the model you’re given by default when loading the page.

A perceptron (single-layer arti cial neural network) with a sigmoid activation function is equivalent to lo-gistic regression. As a linear model, it cannot t some datasets like the Circle, Spiral, and Exclusive Or. To extend linear models to represent nonlinear functions of x, we can apply the linear model to a transformed input (x).

One option is to manually engineer . This can easily be done in the Playground since you are given 7 di erent features to choose from in the FEATURES panel.

Task: You are required to nd the best perceptron models for the four datasets, Circle, Exclusive Or, Gaussian and Spiral by choosing di erent features. Try to select as few features as possible. For the best model of each dataset, you should report the selected features, iterations and test loss in a table. If you also change other hyper-parameters, e.g. the learning rate, you should include them in your report.

For each dataset (Circle, Exclusive Or, Gaussian and Spiral), please include a web page screenshot of the result in your report and explain why this con guration works.

(Note: Don’t worry if you cannot nd a good perceptron for the Spiral dataset. For the other three datasets, the test loss of a good model should be lower than 0.001.)

1.2 Regularization

Forget the best features you have found in last question. You should now select all the possible (i.e. all 7) features to test the regularization e ect here.

You are required to work on the three datasets: Circle, Exclusive Or and Gaussian.

Task A: Try both L1 and L2 regularization with di erent (non-zero) regularization rates. In the report, you are required to compare the decision boundary and the test loss over the three models trained with similar number of iterations: no regularization, L1 regularized, L2 regularized.

For each dataset (Circle, Exclusive Or and Gaussian), please include a web page screenshot of the result in your report and explain why this con guration works.

Task B: We have learned that L1 regularization is good for feature selection. Take a look at the features with signi cantly higher weights. Are they the same as the ones you select in last question? Write down the results you observe in your report. (You can get the feature weights by moving the mouse pointer over the dash lines.)

1.3 Automated Feature Engineering with Neural Network

While we were able to nd di erent parameters which were able to make good predictions, the previous two sections required a lot of hand-engineering and regularization tweaking :( We will now explore the power of a neural network’s ability to automatically learn good features :) Let’s try it out on two datasets: the Circle and Exclusive Or.

Here we should only select X₁ and X₂ as features (since we are trying to automatically learn all other fea-tures from the network). As we have previously seen, a simple perceptron is not going to learn the correct boundaries since both datasets are not linearly separable. However, a more complex neural network should be able to learn an approximation of the complex features that we have selected in the previous experiments.

You can click the + button in the middle to add some hidden layers for the model. There is a pair of + and

• buttons at the top of each hidden layer for you to change the number of the hidden units. Note that each hidden unit, or neuron, is the same neuron from Lecture 17 (possibly varying the sigmoid activation function).

Task: Find a set of neural network model parameters which allow the model to nd a boundary which cor-rectly separates the testing samples. Report the test loss and iterations of the best model for each dataset. If you modify the other parameters (e.g. activation function), please report them too. Can it beat or approach the result of your hand-crafted feature engineering?

For each dataset (Circle, Exclusive Or), please include a web page screenshot of the result in your report and explain why the con guration would work.

1.4 Spiral Challenge (0.5 Extra credit)

Congratulations on your level up!

Task: Now try to nd a model that achieves a test loss lower than 0.01 on the Spiral dataset. You‘re free to use other features in the input layer besides X₁ and X₂, but a simpler model architecture is preferred. Report the input features, network architecture, hyper parameters, iterations and test loss in a table. For simplicity, please represent your network architecture by the hidden layers a-b-c-…, where a, b, c are number of hidden units of each layer respectively.

Please include a web page screenshot of the result in your report and explain why this con guration works.

You are now a neural network expert (on the Playground)!

• Question: Image Classi cation)

An online tutorial might be helpful to you for nishing this assignment, http://blog.yhathq.com/posts/image-classification-in-Python.html

As part of Homework 3, you will be participating in a class-wide Kaggle-style competition. Given a dataset of images of handwritten digits, you will create a model to classify a given image as the digit it rep-resents. You will be competing with your peers to create the model with the greatest classi cation accuracy.

The data set is available as zip.train and zip.test. Each example in the train and test data is a the number in question followed by 256 grayscale values representing a 16 16 image. Values have already been normalized. You can read a full description of the data set in the zip.info.txt le available in collab. Visualization of some samples are provided in the subdir \image digits” in the attached data ZIP le.

Your objective is to develop a model with the highest possible classi cation accuracy for the images.

2.1 Evaluation

The main evaluation metric for this competition is Classi cation Accuracy (%-age of correctly classi ed images). Classi cation accuracy is given by:

_{Acc =} #correct

#total

on the testing dataset.

Your code should generate predicted labels of the testing dataset. Each prediction label should be on its own line!

The results le should follow the following format:

1

9

5

6

2

3

8

4

6

0

2.2 Extra Credit

Top-performing models will receive up to 1 extra credit point on this homework: (The top 20 submissions will get extra credits. We will run your code to generate the predicted labels.)

2.3 Submission

Please submit your source code and PDF report containing your written answers via collab.

In collab, you will nd starter code in number recognition.py. This le will be run from the command line.

python number recognition.py path/to/training data path/to/testing data

Feel free to add any methods or classes to your implementation, so long as your preserve the given command line behavior for your submission.

2.4 About Data

Normalized handwritten digits, automatically scanned from envelopes by the U.S. Postal Service. The original scanned digits are binary and of di erent sizes and orientations; the images here have been deslanted and size normalized, resulting in 16 x 16 grayscale images (LeCun et al., 1990).

The data are in two les. Each line in the training dataset consists of the digit id (0-9) followed by the 256 grayscale values. The testing dataset is identical, but with no digit id.

There are 7291 training observations and 2007 test observations, distributed as follows:

	0	1	2	3		4		5		6	7	8		9	Total
Train 1194		1005	731	658	652		556		664		645	542	644		7291
Test	359	264	198	166	200		160		170		147	166	177		2007
or as proportions:
	0	1	2		3		4		5		6		7		8	9
Train 0.16		0.14	0.1	0.09		0.09		0.08			0.09	0.09		0.07		0.09
Test	0.18	0.13	0.1	0.08		0.10		0.08			0.08	0.07		0.08		0.09

Alternatively, the training data are available as separate les per digit (and hence without the digit identi er in each row)

The test set is notoriously “di cult”, and a 2.5% error rate is excellent. These data were kindly made available by the neural network group at AT&T research labs (thanks to Yann LeCunn).

Figure 1: Description of this data set (zip.info.txt le)

2.5 Discussion of PCA and Logistic Regression

Principal component analysis allows us to reduce the dimensions of our data while retaining as much variance as possible, thus retaining information to separate our data points. Scikit learn provides multiple ways to perform dimension reduction with PCA. For this problem, sklearn.decomposition.RandomizedPCA is recommended.

Please select three di erent values of PCs you choose for PCA and present your prediction error results when using the Scikit logistic regression classi er. Please draw a gure showing how the prediction error changes when varying the number of PCs? Please also include the error discussions about the logistic regression without PCA in the discussion and in the gure.

Since the labels of the zip.test are fake, you surely should not use results from zip.test to discuss the performance.

– Q: can we use prede ned functions like logistic regression, or cross validation from scikit-learn? { A: Yes.

– Q: how many classi ers should be made, and how to deal with using the best one but including all of them?

{ A: You need to submit all the codes about the methods and model selection of the methods you tried. And you need to make sure the main function providing the interface for TA to run as the best one that you make!

{ A: please discuss the results of what you have tried in the written part of the submission.

– Q: Must we implement all the models suggests by the skeleton code? { A: No. The template is just a recommendation what you can try.

{ A. The minima requirement for this coding problem requires your discussion of running the LogRegression(LoG) and PCA+LoG on the datasets.

– Q: Is testing data being released separately and if so when? – Q: If not, will you simply run the scripts as stated in the homework instructions? (without the parameter for model selection, students just only include the one they want run).

13