Naïve Bayes Classifier. Neural Network. Benchmarks. Gradient-Based Learning Applied to Document Recognition by LeCun, Y., Bottou, L., Bengio, Y., Haffner, ...
1.Analysis of Classification Algorithms In Handwritten Digit Recognition Logan Helms Jon Daniele
2.Classification Algorithms Template Matching Naïve Bayes Classifier Neural Network
3.Benchmarks Gradient-Based Learning Applied to Document Recognition by LeCun, Y., Bottou , L., Bengio , Y., Haffner , P. Comparison of Machine Learning Classifiers for Recognition of Online and Offline Handwritten Digits by Omidiora , E., Adeyanju , I., Fenwa , O.
4.MNIST Training Set: 60,000 samples Test Set: 10,000 samples Accuracy: Number of correctly guessed test samples/ 10,000
5.Naïve Bayes classifier
6.Naïve Bayes Classifier Each pixel value (on/off) is independent of any other pixel value Each pixel has a probability associated with being on or off in any given digit class The probability of each pixel is used to determine the probability of an unknown digit being classified in one of the known classes
7.Naïve Bayes Classifier Training set: 60000 digits Test set: 10000 digits Success rate: Abysmal: 08.13% correct classification rate Benchmark: WEKA: Multimodal Naive Bayes: 83.65% 08.13% <<<<<<<<< 83.65%
8.Naïve Bayes Classifier Challenges: Pixel probabilities change according to the shape of the digit Are pixels the best feature set by which to compare different digits? Input size: 28x28 digit image results 786 pixels Requirements for matrix manipulation
9.Naïve Bayes Classifier Improvements Discarding extraneous pixel data Pixel values are mainly contained in a 20x20 matrix Using a binary pixel value vs a range of pixel values (0-255) Edge detection Incorporate feature extractor(s) and evaluate images based on those features
11.Neural Network Type: Feed Forward Training: Back-propagation algorithm Response Function: Architectures: Name Input Layer Hidden Layer Output Layer NN300 784 300 10 NN1000 784 1000 10
13.NN300 Training time: ~17 hours (~52 mins/epoch) Learning rate: Epoch Rate 1, 2 0.0005 3, 4, 5 0.0002 6, 7, 8 0.0001 9, 10, 11, 12 0.00005 13, 14, 15, 16, 17, 18, 19, 20 0.00001
14.NN1000 Training time: ~2.5 days (~3 hrs/epoch) Learning rate: Epoch Rate 1, 2 0.0005 3, 4, 5 0.0002 6, 7, 8 0.0001 9, 10, 11, 12 0.00005 13, 14, 15, 16, 17, 18, 19, 20 0.00001
15.Results After 20 epochs Network Accuracy Benchmark 1 95.30% NN300 75.82% Benchmark 2 75.12% Network Accuracy Benchmark 1 95.50% NN1000 -
16.Benchmark 1 95.30% On MNIST test set as is. 96.4% Generated more training data by using artificial distortions 98.4% When using deslanted images
17.Future Work Further training of NN300 with the MNIST test set has increased accuracy to 84.01% Experiment with hidden neuron count and multiple hidden layers Research other types of neural networks