figure 1: Example of bounding boxes.

Approach/Algorithm

1. Bounding Boxes

One of the first methods we found was the use of bounding boxes. This method characterizes where a character is versus what is the background of an image and then creates a bounding box around each character (see figure 1). First we had to resize each image read in Matlab to be three times its original size in order to make it large enough for the characters to be recognized. Next, was the task of determining where text was in the image compared to what was just background. To do this we used edge detection techniques to isolate the background from the text. Initially the Prewitt and Sobel functions within Matlab proved to be the most effective when using a threshold of 0.5 and 0.09 respectively. However, after testing multiple images with these edge detectors, the Canny method proved to be the most



figure 2: some of the extracted bounding boxes sent to classifier.

effective and accurate when using a threshold of 0.62 amongst the majority of our data set. After edge detection, we used bwareafilt and extracted the connected objects from the image within the range 500-5000. Some of the characters were lost in the initial edge detection stage due to minor edges being lost, that would enclose the character. In order to capture more enclosed characters, we used dilation to expand our found edges. Characters were the majority of the enclosed objects detected however, there were several enclosed/connected non-character objects in the background that were included in these extracted features which we took into account and resolved later using a classifier. After this step, we took the complement of the result and used regionprops to create bounding boxes around the objects found. We were then able to extract the objects within the bounding boxes and save them as individual images. These images were then passed to our classifier which was able to determine character versus non character objects (see figure 2).
   

2. HOG

Once individual characters were found, we extracted each image within every bounding box and sent them to our classifier using im_crop. The classifier was trained to detect characteristics of every character, including uppercase letters and numbers. These learned classifications were compared to the characteristics of every object found within the bounding boxes. To detect these characteristics we decided to use the HOG (Histogram of Gradients) method. In order to implement this method we used several variables including:

Training Images : This variable set the correct path through folders to extract images' names.

Taining Boxes: A 4 X N array of object bounding boxes. 4 represents Xmin, Ymin, Xmax, and Ymax respectively. N represents the number of bounding boxes.

Training Box Image Names: A 1 X N list of the names of the images containing bounding boxes.

Training Box Labels: The object label for each bounding box.

Training Box Patches: A 64 X 64 X 3 X N array of patches for each object.

figure 3: Small example of positive data sets for 'A'.

For the first sprint we spent a majority of the time focusing on making sure our classifier worked for the letter 'A' before moving on to any other characters. In order to align the aforementioned variables with the correct data, we created two data sets: positive and negative. For example, the positive image set for the letter 'A' included a large set of images (about 1000 images each for uppercase, handwritten, and multi-font variations). In contrast, the negative image set included all letters that were NOT the letter 'A' and images containing no letters at all. Once we had our positive and negative sets, all of the images were then resized to be equivalent to one another and all aforementioned variables were matched with their respective data for each letter. Finally, to compute the HOG features, a loop was implemented to traverse all training box patches using the function vl_hog (These results can be seen in figures 4 and 5 below). Once the features of the character were defined, we were able to compare the images from the bounding boxes to this set of features in order to detect what the character within the bounding box was. This was computed by getting a score for the character to see how closely its characterists were related to the characterists found by our HOG function. First, we created an array of scores for the positive and negative images using vl_nnconv. By using this array with the ROC function, we got a threshold for our character (each character contained a different threshold). For example, our calculated threshold was determined to be 25.3115 for the letter A, meaning that if our extracted images score was below this threshold, we could be confident that it was not the character we compared it to, and we could move on to the next character until we found a score above the threshold, signifying a positive match.

We organized our data into separate folders with names corresponding to the index of an array containing all characters we are able to identify. We are able to identify 36 characters in total, including all 26 letters of the English alphabet and numbers 0-9. We then looped through all character files for our positive training data and stored the mean HOG feature map for each character to a cell in an array, each index mapping to their respective character from the data in our organized files. From there we were able to take individual images sent from all bounding boxes and compare them to each of the mean HOG maps. We then saved the score of the comparison to each of the maps in an array where the index also corresponded to the array of characters. We extracted the index of the maximum score from this array. If it passed the threshold for the character at that index we can be confident that the two characters were a positive match.

HOG Features

Results

We started out by testing our algorithm with individual images of characters against their respective correct matching character. We looped through a list of test images and compared these with the HOG features for that image. We repeated this process with images that did NOT match the character we were testing. If the score a character obtained was lower than the threshold it was compared to, we rejected it as being a match and added 1 to the correct guess score. This test was run through several different inputs of characters and overall we found that our positive and negative data sets had an average of a 97.22% accuracy in finding matching characters to their classifying HOG descriptor. We found that our classifier was sometimes confused by images that were very similar in features. For example characters such as 'c' and 'o' had very similar features and would sometimes be misclassified. However by choosing the character that gave the higher score matching to the HOG features, it overcame this obstacle. Through this testing we realized that it is not the quantity of testing data that returns the best results, rather it is the quality. If we had a lot of images that were not high quality, there would be more features added to the HOG classifier that were not useful. We also found there would be a higher acceptance rate and therefore less accurate threshold for characters when we had more input data. Finally, we found that when we used images for our input data that was more closely related to our testing data, we were able to detect features more accurately. Since we obtained our training data first, we had to search for input images that were faced more straight forward than tilted at an angle, with very little or preferably no illumination variation, occlusion, or deformation.

Conclusions