2017 Internship at kLab

Motivation/Background: HSI classification What is HSI? Comparison to RGB The advantages of HSI over other kinds of data Instant, remote, nondestructive data collection Hundreds of data points per pixel What is classification? Introduce using toy data set, like CUB-200 Applications of HSI classification Cancer detection Art authentication Food quality analysis Remote sensing Urban development agricultural monitoring environmental assessment Problems faced High dimensionality Scarcity of labeled data Method: MCAE and Ladder Network Feature Extractors Baseline: PCA Inspiration: Autoencoder Our proposed method: MCAE Classifiers Baseline: SVM Inspiration: Neural network Our proposed method: Ladder network Results : State-of-the-art Comparison Comparison of MCAE against PCA and state-of-the-art Comparison of Ladder Network against SVM and state-of-the-art

Working with the ladder network all day has brought some things to light. Firstly, and most importantly, the model is overfitting like there is no tomorrow. This means that the model memorizes the training data instead of generalizing. Contrary to what high school has taught me, this is undesired. The original model predicts up to 15% more accurately on training data than validation data which is very bad. I've been trying to change parameters to reduce this margin, but I haven't had much success. The main problem is that there are way too many knobs to turn. I could add layers, remove layers, change the sizes of layers, adjust the amount of noise, change weights of costs, change the initial learning rate, change the learning rate function, or change the optimizer. Decisions, decisions.

The first thing I did when I got to the lab was test my changes. I believe the problem I was trying to solve was an exploding gradient, and my solution was gradient clipping. Don't quote me on that though. To my relief, it worked! Turns out there was a very simple solution to what seemed to be like a disastrous problem. I spent the rest of my time in the lab working on various parts of the ladder network. At 2:00, Mingming presented his driving simulator at Slaughter Hall. Watching Gerry drive like a madman and trying it out for myself was pretty fun. It's a cool project, and I think Mingming got some really good feedback/suggestions, which is nice to see. At the end of the day, the interns, REU students, and some staff drove down to the Mees Observatory. I'd never been to an observatory, so this was pretty cool. Unfortunately, we didn't get to see anything because of the weather. It was a good time nonetheless.

Today I made a lot of slow and steady progress. With the paper and the CAE experiment out of the way (for now), I focused on the ladder network code. First, I adapted the code to be object-oriented, so that it would (mostly) work within our pipeline. Every few iterations, I tested the code with Pavia U and MNIST, just to make sure I didn't stray away from the 80% and 98% I achieved earlier. I would guess that failing to do so is what gives rise to hard-to-trace bugs, and nobody likes those. There was one issue where the model would crash and burn around epoch 50, and predict nothing at all. This might be linked to the NaN loss that was occurring in the previous ladder network code, and I think I've found the solution. Whether I have succeeded or failed will be in tomorrow's blog post. I haven't had enough time to run >50 epochs with the fix yet, and everybody likes a good cliffhanger. Tune in next week to find out who shot Nate and the real reason he left to run in t...

"Happiness is when the code works." - poster from the lab I have finally made some progress on the ladder network. With the code from last time, I wrote a method to pass the Pavia University data into the ladder network. There were a surprising number of difficulties, but I got it to work. There were a few problems with how I formatted the data that I didn't notice at first, so I was heartbroken when the network made abysmal predictions. Turns out these algorithms don't learn too well when fed garbage, so I fixed the data and kept trying. Here is the end result. The ladder network was trained on 450 labeled HSI samples and 42,776 unlabeled samples trained over 30 epochs. I'll admit the results aren't great (an SVM achieved 84% on similar data). However, only necessary changes were made to the network, so this is basically the same model that was predicting digits yesterday. There's still room for optimization with this model, so I expect the accurac...

Today, Nate came back from the mountains to work in the lab. Welcome back Nate! I kept on doing the same work from last time. With the CAE experiment v.2 running in the background, I proofread and edited my section of the paper, and I worked with the ladder network for the rest of the day. First, I got rinuboney's implementation of the ladder network in TensorFlow ( github ) working in Python 3. Testing it with the MNIST dataset , I got the expected results (very good). Since we all love graphs, I quickly made one for said results. The idea is pretty simple. The ladder network is trained on 60,000 images of handwritten digits, 100 of which are labeled. The network makes predictions about the labels of 10,000 other images, and we calculate how accurate those predictions are. This process comprises 1 epoch, and is repeated 149 more times. By the end, the network achieves almost 99% accuracy. This is what we're hoping to do with HSI classification, but there seems to be an...

Today I started rereading the ladder network papers and trying to find out why the code doesn't work well. It was a fruitless endeavor; there's just too much that could go wrong. Perhaps the ladder network just does poorly on HSI data, and there is no flaw. Maybe it's as simple as changing some hyperparameters. Who knows? Most, if not all, of the interns came outside for lunch, which was a nice surprise. After lunch I was tired of looking at the ladder network code, so I went back to writing the paper. Taking the feedback Ron had last time, I improved on the SSL section a lot. It's not perfect, but we're getting there. I also had some time to visit the Perform Lab (Titus and Aditi's lab), where I got to see their SMI eye tracker (which was unfortunately malfunctioning), motion trackers, and HTC Vive, which they let me try. Very cool stuff; thanks to everyone at the Perform Lab.