In case we weren’t sure

“Once upon a time, or so the story goes, the American military were developing a computer system that they could train to identify tanks on the battlefield. The approach involved connecting a ‘neural network’ to a camera. The training was to be done using photographs. So the design team went out into the field and took 100 photographs of scenes with tanks in various orientations – out in the open, hiding behind trees, and the like. They also took 100 photographs of scenes with no tanks present. The system would be taught using both positive and negative cases. They split all the photographs into two sets, one for training and one for testing the system after training had taken place. Using the training set, they showed the system pictures of tanks and said, ‘Tank’. They also showed the system pictures without tanks and said, ‘No tank’. Each time the system would first have a guess, and if shown to be wrong would adjust itself. A keen understanding would emerge, it was hoped, of the key features it needed to consider in making the right judgment. From entirely random beginnings the system’s performance improved. It got so proficient that it could give a correct answer most of the time. The next step was to test the system on the remaining photos—the set that it had not yet seen. It behaved extremely well—perfectly in fact, categorizing every photo as either ‘tank’ or ‘no tank’ correctly. The designers decided to commission a further set of photos for more testing. The pictures came back and they were shown to the system. Only this time its performance was abysmal—no better than flipping a coin. It took the designers a while to work out what was going on. It turned out that the original photographs with tanks and without tanks had been taken on different days. The ‘tank’ days happened to be sunny. The ‘no tank’ days had been cloudy. Each time the system was shown a photograph with a tank, it saw bright sunlight, blue skies and shadows. Each time it saw a photograph without a tank, it saw grey skies and an absence of shadows. This was the meaning of ’tank’ it inferred. The designers had developed a sunny day detector, and a good one at that.” – Lawrence Chapin, et al., “Predictive Coding, Storytelling, and God: Narrative Understanding in e-Discovery”