Problems with measuring performance in Machine Learning

With today's advancements in AI we often see media reports of superhuman performance in some task. These often quite dramatic announcements should however be treated with a dose of skepticism, as many of them may result purely from pathologies in measures applied to the problem. In this post I'd like to show what I mean by a "measurement pathology". I therefore constructed a simple example, which hopefully will  to get the point across.

Example: measuring lemons

Imagine somebody came to your machine learning lab/company with a following problem: identify lemons in a photo. This problems sounds clear enough, but in order to build an actual machine learning system that will accomplish such task, we have to formalize what this means in the form of a measure (of performance). The way this typically begins, is that some student will laboriously label the dataset. For the sake of this example, my dataset consists of a single image with approximately 50 lemons in it:

As mentioned the picture was carefully labeled:

With human labeled mask here:

Now that there is a ground truth label we can establish a measurement. One way to formally express the desire to identify lemons in this picture … Read more...

Silent immobility

I have a few AI related posts in the pipeline, but before I publish them (most still need some work), I want to share my recent experience and some thoughts on it.

I just came back from a trip to Europe, a typical summer visit. The trip went fine, children are happy, the whole flight was uneventful. I've spent there a week, back in my hometown visiting friends and family. This time however I decided to pay attention to something different than usual, instead of focusing on stuff that has changed, I decided to seek the stuff that remained the same.

It's been more than 7 years since I moved from Poland to California, nevertheless there are countless things there which seem to not have changed at all e.g particular stores and institutions, my neighbors, bars and coffees etc. Wound up with the constant push for progress, we tend to not see how many things appear to be frozen in time.

Airlines

Now let me get to a concrete example of what I'm talking about: on my way there I obviously took a transcontinental flight, one of the mayor European airlines. A nice and neat Airbus A380 welcomed us at … Read more...