While rereading my recent post [the meta-parameter slot machine], as well as a few papers suggested by the readers in the comments, I've realized several things.

On the one hand we have Occam's Razor: choose only the simplest models for things. On the other hand we know that in order to build intelligence, we need to create a very complex artifact (namely something like a brain), that has to contain lots of memories (parameters). There is an inherent conflict between these two constraints.

Many faces of overfitting

If we have a model too complex for the task we often find it will overfit, since it has the capacity to "remember the training set". But things may not be so obvious in reality. For example there is another, counter intuitive situation where overfitting may hit us: the case where the model is clearly too simple to solve the task we have in mind, but the task as specified by the dataset is actually much simpler than what we had originally thought (and intended).

Let me explain this counterintuitive case with an example (an actual anecdote I heard from Simon Thorpe as far as I remember):

Figure 1. … Read more...

In my career I've encountered researchers in several fields who try to address the (artificial) intelligence problem. What I found though, is that researchers acting within those fields had a vague idea of all the others trying to answer the same question from a different perspective (in fact I had a very faint idea myself initially as well). In addition, following the best tradition of Sarye's law there is often tension and competition between the researchers occupying their niches resulting in violent arguments. I've had the chance to interact with researchers representing pretty much all of the disciplines I'll mention here, and as many of the readers of this blog may be involved in research in one or a few of them, I decided it might be worthwhile to introduce them to each other. Within each community I'll try to explain (at least from my shallow perspective) the core assumption, prevalent methodology, and the possible benefits and drawbacks of the approach as well as a few representative literature/examples (purely subjective choice). My personal view is that the answer to the big AI question cannot be obtained within any of these disciplines, but will eventually be found somewhere between them, and … Read more...

Today we'll step back a bit and consider the psychology of a machine learning researcher when he does his job, a subject which interests me deeply and one that I've already touched in another post.  Some of this comes from my own introspection, as I've been doing machine learning for quite a few years now.

Emails and ML models trigger dopamine

It is a well known fact from biology that little achievements trigger the release of small amounts of dopamine - a neurotransmitter that is believed to be involved in reinforcement learning. The dopamine makes us feel good and also triggers plasticity in certain parts of the brain (likely allowing the brain to "remember" what behaviour lead to the reward). Reinforcement learning however has its issues, since the reward can appear by coincidence and therefore reinforce the "wrong cause". This is very much visible these days with Internet, emails and texts: since receiving an important and rewarding message reinforces the behaviour which lead to it - and that most likely was pressing "get mail" button - we get addicted to checking email! Same applies to social media, texting, and is also the mechanism underlying gambling. In reality rewards … Read more...

There is a widespread belief among the "artificial intelligentsia" that with the advent of deep learning all it takes to conquer some new land (application) is to create a relevant dataset, get a fast GPU and train, train, train. However, as it happens with complex matters, this approach has certain limitations and hidden assumptions. There are at least two important epistemological assumptions:

1. Given big enough sample from some distribution we can approximate it efficiently with a statistical/connectionist model
2. A statistical sample of a phenomenon is enough to automate/reason/predict the phenomenon

Both of these assumptions are not universally correct.

Universal approximation is not really universal

There is a theoretical result known as the universal approximation theorem. In summary it states that any function can be approximated to an arbitrary precision by (at least) three  level composition of real functions, such as e.g. a multilayer perceptron with sigmoidal activation. This is a mathematical statement, but a rather existential one. It does not say if such approximation would be practical or achievable with, say, gradient descent approach. It merely states that such approximation exists. As with many such existential arguments, their applicability to real world is limited. In the real world, we … Read more...

[Note: this post gets updated every once in a while with new pictures]

This is mostly a fun post, though I hope it may trigger some thinking. Since there is much hype about self driving cars, I decided to express myself artistically and draw a bunch of situations that the current technology most likely would not be able to deal with. Some of them might be funny, some might be dangerous. Answer yourself if you want to be driven by an entity which cannot understand these situations and if you are developing a self driving car, let me know in the comments if such situations could indeed be misinterpreted. I'll start with just a few, it takes time to draw them and I'm not a very good artist. Email me if you have ideas for additional situations.

 

 

Artwork 1: Open manhole. Unlike with a regular pothole, driving into open manhole can lead to a disaster. Would a driverless car figure that out? If you consider an open manhole without construction cones an unlikely possibility, take a look at this video.

 

Artwork 2: Stop sign prank. Somebody (bored teenagers?) put tens of stop … Read more...

This post could be considered a continuation of my previous post, "AI and the ludic fallacy."  Bear with me, as this post will make some important yet philosophical points. There are many people who cringe when they hear the p-word –  I have been known to be such a person myself. As with many fields out there, there is a lot of low quality material (namely BS) in philosophy.  However, I have also seen many incredibly insightful philosophy pieces, and it seems to be a helpful discipline whenever one needs to get out of the "box".

This post is about one such box.

An Example Box (as seen from another Box)

A brief digression from AI to illustrate where we're going.  Many successful disciplines such as math and logic operate within a carefully designed box — a set of axioms from which truths are derived. Once a decent set of axioms is established, very elegant and exciting theories can be built. Some of these theories may be successful in modeling physical reality:  classical math built on set theory is so successful that after working within it, one may be tempted to think that reality is the set theory … Read more...