Inside of a nebula

I'm taking a break from AI in this short post, it's time for something more general about the universe [see the last post in this category "what if we had a warp drive"].

In our daily activities we may not notice how lucky we are - we can see the sky. I mean the deep sky, even far beyond our Galaxy. And by looking at those things, we can learn that the Universe is expanding, that there are quasars, active galaxies, large scale cosmic structures, galaxy clusters, cosmic background radiation and many other marvels. We treat all that as obvious.

But imagine the Sun along with the solar system was trapped inside one of the dense nebulas, which there are countless numbers of in our Galaxy. Say we were trapped somewhere deep inside the Orion nebula.  All we would see in the night sky would be the faint pink glow of hydrogen and maybe a few blurred stars shining through the fog.

And best of all, since the nebula is many, many light years across, we could do nothing to see beyond it. Absolutely nothing. Discovering anything about the outside universe would require sending a probe light years … Read more...

The complexity of simplicity - balancing on the Occam's razor

While rereading my recent post [the meta-parameter slot machine], as well as a few papers suggested by the readers in the comments, I've realized several things.

On the one hand we have Occam's Razor: choose only the simplest models for things. On the other hand we know that in order to build intelligence, we need to create a very complex artifact (namely something like a brain), that has to contain lots of memories (parameters). There is an inherent conflict between these two constraints.

Many faces of overfitting

If we have a model too complex for the task we often find it will overfit, since it has the capacity to "remember the training set". But things may not be so obvious in reality. For example there is another, counter intuitive situation where overfitting may hit us: the case where the model is clearly too simple to solve the task we have in mind, but the task as specified by the dataset is actually much simpler than what we had originally thought (and intended).

Let me explain this counterintuitive case with an example (an actual anecdote I heard from Simon Thorpe as far as I remember):

Figure 1.

Who will figure out intelligence?

In my career I've encountered researchers in several fields who try to address the (artificial) intelligence problem. What I found though, is that researchers acting within those fields had a vague idea of all the others trying to answer the same question from a different perspective (in fact I had a very faint idea myself initially as well). In addition, following the best tradition of Sarye's law there is often tension and competition between the researchers occupying their niches resulting in violent arguments. I've had the chance to interact with researchers representing pretty much all of the disciplines I'll mention here, and as many of the readers of this blog may be involved in research in one or a few of them, I decided it might be worthwhile to introduce them to each other. Within each community I'll try to explain (at least from my shallow perspective) the core assumption, prevalent methodology, and the possible benefits and drawbacks of the approach as well as a few representative literature/examples (purely subjective choice). My personal view is that the answer to the big AI question cannot be obtained within any of these disciplines, but will eventually be found somewhere between them, and … Read more...

The meta-parameter slot machine

Today we'll step back a bit and consider the psychology of a machine learning researcher when he does his job, a subject which interests me deeply and one that I've already touched in another post.  Some of this comes from my own introspection, as I've been doing machine learning for quite a few years now.

Emails and ML models trigger dopamine

It is a well known fact from biology that little achievements trigger the release of small amounts of dopamine - a neurotransmitter that is believed to be involved in reinforcement learning. The dopamine makes us feel good and also triggers plasticity in certain parts of the brain (likely allowing the brain to "remember" what behaviour lead to the reward). Reinforcement learning however has its issues, since the reward can appear by coincidence and therefore reinforce the "wrong cause". This is very much visible these days with Internet, emails and texts: since receiving an important and rewarding message reinforces the behaviour which lead to it - and that most likely was pressing "get mail" button - we get addicted to checking email! Same applies to social media, texting, and is also the mechanism underlying gambling. In reality rewards … Read more...

Give me a dataset to train, and I shall move the world

There is a widespread belief among the "artificial intelligentsia" that with the advent of deep learning all it takes to conquer some new land (application) is to create a relevant dataset, get a fast GPU and train, train, train. However, as it happens with complex matters, this approach has certain limitations and hidden assumptions. There are at least two important epistemological assumptions:

  1. Given big enough sample from some distribution we can approximate it efficiently with a statistical/connectionist model
  2. A statistical sample of a phenomenon is enough to automate/reason/predict the phenomenon

Both of these assumptions are not universally correct.

Universal approximation is not really universal

There is a theoretical result known as the universal approximation theorem. In summary it states that any function can be approximated to an arbitrary precision by (at least) three  level composition of real functions, such as e.g. a multilayer perceptron with sigmoidal activation. This is a mathematical statement, but a rather existential one. It does not say if such approximation would be practical or achievable with, say, gradient descent approach. It merely states that such approximation exists. As with many such existential arguments, their applicability to real world is limited. In the real world, we … Read more...

Self made time capsule, part 2.

In my previous post I described the hardware components of my self made time capsule/home server. It consisted of the Intel NUC micro-PC, Netgear managed 1GBps switch and Edimax 802.1ac access point. Here I'll go over the basic config, necessary to achieve the functionality I've mentioned.


I'm using ubuntu 16.04 LTS (Long Term Support). It is a very decent Debian based distribution and works very well on the Intel NUC. In this post I'll assume that the Linux is already installed and all the hardware components are detected by the kernel (I had no issues whatsoever, it worked out of the box). The only issue that may perhaps be a problem on the NUC is when you have secure boot enabled in the BIOS, which should be disabled before you install Linux. Also make sure the boot sequence in the BIOS makes sense. After you install linux, it's a good idea to update to make sure all the installed packages are the latest.

Before we begin the setup it is good to install a few essentials before we screw up our Internet connection:

apt-get install openssh-server
apt-get install git
apt-get install vim
apt-get install dnsmasq
apt-get install vlan



Self made time capsule, part 1

It will not be about AI this time, neither will it be about Sci-fi. It will actually be exactly about what the title indicates. So let's begin.

Since a certain incident in the late 90's involving a 850MB drive I'm quite paranoid about having backup. For many years this paranoia was satisfied with Apple Time Capsule - a handy device that acts as a wifi/router and a network attached storage, which through afp protocol offers time machine service to Mac computers. I have one back in Poland and I had one here in California, until one day in January 2017 all of a sudden the device died. I had this device since 2010 so it served me well for quite a few years (I upgraded the drive to a 3TB in the meanwhile), but still the death was surprising and disappointing.

But what was even more disappointing, was to see the current Apple's offering in that segment.  As mentioned I bough my (back then 1.5TB) capsule in 2010, now it is 2017 and Apple offers... a 2TB Capsule for $299 and a 3TB Capsule for $399. This is ridiculous!

Ultimately, I decided to build one myself, and I'm very happy … Read more...

PVM on the GPU (dev update)

I've mentioned several times that the Predictive Vision Model (PVM) is not expressible in any of the current deep learning frameworks such as TensorFlow or Caffe (not easily or for that matter efficiently at least). This is due to the inherent feedback and multi scale structure. PVM is not an end to end trained system, it is a collection of intertwined sequence learners. That being said, I'm currently working in my free time to bring PVM to the GPU.

I'm not the most experienced person in the GPU programming domain, but I can definitely write a kernel and use Nvidia profiler. So far my results look very encouraging: I can train more than 210 million float32 parameters at 21 fps with Nvidia Titan X based on the Pascal architecture. In other words that is 4.4 billion trained float32 parameters per second. This training performance matches that of deep learning models, where e.g. RESNET-50 with ~25 million parameters can be trained at approximately 100-150 samples/s (single GPU). In fact my GPU utilisation is now close to 97% with most kernels.  To some degree I feel that PVM will be even better suited for GPU implementation than end-to-end deep learning because of … Read more...

What would an autonomous car do?

[Note: this post gets updated every once in a while with new pictures]

This is mostly a fun post, though I hope it may trigger some thinking. Since there is much hype about self driving cars, I decided to express myself artistically and draw a bunch of situations that the current technology most likely would not be able to deal with. Some of them might be funny, some might be dangerous. Answer yourself if you want to be driven by an entity which cannot understand these situations and if you are developing a self driving car, let me know in the comments if such situations could indeed be misinterpreted. I'll start with just a few, it takes time to draw them and I'm not a very good artist. Email me if you have ideas for additional situations.



Artwork 1: Open manhole. Unlike with a regular pothole, driving into open manhole can lead to a disaster. Would a driverless car figure that out? If you consider an open manhole without construction cones an unlikely possibility, take a look at this video.


Artwork 2: Stop sign prank. Somebody (bored teenagers?) put tens of stop Read more...

Outside the box

This post could be considered a continuation of my previous post, "AI and the ludic fallacy."  Bear with me, as this post will make some important yet philosophical points. There are many people who cringe when they hear the p-word –  I have been known to be such a person myself. As with many fields out there, there is a lot of low quality material (namely BS) in philosophy.  However, I have also seen many incredibly insightful philosophy pieces, and it seems to be a helpful discipline whenever one needs to get out of the "box".

This post is about one such box.

An Example Box (as seen from another Box)

A brief digression from AI to illustrate where we're going.  Many successful disciplines such as math and logic operate within a carefully designed box — a set of axioms from which truths are derived. Once a decent set of axioms is established, very elegant and exciting theories can be built. Some of these theories may be successful in modeling physical reality:  classical math built on set theory is so successful that after working within it, one may be tempted to think that reality is the set theory … Read more...

Can a deep net see a cat?

In this post I will explore the capabilities of contemporary deep learning models on the vitally important task of detecting a cat. Not an ordinary cat though, but a sketch of an abstract cat. This task matters because success tells us something about whether a visual system has learned generalization and abstraction  -- at least on par with a 2-year old. This post is inspired by my ex co-worker Peter O'Connor who tried similar experiments on LeNet several years ago. In addition, this post is a continuation of this blog's highly popular "Just how close are we to solving vision?" which to-date has amassed nearly 15,000 hits. Let's begin by introducing my menagerie:

Figure 1. The cat menagerie. From left to right (top to bottom): "abstract cat", rough sketch of a real cat, less rough sketch of a real cat, the "best cat" I could draw, "best cat" inverted.

I made these sketches myself, based on a photo of a cat. NOTE: Whenever you test a deep net (or any other machine learning model), always use new data. Anything you find on the Internet is either already in the training set or soon will be.

VGG 16


Scaling up AI

Caution: due to a large number of animations this post may take a while to load (depending on your connection speed), please be patient and don't reload unless necessary. The animations will likely load before you read the text.

Scalability in Machine Learning

Scalability is a word with many meanings and can be confusing, particularly when applied to machine learning. For me the meaning of scalability is the answer to this question:

Can an instance of the algorithm be practically scaled to larger/parallel hardware and achieve better results in approximately the same (physical) time?

That is different from the typical understanding of data parallelism, in which case multiple instances of an algorithm are deployed in parallel to process chunks of data simultaneously. An example of scalability of instance (definition above) is for example computational fluid dynamics (CFD). Aside from the need to obtain better initial conditions, one can run the fluid dynamics on a finer grid and achieve better (more accurate) results. Obviously it requires more compute, but generally the increase in complexity can be offset by adding more processors (there are some subtleties related to Amdahl's law and synchronisation). For that reason, most of the world's giant supercomputers are … Read more...

Recurrent dreams and filling in

Caution: due to a large number of animations, fair amount of traffic and the tiny size of my web hosting machine, this post may take a while to load, please be patient and don't reload unless necessary.

There has recently been a fair amount of deep learning work on video prediction and generative models that focuses on infusing motion into static pictures. One such paper is e.g. available here:

The approach taken in that  paper was to train a model on a huge amount of data and explicitly separate the task of prediction into (1) the generation of static background and (2) a moving object. As much as this work is impressive, the separation into background and foreground prediction seems a bit unnatural. Given however the nice mesmerising quality of video (and the importance of prediction) I decided to play a little bit with our Predictive Vision Model (PVM) which is also capable of generating such "dreams". For the sake of this post I only trained a very small instance of PVM on a single relatively short video, so the results shown here are mainly illustrative and this is by no means a full blown scientific study.… Read more...

Learning physics is the way to go

In many of my posts I'm directly or indirectly postulating learning of the physics as a way to create a "real AI". The point I'm trying to makes is so obvious, that it actually is not obvious at all and it took me some time to realise it. As with many such obvious/non-obvious things it takes multiple angles before the essence could be captured, hence why I write this blog. I'm trying to express myself in many ways until I hit the explanation that everyone just simply gets. So let me try again in this post:


The world around us is complex. Everything to some degree interacts with everything else, there are lots of regularities but there is also a fair amount of chaos. No two trees look identical, yet we manage to categorise them. In terms of physical language it appears that a good chunk of our reality is a "mixing system" at the "edge of chaos" (or otherwise critical). We therefore cannot very well predict what will happen. Yet I'm postulating prediction as a training paradigm, does this make any sense?

It does and here is why: even with the chaotic world, there are numerous aspects of … Read more...

More thoughts on the self driving car

Recently Tesla had shown a teaser video of their "self driving car" project which immediately drew media attention and swarms of self driving "enthusiasts" to again announce that this is a done deal already (which it is not). Here is the video in question:

Note: above video has been subsequently taken down, I'm now linking to a mirror.

Now this looks very impressive as a demo but there are a few details I'd like to point out before we start saying again that the self driving car is a done deal from technological point of view. Disclaimer: I do like Tesla and I think some of their ideas are great, but their self driving seems a bit premature, somewhat over promised and over hyped.

  • The lighting conditions in a video are perfect from computer vision point of view. Although it is a bit foggy, the illumination is uniform and diffused. There are no hard shadows, flares or ghosts.
  • The lane markings are all clearly painted and visible everywhere.
  • There are no "unusual situations" (see below what I mean by that).

Just a reminder that a self driving car was demoed as a research project in mid 80's at CMU … Read more...