To illustrate the core of this problem, I remade Matts map but used transparent symbols. Click on the image to enlarge.

One could argue wether a map is the ideal visualization of this data set or not, but I will skip this discussion for now. Let's assume that you really want to see the geographical patterns in your data and thus will go for map with circles which area represent crime rates.

What's so bad with overlapping symbols?

In the case of our crime rate map, the main problem is that overlapping circles will be perceived as a connected area. And since area is a cue for crime rate, we connect a large connected area with a high crime rate.

Another problem is that some symbols are completely hidden behind others. In the crime rate map this results in a distorted view on the data since where we're actually looking at the maximum crime rate of a geographic region instead of the average crime rate.

One way to get around these issues is to combine overlapping symbols into a new symbol. Later in this post I will describe this process in more detail. Here's how the cleaned crime rate map looks like.

In other scenarios, a problem of overlapping symbols might be that you don't see how small values would eventually add up to a larger value. A great example is the following map which some of you might well know from Google Analytics. What do you think, from where do the most visitors come from? It's California.

To demonstrate this effect, I mapped the crime statistics again, but this time I used the actual number of murders instead of the rates. Without any cleaning, the map looks like this:

Thanks to the transparency of the symbols you at least get an idea of the data you're probably missing, but still, we are quite bad in mentally adding the areas of overlapping circles. Let's compare this to the cleaned version:

Looks much better, doesn't it? This time, New York wins the race.

The dirty math behind this

Well, actually the math itself isn't that dirty. The basic concept goes like this:

1. Look for intersections between symbols.
2. Combine two intersecting symbols into one.
3. Proceed to 1.

1. Finding intersecting symbols

If you're dealing with circular symbols, the intersection finding is pretty easy. For each pair of circles we need to check if the distance between the center points is smaller than the sum of both radii.

Just make sure to not check a circle against itself . Stop as soon as you found intersecting circles and proceed with the merge step.

2. Merging symbols

For merging of symbols, I prefer keeping references of all original data items (eg cities) in the merged symbol, which is kind of useful for updating the radius. For instances, if you want to merge the murder rates of two cities, you would divide the sum of murder incidents in both cities by the combined population of both cities, which is not the same as simply averaging the murder rates.

The postion of the combined symbol is calculated as the weighted mean of the original positions. This ensures that the merged symbols don't move away too far from it's original center of gravity. The weight itself is defined as:

The new position is then calculated in (lon,lat) coordinates and converted to (x,y) afterwards using your favorite mapping software.

Both steps are repeated until all intersections were removed. You can easily control the granularity of map symbols by changing the radius range for the symbols.

Final thoughts

While this process is trivial from a mathematical/algorithmic point of view, I know that it might be not as easy to implement this using tools like Google Maps. But that's kind of a weak excuse, isn't it? We need better tools for mapping.

To be honest, the first time I saw this map I didn't thought much about it. Ok, poverty is highest in south central of the United States, especially near Mexican border. But recently I used the same data to demonstrate a choropleth map that I created from-scratch and I was really surprised to see a somewhat different picture:

Naturally, I wanted to know where the differences come from and spent some time to investigate. Actually, I think there are two big fails in the Guardian map (which was made using Google Fusion tables).

Don't mess around with your class limits

The values in the poverty data range from 6.6% to 22.7% and the map shows them divided into five classes. If one would compute the exact equidistant class limits between the minimum and maximum value one would come up with the following classes (the gray bar is used to indicate the data range):

I'm not sure if this is the default behaviour of Google Fusion Tables or the editors choice, but the Guardian map used the class limits 6-9%, 9-12%, 12-15%, 15-18% and 18-23%. Due to the round numbers one might think that they are easier to understand than the fractioned numbers above, but this comes at the high price of distorted class distribution:

Note that the fifth class (which shows the poorest states) is blown up while the first class is a bit under-represented. Given the highly political topic, I'd argue that while we're trying to map inequality, we should at least use equally distributed classes.

Don't mess around with your class colors

The second big failure of the map is the choice of colors. This colors were used for the Guardian map:

Obviously there's a large jump between the first and second class and an enormous jump between the fourth and fifth color. The fourth color looks like taken from a completely different gradient and is hardly distinguishable from the third color. Again, I'm not sure if this is some kind of default in Google Fusion tables, but maybe they were just hand-picked.

Instead, in my map I simply used equidistant colors from a HSV gradient:

But, as mentioned in the comments below, even equidistant HSV colors are not the best option. The problem is that humans perception of brightness differs from the arithmetical lightness of HSV colors.

To demonstrate this difference, let's compare the equidistant HSV colors to a hand-picked color scale from colorbrewer2.org:

Quite a different picture, isn't it?

And better think twice about your class count

Another question is why we should use five classes at all. It's kind of interesting to see how "dramatically" the picture changes if one changes the number of classes. Given the fact that we're living in the age of interactive maps that allow us to read data values from tooltips, there's no more reason to be stingy with colors. At least, I think a number of seven classes should be a better trade-of between correctness and color distinguishability.

Be careful when visualizing non-area-related data using choropleth maps

At the end I just want to mention another well-known problem of choropleth maps. The visual significance of a particular geographic region depends on the color value multiplied with the area of that region. Thus, a larger but equally colored region appears more important than a smaller one. Especially when you're dealing with non-area related data, like the poverty of human beings, this might cause additional mis-interpretations. One way to get around this is to use cartograms, which aim to resize geographic regions according to a measurement that has more relevance to the context of the data. For instance, in the next image you can see a Dorling cartogram (where circles represent regions) that sizes the states according to 2010 population. This way, we visually relate the poverty rates to the affected population instead of the affected area.

So, what to do next? For me, the clear answer is that we need better educated map makers and, perhaps more importantly, we need better open source tools for thematic mapping. That's what I'm kind of working on right now..

Update: Jorge Camoes wrote a kind of follow-up post to this one, called The same data, the same map, different stories. Make sure to check it out as well.

In this related article, the creator and Facebook intern Paul Butler explains the process of rendering the map. The most interesting remark he made can be found in the last sentence:

It's not just a pretty picture, it's a reaffirmation of the impact we have in connecting people, even across oceans and borders.

Well, there are some nice things about the Facebook friendships map. I like the colors. I like the density of information. I like the patterns. But there's one big part of the picture missing in the map.

And that's the internet.

After all, Facebook is "just" a website and websites can be accessed through the internet. So, when speaking of "connecting people, even across oceans and borders", we should demonstrate our respect for the internet. To do so, I re-created my old map of the internet:

It shows more than 300k geographic locations of ip adresses, based on the free geo location database by MaxMind. Interestingly, the map shows almost the same "patterns" and complexity that is visible in the Facebook data. Given that, the remaining insight we get of the Facebook map is that Facebook is accessed by people all around the web (except for China).

No big news here.

Sketch names must be at least three characters long

Surprise, surprise, I'm a geek, and as such, I usually prefer shorter file names. So, yesterday I called the first of a planned series of sketches "p0". I started coding a bit and then, after running my sketch, I got this handy little error message:

java.lang.IllegalArgumentException: Prefix string too short
     at java.io.File.createTempFile(File.java:1782)
     at java.io.File.createTempFile(File.java:1828)
     at processing.app.Base.createTempFolder(Base.java:1564)
     at processing.app.Sketch.makeTempFolder(Sketch.java:1112)
     at processing.mode.java.JavaBuild.build(JavaBuild.java:135)
     at processing.mode.java.JavaMode.handleRun(JavaMode.java:176)
     at processing.mode.java.JavaEditor$20.run(JavaEditor.java:481)
     at java.lang.Thread.run(Thread.java:680)

After googling around (the first time), I found this thread in the Processing forum. Someone explained the behaviour as follows.

I noticed that if you use a short name for a script (i.e. "p3"), it won't run.

Surprisingly, his example used almost the same sketch name, which clearly identifies him as a geek as well. I know, this clearly looks like a bug and I instantly filed a bug report.

No built in function for reading CSV files

Next thing I want to do is to read in a CSV file, which is almost always the first step of visualizing not-random data. Unfortunately I remembered that there was no built-in function for doing this in Processing. For instance, in R all you need to type is:

data = read.csv('data.csv')

In Python, opening a csv file is a two-liner.

import csv
data = csv.reader(open('data.csv', 'r'), dialect='excel-tab')

But in Processing you have to do the parsing yourself:

String [] lines = loadStrings("data.csv");
int i = 0;
for (String l : lines) {
   if (i++ == 0) {
       // special treatment for headers
       continue;
   }
   String[] cols = split(l, '\t');
   // do something with the data
}

Separating the data parsing and the data processing is not easily possible. I know that Ben Fry wrote a class Table that handles CSV import, but what I don't understand is why this is not a core functionality of Processing. I mean, Ben Fry actually created Processing, he simply could have included this. The same is true for XML, JSON and many other data formats.

Charset handling while opening files

Now things get even worse. Actually, I didn't managed to parse the CSV file because loadStrings didn't manage to read the file. Instead I got this neat error message (at least without the useless Java stacktrace):

The file "data.csv" is missing or inaccessible, make
sure the URL is valid or that the file has been added
to your sketch and is readable.

I checked that the file is located in the data folder inside my sketch root and I even dragged it in the sketch window to make sure the IDE didn't miss anything. I reduced the code to the loadStrings() call, but nothing works. Googling this error (again) leads me to this thread in the Processing forum. Somebody mentioned that this could have something to do with charset issues and suggests the following:

String[] lines = loadStrings("data.txt", "ISO-8859-1");

As I'm used to deal with different charsets in Python (2.7), I'm kind of used to this and was happy to discover the charset parameter. But, unfortunately this doesn't work (anymore) in Processing 1.5.

The method loadStrings(String) in the type PApplet is not
applicable for the arguments (String, String)

Ok, looks like somebody removed the second parameter. Later in the same thread, Ben Fry explained the current behaviour.

No, *please* read revisions.txt for changes. All files are now treated as UTF-8 by default, to deal with this issue.

I admit that I haven't read the revisions.txt for a while, but I finally looked up the current version for the term "loadStrings", but no match. Guess I have to read it in more detail, sometimes later. Now I want to read in the file. I looked up the encoding and found out that my text editor stored the file as Western (Mac OS Roman), which, for some strange reasons, seems to be the default behaviour of the editor. Converting the file to Unicode finally fixed the bug.

One could argue that treating all files as UTF-8 by default is not as smart as allowing the user to specify the encoding of their files. But the crucial point about this error is the error message. A friendly message like this would have saved me lots of time and nerves.

There was an charset error while reading "data.csv".
Please check that the file is encoded in UTF-8 – and
 *please* read the revisions.txt!

I hope you don't understand this post as a pure rant. Instead I just wanted to document an user experience, through which probably many users will run. Still, I love Processing and all the things that it makes possible. And I will continue using it, although I really hate Java. This is maybe the worst design decisions of all: basing an amazing project like Processing on a language like Java.

Update: I just read that Processing 2.0 will include built-in support for JSON and Tables, which is pretty cool.

Mistake No 1: Not showing the 100 people

In the first part of the graphics the author wanted to visualize that 13 out of 100 Italian are living in poverty. Unfortunately they only show the 13 poor people (red) surrounded by about 6 to 7 non-poor people (gray). This in fact visually implies a much higher poverty rate of about 65%.

The corrected version of this ISOTYPE-kind of chart would look like this:

Mistake No 2: Mapping data values directly to circle radii

This is a very common mistake whenever circles are involved in data visualization. It's been described several times (also by myself), and while there is still some discussion about the best way to map numbers to circle radii (square root vs. perceptual scaling), it is widely accepted that one cannot map them 1:1. This is exactly what they done in the infographic.

Mistake No 3: Mixing several trends into one graphic

The last mistake that I want to point out is that the authors seems to mix up several unrelated trends. As said above, there is a strong difference between the poor south and the rich north of italy. Let's focus now on the age-pie of the circle.

I created a horizontal bar version of it to make the numbers more readable for now. The chart shows percentages, but I asked myself "percentage of what?". If we sum up the values of the South we end up with about 105%, summing up the values for the north results in something below 20%. So it can't be the percentage of the total population living in poverty in that particular region. Seems like they show the percentages of the people living in poverty in Italy. This way they repeatedly mixing in the fact that they are more poor people in the South than they are in the North etc. This makes it impossible to compare the age profile between the different regions. In fact the age profile is quite the same for all regions.

The same error occurs for the other pies as well. So, to give a résumé: The infographic looks very nice but is also hard to read and full of mistakes. There's lot of this stuff out there in these days, way too much to critize them all. Maybe nobody cares..

Fangen wir mal mit den Schweinen an. Relativ normale Entwicklung bis 2009 zu etwas über 55 Mio Schweinen pro Jahr, aber dann in 2010 der Absturz. Hat das vielleicht etwas mit der Schweinegrippe-Pandemie von 2009 zu tun? Eher nicht, denn dann müssten die Schlachtungen schon 2009 zurückgehen. Außerdem gab es keinerlei Ansteckungsgefahr durch den Verzehr von Schweinefleisch. Woher kam dieser starke Rückgang aber dann?

Weiter zu den Schafen, auch hier zeigt sich ein ähnlicher Absturz, diesmal allerdings schon ein Jahr früher. So geht die Anzahl der geschlachteten Tiere von etwa 1,15 Mio. Tieren in 2008 plötzlich auf 200.000 Tiere in 2009 zurück. 2010 wurden laut dem statistischen Bundesamt sogar nur noch 28.000 Schafe geschlachtet. Aber wieso? Und was haben die Deutschen stattdessen gegessen?

Zumindenst letztere Frage lässt sich anhand der seltsamen Daten beantworten, denn 2010 gibt es einen sprunghaften Anstieg der Jungrinderschlachtungen auf saftige 58 Mio (Vorjahr: 31.000, davor konstant bei null). Also Jungrinder statt Schweine? Aber was haben die Deutschen 2009 gegessen?

Auch das ist leicht zu beantworten: Lämmer. Denn bei den Schafschlachtungen gibt es genau für das Jahr 2009 den sprunghaften Anstieg. Den Sprung von 0 Lämmern in 2008 zu über 800.000 Lämmern in 2009 kann ich mir eigentlich nur dadurch erklären, dass Lämmer vor 2009 einfach zu den Schafen dazugerechnet wurden. Was bleibt ist der seltsame Abfall in 2010 (denn den gibts ja bei Schafen und Lämmern).

Machen wir mal weiter mit den Pferden. Ich habe noch nie in meinem Leben Pferdefleisch gegessen, aber scheinbar war das im letzten Jahr der absolute Renner.  Trotz beinahe stetig sinkenden Schlachtungszahlen von 1993 bis 2009 springt die Zahl der geschlachteten Pferde 2010 auf fast 900.000 an, also den hundertfachen Wert der Vorjahre. Es wird immer seltsamer..

Kommen wir abschließend zur letzten Anomalie. Diesmal geht es um die Ziegen, deren Schlachtungszahlen in den letzten Jahren einem relativ kontinuierlichen Aufwärtstrend folgen. Die Anomalie steckt hier in der Schlachtmenge in Tonnen. Diese Schlachtmenge betrug vor 2010  etwa 18kg pro Ziege. In 2010 hat sich dieser Wert allerdings verzehnfacht, so daß man nun scheinbar im Schnitt 168 kg Schlachtgewicht aus einer Ziege bekommt. Was ist passiert? Sind die Ziegen auf einmal fetter geworden? Wohl kaum..

Was ich eher glaube ist, dass das statistische Bundesamt hier einfach totalen Murx veröffentlicht hat. Zwar sind die Zahlen für 2010 als "vorläufige Zahlen" gekennzeichnet, allerdings verstehe ich nicht, wieso die Werte mal in den Keller sinken (wie bei den Schweinen und Schafen) und mal in die Höhe schnellen (wie bei Pferden und Jungrindern). Auch die seltsame Entwicklung der Schlachtmenge je Ziege kann eigentlich nur ein Fehler in den Daten sein, fast so, als ob die Zahlen mit Hundert multipliziert wurden..

Ich frage mich, ob sich die Zahlen eigentlich auch mal jemand vom statistischen Bundesamt ansieht?

Visualization of world-wide internet access taken from the current german "AI Journal"

Thanks to the Mercator projection, the European countries are about twice as large as there are in reality, Iceland appears as big as Spain and Alaska as big as Australia. As it's impossible to map the surface of a sphere to a plane without distortions, every map projection has to deal with some kind of errors. But there is at least one mistake that cannot be tolerated. Can anybody see where Greenland has gone? I mean, it's quite a large country so it's not easy to forget. I think they simply left it out. Who cares about the 57,000 people living there? Who cares about all the people seeing this map in the journal? Who cares about visualization integrity? At least not the editorial staff from the German AI Journal.

So, I want to apologize to all people living in Greenland by completing this post with a map of Greenland in it's real size and proportion. I know you are there!

The geographic position and size of Greenland