David Johnstone

Real estate data visualisation for Victoria

February 20, 2014

For a long time I’ve been thinking about making a tool to help you work out where to live when you don’t know the area well. Put another way, I’ve long thought that real estate websites could do a better job with data visualisation. Anyway, I decided to spend a little bit of time working on this problem, and I now have something to show for it.

Here’s the tool. It shows median sale or rental price, annual growth or rental yield for a variety of property types (all houses, or two, three or four bedroom houses, or all units, or one, two or three bedroom units) in Victoria. It shows all this data on a per-town or per-suburb basis on a colour coded map. In the picture above, the median sale price of all houses is shown, and it is obvious that the inner eastern and south-eastern suburbs are the expensive ones.

There tend to be outliers that mean the prices that most localities are in is compressed to a narrower range of the colour scale, so it’s often helpful to change the minimum and maximum values of the colour scale to see the differences more clearly. In the image above, the minimum was set to $250k and maximum to $1.8M, although there are two suburbs with a medium house price above that.

All the data for this comes straight from realestate.com.au and is current as of now (February 2014). On the tech side of things, I decided to use Bootstrap for once. It works well. chroma.js is used with the colours (the actual colour scale is from ColorBrewer).

On O’Grady and cycling’s dark history

July 27, 2013

It is now widely recognised that performance enhancing drugs were widely used by professional cyclists in the 90s and early 2000s. It seems reasonable to believe that the vast majority of riders of note from this time used EPO or other methods of illegally enhancing their performance. Therefore, nobody should be surprised when it is revealed that a rider who was active in this period used PEDs.

Stuart O’Grady — who has won Tour stages and Paris–Roubaix, worn the yellow jersey and won Olympic gold — is the latest in a long line of “revelations” over the past few years. I am surprised that the reaction against him has been as negative as it is, although based on the comments on Cycling Tips, this negativity has more to do with the timing (just after he retired a year earlier than expected, and at the same time the French Senate report is released that reveals that he provided a “suspicious” sample in 1998), the convenience of his admission (having nobody else involved and only doing it once is the smallest admission possible, and many don’t believe it) and hypocrisy (he has previously criticised Lance Armstrong), rather than the bare fact that he used EPO.

However, I think it is quite unfair to O’Grady and the rest to demonise them for what they have done. I find it very hard to believe that the people involved in cycling back then were any more “evil” than any group of people you’ll find today. The problem was that doping was collectively seen as acceptable (and even necessary), and this made it very hard to resist. Most people today, if they happened to be professional cyclists in 1998, would have doped, because back then, most of them did. If cycling really has turned over a new leaf and today’s peloton is largely clean (and I hope it is), it’s not because today’s cyclists are better people, but that the culture of cycling has changed to make doping far less acceptable.

For this reason, those who resisted drugs back then should be most applauded — I don’t know how many there are, but surely there are some who gave up promising cycling careers for this reason, and others who still competed, but never at the same level. The corollary of this is that I’m very hesitant to harshly criticise… continue reading

Netflix, Amazon and Hulu compared for streaming movies

July 23, 2013

Paying a small price each month to get unlimited access to watch as many movies as you want sounds like a good deal for those who watch lots of movies. Netflix, Amazon Prime and Hulu Plus all have libraries of thousands of movies and TV shows that customers can stream. Here I look at how good the selection they offer actually is.

First up, the IMDB top 250. These aren’t necessarily the greatest movies ever made, but it is a good selection of the most loved and most important films of all time.

The following table shows the availability of each movie through Netflix (streaming), Amazon Prime, Hulu Plus, Crackle, YouTube and Epix according to Can I Stream It?. The final column shows the availability on any of the streaming services. All data was collected in July 2013.


Note: There seem to be more movies on YouTube than this indicates. Other old movies that are in the public domain like Nosferatu can be found, and even more recent movies like The Good, The Bad and The Ugly are there. I’m not sure what copyright law thinks about this though.

We have slightly more than a quarter of these movies available through these streaming services. Netflix comes out the best, with 15% of the titles available. But, to be fair, a lot of these movies are old or obscure or foreign and don’t appeal to most people.

Let’s look at some more recent movies that received critical acclaim. Namely, let’s look at the 58 films that were nominated for Oscars in 2010. These movies aren’t brand new now, and there’s a lot of big movies like Avatar and Up in this selection.


If you want to watch foreign films or documentaries you might be in luck, but other than that, they’re all pretty useless. None of the movies nominated for Best Motion Picture of the Year, Best Performance by an Actor in a Leading Role, Best Performance by an Actress in a Leading Role, Best Performance by an Actress in a Supporting Role or Best Achievement in Directing are available for streaming. There are a handful of unavailable obscure short films and documentaries that might… continue reading

Alt is a simpler and prettier interface to NewsBlur

June 25, 2013

Like many people, I had a go at creating a feed reader to fill the gap that Google Reader is about to leave. However, rather than building yet another feed reader backend, I have made use of the NewsBlur API and created an alternative interface to NewsBlur, called Alt.

I built this mostly to cater for how I use a feed reader, so it wouldn’t hurt to explain what this is: I get forty or so new items a day between the feeds I follow. I normally check my reader a couple of times a day, and all I ever do (with the very occasional exception) is read through the unread items. Therefore, I’ve created an interface that lets me do this in the simplest way possible.

Upon logging in with your NewsBlur account, you get a list of items that automatically loads more as you scroll down, much like how Google Reader works. If you have a high enough screen, it looks something like this:

As new items are seen (i.e., get to the top of the visible page), they are automatically marked as “read”. They can be “unread” by clicking on the eye icon up the top, next to the date. Keyboard navigation for next/previous is supported with j/k. There is a counter up the top of the page that indicates how many unread items are remaining. Clicking on this brings up the list of subscriptions, and from here it’s possible to view any individual feed or group of feeds (click on a folder name to show all items from that folder, or click on the folder icon to expand the folder). There are controls at the top of the page to show all items (rather than unread), sort by oldest first (rather than newest), and show only headlines (rather than full items). That’s basically all there is to it.

If you’re interested in using it, go to altfeedreader.com.

For what I want, there are many advantages and few disadvantages to only creating a frontend, rather than an entire system. I don’t have complex needs, so I don’t need the backend to do any more than what most backends already do — there’s no point reinventing the wheel if you want it to look the same as the old wheel. Furthermore, by not creating and maintaining a… continue reading

Be careful when buying a wide gamut monitor

June 7, 2013

Compared to normal monitors, wide gamut monitors can display a wider range of colours; you get redder reds, greener greens and bluer blues. They’re useful for some people, because the real world isn’t limited to the colours that your computer monitor can display (and neither are printers).

However, for the majority of people who aren’t into calibrating their monitor with hardware calibrators and worrying about colour profiles when printing on $2000 printers, they have a major downside — the web.

If you have a wide gamut monitor and don’t adjust anything, the colours you see will be stretched to fill the wider colour space that your monitor supports. This might sound okay, but there are problems:

  • It can be uncomfortable to view a page with a lot of bright colours — you might often be thinking the designers should have toned it down a bit, but in reality it’s your monitor that’s toning it up and making it much more vivid.
  • You’re not seeing pages as the designers intended — they were probably working with a normal colour space (sRGB), and stretching the colours to a wider colour space can make things that were aesthetically pleasing to clash.
  • Photos look wrong — the photographer might have a proper colour managed workflow and produces photos that are “just right” on his screen and any other screen that is correctly calibrated. Then you put it in a browser which stretches the colours to look quite different to what was intended.
  • Support in browsers for proper colour management is limited, so even if you know what you’re doing, there will be problems.

On Windows, Chrome and IE ignore any colour profiles you might have, and stretch colours in the browser to the full range that your monitor supports, so a rgb(255, 0, 0) red is the reddest red that your monitor can display. The only exception to this is images in Chrome that have embedded colour profiles; Chrome honours these and displays them correctly.

Firefox, my browser of choice, is better, and has a gfx.color_management.mode option that can be set to 1 to enable full colour management. This is almost ideal, except it doesn’t work perfectly — there are some bugs that cause some things to not be colour managed. Take a look at this:

These nine squares are made red in nine different ways (if… continue reading

On the choice of a wife, by John Aiken

September 12, 2012

I was reading a book on the history of marriage, and it mentioned a handful of essays that were written in the US in the late eighteen century on the subject of marriage. Wondering what they had to say about the matter, I turned to Google and found one of them. What follows is the text to “On the choice of a wife” by John Aiken, which forms a chapter in his book, Letters from a father to his son, on various topics, relative to literature and the conduct of life : written in the years 1792 and 1793. It only seems to be available online in this scanned copy of the book or on Google Books. In the course of searching for this, I also discovered H. G. Wells’ On the choice of a wife, which is a much shorter and more amusing document. Anyway, without further ado, here is some late eighteenth century advice on what to look for in a wife.

Dear son,

There is no species of advice which seems to come with more peculiar propriety from parents to children, than that which respects the marriage state; for it is a matter in which the first must have acquired some experience, and the last cannot. At the same time, it is found to be that in which advice produces the least effect. For this, various causes may be assigned; of which, no doubt, the principal is, that passion commonly takes this affair under its management, and excludes reason from her share of the deliberation. I am inclined to think, however, that the neglect with which admonitions on this head are treated, is not unfrequently owing to the manner in which they are given, which is often too general, too formal, and with too little accommodation to the feelings of young persons. If, in descanting a little upon this subject, I can avoid these errors, I flatter myself you are capable of bestowing some unforced attention to what an affectionate desire of promoting your happiness, in so essential a point, may prompt.

The difference of opinion between sons and fathers in the matrimonial choice may be stated in a single position—that the former have in their minds the first month of marriage, the latter, the whole of its duration. Perhaps you will, and with justice, deny that this is the difference between us two, and will assert…

continue reading

Drivers vs. cyclists: the war that should not be

January 20, 2012

The furore over bikes on the road that Shane Warne instigated has not yet gone away. The main development since my last post on mandatory registration and number plates has been the emergence of the account of the cyclist himself and eyewitness reports. It sounds like Warne wasn’t close to being as innocent as he made himself out to be, but this story has become bigger than Warne’s altercation for it has reignited in the media the long-running tension between drivers and cyclists. Bridie O’Donnell — a professional cyclist — wrote a very sensible response to Warne, Miranda Devine wrote an inflammatory piece about the “silent two-wheeled menaces on our road”, Crikey responded to Devine, various cycling groups put out press releases, and thousands more discussed these things online.

Today I want to talk about the way we divide each other into groups. It is a known quirk of humans that we easily form groups and then see other groups as wrong and inferior, just because they are not us. We label them and stereotype them, and sometime after groups are formed and opposition has begun we create logical reasons to support that opposition. This group identification behaviour isn’t completely bad — without it we wouldn’t have sports teams, and more importantly, it makes an incredibly complex world possible to understand — but especially in the connected and pluralistic world we live in today, it leads to all sorts of problems since we are very exposed to “others”. There are countless examples of this grouping behaviour. Many of them are innocuous, such as Android versus iPhone, Ford versus Holden, Star Trek versus Star Wars, road cycling versus mountain biking etc., but others are more serious — think of all the ethnic strife around the world, and racism more generally. Consider also the typical attitudes we have towards four-wheel drive owners, rich people, poor people, accountants, people who live in Frankston… This is not to say that there are no other reasons for these conflicts, but the way we group people together underlies many things.

This behaviour also explains much of the controversy over bikes on the road. There are large sections of the community who strongly dislike cyclists, mostly because everybody around them also dislikes cyclists. There are social commentators like Miranda Devine who are quite comfortable labeling all cyclists as menaces. There are public figures like Shane Warne who… continue reading

Shane Warne vs. cyclists: registration and number plates

January 18, 2012

Shane Warne has managed to get himself embroiled in controversy again. This time it’s with cyclists. Firstly he wrote a series of tweets last week where he called for cyclists to ride in single file, show number plates, and pay registration. Unsurprisingly there were a lot of responses, some of which he retweeted. The classiest of them wasn’t from the wife of a certain famous Australian cricketer suggesting that he “just throw a cricket ball at them if they really annoy u”.

Today he had an altercation with a cyclist which he then tweeted about. It’s not clear what happened exactly (and trying to reconstruct the incident from only one perspective is bound to give an incomplete story), but it sounds as though a cyclist hit his car (as in a punch, not a crash), held onto his car, hit his car again, exchanged verbal abuse with him, stopped in front of him and held up traffic, and finally Warne slowly drove around the cyclist and went on his way. Warne also mentioned that he reported the incident to police and finished by restating his belief that cyclists should pay registration fees and have number plates. The thing I wonder the most about this story is why the cyclist hit Warne’s car: I’ve never heard of a cyclist hitting a car for no reason — such reactions tend to happen because the cyclist barely escaped a trip to hospital moments before, even if the driver was oblivious to the situation. But that’s not the point of this post… Update: here is the cyclist’s account of the incident.

In the discussions that have followed, it is clear that there are a lot of people who don’t understand vehicle registration fees and how they relate to road funding. This is demonstrated by both the common “cyclists don’t pay registration fees to be on the roads, therefore they aren’t entitled to use the roads” and the frequent response “the vast majority of cyclists also drive and therefore do pay”. Let’s look at some numbers.

In 2006–07 $16.2B was collected directly from motor vehicle users. Only $3.9B (24%) of that came from vehicle registration fees. The majority (56%, or $9.2B) came from fuel excise, and the remaining 20% came from a combination of stamp duty, driver’s license fees and… continue reading

Joshua’s long day

December 22, 2011

Joshua 10:12–14 has long been understood to be referring to a day significantly longer than normal. Here, I’m going to explain an interpretation that doesn’t involve any any miraculous physics. Here’s the passage in question:

At that time Joshua spoke to the Lord in the day when the Lord gave the Amorites over to the sons of Israel, and he said in the sight of Israel,

“Sun, stand still at Gibeon,
and moon, in the Valley of Aijalon.”
And the sun stood still, and the moon stopped,
until the nation took vengeance on their enemies.

Is this not written in the Book of Jashar? The sun stopped in the midst of heaven and did not hurry to set for about a whole day. There has been no day like it before or since, when the Lord heeded the voice of a man, for the Lord fought for Israel.

As far as miracles in the Bible go, Joshua’s long day is one of the biggest. Since the sun and moon moving across the sky are caused by the earth’s rotation, stopping them in the sky requires the earth to stop rotating, and if you’ve ever been in a decelerating car, you’d know you can’t stop an object without affecting things sitting inside that object. In the earth’s case, there are some massive bodies of water that wouldn’t be very happy if you suddenly stopped the basins they normally stay in. Bearing in mind that, at the equator, the surface of the earth is moving at about a thousand miles an hour, it’s not hard to see such an event causing waves that would destroy the world. And this is to say nothing of the forces on the tectonic plates (expect massive amounts of geological activity), winds stronger than any hurricane all over the world, and the same force applied to you and everything around you. That said, God is powerful enough to stop all these things from happening, but that’s why this event would be a very big miracle.

However, there are reasons to believe that such an event is not what is being described here. Rather, according to John Walton’s essay “Joshua 10:12–15 and Mesopotamian Celestial Omen Texts”, Joshua is asking the Lord to make both the sun and the moon visible on… continue reading

James, Paul, justification, faith and works

November 27, 2011

James 2 and what it says about justification by works and faith made a lot more sense to me when I read this (in Carson and Moo’s “Introduction to the New Testament“):

James’s teaching in chapter 2 may be harmonized with Paul in at least two different ways. The first, and more popular of the two, argues that James is using the verb “justify” in the sense of “vindicate before people” (the verb is used this way in, e.g., Luke 7:29). Paul and James, then, are talking about different things: Paul of the declaration of our righteousness, and James of the demonstration of our righteousness. Another possibility is to take “justify” in James to mean “vindicate at the last judgment,” a force the word often has in Judaism (see Matt. 12:37). On this view, both Paul and James are referring to the sinner’s righteousness before God, but Paul is focusing on the initial reception of that status and James on the way that status in vindicated before God in the judgment.

I’d heard lots of explanations of the passage before, yet I’d never thought that James could be using “justify” in a different way to when we talk about “justification by faith alone”. It’s hard to read those big theological words without importing all the usual theological meaning we normally attach to them. I think it’s interesting that in the ESV, Luke 7:29 contains “they declared God just”, but the footnote says it’s literally “they justified God”.

Also, the reason for why James talks about faith and works and justification in this way could be because Paul’s teaching on justification by faith alone has gone out into the church, but has been misunderstood by some, and that misunderstood teaching is what James is writing against. It makes a bit more sense that James is responding to a misinterpretation of Paul’s teaching, but before he has a chance to clarify with Paul what his actual teaching is (at the Jerusalem Council in Acts 15), than that James is writing with full knowledge of what Paul’s teaching is.

1 2 3 4