Archive | Technology RSS for this section

A Short Quant Analysis of Video Game Ratings

A kind Redditor made a database of all video game reviews from IGN, a gaming website.  Obviously I couldn’t let this go to waste, so I popped it into R to see what could be done.  First off, a histogram of game scores reveals two obvious biases:

scorehist

 

First off, there seems to be a substantial upward bias in assessment.  As a customer, I’d like a nice normal distribution centered around five.  This is centered around 6.9 and skewed to the right, which seems to validate the intuition that reviewers are often overly generous with games.  However, this could reflect the data-generating process: reviewers do not randomly select the games that they review, and may be non-randomly sampling a normal distribution.  However, the second problem is that reviewers seem to operate by flawed heuristics – the obvious big huge spikes are at round numbers and at .5, suggesting that reviewers are operating within a constrained space.  I would suggest we could probably eliminate ratings more finely-grained than a half-point, because this suggests it’s not tremendously informative.

As a gamer, as opposed to a social science measurement guru, I’m also curious about something else – are some platforms systematically better than others?  The answer seems to be, “not really – but some platforms are worse”.  See the graph below for the effects – basically, the line represents the average mean score, and the distance from the middle reflects whether games on that platform are better than the average (right of the line) or worse (left of the line).  Most are worse!  There are few platforms with positive and statistically significant positive effects on predicted score – interestingly, Macintosh is one of them.  Either Macs are a better gaming platform than given credit for – or only good PC games get ported to Mac.  However, there are a bunch of negative effects from platforms, and as you see below, there are a number of categories that produce truly awful games.  DVD interactive “games” are rated horribly, which shouldn’t be surprising – but Wii games, Game Boy games, and Nintendo DS games are – the coefficients on those are large, negative, and statistically significant.  I wonder if that’s because their games are targeted towards kids.

systemplot

Click to embiggen

 

The data is kinda neat, but limited.  It’d be nice if it were dated, time series data – I’d love to examine whether ratings are changing over time. Eyeballing the chart of which systems do well and poorly, I doubt it.  Curmodgeons love to claim ratings are too generous and it was better back in the day, as curmodgeons claim to do.  I suspect the data would prove them wrong.  Next step might be learning how to use this hacked-together Metacritic API

Drone Valley Will Come…Eventually

Noah Smith, discussing Andreesen’s call for a “Drone Valley” tech-deregulation zone, suggests that a new emphasis on deregulation is bearish for the future of technologically-driven productivity:

In the 1990s and early 2000s, we had a massive technology boom — productivity accelerated, and our lives changed in big ways. That boom happened without the need for a lot of deregulation — computers were these cute little boxes we stuck in our house, and the Internet didn’t really alter the physical landscape. But that happy situation was a historical anomaly. In the past, big technological change required big changes in regulation and law and the shape of society. Those changes brought costs. Think of how cars required expensive roads and safety laws, and made our cities noisier, more polluted, and less walkable.

Tech’s focus on deregulation is a sign that the free lunch of the computer boom may be coming to an end. In fact, that unhappy hypothesis is borne out by the numbers. John Fernald, an expert on measuring productivity at the Federal Reserve Bank of San Francisco,recently found that productivity in the IT sector, and in sectors that use a lot of IT, has slowed down a lot since the early 2000s (well before the financial crisis).

If true, this suggests there is in fact a supply-side story to the US’s lackluster economic performance of the last fourteen years.  It’s not about incentives to invest or work – I think there is little economic benefit to further cutting taxes.  Instead, the supply-side constraint is all about regulation and innovation.  Drones are, today, an edge case of purely academic interest – but the case of AirBnb and Uber are much more obvious.  Incumbent interests are working regulatory bodies to prevent new sources of economic activity.  So too is the current housing crisis in many of America’s major cities, where rental pressure is escalating extremely rapidly while regulation totally clamps down on house-building.  In both cases, incumbents (homeowners, cab companies) reap benefits to their asset values (taxi medallions, houses) while generating large negative externalities.   That being said – one of the reasons this seems so plausible is that the stories of Uber and urban home prices are so well-known and understood, but it’s pretty hard to think of some other comparably important issues where anti-innovation regulation is putting severe supply-side restrictions on growth in place.  And only one of those is that important – taxis are highly visible to urban writers, but hardly an enormous part of the US economy.

It just doesn’t seem plausible that anti-tech regulation is a massive drag on the United States economy.  If we invent cornucopias, that may well change, but as of today these regulations mostly interfere with technologies that are not yet ready for large-scale commercialization.  There is a circular element to this logic, of course; perhaps more or these innovations would be ready for commercialization if not for regulatory hurdles tamping down investment.  However, the circularity argument isn’t that strong – first off, Silicon Valley VCs are pumping plenty of money into drone tech today, and secondly there are many municipalities that will be ready to offer some regulatory arbitrage once these technologies are truly ready.  Some deregulation may well lead to positive externalities, but I’m fairly confident that if this is truly a huge issue it will be resolved sooner or later.  When drone delivery tech is actually ready for use, somewhere will be ready for it.  It seems more plausible that a lot of the truly ground-breaking innovations that will transform the US economy just aren’t ready yet.

People are not known for inventing useful things and not using them.  I would look elsewhere to explain US stagnation – such as monetary policy, fiscal policy, and an enormous financial crisis some of you may remember.

It Doesn’t Matter if Advertising Works or Not

According to internal eBay testing, no.  Randomized controlled trials with Google search engine marketing provided little to no results, and it seems like a large portion of their advertising budget is just wasted.  Their budget mostly goes to show ads to those who would still have ended up at eBay anyway.  As Tim Fernholz notes, there are a few gigantic caveats:

  • Your mileage may vary. eBay is a gigantic retailer, and if somebody has not heard of eBay in the year 2014, they are not a potential eBay customer – they’re also probably not Googling things.  This isn’t true if you’re a nobody, where effective search-engine marketing is often the single best way to spread awareness of the product.
  • Long-term effects might be important.  The tests show only immediate results – whereas in the long term, cutting down advertising might hurt sales a great deal.  But running even very short-term RCTs at a company like eBay can be very difficult, it’s unlikely they’ll be able to do this for a year.

I would add a further point: eBay’s advertising makes competing with eBay more costly.  Partially there is just the opportunity cost – your customers see ads that aren’t yours.  More importantly, you’re helping your competitors’ bottom lines.  Search engine ads are allocated via a competitive marketplace with competing firms bidding to place advertisements.  Every bid that eBay doesn’t place makes the marketplace less competitive and allows their competitors to place ads more cheaply.  This in turn makes their advertising more cost-effective.  In many areas, for example online auctions, eBay is such a huge player that the spot prices for ads will probably dive a great deal, juicing their competitor’s return on investment.  In the long term, that’s a serious competitive threat, which it is entirely within eBay’s power to avoid.  Even if the return on advertising to them is low or negative, it might still make sense for eBay to spend billions on search engine marketing simply because they are the biggest and can best afford the cost.

Search engine marketing is an asymmetric weapon in a number of ways.  It is more useful to the new and weak, and can drive growth very quickly.  But when competing with an incumbent in a large category, it may well be too expensive.  It also suggests that new consumer-facing startups competing with digital-native incumbents (e.g., eBay) will face systematically high marketing costs that will require massive amounts of capital.  Interestingly, despite this logic and some high-profile news to the contrary, there does not appear to be a long-term upwards trend in the size of venture capital funding rounds.  This might be an issue of incomparability – for example, perhaps in recent years staff-heavy enterprise startups have been supplanted by thinly-staffed consumer startups that are plugging a greater share of money into advertising.  Impossible to say with the data publicly available.

If you are trying to strategically decide what kind of company to start or what market to enter, the takeaway seems clear – the idea of easily scaling up to competitive size with an established incumbent through SEM is probably an illusion.  You will face systematically higher costs than you expect, and will need to deploy more of your capital than you think to advertising instead of staffing and product.  As for Google, it doesn’t seem like they should be that concerned.  The logic of the situation clearly suggests that even if advertising doesn’t work, the money should keep flowing in for the foreseeable future, either from established firms or heavily leveraged VC-backed startups but mostly both.

The startup world: an extremely elaborate mechanism to redistribute teachers’ retirement money to Google.

The New Managerial Revolution

With the summer here and the pace of academia slowing down a bit, I have had time to do some pleasure reading.  One book I have just launched into is American Colossus, a history of the American Gilded Age.  It’s excellent so far, quite engaging if a little light on the linear explanatory broad-strokes history.  One aspect of it I was struck by was, however, the description of the railroads.  Brand mentions, as have other histories I’ve read, the way that the complexity of modern railroads drove the emergence of the modern enterprise structure.  Joint-stock companies are much older, but they reached their modern structure only with the railroads – and many other innovations such as long-term capital management, regular schedules, and use of statistical management were only developed because the sheer complexity and size of the railroads mandated with them.

It matches up quite interestingly to this story about the use of Silicon Valley methods in revamping Healthcare.gov.  Small teams, rapid testing, and outsourcing much of the technical backend to the cloud. Yes, Silicon Valley can be much too self-congratulatory and yes, most of their methods are quite inappropriate in a governmental context where the needs are high and the costs of even a single slip-up are devastating.  A fatal error in your laundry-delivery app inconveniences some rich professionals for a day, while a fatal error in public-sector software can mean that people die.  That being said, there are still some parallels between the two – the railroads transformed American business by developing the techniques necessary to deal with large amounts of capital, whereas Silicon Valley is transforming American business by developing techniques necessary to do more with less capital.

If I were to bet on what the economy looks like in 2050, it would be an economy doing more with less stuff.  It’s about more than just replacing server clusters with AWS – it’s about applying the same model to physical stuff.  Airbnb and Uber are demonstrating this quite well – taking advantage of capital that ten years ago simply sat idly.  Rooftop solar panels promise the same – generating enough electricity on site to power a home without the massive infrastructure of coal plants and substations underlying the 20th-century energy grid.  You sometimes hear the complaint, in the financial media, that there is too much capital chasing too little returns – that this is the reason that institutional money is flooding into VC funds that in turns floods into silly social apps with no business plans.  If software truly is eating the world, this is what we would expect – that the material abundance of society increases while the need for capital declines, and the standard of living increases more rapidly than its material abundance.

In a world doing more with less, we would expect this complaint to become a universal feature – a paucity of ways to productively deploy capital, and declining returns to it.  Maybe it’s finally time to read this Piketty book…

R in the Cloud

So R is a great language for statistics, and Rstudio is a great environment in which to run it.  It’s a powerful set of statistical tools, but it does have one limitation – memory.  When running it on your desktop, it doesn’t make incredibly efficient use of memory and you can run into problems when dealing with reasonably large data sets like internet data.  However, there is a neat way around it – moving R to the cloud.  Amazon is best-known as a bookseller with some associated ecommerce on the side, but it also happens to have a gigantic business on the side called Amazon Web Services.  It allows you to rent server capacity on demand, which can basically give you unlimited computing power whenever you need it.

Yhat (an analytics company) provides an excellent guide to setting up R on Amazon Web Services.  It seems pretty complex when you first look it up, but once you get over your fear of entering things via the command line it’s quite easy. Don’t be intimidated!  It took me about twenty minutes to set up an instance, with great results.  Rstudio runs in the browser, and looks just like it does on the desktop, and once you figure out how to upload files it works exactly the same.  Even better, it’s a lot faster than it is on your desktop because it uses memory more efficiently – and while it’s running, it doesn’t totally destroy your computer’s performance.  Even using the introductory “free” server, which has lower specifications than my laptop, it’s faster than my laptop.  And upgrading to one of the “high-end” servers – which are obscenely powerful – costs a few cents an hour and you only get billed when it’s actually in use.  Here’s what it looks like in action, just a simple browser window with R inside:

Screen Shot 2014-06-04 at 2.55.53 PM

In short – upgrading to R on AWS is a pretty easy step that can really upgrade your data analysis game.  It provides an arbitrarily large amount of computing power that can allow you to take on the projects that were too much to handle before.  Even the free tier is great, because it lets you offload long-running jobs to the server while still using your computer.  I’d highly recommend it.

What Solar Power Tells Us About Ukraine

Awesome and welcome news about the price of solar power – it has now reached price parity with fossil fuels in certain markets.  Today it’s developing economies in Asia – but it’s coming for everywhere else too.  The march of solar power continues apace, and short of a shocking and unpredicted slowdown in its development it will very quickly become the obvious choice for new power generation installation.  As Ambrose Evans-Pritchard points out in the Telegraph, a secular decline in oil demand and thus prices will completely hammer rentier petrostates like Saudi Arabia, the Gulf States, and yes, Russia.  For the last forty years or so, oil was the only thing propping up the economy of Russia and the USSR before that.

At the margin, this should probably make us more frightened about Russia’s intentions in Eastern Europe.  Vladimir Putin is not a stupid man, and unless his intelligence service is completely dysfunctional, they have likely been mulling over the alarming price charts of solar for the past four or five years.  As oil demand starts to slacken, it will completely destroy the government finances and military capability of the Russian state.  Without an endless river of oil money, it might prove difficult to keep buying off elites and regular Russians.  It will certainly prove difficult to maintain its relatively high degree of regional military power, especially after the extremely expensive professionalization and modernization reforms of the past few years.

Russia might at this moment be more relatively powerful than it will be for the foreseeable future – or ever again, perhaps.  Steven van Evera writes that we should be concerned about wars as great powers decline, in order to seize advantages before they disappear.  It’s certainly hard to imagine that Russia will ever be better-positioned for a geopolitical power grab against its neighbors than it is in April, 2014.  If Putin wants to regain Novorossiya, the clock is ticking.

Statistical Inference & “Running the Numbers”

I identify a lot with Frederik deBoer, who has a liberal arts background and is getting a crash course in quantitative methods as a grown-ass adult.  He has a nice piece on “Big Data” and its broken promises.  One of the most important is the simple fact that basically anything is “statistically significant” in a large-enough dataset, for which he offers a pretty good layman’s explanation:

Think of statistical significance like this. Suppose I came to you and claimed that I had found an unbalanced or trick quarter, one that was more likely to come up heads than tails. As proof, I tell you that I had flipped the quarter 15 times, and 10 times of those times, it had come up heads. Would you take that as acceptable proof? You would not; with that small number of trials, there is a relatively high probability that I would get that result simply through random chance. In fact, we could calculate that probability easily. But if I instead said to you that I had flipped the coin 15,000 times, and it had come up heads 10,000 times, you would accept my claim that the coin was weighted. Again, we could calculate the odds that this happened by random chance, which would be quite low– close to zero. This example shows that we have a somewhat intuitive understanding of what we mean by statistically significant. We call something significant if it has a low p-value, or the chance that a given quantitative result is the product of random error. (Remember, in statistics, error = inevitable, bias = really bad.) The p-value of the 15 trial example would be relatively high, too high to trust the result. The p-value of the 15,000 trial example, low enough to be treated as zero.

With a big dataset, everything has a very low p-value.  When you run it through most predictive models and test it on every possible variable, you will get the result that every independent variable impacts the result, and the p-value is low enough you can discard the possibility that it is a result of random chance.  This is especially troublesome when your sample, or n, is extremely large yet represents only a small portion of the overall result.  You will not face the issue that your results are driven by random variations of the sample, but you are very likely to face the problem that your sample isn’t representative.  If there is some selection bias in what datapoints are sampled, you could be producing very poor models of reality.  But judging whether your data is representative is difficult, because it’s difficult to make statements about missing data.  For really big datasets, making sure that your data is representative is a devilishly difficult and extremely important problem.

The promise of “Big Data” is that “n=All”, a common statement amongst its proponents.  Yet this is rarely the case, often because the data capture is process and often because of computational practicality.  The most fancy machine learning models in the world won’t help you if your data is of poor quality, and pretty simple methods like linear regression can be incredibly powerful when faced with the right methods.  Fancy machine learning techniques do have their uses, though – for example, regression offers little help in the problem of “feature selection“, or choosing which variables to use in constructing a predictive model.  This is a pretty important problem when you are looking at a problem without strong theoretical avenues of investigation.

The most important lesson I have learned from my intensive study of statistics is that certainty is elusive.  To people with no serious quantitative training (including myself nine months ago), you imagine that statistical inference works like high-school math problems.  You “run the numbers”, and an answer pops out.  But this is almost never the case, and successful inference involves many layers of judgment in data gathering, data processing, and model-building.  There are definitely wrong answers – for example, when faced with a problem that calls for a prediction of probability, you can’t assume linearity.  If you do, some predicted values might be negative and negative probabilities are a nonsensical idea.  But answers can’t be “right”, they can only be defensible.  And beyond that, reasonable intelligent people can disagree violently about which answers are defensible.

Learning statistics has been invigorating as I realize just how much is possible with a dataset and an old laptop, and humbling as I realize that statistical investigation is a lot more difficult than learning the commands in R.

Towards an Empirical Identification Strategy for Time Travelers

So one false lead has been exposed.  We know that time travelers aren’t using Facebook.  But we don’t really know this, because the researchers used just about the laziest strategy possible.  They searched Facebook for posts containing the words “Pope Francis” and “Comet ISON” before either of those were named.  I hope I don’t need to clarify the idiocy of this, but just in case: if there are time travelers, they don’t want to be found.  The absolute easiest way to get caught as a time traveler is to make the idiotic mistake of referring to a unique event before it happens.  It should be obvious why time travelers don’t want to get outed, but the main one is that the US government might take an unhealthy interest.

However, if there are time travelers, how could one go about spotting them?  The fact that they’ll likely be pretty averse to getting spotted should complicate things.  Any dead giveaways will be anathema.  However, if we assume that time travelers are trying to avoid distorting history and creating time paradoxes, that might create some characteristic patterns.  But there are a few things that might be worth looking at:

  • Alpha in large public markets:  Alpha is investment returns above and beyond the market benchmark. Basically no one earns alpha consistently.  A lot of those that do are those in relatively illiquid markets – for example, Warren Buffett buying up whole companies.  But you can’t do that if you want to avoid making waves in history.  So we’d need to look for people who are earning massive returns, but only in markets large enough that they won’t affect the actual behavior of the markets.
  • Unusual isolation: Again, if people want to avoid altering history they’re probably taking serious measures to avoid doing just that.  Isolated houses, dark glasses, general distance from social interaction.  Most likely not employed, and may be described as “independently wealthy”.
  • Interest in nature: If we assume that disturbing history is verboten, this is the main reason to time-travel.  Climate change is only getting more serious, and the climate when time travel is finally invented may be unrecognizable.  The future’s naturalists will be excited to come back to gather pristine samples.

So…healthy investment returns, unemployment, isolated lifestyle, and probably an “amateur” naturalist.   If I were out data-mining for time travelers, which I am not, that’s where I’d start looking.  Unless of course the assumption about changing the past is wrong, in which case all bets are off and Elon Musk is obviously an alien time-traveler.

Deceptively Exciting Headlines

I happened across a story today that NASA has begun work on an honest-to-god warp drive.  This is, needless to say, awesome.  This doesn’t go faster than light, which as far as we know is still impossible.  Instead, it would actually warp space by creating a bubble in space which would propagate arbitrarily fast.  The big breakthrough is that NASA has discovered that it wouldn’t take a ball of exotic matter the size of Jupiter but would require a scanty 1,000 pounds.

Except we can’t really create exotic matter.  Warp drive is still not happening anytime soon. Sad.

Simple Answers to Questions About New Technology

Via XKCD:

Frederik Pohl said that, “A good science fiction story should be able to predict not the automobile but the traffic jam”.  In other words, predicting the iPhone isn’t exciting – Star Trek saw it coming forty years ago.  Star Trek definitely never saw Snapchat coming.  It’s why good science fiction is so darn interesting – the rare moments where someone sees the new ways that human’s traditional absurdity will keep on keeping on.