Did one invention lead to the decline of newspapers?

Did one invention lead to the decline of newspapers? What is economic myth and what is true?

Don’t get me wrong. The decline of newspapers is NOT an economic myth. The business continues to lose more revenue each year, and so does other advertising-supported media, such as magazines. Much of this happened in the last decade. It is  unclear if executives at any newspaper have any good strategic choices. None of this is a secret.

But one invention and one firm did not produce this outcome. The typical story blames Google and Internet search, and does so a little too blithely. There is a grain of truth in the common story, but it misses a lot. While it is correct that the firm to profit the most from this trend is Google, Google alone did not kill the newspaper.

This post is not an apology for Google. Rather, it presents a broader economic history than typically found in public discussion. As more issues get debated in public it important not to premise decisions on a distorted view of how we got here. 

More than one invention

What is killing newspapers? It resulted from the accumulation of several inventions and several firms.

First of all, let’s get something out of the way. News papers are suffering from two problems, loss of readers and loss of advertisers. The former can cause the latter, and has to some extent, but loss of advertising can happen for other reasons as well, and did.

Craigslist is probably more responsible for the decline in revenue than any other factor. Their ads are much cheaper. Lots of people use them. Craigslist had a direct and significant effect on ad prices in newspapers. If you are curious, there is some recent good research documenting the effect Craigslist had on newspapers by Rob Seamens and Feng Zhu.

By the way, so did Monster.com and all the other sites for job seekers. Classified ads for jobs used to be a big part of what newspapers provided.

There is a lot I could say about loss of classified ads others than through Craigslist, but since is the short version of this story, I will not. Just remember that classified ads used to be an easy source of revenue and are not anymore.

Now that that is out of the way, let’s focus on other topics, specifically, on ad-supported search on the web. That has taken BOTH readers and ads away from newspapers.

Ad-supported search is easy to misunderstand. Ad-supported-search arose as a result of several trends, and these should not be conflated.

That gets us to Google. As is well known, Google sells ads by auction, where the “auction” is really a bidding game between advertisers for keywords. Advertisers can scan lists of keywords and bid to have ads placed next to a search when a user enters the most common keywords.

Where did this market come from? Let’s review the highlights.

Google was founded by a couple PhD drop outs from Stanford. Larry Page and Sergey Brin met in grad school, experimented with a search engine using page-rank and started a business, eventually hiring Erik Schmidt as CEO.

Here is a relevant fact that is often forgotten. Were Page and Brin trying to compete with newspapers at first? No, at first they were selling a search service to others.

Look, in the late 1990s most of the mass market firms thought search was not very valuable.  Most of the computer scientists thought search was valuable (and still do).

As of the late 1990s the computer scientists were losing this argument because users did not use search engines very much. Users went to sites like Yahoo! and walled-gardens like AOL, who organized the online experience for the mass market. Search was for nerds, not the mass market.

Frankly, the computer scientists were losing this argument for a while in the most recent decade too. Most people forget that AOL had its peak year in 2003 and did not switch to being an advertising supported business until late 2006, which is about the same time that Google began to surpass Yahoo in popularity.

<digression> Here is a digression on my main point. Google is really a recent phenomenon by mass market standards. Only technologists think of Google as an old company because technologists are early adopters of technical services, almost by definition. Moreover, most technologists never liked AOL, and did not give it much credit for bringing less technically skilled readers to the Internet. <end of digression>

Back to the main point, Larry and Sergey were just trying to sell search services in 1998. That was it. And they had a hard time finding any buyers. Eventually Yahoo became a customer, but let’s not exaggerate. Yahoo signed a contract to have Google do search for them because Yahoo’s executives believed — just like everyone else — that search was not of strategic importance for their business. It was so unimportant that Yahoo could  outsource the service.

Why did Yahoo go to Google over others? The common answer is “page rank,” because it worked better at finding relevant answers. Another importance facet was Google’s speed. User’s do not like to wait. Most research shows that online users are super-impatient. Google got those two things right.

<digression> Yes, there were other design features that were appealing too. But this is the short version of the story, so let’s move on. <end of digression>

Let’s consider for a moment why Larry and Sergey went to Stanford to get graduate degrees.  Both were interested in information theory and practice. Their advisers were too. Their advisers got their money came from the National Science Foundation (NSF), a federal agency supporting research. Indeed, NSF had helped grow the Internet backbone and many other facets of the computer science that grew into the Internet, so NSF views itself (rightly) as one of the parents of the Internet.

Did the NSF want to start a new firm like Google? Of course not. NSF was funding initiatives to improve information retrieval and storage, a long standing topic in computer science. But for years such efforts had made only incremental progress because everybody focused on information retrieval in libraries. When the web started to explode, these initiatives were taken up again, and reconsidered, and re-imagined for the web. NSF actually did its job, recognizing a new opportunity, moving funding to it, and seeding research.

Did NSF know what would come from it? No, that is not the way federal funding works at the NSF, thankfully. NSF gives its scientists considerable discretion to apply the research as the scientists sees fit, so long as it stays within the domain of the topic outlined when the research was funded.

Let me say it this way. The page rank algorithm that Larry Page and Sergey Brin employed for their first search engines was standard mathematics by the late 1990s, invented many years earlier. They recognized its application to the web. That was their initial novelty. NSF could not have foreseen that, but NSF should get credit for funding general research in this area, which had enormous economic potential. The payoff to society in this case far exceeded the money spent.

Selling ads

Notice that this story has said nothing about selling ads. That omission reflects historical facts. Larry and Sergey were not selling ads at the beginning. That came later.

How did it come about? Like many marketplace innovations, the initial inventors of ad-based search got it wrong. Imitators profited.

Specifically, Google did not invent keyword bidding. Goto.com (later called Overture, and much later bought by Yahoo!) pioneered the idea of a word auction, and led the category for years. Google imitated Goto.com at first, them made it better.

Goto was a venture backed business, a product of Silicon Valley’s system for funding good ideas. Today not many people remember Goto.com except those who lost money on the investment, and the VCs who were very frustrated with the management. In Valley-Speak the local lore today refers to Goto as just another firm who had the right idea and at the right time but did not execute on it.

We know in retrospect that Overture got imitated by Google and eventually lost out in head to head competition. Why? Overture faced three problems: Bidders gamed the pricing of the first-price auction; porn sites found ways to outbid others when users did not want to see their ads; and advertisers did not like paying for impressions because they could not measure the success of their ads.

Despite those issues, Overture actually made a lot of money at first. The porn sites spent a lot of money on ads, after all.

Google came up with three key innovations, one was the use of second price auctions. This stopped a lot of gaming of pricing by bidders. This was such a great innovation that Overture soon imitated it. Notice that there is nothing technical about this invention. Rather, it is an adaptation of processes.

The second innovation had a similar spirit. Google paid advertisers for user clicks, not impressions. That gave advertisers the ability to measure what business their ads generated. There is some technical work in that, but not much. This came about because Google was trying to serve advertisers better. Overture imitated this one too.

The third innovation was the most clever, a ranking of the “quality” of the web site. Such ranking effectively devalued the ads coming from the porn sites <digression> which made Google MUCH better for most users — just ask anybody with kids <end of digression>.

That quality ranking was both a technical invention and good business.  As business, it did not let the money from advertisers get in the way of the the quality of the search for users. Google put the users first and let the advertisers follow users. Eventually it also gave web sites incentives to improve what they did, and get higher quality ratings.

<digression> To my economist friends…. an economist might be tempted to say: Any good two-sided platform model would tell you that… Well, no, that is historically wrong. Some of the great work on the economics of two sided markets came after Google’s success, and did not drive it. Theoretical economists have been the followers of events here, mostly. The true originality has been in the market place. <end of digression.>

Overture never matched the quality ranking. That is because Google actually had to do some very difficult inventing. The computer science of an automated quality ranking algorithm for the entire web is just plain hard. Real hard. Eventually Overture lost out.

All this took place in the context of competition between search engines.  I am merely telling the short version of this story and leaving out all the inventions that turned out not to work.

Notice, as well, that the quality ranking is a part of Google’s secret sauce. It is unsurprising that they hesitate to  be entirely forthcoming about how it works. Google claims web sites would game a transparent process, so a little murkiness is appropriate. Even if all that is true, it is also not surprising that some web sites are suspicious of Google. They have to live with this quality ranking, take it or leave it.

As a digression on that digression, if you really want to know more then watch Hal’s Varian’s explanation of how the auction works. As part of that, he explains a bit about how the quality ranking works. (Be forewarned: It is ten minutes long).

The entire ecosystem

A little later, Google invented one more thing, placing ads in designated places inside blogs, usually in little rectangle windows. A site can arrange this with Google, and Google splits the money from the ads with the site.

That invention is clever. Once again, it combines a bit of technical cleverness with a bit of imaginative energy. The computer science was not easy, but this should be understood as more than merely a technical invention. It involved a lot of entrepreneurial imagination and it involved building an organization behind it.

It also was important. It was responsible for fueling a big part of the boom in sites like Huffington Post, and others who have a little space to rent next to their blog posts.

That started to happen in 2003 and by the middle of the decade it had spread madly. “Ads by Google” became a common statement all over the web. Many little and big sites began using Google’s ad services, paying only for clicks. It was convenient, especially for smaller sites, generating enough revenue to support an inexpensive site.

It changed the ecosystem of the web. The long tail of ad-supported web began to depend on it. Many niches sites expanded accordingly. Once again, this is a long story, and I am merely giving the short version of it. Just suffice to say, however, this was another source of substitutes for news, and it drew readers away from newspapers.

What led to the decline of news?

Look, this post has a point. By now it should be obvious: One invention did not lead to the decline of newspapers, and one firm did not do it either. The loss of readers and the loss of ads came from the accumulation of a number of events.

Who is responsible? Let’s count. We have blamed Craigslist, other online classified sites, Overture, NSF funding, Silicon Valley’s ecosystem, the efforts of many clever computer scientists, and the efforts of many bloggers.

And that is just the short version of the story. The problems with newspapers did not arise at the hand of a single invention or a single firm. It was a gang.