I think your issue is here:
"Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not."
Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers.
Comcast's paying customers are requesting data from the Internet, data that Comcast doesn't have on its network. To get there those bits travel over Level 3's (or Cogent, or another major provider) network. The issue at hand is not one of Level 3 (or Netflix) trying to PUSH data onto Comcast's network. It is one of Comcast's customers trying to PULL data from Level 3, over a connection that Comcast refuses to upgrade.
Part of the issue is the monopoly most broadband providers have in the US and Canada.
If you had two truly distinct cable options, say Comcast and TascCom. Your friends with Comcast say Netflix sux, low quality, lots of problems. Your friends with TascCom say netflix is awesome, great HD content, too bad there's only like 8 3D movies! You'll sign up for TascCom. Comcast's board will get angry they're losing customers, and fix their interconnects. But it's all a dream, there is no TascCom, there's only Comcast.
Yeah, this does seem to be a core issue. How can we make it easier for people to start last-mile ISPs? It seems like a pretty capital intensive business to get involved in :)
> The core issue isn't at the last mile, but in connecting a last mile network to the internet?
Not really. The cost of the interconnect between e.g. Comcast and Level 3 is immaterial in the cost of operating a network. The only reason there is so much contention there is that it's a choke point where Comcast can try to put up a toll booth.
Those things are built with public money, not private money, which introduces problems of its own. Once you start using public money to build infrastructure, politics determines the level of spending rather than actual demand.
The American Society of Civil Engineers estimates that we have $3.6 trillion in delayed maintenance and underinvestment of our core infrastructure (water, sewers, bridges, power lines, etc): https://www.wsws.org/en/articles/2013/04/08/infr-a08.html.
I'm not one of those people that believes we shouldn't have public infrastructure, but I do think you have to be cognizant of the trade-offs involved. Take something like Amtrak. Amtrak has an almost $9 billion maintenance backlog on the Northeast Corridor: http://usa.streetsblog.org/2011/06/15/house-plan-to-privatiz.... The NEC is the only part of the whole system that generates an operating profit. A private company would shut down the rest of the network, and try to make the NEC service as attractive as possible for customers. But in a regime where politics decides where the money goes, the operating surplus generated by the NEC instead goes to funding money-losing lines in the rest of the country.
Road building, too, is the result of distorted incentives. As an urbanite who doesn't like to drive, I would spend $0 on highways designed to get suburban commuters into the cities, and spend that money on public transit instead. Surbanites, of course, feel differently. Who decides how that money is allocated? Not the market, but the political system, which at the national level systematically over represents rural and suburban votes.
The situation with telecom companies isn't ideal, but I don't think the dynamics of the telecom market are amenable to the kind of broad political consensus necessary for a successful municipal service. Take water, for example. Everybody needs roughly the same amount of water, and is satisfied with a relatively similar level of water quality. Meanwhile, I'd bet 95% of people would be perfectly happy with 5 mbps service, while a small minority wants gigabit. Do you think the political system is set up to make that small minority happy? If there is anything to learn from how municipalities handle public infrastructure is that when you put it to a vote, the voters will spend as little as possible to get the minimum acceptable level of service. That's exactly what happened to our power and water infrastructure.
Meanwhile, I'd bet 95% of people would be perfectly happy with 5 mbps service, while a small minority wants gigabit.
Historic and present demand are terrible ways of predicting future demand when it comes to technology. Most people don't know what they want because they're living in yesterday, but wait five years when they see what their early adopter neighbors are doing, then suddenly everybody would be happy with gigabit service, and who needs to upgrade to 10GbE anyway?
You could easily have said that nobody would want more than 128kb/s ISDN, because nobody does anything more intensive than download music, check e-mail, and watch flash animations. Faster speeds made newer services, services that are used by very ordinary people (like Netflix, Hangouts, Skype) possible.
Yet when you subject infrastructure spending to the political process, the best you can hope for is keeping up with present demand. Early adopters get no traction when they're proposing spending public money.
I want enough bandwidth to stream three HD 1080p movies at the same time (there are three people in my house) with enough left over for VOIP Phone and normal internet. I currently pay $80/month for that. ($50 for internet, 28+change for VOIP). I am willing to pay $120/month (that's frankly $4/day, not enough to really matter) for that. Can I buy it? No. Because no competition.
Why? Because utility monopoly. Why? Because lawyers and lobbyists and clueless and corrupt politicians.
Cable companies are not utility monopolies. The granting of exclusive franchises has been illegal since 1992. Do they have lawyers and lobbyists? Sure. So do Google and Netflix and Facebook and Yahoo.
Companies don't give you what you want because: 1) its expensive; 2) they can make higher returns with that money elsewhere. You can blame lawyers and lobbyists all you want, but the bare fact is that what Facebook paid for WhatsApp would pay for all the lobbying capacity of the top 10 DC lobbying firms for 60 years. That's a really weak argument to lean on when the interested tech players have so much money and lobbying is so cheap.
Just look at what Google is doing with Google Fiber. They're demanding massive regulatory concessions, and still don't seem to be positioning it as a money-making business. If building fiber was a good use of capital, why would internet companies sit on the sidelines and demand someone else do it?
I appreciate your insight, but it really is still a political battle.
The reason why is because companies have to work with local governments to get construction permits to build out the access network. There are huge barriers here that prevent individuals and small companies from actually getting that work done. One of those barriers is the fact that even local governments have problems with bribery and corruption (you don't say!).
Google Fiber is demanding those massive regulatory concessions because they must! They're asking governments to bend over backwards for them because they recognize how hard it is go get through the bureaucracy. That's why they put on this big campaign to get the people to talk to their local representatives and essentially beg to be saved from the monopolies.
Take for example, Google Fiber versus Sonic.net. Sonic's been trying to build out their fiber network for a long time. However, since they're small it's much more difficult for them to work with the local governments and as a result their fiber rollout has been slower despite consumer demands.
Please, show me the bribery and corruption that's creating these political hurdles. Otherwise, that's a strong accusation to be making without proof. If we're talking about good old fashioned lobbying, however, then the fact is that the internet companies have more than enough money to overwhelm the cable companies' lobbying efforts. Lobbyists are up for the highest bidder, after all. Yes, small companies can't afford it, but anyone with the money to realistically build the infrastructure in the first place can. Certainly, the companies with the most at stake (Google, Facebook, Netflix) can. It makes no sense to invoke money and lobbying while ignoring the fact of who has the deepest pockets in this game. Remember, Google is more profitable than Comcast and TWV combined, and has double the profit margin. Facebook's profits are about 2/3 as much as TWC.
The regulatory hurdles that Google is demanding relief from are (largely) not the result of lobbying. They're the result of the dysfunction of municipal politics. Right now, NYC's mayor is attacking Verizon because poor people can't afford FIOS (at $75/month). He's hired a civil rights lawyer to get into the issue. Is it any wonder companies aren't interested in building fiber? Is this the result of lobbying (or corruption and bribes) or predictable political forces?
At bottom, none of the screeds on this subject address the simple fact: the internet companies aren't rushing to build fiber, or lobbying to get permission to build fiber, or publicly demanding deregulation so they can build fiber. They're trying to get Comcast, Verizon, etc, to build fiber. To this day, Google positions Google Fiber as an effort to shame the ISPs, not a worthwhile business venture standing alone. What does that tell you about the monetary incentives at play?
> The regulatory hurdles that Google is demanding relief from are (largely) not the result of lobbying. They're the result of the dysfunction of municipal politics.
Isn't that basically what caust1c said? It looks like the two of you are in violent agreement. :-)
It's simple to get AT&T, Verizon, etc. to at least promise to build fiber; start a plan to deploy municipal fiber (if that's even legal in your jurisdiction). Any time that happens fiber is "Just a year or two away, so this network isn't needed!"
You said it right there. Look further down the financials. Comcast net income is $1.8B, Google is $3.4B. Time Warner Cable (I assume that's what's meant by TMV, although their ticker is TWC) is $0.4B.
But net income isn't a good comparison, because things like spending on R&D and infrastructure are subtracted before you get to it; they don't come out of net income, they come out of gross profits. So Google does not have significantly more funds available for those things than Comcast does; it has less.
And yet telcos weren't happy with the regulated profit that unbundled DSL provided them. Either they're just lying (which I consider entirely credible) or it's not that simple. For example, if the regulated profit is less than in an unregulated scenario, "shareholder value" thinking will argue for deregulation. Also, there's no profit if the regulated cost-plus price is so high that nobody buys it (what if the cost of FTTH is over $100/month?).
Who wants to spend billions on infrastructure they are forced to lease out at wholesale rates to competitors?
The government should, but they wouldn't lease them out to competitors, but providers who happen to be in competition with each other. Make the last mile a utility, like sewers.
Once the FCC finally takes the brakes off unlicensed wireless, rural and lower-density urban customers will see a competitive market of WISPs. They've been dragging it out for years, though, so who knows when that will happen?
Thanks a lot for your response (and to the others who responded as well, appreciate it!). What you're saying makes sense, but just to play devils advocate...
Disclaimer: The blog I'm about to post was linked from Comcast's blog. They claim he is independent, I know nothing about him.
"Netflix could use multiple providers to connect to ISPs and could also use third party CDNs like Akamai, EdgeCast and Limelight, who are already connected to ISPs, to deliver their traffic. In fact, this is how Netflix delivered 100% of their traffic for many, many years, using third-party CDNs. Netflix likes to make it sound like there is only one way to deliver videos on the Internet when in fact, there are multiple ways."[1]
It would seem to me that if Comcast is using all the bandwidth they've purchased from Level 3 than it's a simple reality for them that they'll need to increase that bandwidth or they'll see reduced service. I don't see why this is explicitly either side's problem. It seems like it would be great if they could figure out a mutually beneficial situation, but I guess I fail to see the moral hazard. If Comcast doesn't want to upgrade their network who cares? Barring, of course, the monopoly argument, I see no moral hazard here.
I also see no reason that this has anything to do with net neutrality. It seems like Netflix is simply so big they have to act more like an ISP than they used to.
Couple of points: Comcast isn't buying bandwidth from Level 3. "Settlement-free peering" means no money is exchanged. The reason they don't charge each other is because it is mutually beneficial; Comcast gets to charge it's customers for access to content that passes through Level 3 and Level 3 gets to charge it's customers for access to the Comcast subscribers.
The problem is that there is only a market solution for the Level 3 customers' side; if Level 3 can't accept all the traffic that their customers are sending it, they have other options with who to connect to, as there are other backbone providers. It is in their financial interest to maintain their network capacity, or their clients will go elsewhere.
Comcast, on the other hand, has no market pressure. If they can't handle all the data that their customers are trying to access, their customers have no other option. They are stuck with Comcast.
And therein lies the rub; what both Level 3's and Comcast's customers want is access to each other. Neither company can provide that service on their own; Comcast doesn't have a global network that can send data around the world, while Level 3 doesn't have a last-mile network that connects to all the end users. However, only Level 3 has to worry about losing its customers to a competitor. Comcast can essentially hold all parties hostage; they can demand payments from every party: end users, Level 3, and Level 3's customers. Of course, holding everyone hostage hurts everyone, including their own customers; however, since their customers have no other options, they aren't really in danger of losing them. They can hold out longer than Level 3 can, because if Level 3 doesn't agree to pay Comcast's ransom, someone else might, and Level 3 will lose their customers to whoever pays Comcast's ransom.
> "It seems like it would be great if they could figure out a mutually beneficial situation, but I guess I fail to see the moral hazard."
This article in particular is making the argument that at the level that L3, Comcast, AT&T, Netflix and others are operating, it's almost always mutually beneficial to peer. The article also makes the point that the capital expenditure is negligible compared to the benefit to both sides.
The moral hazard here is that Comcast is (shrewdly) taking the bet that their customers won't blame them for what appears to be Netflix's reduced performance. If there was actual competition in the local ISP space, customers would catch on that the performance is only worse on Comcast's network, but that's not the world we live in.
> Barring, of course, the monopoly argument, I see no moral hazard here.
You can't just ignore the monopoly argument. There's nothing inherently unfair about a monopoly, but if you use it to extort money from people then its no surprise that people call you out on it. In the Good Old Days, you'd see monopolies try to squash their competitors. This is more nuanced than that. Monopolistic-extortion-action-at-a-distance, if you will.
You're right, I'm just trying to disambiguate it from the net neutrality argument, which doesn't seem to actually be relevant here (as Comcast is treating all traffic through those links poorly, not just Netflix traffic). If it is truly about the monopoly, regulators have different (and thankfully more powerful) tools to change Comcast's behavior, whereas using net neutrality seems like a losing (and misleading) fight.
The mutual benefit still exists. Comcast benefits from peering with Netflix because it allows Comcast customers to access Netflix. Comcast is just playing chicken because they know their monopoly position allows them to hold out longer than peers with actual competitors.
> You're right, I'm just trying to disambiguate it from the net neutrality argument, which doesn't seem to actually be relevant here (as Comcast is treating all traffic through those links poorly, not just Netflix traffic).
The network neutrality issue is that Comcast is intentionally causing congestion on all the links that Netflix traffic could possibly use. The traffic for Comcast's own competing video service offerings don't have to traverse the congested links, nor does traffic from anyone who Comcast provides with a separate uncongested link in exchange for a toll or some other business consideration. That's practically the definition of a network neutrality issue.
Not upgrading their connection is not the same thing as intentionally causing congestion. Especially when the connection would be fine if it weren't for the massive share of Netflix traffic.
The massive Netflix traffic that I, as a Comcast subscriber, is requesting.
If we're all all drawing too much data over the network, upgrade it and charge us more! I have no higher bandwidth options than Comcast's cable internet service. Huge lack of competition in California.
There's a huge problem with the monopoly status of Comcast in this case: I'd dump an ISP for not quickly (ie, in a 3-4 month span) working something out with Netflix if I could.
Comcast's monopoly status is greatly impacting their negotiating position, and is core to this issue.
I mean, I'm trying to get a deal worked out with Netflix where they send me some information. For various technical reasons, this deal needs to go through a couple middlemen. Literally the only middleman I can hire is refusing to work out a deal with them that doesn't give an unusual benefit to that middleman (while Netflix's middleman is offering the standard package), and is screwing me in the process. I'd take literally any other middleman if I could, but this particular middleman has political connections, and stops that from being possible.
If the ISP is letting a connection become saturated in order to place pressure on a third party provider to provide settlement, they are doing so at the cost of any other providers running over those trunks. Since switching to AT&T, I've noticed decreased Netflix performance and sometimes Imgur loads like I'm on dial-up again. I haven't checked the routes but it has me wondering if it's shared congestion. To your point, I'm not sure this is a moral hazard yet but it does seem like one of those things that could be very bad for the customer and the ISP can simply get away with it due to lack of competition.
I have two net connections, and both are generally reliable, but I've noticed that the DSL connection (over Windstream) is very slow for certain sites like imgur and tumblr and AWS, so I've set those to use the cable modem (Time Warner) connection, where they work great.
Since I have 2 connections I definitely have competition in my area. (Wireless is another option.) So I don't get why if a connection has issues, it's always the DSL one, and always with certain other sites.
Because the two have different physical limitations. DSL is strongly limited by distance to a cabinet, and most people can't get any better than 7mbit/s, 20 if you're lucky. Thus, DSL provides no pressure on cable, and the only reasons people use DSL are because they get it bundled with their phone and TV service, or because they got fed up with the cable company and switched to a lesser service out of frustration.
In areas where something actually competitive is available, cable companies rush to improve bandwidth, or offer two-year contract deals at undercut prices to try to outlive the competition, then jack the prices back up when the competition goes bankrupt.
> only reasons people use DSL are because they get it bundled with their phone and TV service
Some people use DSL because they're happy with DSL. DSL can be had for cheaper, and many people do not need the higher speeds of cable. HN readers and other geeks who dominate these online discussions assume that everyone is like them and that they need the fastest speed possible while sneering at any available alternative that is "just" 7 megabits. 7 megabits gets you instant page loads on any website. Sure it't not as good if you want to stream four movies to your house at once...but not everyone needs that.
The fact is that Comcast is right when they point out that they do face competition, not only from DSL but also from other services such as wireless. That you and many other people do not like these alternatives does not mean they do not exist. Many people find them to be perfectly adequate and for them the other services are the best value.
Honestly, I've had one connection or the other completely fail, and (aside from the weird issues above) I sometimes haven't noticed for a few days, despite heavy Netflix streaming on 2 devices. Whatever is going on in my local market, to me it seems to be functioning well.
> I also see no reason that this has anything to do with net neutrality. It seems like Netflix is simply so big they have to act more like an ISP than they used to.
When Comcast let links to peers get saturated it is a network neutrality issue. Those services they provide will perform and competitors will not. Those who pay for access will also perform better than those who don't pay. It is by definition not neutral.
There's nothing wrong with the action of not delivering traffic, except that it goes against the spirit of their contract with their customers and the obvious damage it will do to capitalism if their customers have no option to leave for a better company.
> Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers.
Only, I'm sure, because it's untenable to do that. Yet. Cable execs really, really like their model of charging people for channels; it lets them extract maximum revenue. If they can do that with the Internet, they will. It may be difficult to do that on the consumer side yet, but what they're doing with Netflix is basically a complicated way to sneak more Comcast charges on to their customer's bills. In this case, via the Netfilx bill.
You are downvoted not because of emotion, but because your comment is not germane to the discussion. The reason the GP phrased it the way he did is to make it clear whose volition is at play in the data transfer. The technical details at the TCP level couldn't matter less.
This would all be fine and good if Level3 didn't make money from selling bandwidth. Comcast doesn't have to buy a connection from Level3; Level3 will give Comcast a link for free because they know they can turn around and charge their own customers for access to it. Comcast is really just cutting out the middleman here; and that scares Level3.
They could only cut out the middle man if they built their own global network; they don't have any fiber running to Europe or Asia or elsewhere. If there were no middle man, Comcast customers could only reach other Comcast customers.
But they are cutting out the middleman with respect to large traffic sources. Netflix won't be the only one to pay for an interconnect; AWS, Google and Facebook will likely follow suit shortly. Take that business away from the transit providers and it hurts.
But they cant cut out the middleman. Comcast doesnt have a direct connection to netflix or anywhere else on the internet.
I think Level 3 should straight up shut off their peering agreement with comcast and watch how all the last mile customers blame comcast for the issue, until such time as comcast stops playing games.
That assumes they blame Comcast. Maybe they blame Netflix for not working.
If you get to set the narrative such that one side or the other as the only ones who gain from the interconnection, this becomes trivial, but the hard facts are that both sides want the interconnection, but to try as hard to pretend that the other side is the more desperate one in the deal.
> All of their customers will still have to go through middlemen.
Not if they're watching Netflix. As I read the deal, Comcast and Netflix will be directly connected, so if you're a Comcast customer watching Netflix, your packets only traverse Comcast's and Netflix's networks; no third parties are involved.
But the Internet doesn't work that way: if Level3 shut off their peering with Comcast, Level3's customers would just find another transit provider and route around L3 because those companies are still in business to make money. They don't care about squabbles between Comcast and L3; they just want their monetized traffic to get delivered. The last-mile customers probably wouldn't even notice; but Level3's customers would likely be forced to pay more for transit from another company. But they would still pay.
Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers.
Comcast's paying customers are requesting data from the Internet, data that Comcast doesn't have on its network. To get there those bits travel over Level 3's (or Cogent, or another major provider) network. The issue at hand is not one of Level 3 (or Netflix) trying to PUSH data onto Comcast's network. It is one of Comcast's customers trying to PULL data from Level 3, over a connection that Comcast refuses to upgrade.