ISPs Report Breakdown in FCC’s Broadband Measurement Group; ‘Open Measurements’ Questioned

Several ISPs spoke of a breakdown of trust within the group, which they trace back to a flawed period of data collection related to problems at some Measurement Lab nodes in March (CD May 7 p8). That’s the server platform the FCC uses for its broadband measurement, run by the New America Foundation’s Open Technology Institute, Google and other academic researchers. An M-Lab official agreed the atmosphere has grown increasingly contentious since then, driven partly by disputes over how and when to release the flawed data to the research community, as well as by heightened rhetoric that the M-Lab platform is inherently unstable and requires the implementation of redundancy measures by the ISPs. “Whether trust has broken down or not, the real question is, can we carry forward?” said M-Lab engineer Thomas Gideon of the Open Technology Institute. “Can we serve the public interest?"

ISPs and M-Lab have repeatedly praised the commission for its commitment to openness and transparency. And all agree that the collaboration between the otherwise often adversarial stakeholders has been wildly successful. It’s not easy to accurately measure broadband across all kinds of different platforms, ISP officials said. Because of the level of cooperation among all the ISPs, the FCC has been able to release two comprehensive reports with results that have been widely touted by those service providers that scored well in what their actual speeds were versus what they advertised. But some ISPs said they worry that the formalized principles of openness won’t do much to repair the trust within the group.

"We are formalizing our policies regarding openness, transparency and collaboration,” read a slide presented at the last speed test stakeholder meeting (http://xrl.us/bnjbeu). The draft list of principles said in part that measurements taken for purposes of the measurement report would be collected from servers “not under control” of the measured ISP; at least one measurement server “supporting open research initiatives” and unaffiliated with the ISPs subject to measurements is “expected to be used” to gather measurement data; and other measurement servers may be used for limited audit purposes. “We encourage validation and cross-checking of measurement results and practices,” one slide read.

FCC Chief Technologist Henning Schulzrinne told meeting participants that the principles expressed in the draft document were ones that the commission, broadband providers and others had “consistently adhered to,” according to the FCC’s ex parte filing describing the meeting (http://xrl.us/bni7dk). He explained that a more detailed description of the principles might lead to a better public understanding of the commission’s commitment to openness.

But the ISPs balked, according to an M-Lab ex parte filing describing the meeting (http://xrl.us/bni52t). M-Lab described the ISP response as “emphatic resistance to both the principles and to transparent processes and scientifically verifiable measurement.” The opposition was directed most strongly at principles explicitly foregoing the use of ISP servers to collect data that would be used in the FCC’s official reports, M-Lab said. This surprised M-Lab, which stated that even the FCC said the exclusion was not a change from current operating procedure.

The M-Lab ex parte is “not an accurate description of the meeting,” AT&T Vice President Hank Hultquist told us. “The suggestion that ISPs are resistant to principles of openness is an absurd characterization given the long history of the ISPs’ involvement in the collaborative testing and measurement process,” he said. ISPs generally didn’t have a problem with the principles themselves, said several ISP officials who were either at the meeting or had direct knowledge of the events. The principles might not have perfectly portrayed the state of play, but they were fine as a discussion document, the ISP officials said. ISPs were concerned about articulating new policy after two successful broadband measurement reports had already been released, they told us. Some said they wondered whether formalized principles would be a distraction from the group’s core purpose: Planning for upcoming broadband tests.

Some ISPs felt that implementing these principles now might create the impression that previously something had been wrong, officials said. And the introduction of formal principles would necessitate closer inspection of the language, they said. What exactly does “control” mean? Should there be a public notice and comment cycle? Ultimately, several ISPs thought this new discussion about principles would be a distraction, especially if the policies start feeling like regulation, an ISP official told us. ISP representatives said that instead of working on a statement of principles, shouldn’t the group be spending time focusing on the details of future tests?

It’s not a distraction, Gideon said. “It’s a starting point. It’s to make sure we're all clear that there’s an explicit agreement on where we're starting from,” he said. Gideon said he was surprised by the amount of pushback the principles received. Based on M-Lab’s conversations with FCC officials, he didn’t expect it. “If that’s the way they feel, they're entitled to feel that way,” Gideon said. “But distraction or not, if we're all committed to openness and transparency, and we're all committed to a valid and verifiable outcome, then this should be a non-issue.”

'Problematic’ Disclaimer Language

Everything was fine with the ISP/M-Lab relationship until problems were discovered with the March M-Lab data, ISP officials told us. FCC contractor SamKnows, a U.K. broadband performance measurement company, found evidence of congestion in late March, including throughput issues to the New York and Los Angeles test nodes. The parties decided to restart the month-long data collection to gather information for the 2012 report. That’s when M-Lab started acting differently, the officials said. The lab started filing its own ex partes detailing what happened in the meetings from its perspective. Meetings became more adversarial, said ISP and M-Lab officials. They said the collegiality that had marked the first year and a half started to evaporate.

M-Lab admits there were problems at some of its nodes, and agreed the March data should be set aside for purposes of the report. But differences arose over what to do with the flawed data. M-Lab wanted it released, because it still had value to the public interest and researcher community, Gideon said. Others, like USTelecom and the NCTA, worried the data could be abused by parties who might make inaccurate comparisons of carriers’ performance (CD May 30 p1). Ultimately the group decided to release the data with a disclaimer. An FCC ex parte filing had explained that the commission’s commitment to openness would be “compromised” if the data weren’t released -- and the data release policy of M-Lab also required release of the data, the agency said.

The generally agreed-upon disclaimer said that server issues in New York and Los Angeles “distorted test results for a significant number of panelists across various ISPs participating in the study,” and was “unreliable and flawed.” That wasn’t good enough for M-Lab, said Gideon, who said the disclaimer used “very loaded terms. Biased terms.” And there were delays in getting access to the data, including an embargo period of up to a year to let the FCC produce its report. “It took weeks upon weeks to get an NDA that was executable to everybody’s satisfaction,” Gideon said of a non-disclosure agreement. This delayed M-Lab’s ability to analyze its own data, and to offer an empirical analysis to bolster the disclaimer it wanted, with sound scientific footing, he said.

M-Lab continued to press for different language, but the “problematic disclaimer” was pushed through, Gideon said. “There was a lack of cooperation,” he said. “There was no receptive response. There really was no engagement on that.” An FCC ex parte filing said simply that “participants discussed the need for appropriate disclaimers that would acknowledge the deficiencies of the data.” To M-Lab, the official ex parte filings have a “sort of broad stroke characterization that doesn’t capture what we think is critical,” Gideon said.

The dispute over disclaimer language was “part of a pattern,” Gideon said. Then SamKnows came with observations about certain problems with the April data. An interconnection problem in Mountain View, Calif., caused six days’ worth of data to be compromised, leading the commission to throw out data for the second month in a row (CD July 2 p3). This was unrelated to what happened in March, Gideon said. But because the April event was characterized as a “platform event,” it’s “fed into a rhetoric of claiming that the M-Lab platform is unstable,” he said.

Dueling Ex Partes

So M-Lab started filing its own ex partes. Its July 30 letter pulled no punches, speaking of ISP resistance to letting anyone besides the FCC file the meeting ex parte, and accusing them of having an “overall objection to transparency, open and documented processes, and to the Commission’s restatement of principles that have not changed since the beginning of the program.” The letter spoke of the “ire and raised voices that characterized much of the behavior by ISP representatives during this meeting,” behavior that was “abusive to other members of the collaborative” and “both unethical and unprofessional."

It’s that kind of language that makes ISPs wary of giving every participant the ability to file its own ex parte, officials told us. There was no yelling or unprofessional conduct, they said; people had to speak up to be heard over the speakerphone. ("People don’t get red in the face when they're speaking to a speakerphone,” Gideon countered.) Back and forth filings filled with different accounts of what happened in the meeting are “counterproductive, and really distracts what the group has really been intended to do,” an ISP official said.

The FCC’s ex parte documents the results, not the process, and that’s a good thing, the ISP official said: In the course of the discussion, people might ask if ISP servers should be used, or question whether Google’s new fiber Internet service in Kansas City, Mo., makes it an ISP, and what that means for how to measure things fairly going forward, given that Google collaborates with M-Lab. “That’s the sort of free and open discussion that you can have in that closed, trusted environment,” the ISP representative said.

The question now is how to deal with the “hopefully very uncommon scenario where irrecoverable data loss puts the publication of the report in jeopardy,” Schulzrinne told us. The group is discussing how ISP-run servers may be used as part of the testing process, he said. The principle Schulzrinne presented to July’s 25 broadband speed stakeholder meeting states that measurements to be used in the report will be collected from measurement servers “not under control of measured ISPs.” M-Lab’s ex parte filing said this is how it’s worked all along. Gideon said the principle contradicts another circulating proposal that would allow a certain pool of servers to be either the “best research platform servers,” run by M-Lab or some other entity, or the “best ‘public’ ISP-provided server."

"We are trying to identify ways to add redundancy to the system,” Schulzrinne said. Various “engineering discussion documents” submitted to the broadband speed test group to address the failure-mode operation of the overall system “will evolve over time and aren’t FCC policy documents,” he said. “As we have in the past, we look to the group to arrive at consensus that maximizes the credibility and reliability of the data."

The different parties still have a willingness to collaborate, said an FCC official who attended the recent stakeholder meeting. As the measurement process assumes a higher visibility, it brings out stronger feelings among the participants, the official said, but everyone still wants the process to continue. An ISP representative said he’s not sure how to regain the trust that originally existed. That trust requires everybody to agree on a way of operating, and any one party has the ability to blow it up, he said. “This demonstrates how unlikely it was that it ever worked in the first place.”