States Question NTIA Using NIST Model for FirstNet Funding Analysis

An NTIA spokesman said the agency based funding amounts on a technical model by the National Institute of Standards and Technology that considered many factors. NIST didn’t know NTIA used its analysis (NISTIR 8039) for developing funding determination, the NIST study’s author told us. The debate raises government transparency issues, especially for a project this important, said Sunlight Foundation Deputy Director Alex Howard. NTIA using the NIST study without consulting the researchers isn't an ideal process but also not unusual, a University of California-Berkeley geography professor said. FirstNet and AT&T didn’t comment.

Coverage in federal lands is critical, said Colorado Single Point of Contact Brian Shepherd. Federal land has “little to no population but … tremendous recreational usage,” he said. Colorado’s White River National Forest is one of the most visited parks, and public safety often must enter federal land to save hikers and others who run into trouble in the mountains, he said. Shepherd doesn’t believe NTIA or NIST included coverage on federal lands, leading to NTIA assigning proportionately less money to states with more federal land. Washington state shares Colorado’s concern about federal land and raised it with NTIA, said a spokeswoman for Washington Gov. Jay Inslee (D).

“NTIA stands by its decision to use the technical modeling approach because we believe it represents a reasonable and fair means of establishing the state funding allocations,” the spokesman said. To determine FLDs, NTIA used an estimate of the number of towers in each state that would be needed to achieve baseline coverage objectives, he said. The estimate was based on technical modeling from the 2015 NIST study, he said. That model looked at factors including how radio-frequency signals transmit and distribution of potential users, but not each state’s percentage of federal land, the representative said. We asked NTIA Friday to provide records detailing how it used the NIST data; the agency said it would treat our inquiry as a Freedom of Information Act request, and we also filed a FOIA which was received Monday.

Shepherd said the NIST model failed to include areas with population of less than 5 people per square mile, a classification that “essentially eliminates federal land.” The NIST study termed areas with such a low average population density as “rural low population.” The NTIA spokesman disagreed that those areas weren’t included, saying NIST used the population definitions in its study to differentiate propagation analysis and determine coverage requirements.

States Concerned

A comparison of Colorado and Alabama shows the federal-land issue, Shepherd said.

NTIA assigned a maximum grant of $88.5 million to Colorado, which is 66.5 million acres with about 36 percent of that federal land (23.9 million acres), according to Ballotpedia. Excluding federal land, Colorado has 42.6 million acres. NTIA set a maximum grant of $105 million to Alabama, which is about half the total size and excluding its 3 percent federal land is 31.8 million acres. “It appears the NTIA basically said ‘we're not going to cover federal land,’” Shepherd said. The Alabama Law Enforcement Agency has "no insight into how NTIA calculated Alabama’s FLD," said a spokesman.

We found examples that might not support Colorado’s theory, or at least show other factors. NTIA gave $158.9 million to Washington state, which is 42.7 million acres and 29 percent federal land (12.2 million acres). Comparing it to Colorado might support Shepherd’s theory, since Washington has half the federal land as Colorado. But NTIA gave more money to Washington than to the comparably sized Oklahoma, which received $68.8 million and has only 2 percent federal land. A big difference between Washington and Oklahoma is population -- Oklahoma has 3.92 million, Washington 7.3 million.

It’s not a “straight-line causation” and “there are likely multiple factors at play,” said Shepherd. The Washington-Oklahoma comparison doesn’t invalidate the concern, he argued. “States with high federal land, but also with dense metropolitan areas might ‘equalize’ a bit.” States “that got dinged the most,” including Montana, Nevada and Alaska, have “tremendous amount of federal land but also lack high density population,” he said. Washington, California and Colorado, “while containing a lot of federal land, also have population centers that helped offset that,” he said. Colorado and Washington may have gotten more money than Oklahoma despite having more federal land, but they also have much bigger metropolitan areas, he said.

Colorado and Washington are deciding whether to opt out of FirstNet. Colorado collected responses to a request for proposals earlier this year, and Washington issued a joint RFP with Oregon Friday. Colorado is continuing its review, Shepherd said: “We have provided the information to our two bidders and asked them to update any financial elements of their bid so we'll see what kind of impact it has.” The NTIA funding “is one of many factors the governor will consider when making his decision,” the Washington state spokeswoman said.

NIST Surprised

NIST didn’t know NTIA used its study to do funding analysis, said Nada Golmie, NIST Wireless Network Division chief of the Communications Technology Laboratory. “We were not aware of this nor familiar with the parameters and coverage objectives they used to produce the estimates,” said Golmie, one of the study’s authors. How the method is used is beyond NIST’s control and it’s not appropriate for the agency to comment on whether NTIA was justified, she said. The NTIA spokesman said no coordination with NIST was necessary because the NIST model is public and that agency possesses expertise and neutrality.

“We worked on the coverage analysis to better understand the performance trade-offs in terms of coverage and capacity and the impact certain parameters will have on the overall coverage and capacity," said Golmie. "We developed a method to conduct the analysis for the nationwide coverage and documented our results in the NISTIR 8039.” The NIST study wasn’t required by statute, she said.

NIST’s method “is simply an approach to conduct a nationwide coverage analysis given target performance objectives,” Golmie said. “Our goal was not to produce an absolute number of towers per area, but rather to understand the parameter implications and performance trade-offs in achieving coverage objectives.” The method is applicable to analyzing coverage for federal land -- and all other types of areas -- so long as coverage targets and parameters such as recreational users are considered, she said.

Once a model or study is out in the world there's nothing to stop someone else from using it "for some proximate purpose" that may not be what the original researchers intended, said David O'Sullivan, a UC-Berkeley associate professor specializing in geography, simulation and modeling. "In an ideal world, the people re-using a study would ask the originators about fitness for purpose," or build bespoke models to collect exactly the right data for their purpose, he said. "But people are busy, under time pressure [and] it can be hard to find the right person."

“Any time there’s a decision-making process about how a federal agency is allocating funds, a.k.a. taxpayer dollars, then people want to have some transparency,” Sunlight Foundation’s Howard said. It’s not inappropriate for one federal agency to use another trustworthy agency’s research, but it also should give “a full accounting of what else was taken into account, by whom and what their analysis of that data was,” he said. Process transparency is especially important for an issue like public safety communications that everyone agrees is critical to get right, Howard said. “Getting this wrong could have some bad consequences downstream for people we do not want to be without connectivity.”

It’s “always useful” to talk to the researchers behind a study one uses, Howard said about NIST apparently not knowing about NTIA using its 2015 model. “Any time you can talk to the person who did the study is an opportunity to understand what’s behind the paper … the strength of their conclusions, their trust in their own analysis, and whether anything else has happened since then that’s caused them to question it.”