Between the Poles

Streamlining connectivity for all with open, decentralized data

Guest post by Wade Anderson, Vice President, SMB, IQGeo

5G is predicted to have the same economic impact on the US as commercializing the internet did 25 years ago. The technology is the motor of a new data economy built around artificial intelligence, autonomy, cloud computing, and the Internet of Things. 5G is essential to produce and transport data and new knowledge faster and more seamlessly to support an increasingly digital and interconnected economy. The proliferation of IoT applications across industries is showing no signs of slowing down, presenting a huge opportunity for business growth. According to GSMA, the global IoT market will be worth $900 billion in revenue by 2025 – an almost threefold increase on 2019.

With global 5G adoption expected to almost triple in 2021, the distribution of telecoms networks will need to grow to match this projected demand. The US has ambitious plans to achieve nationwide 5G coverage through a combination of state incentives and market forces. The Biden administration pledged $100bn to improve broadband infrastructure across the nation, with the ultimate aim of providing high-speed broadband connectivity for all Americans by 2029. As a result of the COVID-19 pandemic, ensuring connectivity for all has risen further up the agenda for the telecoms industry and for municipalities, government agencies, universities and corporates that support their own fiber and 5G networks.

Major US telecoms providers are deploying superfast Ultra-Wideband 5G services across cities, stadia, and airports. The US is also ahead of many other advanced countries in allocating high-band and low-band spectra for 5G. North America has the world’s largest share of the market in ‘dark fiber’, unused additional capacity that could help share and spread 5G coverage. Yet the breadth of coverage is patchy and inconsistent and lags behind competitors such as South Korea and China. There has been growing frustration with the pace of the rollout across much of the US. There are also concerns over the lack of market diversity and the limited number of vendors following the US ban on Huawei. There remain major challenges in filling gaps in provision, particularly in remote and rural areas where major new infrastructure investment from large providers is unviable. Currently, it is estimated that over a third of Americans living in rural areas lack access to a quality standard of broadband.

Yet the expansion of high-speed broadband to all states and regions is essential to ensure equality of connectivity. To reach rural or small-town locations we cannot rely solely on major telecoms operators but need a diverse market of innovative providers, Wireless Internet service providers (WISPs), and organizations such as transport, governmental, and commercial operators willing to lease out bandwidth. These smaller providers can help complement large networks to reach under-served areas that may not be commercially viable for major infrastructure investment.

Tackling information silos to accelerate rollout

A major challenge that organizations continue to face is the existence of information silos which prevent organizations from planning the most efficient and effective paths to new markets and wider coverage. There is often a lack of structured network documentation as many smaller operators have not implemented a System of Record (SoR) which identifies an authoritative data source for each data element. In addition, there continue to be silos between workflows such as maintenance and construction and between datasets such as geospatial data and data such as asset records, repairs, or upgrades. Providers need to be able to integrate everything from meteorological to economic data with internal processes and workflows to spot new hazards or opportunities.

For example, data from the nearby natural or built environment, such as trees that could obstruct signals or tall buildings that could host 5G antennae, is crucial to maintenance schedules and planning new provision. Similarly, integrating accurate geospatial information on existing networks with data on nearby infrastructure would help organizations identify opportunities to lease out ‘dark fiber’ to others, thereby organically increasing nationwide 5G networks. Open, decentralized internal data is critical to speed up the deployment of 5G and foster a healthy, competitive culture across the industry. A diverse, dynamic telecoms market able to collectively achieve nationwide coverage will require more open and dynamic telecoms data.

To facilitate more agile broadband infrastructure planning, organizations must develop system of record (SoR) strategies and break down internal silos by encouraging efficient and joined-up data sharing. Operators should make their internal geospatial network data accessible to all departments so they can seamlessly incorporate external data such as street maps or satellite data into all their survey, construction, and maintenance processes. Geospatial network data forms a ‘digital thread’ connecting all external data to a specific location in the network and thus revealing the subtle interdependencies between networks and their local environment.

Opening up their internal systems so that disparate departments and divisions can seamlessly incorporate relevant external data into all processes. For example, IBM Weather Group’s LIDAR and satellite data on vegetation cover could be overlaid onto geospatial network data to help companies plan new 5G antennae to minimize signal obstruction from nearby vegetation. Similar organizations could open their datasets to telecoms companies to help create a complete picture of the hazards and opportunities to network expansion. By integrating external data with internal network information, new market entrants could see opportunities for efficient deployment of broadband into new geographies and accelerate the process of ensuring nationwide coverage.

A flow of telecoms data between departments enables open, decentralized information sharing. Open-source, geospatial data supports smaller and larger telecoms players to maximize the reach of high-speed broadband, addressing the growing digital divide of nationwide connectivity.

The ‘dark fiber’ opportunity

Accurate, real-time geospatial data can also help organizations spot opportunities to lease out bandwidth. Many organizations are sitting on goldmines of unused, ‘dark fiber’ network capacity that could be sold to companies or leased to nearby communities to increase coverage. We helped a large university in the western US audit and update their network asset records and uncovered over 200 underutilized ‘dark’ broadband fibers, helping streamline their network and identify extra capacity. They also used advanced geospatial information systems to predict future capacity and find the most efficient pathways to coverage across their campus.

The same data could be shared with telecoms operators or other organizations to help plot the most effective paths to extend coverage using existing capacity. North America already leads the world in the ‘dark fiber’ networks market and has an opportunity to leverage this to accelerate the nationwide 5G coverage without new infrastructure spending. Discovering unused extra broadband capacity in this way helps to increase efficiency for large and small operators alike by allowing them to extract maximum coverage from minimal resources and grow revenue potential in the process.

Other companies are digitalizing and decentralizing their network data and making it accessible to everyone from field technicians to call-centre staff. These companies are moving away from traditional centralized GIS, siloed spreadsheets, proprietary apps, or paper maps towards ‘open’ network data that can be accessed from smartphones and tablets or exported through open APIs. This means that data on current networks can be easily merged with other datasets to spot opportunities to maximize coverage with minimal resources and risk.

Achieving nationwide 5G coverage and the digital transformation of the economy will require a parallel digital transformation of telecoms, fiber and 5G operators. Companies will need to embrace an open, transparent approach to capturing and sharing data while creating the ability to capture and integrate data from across all departments and external sources to develop a holistic picture of the risks and rewards across every region. This new digital transformation will demand the democratization and decentralization of all organizational data so that every operator can see the clearest and quickest route to reaching under-served communities across the country.

U.S. FCC adopts new rules to protect net neutrality

February 26, 2015 the U.S. Federal Communications Commission (FCC) adopted new rules  to protect net neutrality.  This is in response to a court decision in April 2010, when the U.S. Court of Appeals for the D.C. Circuit decided that the Federal Communications Commission (FCC) did not have the authority to impose net neutrality regulations on Internet providers.  The decision was made in the context of a court case against Comcast for throttling certain internet applications.

The new rules reclassify broadband Internet access service as a telecommunications service under Title II of the Communications Act of 1934. Title II applies to "common carriers" which in 1934 meant telephone companies providing voice communications.  It explicitly prevents common carriers from discriminating among users to “make any unjust or unreasonable discrimination in charges, practices, classifications, regulations, facilities, or services.”  The new rules also cite Section 706 of the Telecommunications Act of 1996, which a court has interpreted as  an independent grant of authority to the FCC that supports the adoption of open Internet rules. The new rules apply to both fixed and mobile (wireless) broadband service.  Net Neutrality has been strongly supported by the Obama administration and content providers like Google.

Referred to as "Bright Line Rules", the new rules specifically prohibit blocking, throttling, and paid prioritization by broadband providers.

No Blocking

A person engaged in the provision of broadband Internet access service shall not block lawful content, applications, services, or non-harmful devices, subject to reasonable network management.

No Throttling

A person engaged in the provision of broadband Internet access service shall not impair or degrade lawful Internet traffic on the basis of Internet content, application, or service, or use of a non-harmful device, subject to reasonable network management.

No Paid Prioritization

No “fast lanes”.  Broadband providers may not favor some lawful Internet traffic over other lawful traffic in exchange for consideration of any kind.

The Open Internet Rules and Order can be found here.

FCC classifies broadband as a telecommunications service to protect net neutrality

In April 2010, the U.S. Court of Appeals for the D.C. Circuit decided that the Federal Communications Commission (FCC) does not have the authority to impose net neutrality regulations on Internet providers.  The decision was made in the context of a court case against Comcast for throttling certain internet applications.

Chairman Tom Wheeler of the FCC has just proposed new rules governing the internet that would strengthen the FCC jurisdiction over the Internet and enable it to set rules that protect net neutrality.  The new proposal by the FCC would reclassify broadband Internet access service as a telecommunications service under Title II of the Communications Act of 1934. Title II applies to "common carriers" which meant telephone companies or voice communications.  It explicitly prevents common carriers from discriminating among users to “make any unjust or unreasonable discrimination in charges, practices, classifications, regulations, facilities, or services.”  The Chairman of the FCC also cites Section 706 of the Telecommunications Act of 1996, which a court has interpreted as  an independent grant of authority to the FCC that supports the adoption of open Internet rules.

The proposal states that mobile broadband is included under Title III of the Communications Act the effect of which is to classify mobile broadband as a telecommunications service.  This means that both wired and wireless Internet would be covered by the proposed net neutrality rules.

Michael Byrne receives national award for development of the National Broadband Map

Michael Byrne, who was Geographic Information Officer (GIO) of the Federal Communications Commission (FCC) and led the development of the National Broadband Map (NBM), has just been awarded the Citizen Services Medal for creating a series of online maps (National Broadband Map and derived maps) and geospatial visualizations that helped people make informed decisions about the country’s communications systems.  It was developed in seven months from a standing start, launched Feb17, 2011, and got half a million hits in the first ten hours it was up.

The Service to America Medals are awarded every year to federal employees who achieve amazing results and epitomize public service.

The National Broadband Map allows users to use a web tool to search broadband availability across the United States and compare real download speeds to advertised broadband performance. All of the data is available for download. The map was part of an FCC stimulus (American Recovery and Reinvestment Act of 2009) initiative to improve broadband coverage/speed across the U.S.

The development of the National Broadband Map is a fascinating story.  The business objectives of the National Broadband Map, which were were mandated by Congress, included searchability, interactivity, responsive on-line access and a very tight development timeframe.

The data collection part of the story was a challenge because the FCC had to rely on the states to collect the data.  The FCC ended up with data on 3500 broadband providers, submitted by 50 states, 2 territories and the District of Columbia.  Michael's team had to take all this data in many different formats and load it into the NBM database. The end result was a database covering 1650 broadband providers with 25 million records showing where broadband Internet service is available, the technology used to provide the service, the maximum advertised speeds of the service, and the names of the broadband providers.

Michael and his team's biggest technical challenge was the tight development timeline.  The second major challenge was ensuring fast user response time. 

Seven months is an incredibly tight development timeline for any enterprise system.  Early in the development cycle, the technical team, which was comprised of only eight people including Juan Marín Otero, the lead geospatial architect, decided that a RESTful architecture and open source geospatial software were the only way they were going to complete the project within the timeline because they recognized that they had to communicate directly with the developers if they were going to resolve issues in days, rather than months. 

After deciding on the open source software and open data stack, the system was developed in four and a half months.  It survived a deluge of traffic in its first hours and days.  The first day the map went live, the site received 158 million hits.

The open source software and open data stack used for the NBM includes

• Apache Web Server
• AspectJ
• Cloudmade
• EHCACHE DX
• jQuery
• Geolytics
• GeoServer
• GeoWebCache
• Google Analytics
• Google Geocoder
• Google Maps API
• Hibernate
• Hibernate Spatial
• HSQLDB
• Jersey JAX-RS RI
• Junit
• Mapbox
• Maven
• Mockito
• MySQL
• OpenLayers
• OpenStreetMap
• PHP
• Spring Framework
• Wordpress

Open source geospatial developers will recognize OpenLayers, Mapbox, GeoServer, GeoWebCache, and Hibernate Spatial (on MySQL) software, Google Maps API and Google Geocoder services and OpenStreetMap geodata.  The software incorporates many Open Geospatial Consortium (OGC) geospatial standards.  For example, Hibernate Spatial uses the Java Topology Suite (JTS) as its geometry model. JTS is a Java implementation of the OGC Simple Features Implementation Specification for SQL v. 1.1 (SFS).

Obama reaffirms support for net neutrality

A central achievement of former FCC Chairman Julius Genachowski was to put together a set of regulations that required Internet providers to treat all traffic the same.  But the D.C. Circuit Court of Appeals while affirming the FCC’s power to regulate the Internet, overturned these FCC regulations that keep Internet providers from blocking or slowing down access to certain websites.  The court said the FCC has authority over Internet providers, but exceeded its jurisdiction by regulating Internet providers as heavily as it regulates telephone companies.  In practice this means that Comcast, Verizon and others can throttle certain Internet applications such as BitTorrent or charge sites like Google, Facebook and Netflix for different speeds.  The new FCC Chairman Tom Wheeler has advocated for net neutrality rules that include "fast lanes".

The President has made clear since he was a candidate that he strongly supports net neutrality and an open Internet.  Last week President Obama reaffirmed his position,

“One of the issues around net neutrality is whether you are creating different rates or charges for different content providers. That’s the big controversy here. So you have big, wealthy media companies who might be willing to pay more and also charge more for spectrum, more bandwidth on the Internet so they can stream movies faster.

I personally, the position of my administration, as well as a lot of the companies here, is that you don’t want to start getting a differentiation in how accessible the Internet is to different users. You want to leave it open so the next Google and the next Facebook can succeed.”

The President's statement has been quoted and supported by an open letter from a number of organizations including Mozilla, Etsy, Kickstarter, ACLU, and the Rural Broadband Policy Group.  The Internet Asssociation, which includes Amazon, ebay, Etsy, Facebook, Google, Yahoo, Uber, Paypal, Vimeo, Netflix, LinkedIn, Salesforce, Monster, Groupon and other internet technology companies, is also  record supporting net neutrality.

Infrastructure quality key factor determining location of real estate investment

The results of an analysis of a survey of 241 public sector officials and 202 senior-level real estate executives (developers, investors, lenders and advisors) conducted by the Urban Land Institute and EY have just been released as a report Infrastructure 2014: Shaping the Competitive City.

The survey was conducted in January 2014 and included public and private respondents in large and mid-sized cities across the globe, but primarily in the United States, Europe and Asia Pacific.

88 % of respondents rated infrastructure quality as the most important factor influencing the location of real estate investment and development.  Consumer demand was the second most important factor. Among public respondents infrastructure quality was rated as the most important influencer  (91 %). Private sector respondents assessed consumer demand and infrastructure quality as the most important factors (90 % and 86%, respectively).

The most important types of infrastructure determining infrastructure quality as determined by asking  repondents to prioritize infrastructure improvement were public transit, roads and bridges, pedestrian infrastructure, telecommunications including broadband, water and wastewater, and energy. 

1. Improved public transit services (bus and rail) 78%
2. Improved roads and bridges 71%
3. Improved pedestrian infrastructure 63%
4. Improved telecommunications infrastructure 58%
5. Better passenger connections 58%
6. Better water and wastewater infrastructure 52%
7. Better energy infrastructure 51%

 

U.S. has fallen to 16th in the OECD in broadband access

A very interesting article by Brendan Greeley in Bloomberg Businessweek points out that with respect to broadband penetration in the OECD, over the past decade the Unites States has fallen from 10th to 16th.  By 2013, France, Norway, the UK, Germany, Luxembourg, Finland and New Zealand have passed the United States. 

Back in 2003 South Korea was way ahead of everyone.  It had opened up the last mile of its existing cable and phone companies, and offered inexpensive loans to companies to invest in more fiber.  After that there  there were a number of countries including the U.S. at about 10 percent.  France, Germany, and the U.K. were under 6 percent.

At that time, the U.S.had an advantage - every house in the U.S. had a copper telephone line and almost every house had cable as well - which was not the case in the E.U.

By 2013 all the countries that are ahead of U.S. have either opened up the last mile to compeition or subsidized infrastructure by government or both.

• Switzerland opened up the last mile of its phone monopoly in 2007.
• Under a directive from Brussels, all EU countries have opened up the last mile for all monopolies.
• Sweden, in addition, aggressively invested at both the national and local level in rolling out more fiber.
• The U.K. split British Telecom into two separate companies, one that sells network access and another in the very competitive retail communications market.
• In France, in 2003 there was a single state monopoly France Telecom. Since then France opened up the last mile to open competitiion.  In 2013 France is now fourth in the OECD rankings.  It used to be 14th in 2003.  Brendan Greely mentioned Free, one of the startups that took advantage of open access to France Telecom's telephone lines and which offers 1-gigabit DSL, free calls to 108 countries, 197 channels, and a set-top box with a game console and a Blu-ray player—all for €39 a month ($53.60).  And without any special regulatory concessions, Free has also started investing in its own super-fast fiber connections to homes.

The problem is that the last mile of broadband is expensive: Two alternatives are either the government contributes via subsidies or the regulator forces the existing phone and cable monopolies to open up the last mile on a wholesale basis, enabling new companies to compete.  But the FCC in the U.S. has been trying a third alternative, an experiment to see what happens when you encourage monopolies in return for infrastructure investment.  The FCC has.allowed the existing phone and cable companies to hold on to their own last miles, competing only with each other and reinvesting their profits into more infrastructure.  Brendan Greeley's perspective is that the FCC's experiment has failed.   The U.S. has dropped to 16th place in the OECD and all the countries ahead of the U.S. in 2013 have either opened up the last mile or subsidized infrastructure by government or both.

D.C. Circuit Court of Appeals overturns FCC's net neutrality regulations

In April 2010, the U.S. Court of Appeals for the D.C. Circuit has decided that the Federal Communications Commission (FCC) does not have the authority to impose net neutrality regulations on Internet providers.  The decision was made in the context of a court case against Comcast for throttling certain internet applications. Net neutrality is supported by Vinton Cerf of Google and others and opposed by carriers like Comcast and Verizon.  The FCC in its role as regulator of interstate communications did have other options, for example, imposing common carrier-style regulation.  The FCC decided to classify Internet access as an “information service” rather than a “telecommunications service,” a classification that allows for more regulatory intervention and includes traditional telephone companies.

A central achievement of former FCC Chairman Julius Genachowski was to put together a set of regulations that required Internet providers to treat all traffic the same.  Now the D.C. Circuit Court of Appeals while affirming the FCC’s power to regulate the Internet, has overturned the regulations that keep Internet providers from blocking or slowing down access to certain websites.  The court said the FCC has authority over Internet providers, but exceeded its jurisdiction by regulating Internet providers as heavily as it regulates telephone companies.  In practice this means thet Comcast, Verizon and others can throttle certain Internet applications such as BitTorrent or charge sites like Google, Facebook and Netflix for different speeds.

In response to the rulling the new chairman of the FCC has said

...the Court of Appeals has ruled on the Open Internet (Net Neutrality) Order upholding the Commission’s authority to act under Section 706. The FCC is not going to abandon its responsibility to oversee that broadband networks operate in the public interest. It is not going to ignore the historic reality that when a new network transitions to become an economic force that economic incentives begin to affect the public interest. This means that we will not disregard the possibility that exercises of economic power or of ideological preference by dominant network firms will diminish the value of the Internet to some or all segments of our society.

At the present time, the FCC has not elucidated specifically what it intends to do.

Sam Pitroda on harnessing technology to transform Indian society

India is projected to become the world's 5th largest economy by 2020.  The IEA estimates that India will require 600-1200 GW of new electric power generation capacity by 2050.  But 30% of India's power generation is lost to "aggregate technical and commercial" (AT&C) losses. To address this challenge, the Union Government created the India Smart Grid Task Force, chaired by Sam Pitroda, advisor to the Prime Minister.  Sam Pitroda has a reputation for making change happen.  As an advisor to a previous prime minister, Sam Pitroda spearheaded the telecom revolution in India which in 20 years transformed India from a country with 2 million phones to 900 million phones.  He is a strong believer in the power of  technology for enabling change for the benefit and betterment of society.  Today at the ESRI User Conference Pitroda reiterated this theme.

Pitroda ponted out that India is a complex country, both geographically and with respect to the diversity of its people.  Technology has been the key driver in India since 1947 when the British left.  Agriculture, telecommunications, higher education, energy, and space technologies were among the initial focus areas. India is now able to feed 1.3 billion people and send satellites into space.  20 years ago there were 2 million telephones for about 800 million people. Now there are about 900 million phones, mostly wiereless, in India as a result of the telecommunications revolution that Pitroda helped to initiate.

The challenge that India is facing is 400 million people below the poverty line. Every national policy has to take this economic pyramid into account. India is also a very young country with 550 million people under the age of 25.  These demographics means that the Indian development model has to be different from the Western model.  It is simply inconceivable that every household in India (and China) will have a a car, pay $30,000 per year for education, and $70 000 for a bypass operation as in the United States.

The primary challenges that India faces are social equity, lessening the disparity between the affluent and the poor, urban and rural, educated and uneducated, reinforcing Indian democracy, and demography - creating economic opportunities for the young. 

In Pitroda's view the key to meeting these challenges is information.  People are poor because of a lack of information.  Access to information is essential for making democracy work. Information is also a great leveler which helps reduce disparity.  It is also going to help provide the 15 to 20 million new jobs that India needs to create every year.

Already most Indians have access to voice communications.  The next challenge is data communications.  India and tthe US jointly developed an open platform for sharing government data that is being made available to other countries.  At a cost of about $2 billion, India has developed a Knowledge Network linking Indian universities and technical institutes.  The next step is to create a fiber optic network linking Indian cities, towns, and villages, at an estmated cost of $7 billion over the next 18 to 24 months.  The long term objective is to empower over a billion people.

A key part of the the IT platfom is developing geospatial infrastructure including data and technology.  The XI Five-Year Plan mandated the use of geospatial technology in new mission-critical projects. The objective is to provide a seamless, nationwide, geospatially-enabled database, which will serve as a platform for data sharing amongst various national government departments and state governments. 

Several years ago it was realized that just about every department has its own base map.  The Water Department, the Revenue Department, the Gas Authority all had different base maps with different boundaries.  Information is power and no one wants to share.  It was recognized that it was essential to get everyone working together to agree on one common standard base map.  It took a year and half to achieve this.  It is not easy to get 30 different states some of which have populations of 200 million to agree on common standards and this requires changing people's mindset.  From Pitroda's perspective the problem is a 19th Century mindset, 20th Century policies, and 21st Century technology.  The solution is to get the younger generation to think differently.

Pitroda's vision is that a national information infrastructure based on a common geospatial platform will help make government more transparent and accountable, reduce corruption, reduce the waste of resources, empower citizens and provide a foundation for a uniquely Indian approach to innovation and economic development that will reduce disparity and create jobs for the young Indian workforce over the next decade or two.  Based on the Indian experience with revolutionizing communications, Sam Pitroda is optimistic that access to information incorporating geospatial techology will transform Indian society.  This is an incredible vision for transforming the lives of 1.3 billion people.

FCC shows latest U.S. broadband deployment progress in an interactive map

I've blogged about the FCC's broadband map before. Last year the National Broadband Map was launched with more than 25 million records showing where broadband Internet service is available, the technology used to provide the service, the maximum advertised speeds of the service, and the names of the broadband providers.

The FCC has just released an interactive, web-based map illustrating the broadband deployment results reported in the FCC's Eighth Broadband Progress Report. 

 

It shows census block areas of the United States with and without access to fixed broadband of 3 Mbps download and 768 kbps upload.  You can zoom into the map or pan around and explore details of broadband availability in each county in the United States.  These details include population and income numbers from the census, and the percentage of each county that has access to the major fixed technologies providing broadband service.  Perhaps even more importantly, the map provides full access to anyone with a web browser to the raw data used in the map and the Report.

A few other important points about the map.

• Accessibility: no software barriers, terms of service, or other restrictions prohibit use
• Machine readable: the map data is stored in a SQLite file.  SQLite is an open source project.
• Non-proprietary format: no other software is required to look at or analyze the data
• Open IP (License free): data is not subject to any further licensing restrictions.