Location-enabled Society: Mapping tweets using real-time big data technology and privacy

In a very appropriately titled session Taking Society's Pulse in Real-time at the Creating the Policy and Legal Framework for a Location-enabled Society conference in Boston, one item really got people's attention focussed on the issues that arise from combining location with personal data. TweetMap is a technology that visualizes a large number of tweets on a map of the world. This is not the first time I have seen tweets mapped.  But what really got people's attention is that a speaker showed how to drill down to your neighbourhood and see what the people next door are tweeting about.  Of course you can do this without a map by simply following your neighbors on Twitter.  But somehow mapping it makes it seem more intrusive, even invasive to some people. In another demonstration, Tweetmap was used to show how a person sailing in the Caribbean could be tracked as they tweeted from their boat.

To be tracked geospatially on Twitter requires that you explicitly agree to geolocation tracking for Twitter on your iPhone or Android device.  For example, on an iPhone the geolocation option allows you to make your location known to others. Apps such as Facebook or Twitter for iPhone can report your location as well as the messages you post.  The iPhone requires that these applications obtain your permission before they can be enabled.  Apparently only about 2-4% of smart phone users, mostly younger folks, agree to this.  After seeing a Tweetmap drill down, I would expect that some people might have second thoughts about agreeing to the gelocation option. And in the not too distant future this will be possible in real-time on inexpensive hardware with open souce software.

TweetMap is an instance of MapD, a massively parallel database platform being developed through a collaboration between Todd Mostak, (a researcher at MIT), and the Harvard Center for Geographic Analysis (CGA).  Currently the tweet database starts on 12/10/2012 and ends on 12/31/2012 and contains 125 million tweets.  The database can be queried by time, space, and keyword.  The ultimate objective is to be able to query billions of tweets live in real-time.  In this case real-time means from tweet to tweet on a map in under a second.

MapD is a general purpose SQL database that can be used to provide real-time visualization and analysis of very large data sets.  MapD uses commodity Graphic Processing Units (GPUs) to parallelize compute intensive tasks such as querying and rendering very large data sets on-the-fly.  MapD runs on inexpensive hardware ( ~$1000) and is intended to be open source.