I’ve been learning about “Big Data,” which is a hot topic these days. One thing I learned about it is that for every person who is qualified to get a job as a big data analyst their are 100 jobs! In other words there are not enough people to process the ridiculous amount of data in the world, and the job security and money is good if you can do it. To prepare for the possibility that this is something I would want to do I am taking an online Intro to Statistics course at Udacity.
I also learned just how much data we are talking about. According to a lecture I watched on the Ted youtube channel, the amount of data created in the year 2004 was equal to the amount of data ever created in the history of humankind from Dec 31, 2003 back to the first time any information was recorded. We continue to double the amount of data in the world. Companies and scientists are trying to make meaning out of some of those numbers. In this fascinating talk on Edge.org (below), Geoffrey West explains how he took data about biological beings, cities and companies and showed that many of the characteristics of each of these things scale in relation to each other. In other words, in a mammal, the length of all veins in the animal is directly proportional to the weight of the mammal, regardless of species. The same is true of length of roads/ population of a city, and the resulting proportions are the same. He explains it way better than I can. I was really blown away by the idea that biology and data analysis can be used to understand cities!
I also was very impressed by another talk on Edge.org (actually quite a few, I have added Edge.org to the resources list, and thank you very much for the recommendation Grandpa Hans!). In this talk, Alex Pentland, Director of the Human Dynamics Laboratory MIT Media Lab, talks about how we can use big data to redesign many of the systems we use every day to function more intelligently and more humanly. He talks about the issue of who owns our data and how it might be regulated. He also talks about how big data can enable the owners and analysts of that data to track and control people to a unimagined degree, and why he thinks the future will go a different way:
You could build something that to a first approximation would be the real evil empire, and of course people are going to try to do that. At the same time there are some elements of these forces that are really promising. For instance the architectures tend to be things that have no central points, which means that there is no place for the dictator to grab. They have to actually go to every house to do it…. It tends to dissolve the power of the state and the organization. Because you can build things that are far more efficient if they are distributed without the hard boundaries that you see today. That means that the service oriented government, or the service oriented organization will tend to have better services for less price as opposed to the one that tries to own the customer or control the citizen. So I expect to see that these kinds of hard boundaries will dissolve because there will be competition from things that are better that don’t have the hard boundaries.
In this he explained and expanded on something I have been thinking about a lot, without being able to explain it very well. My thoughts on this were energized by the idea of wifi mesh networks, which is a true peer to peer internet that doesn’t rely on fiber optic cables or large servers, but allows individual computers to talk to each other and exchange information. I realized that if the internet can be designed without a “place for a dictator to grab” probably other things could to. The internet is one place where the NSA has grabbed us for the time being. We are also “grabbed” by energy companies because we are dependent on them for our everyday needs. If every house has its own solar electricity set up, the powers that be lose that hand hold as well. Same goes for big banking vs community banking, and the Federal Reserve vs. crypto-currency. I always suspected that this would be not only a more just way of doing things, but also be far more efficient than top-down control, and Pentland expressing that the data supports this is very encouraging.