Information overload and futility

I read recently about a guy who is keeping a video diary of his whole life. The idea is to wear a digital video camera and have it turned on permanently, streaming all the minutest details of his life to banks of hard drives. I’m glad technology now allows that kind of (useless and wasteful) stunt. Computers are becoming more powerful and more wearable each quarter, with so many cores, we don’t even know how to program them efficiently anymore. The Internet is bigger, faster and more ubiquitous, roughly 110 million websites these days, and 550 billion web pages in the so-called deep web. Hard drives have come a long way from the 80 lbs beasts they were originally (see them at the Computer History Museum, www.computerhistory.org), offering now 1.5TB in a 3.5 inch format. All those technologies are colluding to facilitate ever more massive data collection, storage, processing and querying efforts. We have entered an age of peta-scale (10^15) computation (IBM’s RoadRunner reached a peak of 1.7 peta-flops in May 2008) and peta-scale data sets which I’m finding hard enough to comprehend (and work with) when the data is meaningful and useful, but downright overwhelming, burdensome and even wasteful when the information is useless. Huge databases of genes and pathogens instantly accessible via the web are undoubtedly a significant progress. But chronicling Joe Schmoe’s life second by second with an always on video stream? In the end, I feel it destroys precious environmental resources, indirectly, invisibly and all too easily from the comfort of your armchair. YouTube has 84 million clips, and 13 hours of fresh videos are uploaded every minute. Are all those videos significant? Do I want to know about the latest silly prank of a teenager who doesn’t have anything better to do? Well, maybe actually. I know people who spy on their teenagers via the traces they leave in Facebook and YouTube. By the way, beware of errors, misinformation and manipulative propaganda when you access all this information. Not everything is accurate or neutral in this ocean of information that the Internet has become. I also have mixed feelings about the iPhone application that allows you to know in real time what your buddies are up to. Do I really want to know? Maybe not. Maybe they can have their life and maybe I don’t have any desire to know what they do each second of their lives. What are we going to talk about when we meet, if I already know everything they’ve been up to lately? Lastly, good old email has also become a prime contributor to information overload. They say the US economy wastes billions of dollars every year when we have to spend (too much) time wading through mountains of not so relevant emails, that were only to easy to generate and send. I have quite a few friends pulling their hair every day because of the hundreds of emails they have to read daily, and I’m not even talking about spam. Technology has made it easier to propagate information, almost too easy and seemingly inexpensive, but we should also think about the relevance of the information we post, and about the environmental footprint of it all.

TV sucks

We don’t have TV anymore! We should celebrate. We canceled our subscription to cable TV, and yesterday evening at 9:10 pm, the curtain fell and it was over. It was long overdue. I couldn’t stand the commercials anymore, and the sheer stupidity of most programs. I didn’t want to pay for a bundle that included QVC and other nonsense. I would have paid for a few select channels, like PBS, the History channel, and maybe TCM. But they don’t sell TV that way. Talking about the programs, the other day, I was shocked to see how insensitive the commercials can be. It was in the middle of Intervention, and the family of an alcoholic was trying to rescue their beloved one (OK, it is strange already that this kind of situation can be made into a spectacle), but zap! a commercial for Kentucky Fried Chicken cut in! I was disgusted. It didn’t make me feel like eating KFC at all, by the way. The channel probably sold the wrong time slot to KFC. In any case, I don’t want to pay $60 a month for that kind of crap, I have better things to do with my life and my money.

Open source

So there is this practice where software is free, the source code is readily available, and companies can make money on services, training and consulting rather than licenses on the software itself. I have mixed feelings about open source software. I don’t have a problem with proprietary software. It seems ok to me that some company would invest to develop a product, and would enjoy the income from licenses. At the office, my colleagues say that whether you like it or not, open source software is upon us. Maybe. They say that whatever software a company can develop, it will be replicated in short order by an online community of open source developers. Maybe not. It might be non-trivial to roll out really difficult software, like optimized linear programming solvers, for example. Probably open source software has been made possible by the fact that writing and distributing software might not be as capital intensive as producing cars, for example, which requires plants, machines and other capital items. Maybe it’ll spark a new kind of economy. But it’s not obvious to me that software should automatically be free, whether like free speech or free beers.

Monday blues

Monday morning, 8:00 am, and I’m by myself in the office, watching the coastal fog very slowly recede over the hills. At 8:00 am, I’m usually alone in the office. Geeks are not a matinal species. The office usually starts filling in after 10 am only. There are often checkins into our source control system at 1 am though.

The startup I work for is in the same building as a few financial outfits, a branch of Morgan Stanley, a Bank of America, and more on the second floor. The necktie wearing people I crossed on that floor this morning were bleary eyed and “wandering around in a daze”, as they reported in the New York Times. Lehman Brothers in bankruptcy, Merryll Lynch bought out by Bank of America. Don’t worry, the windows don’t open in this building, the branch managers on the second floor won’t be able to defenestrate themselves.

It’s weekly report time for me, and in this blog, I want to write about geeks and irrationality. I thought a bunch of seasoned software engineers would be a pretty rational crowd, but it doesn’t appear to be so. We are building a really complex system here at company X and in good old Cartesian fashion, I thought it would be efficient to decompose the system into modules and sub-modules, and test those separately. In 1637, Descartes was already engineering that way. His second precept in the Discourse on the Method is “to divide each of the difficulties under examination into as many parts as possible, and as might be necessary for its adequate solution.” Alas, my colleagues haven’t read the Discourse on the Method (probably too old), and they insist on putting the big, complex system together and then testing it with “real data”. They feel that testing is not worth doing if it doesn’t use “real data”. I agree with that, but I still want to decompose into modules, have clear specifications for the modules and write tests against those specifications. I’m concerned that if we throw everything and the kitchen sink into the System and then turn on the ignition key with “real data”, the System will simply not perform adequately, and we’ll have one giant Mess to dig into to figure out what is going on. I’m pretty sure we’ll be worth off if we can’t analyze our system in terms of simpler modules. Maybe there isn’t enough pressure on us software engineers to get it right the first time around. In other engineering disciplines, when building an Airbus for example, they can’t overlook their subsystems till it’s time to take off. Or maybe I should change my view of testing. For the components I write, I shall henceforth not test them but simply throw them into the System, and wait till the System flounders miserably. It’ll still be time to write specifications and to test then, right? In the meantime here is a law of software engineering:

Complex Software System + Real Data + No Analysis into Modules = One Giant Mess.

Culture of fear

I’m tired, and nothing works at the office today. I like Fridays though, I tend to be productive on Fridays, much more so than on Mondays. Mondays don’t work for me. Anyway, I was reading the news yesterday, and came across that tidbit in the warning ahead of Hurricane Ike: “Persons not heeding evacuation orders in single family and one or two story homes will face certain death.” Somebody went overboard at the National Weather Service, apparently. Maybe they are inflating the language just to make sure people get scared and evacuate. Then this morning I heard a report on NPR about Ike: in a dramatically raspy voice, the reporter was saying “if you wanted to hit the US where it hurts the most, you couldn’t choose a better spot”, talking about the oil refineries in the path of the hurricane. I guess you needed some forward looking imagination to try and come up with something scary that time: Ike is now only a category 2, after all. I think we live in a culture of fear. On CNN, in between news about hurricanes and other disasters, they now have segments about survivors of various tragedies: how Mrs X. survived a shark attack, how Mr. Y spent two months on a drifting iceberg in the middle of the Atlantic, whatever, as long as you get scared next time you board a plane or a ship. In fact, cultivating fear in the audience is a bona fide genre on TV, with series on mega-catastrophes, and ads that tell you (in black and white) that you could suffer from disease X or Y in the future if you don’t (buy now! and) swallow (in color) their drug. Culture of fear and culture of violence too actually, where some make the argument that if everybody was armed, nobody would get hurt. That will be the topic of another posting. In the meantime, I’m canceling my cable TV subscription, and I’m very careful about how much radio I listen each day, even NPR. It’s on the Internet too, but it’s easy for me to discriminate there and not be taken by that culture of fear. I’m wondering what it says about the country in terms of psychology and sociology. And who benefits from that climate of fear? Is it the various merchants who peddle their drugs in the ads? Ah, I know! Yes, fear sells. Probably because it suppresses thinking. The only good consumer is the non-thinking consumer. If fear suppresses thinking, let’s use fear to turn the masses into good consuming masses. Or is there something deeper? Well, I have to watch out not to fall into meaningless generalities. Not everybody is afraid all the time, for sure. But it’s in the media, that’s for sure too.

9/11

Such strange times. It’s 9/11 today. The country is at war, yet here in the Silicon Valley, you couldn’t notice it. Except that there are more defense related contracts, and that’s good for business, I guess. In fact, people are still very much into materialism. A new line of iPods was released on Wednesday. At the coffee shop next to the office this morning, I noticed so many iPhones. There was a geek, no doubt thinking he was so cool, who couldn’t help but touch the iPhone on his hip every now and then. While he was ordering his decadent triple-latte-no-foam-with-room-on-top, he caressed the precious jewel 3 times in 1 minute, I counted. Should this poor soul seek help for some form of an obsessive disorder? He is not alone, unfortunately, although usually, the severity of the disorder is less acute. At the office, almost everyone has now ditched their previous cell and bought an iPhone. What for? Me, I’m thinking about a $60/year pay as you go plan, no thrills included, no Internet, no text, nothing. I don’t use my cell, except for emergencies. I don’t care about the latest gadgets. The iPhone is definitely very cool, but I don’t need it. Or if I had one, I would spend too much time playing with it at first, then that would pass, and then I would start wondering why I spent so much money on the device. And I would have a hard time reconciling my behavior with what is going on in the rest of the world. Our world is so fractured. There is a lot of misery, yet here we bathe in materialism and nobody seems to care what is going on elsewhere. It feels so strange to me.

K-cups

Another day in the Silicon Valley... here we are, in the office of a computer science startup that shall stay anonymous. We just got a new coffee machine, actually, the n-th one in a series of new coffee machines, our office manager being under the influence of a coffee machine broker who desperately wants us to buy one of his devices. We keep sending the machines back, because they are usually not respectful of the environment enough, with all those little coffee bags or K-cups that you have to insert and then throw away each time you make a cup of coffee. There was a machine we all liked, that would grind the coffee beans on the spot instead of relying on wasteful little bags, but it was too expensive. Too bad, it was a programmable device, and some in the office had enough time on their hands to start hacking into it and commandeer it to produce unbelievably strong espressos. 

Anyway, the latest machine we are trying out uses plastic “K-cups”, that you have to insert, and remove, yourself. That is... we have lots of very smart people at the company, and some have even figured out that they can insert the K-cup themselves, but let others remove it for them! I was particularly annoyed with that, so this morning I sent an email to the whole gang asking if they could please clean up after themselves. I sent that email around 9 am, and at 10 am, I was back at the coffee machine for another Colombian Black Gold Extra Bold, and lo and behold, a French Vanilla K-cup, used of course, was there in the machine! I tried again at 11 am, and another K-cup (still French Vanilla) was in there, used again! So, people just didn’t care about my email. My take on that trivial episode is that people don’t care about being polite, or that I have a different notion of what it means to be social and polite. Geeks are not known to be a polite kind, after all. In my darker moments, I interpret the incident to mean that some are taking advantage of the fact that we are not looking for fingerprints on those K-cups to express their darker side, Shadow as Jung would call it. I'm glad they are not repressed. Sigh. Whatever.