Regular Expression

Ship Code When It Is Ready

2012-01-01T15:42:00.000-08:00

My previous jobs had what I consider a fairly standard month long release cycle. The first three weeks of the were spend writing code. On the fourth week the code was dropped to QA where they would hammer on it, find bugs, and have developers fix them. At the end of the fourth week the code shipped.

I distinctly recall the final week before every release having an uneasy feeling in my stomach. Even though I had thoroughly tested my code and QA had signed off on it, I always worried that it would break horribly in production. Did I miss a corner-case? Was someone integrating with my code going to break it? Had there been regressions in the mad rush to get fixes in before the release? I’ll never forget that feeling....

That is why I was really surprised when I started working at turntable.fm. I distinctly recall when I finished my first (tiny) feature. After quadruple-checking the code to make sure all the digital i’s were dotted and t’s crossed I sent it off to code review. Another developer took a look and said “Yep, looks good.” I naively asked, “when is the next release?” To which he responded “Right now.” Uh oh, I was getting that feeling again... but just a minute later the code had been deployed and there was my shiny new feature staring me in the face, working just as I had hoped. It was a special moment.

This had a profound effect on how I think about deploying code. I often get asked what is our release process, to which I reply “we ship when the code is ready.” It’s just that simple.

I’ve observed many benefits of this release strategy:

Instant gratification. There is nothing like the constant positive reinforcement of frequently shipping shiny new features. As a developer this kicks my motivation into overdrive.
You’re users get a constant stream of new features rather than having to wait for what amounts to a “monthly feature dump.” After you push new features and ask your users if they can spot them, you’ll be surprised at how quick they identify them.
When each feature is pushed separately it is a lot easier to gauge your users response. Just like with A/B testing, to get a clear response you want to minimize all other factors.
All the integration and testing is done by the developer shipping the feature. This means if anything goes wrong its obvious who should be investigating the issue.
If there are problems it is easy to roll-back since your change is that only thing that got pushed.
Debugging any issues with the code is easier since the code is still fresh in your mind. Getting all of the code you wrote a week (or a month) ago back in your brain can take longer then you would expect.

Do you ship code when it's ready?

Managing Talent: A Running Metaphor

2011-11-27T19:23:00.001-08:00

When I first joined the cross-country team in high school I naively thought that running was all the same, you just ran. Then came the first day we ran hills. It was then that I realized that running has a lot more technique and skills then were initially apparent. One of the most important things I learned was how run down hill.

On that first day of running hills our coach told us to “really work it going up the hill and recover on the way down.” After reaching the top of the hill I began my “recovery” back down the hill (i.e. jogging slowly). However, I noticed that everyone else was flying passed me on the way down the hill. Not only that, but when they reached the bottom they seemed even more rested and recovered than I was.

After practice I asked one of the guys on the varsity team why everyone’s “recovery” speed was so much faster than mine. In true Southern California surfer fashion he said “Dude, you just got to learn how to let your legs go.” Let my legs go, what the hell does that mean? Is that some sort of weird surfer zen thing?

What he meant was, when running uphill you really have to churn your legs to keep moving. However, when you reach the top and start going down, your legs are already churning. With the help of gravity, your legs will keep churning unless you actively try to stop them. See also: Newtons first law. It is the same effect you get as when you drive down hill and put your car in neutral.

Once you figure out how to “let your legs go” you just have two things to worry about. The first is that you regulate your speed. Your legs can take you far faster than you can safely go, so you should constantly be regulating your speed to go as fast as is safely possible. The second thing is avoiding obstacles. It only takes a small rock or divot to catch you off guard and send you tumbling down a hill. The further away you see the obstacle the easier it is to avoid.

Managing talented people fundamentally uses the same technique as running down hill. Imagine your team as the legs and you, as the manager, are the head. When you have a talented team, they should be able to move incredibly quickly. While their ability to move fast is a good thing, it is your job as the manager to make sure that your team is going as fast as they can while still being safe. If you allow the team to move too fast they will eventually trip and fall. Hopefully its just a scrape or a bruise and they can get back at it. But sometimes the injury can be really serious and the recovery process is a long and painful one.

Avoiding obstacles is the second skill you need to master. As the manager you are the vision. You see whats coming both short and long term, it is your job to steer the team appropriately. You have to make sure you are tracking what is right in front of you as well as what is on the horizon. Its important to remember that the further away you see an obstacle, the more smoothly you can steer the team around it. If you are really good, your team wont even notice the small course correction you made a while back that let you “effortlessly” avoid the giant bolder that would have eventually blocked their way.

If there is one piece of advice I can give you about managing a talented team it is: “let your team go.”

Eatability Testing: Why don’t more restaurants do it?

2011-11-27T19:09:00.001-08:00

Two things I’ve observed about people in New York is that everyone jaywalks and no one cooks. Everyone eats out all the time. However, since most people are busy, a large amount of food is served via takeout and delivery (as opposed to eating in). This presents a challenge that few restaurants seem to realize: the long delay between when the food is prepared and when it is actually eaten.

How many times has this happened to you, it is late on Sunday evening and you call one of your favorite restaurants for a little delivery. You pick your favorite item off the menu, place your order, and wait in anticipation. Thirty minutes and a cash exchange later and food is on your dinner table. You pop open the styrofoam container, grab a bite, and yuck, it’s cold.

Across the street from where I work at there is a fantastic Vietnamese sandwich shop called Num Pang. Their menu is filled with tons of delicious sandwiches like ginger barbecue brisket and hoisin veal meatballs. The trick with eating Num Pang is that you have to eat it immediately. In the 10 short minutes it takes to get your order and walk back to the office, the bread has already started to become soggy. If someone happens to catch you in the hallway you might as well say goodbye to your sandwich and hello to a (not so) hot mess.

This begs the question: why don’t more restaurants and takeout joints test the experience of eating food from their establishment? Prepare an order, put it on the counter for 20 minutes, and then have the chef eat it. Better yet, have the delivery person take it out with an order and then bring it back. I’d imagine most chefs would be surprised at the “presentation” and taste of their food after it has taken a bike ride through NYC in the winter. What was once a nice pad thai dinner will likely have turned into a cold ball of yuck.

Why don’t more restaurants do this? I think many restaurants are in for a rude awakening when they start eatability testing their menu.

Here are some of the most common missteps I’ve seen:

If you are serving something on bread (hamburgers, sandwiches, etc) then any sort of sauce or liquid on it is going to make the bread soggy within 5 minutes. If you can, just put the sauce in a container on the side. I’m sure some health-conscience customers would appreciate it as well.
If you have to serve it with sauce, make sure to put the sauce as far away from the edges of the bread as possible. Otherwise, the sauce leaks out of the sides and make the top and bottom of the bread soggy as well.
Separate hot and cold, just like you would at the grocery store. Styrofoam insulates surprisingly poorly, so packing a cold dessert on top of a hot soup is a bad idea. Account for leakage. Some well placed napkins can go a long way.
Account for the cooking and cool down that happen during travel. Talk to any chef in a food competition and they will tell you that residual heat can have a large effect on the taste of a meal. Account for the 10-20 minutes of delivery time for all to-go orders.
Test & iterate. There will always be surprises, so eatability test with different items on the menu to make sure that it is as good as you expect it to be.

Soundtracked: Understanding how we consume music

2011-07-10T22:32:00.000-07:00

Lately I’ve become fascinated with music, particularly how people talk about and discover new music. For the last few months I’ve been talking to just about anyone and everyone about music and I’ve learned some fascinating things which I plan to write about in a few blog posts. Here is the first set of observations I’ve made about how we consume music.

Music genres are useless

Just as an experiment, go ask a few people what type of music they listen to. Chances are they responded with something frustratingly generic like “I listen to rock music” or “I like hip-hop.” Given that, do you feel like you have a strong sense for that persons musical taste? Didn’t think so. While most people can describe their music tastes in terms of genres, for the person hearing those genres, they typically have very little meaning. This is becuase high level genres have become so overloaded that they could mean practically anything. The Foo Fighters (stadium rock), Pink Floyd (progressive rock) and The Decemberists (folk rock), and fall under the “rock” genre, yet in terms of sound they are miles apart. That is why I no longer ask people what type or genre of music they listen to, it's useless.

Finding out what people actually listen to

Now that we know music genres are useless, how do you actually ask someone what type of music they like and get a useful response? Initially I tried asking people what their favorite bands/artists were. Interestingly, I found that people struggled mightily with this question. There was often a good minute of hems and haws before I would get even a single band. After a lot of thought many people would end up with five or so bands and then end by saying something to indicate that they weren’t happy with their response like “yeah, I guess that is it...” Not happy with the result, I experimented with a few other questions.

The most effective way I’ve found of is asking them what are the last few bands/artists they’ve listened to. Most people have no problem rattling of the last three or four bands they listened to, and while its a biased sample, it's far easier for them to answer and much more useful then genres. For example, if someone tells me the last few artists they listened to were Fleet Foxes and The Freelance Whales I’d immediately know they they like softer indie rock, with great vocals, lots of harmonies, and acoustic guitars. It’s so much more accurate and useful then had they said “I like indie rock.”

New is a relative term

When people are sick of their current music they will often ask their friends if they know any “new music.” Most people interpret to mean any music that has been recently released. The reason they ask for this is that they want music they haven’t heard before, not because they have some aversion to older music. That's why even when people are asking for new music, you can generally interpret that to mean “new to them” rather than new in terms of release date. The beauty of music being “new to them” is that even if you know something is old, as long as they haven’t heard it yet, it is new to them. For example, when people ask me for Taking Back Sunday-style rock music I’ll recommend We Are Scientists - With Love and Squalor. Even though the album was released in 2005 the band was still pretty small at the time so most people didn’t (and still don’t) know about them. Score one for the relative new-ness of music.

That is all for now.

Web Application Caching Strategies: Generational caching

2011-06-05T20:33:00.000-07:00

This is the second in a two part series on caching strategies. The first post provided an introduction and described write-through caching.

In the previous post in this series on caching strategies we described write-through caching. While an excellent strategy, it has a somewhat limited application. To see why that is that case and how generational caching improves upon it, let's start with an example.

Imagine that you are creating a basic blogging application. It is essentially just a single “posts” table with some metadata associated with each post (title, created date, etc). Blogs have two main types of pages that you need to be concerned about, pages for individual posts (e.g. “/posts/3”) and pages with lists of posts (e.g. “/posts/”). The question now becomes, how can you effectively cache this type of data?

Caching pages for individual posts is a straightforward application of the write-through caching strategy I discussed in a previous post. Each post gets mapped to a cache key based on the object type and id (e.g. “Post/3”) and every update to a post is written back to the cache.

The more difficult question is how to handle pages with lists of posts, like the home page or pages for posts in a certain category. In order to ensure the cache is consistent with the data store, it is important that any time a post is updated, all keys that contain the post need to be expired. This can become difficult to track since a post can be in many cache keys at the same time (e.g. latest ten posts, posts in Ruby category, posts favorited by user 15). While you could try to programmtically figure out all keys that contain a given post, that is a cumbersome and error-prone process. A more clean way of accomplishing this is what is called generational caching.

Generational caching maintains a "generation" value for the each type of object. Each time an object is updated, its associated generation value is incremented. Using the post example, any time someone updates a post object, we increment the post generation object as well. Then, any time we read/write a grouped object in the cache, we include the generation value in the key. Here is a sequence of actions and what would occur with the cache when performing those actions:

By including the generation in the cache key, any time a post is updated the subsequent request will miss the cache, and query the database for the data. The result of this strategy is that any time a post object is updated/deleted, all keys containing multiple posts will be implicitly expired. The reason I said implicitly expired is that we never actually have to delete the objects from the cache, by incrementing the generation we ensure that the old keys are simply never accessed again.

Here is some sample code for how this could be implemented (in psuedo-Python):

After implementing this strategy in multiple applications I want to point out a few things about how it works and its performance properties. In general I have found that using this strategy can dramatically improve application performance and lessen database load considerably. It can save tons of expensive table scans from happening in the database. By sparing the database of these requests, other queries that do hit the database can be completed more quickly.

In order to maintain cache consistency this strategy is conservative in nature, this results in keys being expired that don’t necessarily need to be expired. For example if you update a post in a particular category, this strategy will expire all the keys for all the categories. While this may seem somewhat inefficient and ripe for optimization, I’ve often found that most applications are so read-heavy that these types of optimization don’t make a noticeable overall performance difference. Plus, the code to implement those optimizations then become application or model specific, and more difficult to maintain.

I previously mentioned that in this strategy nothing is ever explicitly deleted from the cache. This has some implications with respect to the caching tool and eviction policy that you us. This strategy was designed to be used with caches that employ a Least Recently Used (LRU) eviction policy (like Memcached). An LRU policy will result in keys the with old generations being evicted first, which is precisely what you want. Other eviction policies can be used (e.g. FIFO) although they may not be as effective.

Overall, I’ve found generational caching to be a very elegant, clean, and performant caching strategy that can be applied in a wide variety of applications. So next time you need to do some caching, don’t forget about generations!

Web Application Caching Strategies: Write-through caching

2011-06-05T20:25:00.000-07:00

You have probably heard about all of the big websites which heavily rely on caching to scale their infrastructure. Take a look at the Wikipedia entry on Memcached and you will find a veritable who's who of big internet companies. While we know these websites do caching, I have found very little written about how they actually do caching. Unfortunately a lot of caching is so particular to the individual website that it isn't very useful for most developers. There are however a few overarching caching "strategies" which if done correctly will guarantee you will never get stale data from your cache. In this series I'm going to discuss two of these strategies, write-through and generational caching.

The first and simplest strategy is write-through caching. What happens is when you write to the database, you also write the new entry into the cache. That way, on subsequent get requests the value should always be in the cache, and never need to hit the database. The only way a request would miss the cache is because a) the cache filled up and the value was purged or b) server failure. Here is some sample code in Python for how this works. I'm using database.get/put/delete as shorthands for SELECT/INSERT/DELETE in your database of choice.

The strategy itself is really simple to understand and for many workloads can result in dramatic performance improvements and decreased database load.

While a simple and clean caching strategy there are a few things you should be aware of to avoid some common issues when implementing this strategy.

Often times people will cache database objects by using the database id as the key. This can result in conflicts when caching multiple types of objects in the same cache. A simple solution for this is to prepend the type of the object to the front of the cache key (e.g. “User/17”).

Next, for any put/delete operations to the database it is important it check that those operations completed successfully before updating the cache. Without this type of error checking you can end up in situations where the database update failed but the cache update happened anyways, which results in an inconsistent cache.

While this strategy is effective for caching single objects, most applications fetch multiple objects from the database (e.g. get all books owned by "Joe"). For a strategy which handles multiple objects, see the next post in the series about generational caching.

2011 PaaS Predictions: Winners and Losers

2010-12-25T21:21:00.000-08:00

This past year has been an interesting one in the cloud space, and particularly for Platform-as-a-Service. While PaaS has certainly increased in popularity over the past year, I think it is still at the bottom of what is going to be a hockey-stick like growth curve. That obviously means that 2011 is going to be an even bigger year for PaaS. Given that is the case I wanted to put down my predictions for what I think it going to happen in the space over the next year.

Heroku

Heroku’s $212 million dollar acquisition by Salesforce was probably the biggest news of the year in the platform space. Not only is it a huge win for one of the most popular platforms, it is also a validation of the fact that the space is going to be huge. However, any time a small agile start up gets bought up by a giant corporation there are obvious concerns that they will lose their agility and get out of touch with the customers who made them successful. Given what I’ve seen from the Heroku guys so far I think they have a strong enough vision and understanding of their customers that they will be able to fend off the temptations of SF execs to meddle with their newly acquired toy.

I also suspect that we will see Heroku start getting a little more enterprisey in their offering over the next year. Unfortunately I don’t think they will see a lot of uptake since most enterprises move slower than molasses on a cold day.
Overall: OK

Google AppEngine

In addition to having incredibly liberal quotas, AppEngine has long been one of the most feature rich platforms available. Yet, there have been two main issues which really held back what would have otherwise been incredible growth, performance and datastore.
One of the most common complaints I have heard from developers using AppEngine is poor performance. This issue came to a boiling point earlier in the year when datastore latencies got so bad that Google stopped charging for Datastore CPU usage. The datastore issues were subsequently resolved and Google was able to deliver a substantially faster datastore API. Following this issue I think over the next year Google will start drilling down on other performance issues and deliver a much more performant and stable platform.

The other main point of friction for datastore has been the fact that it relies on a NoSQL like data model. While NoSQL is all the hotness right now, the vast majority of developers are still not comfortable using non-relational data models. That is why the announcement of a SQL database as part of AppEngine for Business came with much celebration. While it is still in private beta I would suspect that we will see it publicly released this year which will bring in a new wave of customers ready to take advantage of the large quotas and familiar SQL tools.
Overall: Very good

Djangy

While AppEngine is a great platform for Python applications there has been pent up demand for a Python/Django version of Heroku. New startup Djangy looks to be the first viable attempt at filling that gap. Looking at their documentation they seem to have taken the “Heroku for Python” manta very seriously, and that is a good thing. While Djangy is currently in private beta I’m guessing that they will be gearing up for public launch in the next 3-6 months. After that I think we will see quite a few Pythonistas migrate over to Djangy so they can get back to using the tools they are familiar with (Full Django + SQL).
Overall: Very good

DotCloud

DotCloud is a new Y-Combinator startup which aims to bring more flexibility to the platform space. Rather than a platform being married to a single language and database technology they aim to allow you to mix-and-match many languages (Ruby, PHP, Java, Javascript) and database technologies (MySQL, PostGREs, Redis, MongoDB). It is the choose your own adventure version of cloud platforms. Unfortunately I think using Dot Cloud will be just that, an adventure, and not the good kind.

As I discussed in my previous blog post, fragmentation is a huge issue for platform providers. I was discussing fragmentation within the context of a single language but the problem absolutely explodes when you start mixing in multiple languages with multiple databases. Consider the fact that even Google, a company with a tremendous amount of technical resources, only supports two languages and one database. While their vision is great one, I think they are going to run into some issues. In particular, they are going to reach a very unfortunate fork in the road. One path will be to only do barebones support for each language/database and risk frustrating customers with their lack of options. The alternative path will be to provide more in-depth support for each component and have to live the burden of regression testing the myriad of combinations possible, destroying their agility. Frustration seems inevitable either way.
Overall: Poor

Platform as a Service (PaaS) Fragmentation

2010-12-24T23:06:00.000-08:00

You will find no shortage of cloud pundits around that will tell you that the Platform layer is the future of the cloud stack. The thought is that in the future we will be able to forget about the menial tasks of managing servers and just worry about developing applications. While a pleasant vision, let’s take a walk through the cold harsh reality.

In the PaaS space there is a big elephant in the room, it goes by the name fragmentation. Stop and think about it for a second, how many popular programming languages are there? How many popular frameworks does each one of those languages have? That's not to mention the hell that is versioning of both of those things.

Lets look at a quick example to illustrate this point. You’ve decided you want to create a platform, the first question is then what language you want your platform to support. After considering the options you decide on your language of choice. Depending on which part of the language release cycle you are in you are probably looking at a decent split between the current version, the previous version, and the experimental new version. Unless you get lucky in terms of timing or are particularly aggressive in deprecating versions, you are probably going to want to support all three versions. We see this exact trend in both the Ruby and Python worlds. Note that I haven’t even considered any alternative runtime implementations (e.g. Ruby Enterprise Edition, Python Unladen Swallow). Next is the choice of web framework(s). Almost every popular web programming language has a myriad of web frameworks to choose some. You have the most popular framework (Rails, Django), the minimal framework (Sinatra, web.py), and a whole bunch of other long tail frameworks. Depending on the language you could easily be looking at 3-5 viable contenders for frameworks, not to mention versioning. I could keep going, but I think you get the point.

This ties into a recent Twitter conversion where @georgevhulme posed the question:

“Who will win the Paas battle next year, and become the dominate platform?”

In my opinion, the answer to this question is none. No single company will become the dominant platform in the next year, or even next five years for that matter. The PaaS space is simply too fragmented for any one company to own a substantial portion.

Better Cache Busters (aka Asset Timestamps)

2010-09-16T22:01:00.000-07:00

If you read any article on web performance they will almost undoubtedly mention expire-headers. They effectively tell the browser not to bother asking the server if an asset (e.g. an image or stylesheet) has changed until the expiration time. Ideally they should be set for many years in the future so that client browsers aren’t spending tons of time waiting for 304 Not Modified responses. The problem with expires headers is that when the file changes you need some way to signal the browser to fetch the updated file. The general trick is to change something in the URL which makes the browser think it is a new file and hence fetch it from the server. This is often called a cache-buster. That is why when you pop open Firebug you will often see files named puppies_3123141.png or styles.css?12345678.

Now that we know something needs to be added to the URL, the question becomes what should we add? There are three properties that we are really interested in:

1. The value should (only) change when the contents of the file changes
2. The value should be consistent across different machines
3. It should be fast to compute

The first possibility is to take the approach used by Ruby on Rails which is to use the modified time of the file. While this satisfies property 3, it does not work for 1 or 2. It breaks property 1 because appending an empty string to a file causes it modified time to update without the actual contents of the file changing. Property 2 is a much larger issue which many people have to face. Static assets are generally served from multiple machines, that means the modified time needs to be consistent across all of them. This is very difficult to achieve, particularly when the files are under version control. While modified time isn’t a great solution, it does bring us to another possibility, version control ids.

One of the great things about version control systems is that in order to give you a reference to a particular commit, unique ids have to be generated for each one. Given that is the case, we can simply use the current commit id (or hash for you git’ers) as the cache buster. We are currently on commit X, I update a file and commit it, we are now on commit Y. Since all machines will be updating their files from version control, everyone should be on the same commit id. In terms of speed, its not necessarily the fastest (especially if you are using SVN) but it only needs to retrieved once since it is the same for all files. The problem with commit ids is that it violates property 1. When a single file changes in the repo, every files’ id changes. That means that every time you deploy new code for your webapp, each client is going to fetch all the files again, even if all you changed was a README. Getting better, but still leaving something to be desired.

The last possibility I am going to talk about is an oldie but goodie, the MD5 hash. The cache buster of an asset is simply the MD5 hash of the asset itself. It satisfies property 1, and as an added bonus if the file is ever changed and then rolled back the MD5 hash will roll back with it (git reset anyone?). Property 2 is no problem, the contents of the file is the same across the different machines, hence so is the MD5. The only thing left is property 3, speed. Clearly computing the MD5 is going to be more time consuming than fetching modified time. However, just about every language has a standard hash library written in C for computing MD5’s and its pretty darn fast. The only place I could see this being an issue is if you have very large files or a extremely large number of them. Even still, you can just write a deploy script that precomputes all the hashes beforehand.

Overall, using MD5s as a cache buster give you all of the nice properties you could want in a cache buster with very little drawback. I went ahead and wrote a monkey patch for Rails that changes the asset id method to use MD5’s, the source code is available here (http://pastie.org/1164279). Enjoy busting caches.

Twitter is a bar and other Social Network metaphors

2010-06-16T11:26:00.000-07:00

I have spoken with a few people recently about social networks and how they compare. In explaining it, I’ve settled on a few metaphors that most people seem to understand pretty intuitively. Here goes...

Twitter is a bar

Twitter is your neighborhood Irish bar on a Friday night. It’s busy, noisy, and there are tons of conversations happening simultaneously. This makes Twitter a great place for short, and casual conversations, particularly since there are a bunch of people around to talk to. With the many conversations happening within ear-shot, it’s easy to catch something of interest and hop in to another conversation at a moments notice.

However, the raucousness makes Twitter a difficult place to have an extended conversation. It’s certainly not the place to (at least easily) discuss the intricacies of foreign policy or War and Peace. The people who try to engage in these conversations generally are quickly frustrated and leave for other establishments.

Facebook is a coffee shop

Unlike the raucous and noisy crowd of Twitter, Facebook presents a much quieter and civilized environment akin to that of your local coffee shop. Plush and comfortable surroundings make Facebook a nicer environment for conversing with friends. Conversations can flow freely but be careful what you say since your parents or kindergarten teacher may walk in at any moment. While some people are there to chat with friends, others are simply hanging out and taking advantage of the free amenities (e.g. photos).

LinkedIn is a library

Last but not least, LinkedIn is your local library. Walking into the library the silence is almost deafening. The whispers of others serve as reminders that a few people occasionally visit this establishment. The library’s biggest asset is that it’s filled with valuable and accurate historical information. Yet, looking as this information you get the sense that it’s stale and rarely updated. While you may not be able to find out the latest news trends, if you want to know where your competitor’s CEO went college, this would be a good place to look.

So where do you want to hang out?

Killing Multithreaded Python Programs with Ctrl-C

2010-05-17T20:54:00.000-07:00

If you have ever done multithreaded programming in Python you have probably found it frustrating that you can't simply hit Ctrl-C in the terminal and have it exit like a normal Python process. Instead you have to put the process in the background (Ctrl-D) and then either "kill %%" or kill the PID. The good news is that it doesn't have to be this way. After experimenting a bit I finally figured out why it doesn't work normally and what you have to do to make it work.

Normally when I write a threaded program in Python it looks something like this...

The problem is with this program is that if you hit Ctrl-C it doesn't do anything. The reason is that join() is a blocking operation. As a result the process will only receive the signal for Ctrl-C when join() becomes unblocked, which in this case will never happen.

In order to handle Ctrl-C with multiple threads you can use the following code:

This code does a few things differently in order to make it handle Ctrl-C as we would like. First, instead of using join() we use join(timeout) which tries to join a thread but will timeout if it does not occur after the timeout elapses. This allows the main thread of execution to continue doing other things, in particular waiting for a KeyboardInterrupt to be thrown which is what Ctrl-C raises. Since join will return upon timeout, we need to keep any threads which aren't None and respond to isAlive().

The next thing is that if child threads never return or take a really long time you need a way to notify the child that it should die. This is accomplished by the kill_recieved flag in the Worker class. When that flag is set by the parent process the child knows that it should finish up what it is doing and return.

The last thing is something that caught me off guard a bit. Initially, in the main() while loop I was trying to catch all exceptions that came up by using "try...except Exception:". As it turns out Exception does not include KeyboardInterrupt, meaning that Ctrl-C's that are raised in that block will not be caught. If you instead use "try...except KeyboardInterrupt:" or just "try..except:" it will work as you expect it to.

So there you have it...how to exit multithreaded Python programs using Ctrl-C.

Using Memcached as a Distributed Locking Service

2010-05-03T16:29:00.000-07:00

One of the beauties of memcached is that while its interface is incredibly simple, it is so robust and flexible that it can do nearly anything.

As an example, I am currently working on a project where we need to do distributed locking. The reason for this is that we are running into what is generally know as the lost update problem. Basically, we need to read an object, update it, and write it back in a serialized fashion in order to ensure that no updates are lost. The easiest solution is to have a lock which must be acquired in order to do the read/update/write operations. Locks are great with a set of threads but once you break out of the context of a single machine, you need something distributed. While there are certainly other library or complex pieces of code that do this, I find this to be a pretty elegant solution for a set of nodes which have access to the same memcached servers.

Provided below is a class called MemcacheMutex which provides the standard acquire/release mutex interface but with the twist in that it is backed by memcached.

There is one big caveat. Since memcached is not persistent it is possible that the lock could get evicted from the cache. This could result in a case where P1 acquires the lock, the key gets evicted, then P2 is able to acquire the lock even though it has not been released by P1. If your memcached is doing tons of operations or you are holding onto the lock for really long periods of time then this could become an issue. In that case, you should use persistent version of memcached, memcachedb.

The Cloud Support Conundrum

2010-04-16T15:28:00.000-07:00

Two nights ago I read an interesting tweet by @jclouds (Adrian Cole) that I think deserves some discussion. It is shown below:

For the record RightScale supports Rackspace, GoGrid, and Eucalyptus in addition to AWS. However, I think there is a bigger issue at hand which is how many clouds should cloud management platforms really support?

I'm going to take the opposite stance on this issue. I actually think it is quite impressive that cloud management platforms in general support as many clouds as they do. From a financial perspective you only really need to support one cloud (AWS) since it is dominates usage by far. However, I think most people agree that putting all of the eggs in one basket is foolhardy. So the next logical move is to support the next biggest player in the market, Rackspace. That is why support for AWS and Rackspace is ubiquitous across all cloud management services (RightScale, enStratus, cloudkick). At that point even if one of your legs gets kicked out from under you, you still have another one to stand on.

Aside from AWS and Rackspace, there are a smattering of "other" clouds for which adoption and usage has been small. Case and point, in addition to AWS and Rackspace, the three aforementioned cloud management services support a total of nine additional clouds. Yet, the only overlap across those nine clouds is RightScale and Cloudkick supporting GoGrid. The rest are unique to that particular service.

While I would like to think this that the divergent opinions in cloud support are a result of too many awesome clouds being available, that is simply not the case. Instead, these platforms have the unpleasant task of trying to find clouds that are sufficiently feature rich enough to support actual usage. While most clouds seem to be improving in providing the endemic cloud features, this process can be slow and leave much to be desired.

The alternative is customers asking management platforms to support some of the other clouds. While this can and probably does happen, I can't imagine it happens in a sufficient enough quantity (or with sufficient $) to make it a worthwhile venture. If you are ask customers they would love it if you added support for <some obscure cloud only they use> or even better <some cloud their son is building as a science fair project> but lets be realistic, it's not gonna happen.

With that being said, I do think there will be more diversity in cloud offerings, just not as quickly as most cloud folks want it happen. At this point the focus should be getting people onto the cloud, not a cloud.

Unknown Substitution Cipher Interview Question

2010-03-28T20:45:00.000-07:00

You are given a file containing a list of strings (one per line). The strings are sorted and then encrypted using an unknown substitution cipher (e.g. a < c, b < r, c < d). How do you determine what the mapping is for the substitution cipher? The unencrypted strings can be in any language.

After a few questions the interviewer clarified and stated that the cipher was a simple substitution cipher where every letter in the alphabet is mapped to one and only one other character in the alphabet.

Given this problem the simplest solution (not counting brute force) would be to do frequency analysis. If the strings contain words or phrases in a popular language there is a reasonable shot that frequency analysis would at least get you started figuring out the cipher. After discussing this and how it would work the interviewer told me "you do frequency analysis and cannot draw any conclusions, what do you do then?"

Ah, back to the (metaphorical) drawing board. The next big clue is the fact that the strings are sorted. This type of interview question is generally worded very specifically so when what seems like in extraneous factor is "thrown in" it probably means something. In this case the fact that the strings are sorted tell you something, in particular, precedence of the letters. For example, given the following strings (which are sorted in ascending order)

ggcaa
ggpqr
grfzu

you can start to learn about the ordering of the encrypted letters. For example, in the above example we know that c < p from the first two strings and g < r from the second two strings. We can only use the first letter that differs from any two pairs of strings, everything afterward is inconsequential. Doing this over the entire file you would get a set of relationships (e.g. c < p, g < r, q < z, z < t). The next questions are then "what can you do with this data and what is the most appropriate method for storing it?"

This is where things get tricky. After going through some of the common data structures it seemed like a tree, in particular a binary tree, might be the right fit. The problem with a binary tree is that you don't necessarily have all the required information to create the tree. For example, say the root of the tree was the letter Z and it had a left child Q. Now say you are trying to insert the letter G. You may know that G < Z so it needs to be in its left sub-tree, however you may not know what the relationship is between G and Q. In going through the list of strings, you are not guaranteed to have all relationships between all letters.

Without all the relationships you are forced to fall back to a graph structure. Each letter becomes a node in the graph and each greater than relationship becomes a directional edge in the graph (g < y would have an arrow pointing from node g to node y). With the graph populated with all the edges, now what do you do? The interviewer assured me that each node is part of the graph (i.e. there are no partitions).

This leads to one of the tricky parts about graphs. Since all nodes are equal, where do you start? There is no "root" node like there is in a tree. In this case you want to start with the node with no incoming edges, this is the smallest letter. Such a node has to exist since there cannot be cycles in the graph. You can then delete that node from the graph and then repeat this process until there are no more nodes in the graph.

Now for the fun part, what is the complexity of this algorithm? Finding the smallest node in the graph is N complexity for the first node, N-1 for the second node, and so on which leaves you with N+(N-1)+(N-2)+...+1. This is can also be written as the summation from 1 to N of i which equals (N*(N+1))/2 (see here if you don't believe me). This means that the algorithm is O(n^2).

Enjoy.

NoSQL: The Paradox of Choice

2010-03-02T23:41:00.000-08:00

I have been watching a lot of TED talks recently and while there are a ton of excellent talks, the one I want to talk about today is called "The Paradox of Choice" by Barry Schwartz (he has a book by the same title). While watching his talk it struck me how relevant the topic is to NoSQL databases. I want to talk about these connections and their implications.

More != Better

I want start with an idea that many people know intuitively but is still worth calling out explicitly. Having more options is not necessarily a good thing, in fact, in many cases its a bad thing. In his talk Schwartz cites a study which showed that for every additional 10 mutual funds offered by an employer, participation went down by 2%. As the number of options increases, the amount of time and effort required to make a decision increases significantly, to the point where you are put into a state of paralysis as a result of the myriad of options. For all intents and purposes the birth of the NoSQL database came with the publication of the BigTable paper by Google and the Dynamo paper by Amazon circa 2006. Since then, the number of NoSQL databases has gone from two, to five, to 29. That's right, http://nosql-database.org/ currently lists 29 different NoSQL databases each of which has slightly different feature set and benefit/drawback trade offs. Good luck picking the right one.

More Options => Higher Expectations

When many options exist it is only natural for us to expect that one of them hasto have the features you are looking for. With only one option it doesn't matter if it has the features you want since you don't have a choice. Prior to the NoSQL movement there was only one game in town and it was the relational database (MySQL/PostGREs/MSSQL), so you had no choice but to grin and bear it. However, now that there are almost 30 options (there probably will be by the time I'm done with this post) one of them has to have the right mix of peanut butter and chocolate to fit my tastebuds. Unfortunately this rarely turns out to be the case.

Higher expectations not only apply to features, but also performance. The promise of infinite scalability will draw a lot of eyes and ears, but in order to win them over you have to show users tangible benefits. I cant count how many blog posts I have read about people/companies who are using MySQL as a key-value store because it is faster then the key-value stores themselves. In order for NoSQL databases to win, users need to be able to benchmark the database and be impressed by its performance.

What have we learned from this? For all the hype it is getting, NoSQL is in a rough spot. The number of options is large and there aren't clear winners in any category. A few of the databases seem to be rising to the top (Cassandra for instance), but until enough people get behind a relatively small number of databases (I say pick 3) the knowledge, tool support, and amount of helpful information available online is going to be to painfully low.

Peering In The Clouds

2010-02-16T21:55:00.000-08:00

An idea that I have been kicking around in my head for some time now is that of cloud peering. This idea comes from the networking context whereby big ISPs will negotiate a peering agreement that allows customers on the two networks to communicate with each other. While the big ISPs are cut-throat competitors they understand that if they do not work together that the Internet would turn into large islands where customers can only communicate to others on their island.

Here is what Wikipedia has to say about the benefits of peering:

Peering involves two networks coming together to exchange traffic with each other freely, and for mutual benefit. This 'mutual benefit' is most often the motivation behind peering, which is often described solely by "reduced costs for transit services". Other less tangible motivations can include:
Increased redundancy (by reducing dependence on one or more transit providers).
Increased capacity for extremely large amounts of traffic (distributing traffic across many networks).
Increased routing control over your traffic.
Improved performance (attempting to bypass potential bottlenecks with a "direct" path).
Improved perception of your network (being able to claim a "higher tier").
Ease of requesting for emergency aid (from friendly peers).

Are those not things that every cloud infrastructure provider would love to have more of? More redundancy/capacity/performance seems compelling to me.

The way I would envision such a system working would be for two infrastructure providers to agree to provide free bandwidth between their clouds. For why this is important, think about the cloud today. When trying to transfer data between clouds what ends up happening is you charged going in both directions (i.e. out of one cloud, into the other). Say for example you were running an application in the Rackspace cloud but doing nightly DB backups to S3 for redundancy, you would end up paying 22 cents/gig to Rackspace for the outbound traffic and another 15 cents/gig for traffic into S3. At 37 cents/gig you could quickly start racking up a nice bill just for bandwidth. Cloud peering could alleviate a lot of this cost while attracting more customers to the cloud by reducing the concern over lock-in and outages.

This strategy seems particularly attractive for some of the smaller players which are looking to grow market share. It will be interesting to see if anything like this develops in the future.

Designers Guide to Web Performance

2010-01-24T22:42:00.000-08:00

I have found myself reading quite a bit about usability and web design recently. While I have learned a lot about design I have also learned that there is a large portion of the design community which is not terribly familiar with how they can improve performance of the websites they are creating. While a pixel perfect layout is a beautiful thing, no one wants to wait 20 seconds for it to load. Since the web design community has taught me so much, I wanted to give back by writing this post on some simple techniques for improving site performance.

Welcome to Web Performance 101, here is your first assignment:

The above is a waterfall chart which shows how long the pieces of a particular website take to load. To be nice I've blanked out the website URL's but I left the file extensions since they will come in handy later. What this chart shows us is that initially the browser made a request for the index ("/") page and 1342 ms later it had received all of HTML for that page. One second in the browser makes a request for the first javascript file which takes 585 ms to load. So on and so forth for about five seconds. While there was more to the chart I cut it off for brevity.

Progressive Rendering

You probably have noticed that often times your browser starts drawing the website before the entire thing has loaded, this is called progressive rendering. This is really nice because if done correctly it can make your website "feel" much faster for end users. In order for the browser to start drawing the page it needs two things, the HTML and the CSS. Until the HTML/CSS are downloaded your users are going to be staring at a blank white screen. This is why it is really important to put CSS in the head of the document before javascript and images. In the chart above there is a vertical green line at about 2.25 seconds, that is when the browser could start rendering the page. Had the designer of the site put the CSS before those five javascript files the page could have started rendering at 1.5 seconds or even earlier. Its such a simple an easy change, that there really is no reason not to.

Javascript: The Performance Killer

While every website is going to have a waterfall pattern the goal is to have a very steep waterfall. The further the bars extend out to the right of the graph, the longer the site takes to load. Hence, we want to shoot for a very steep and short waterfall which should translate into a fast site. Unfortunately this chart looks more like stairs then a waterfall, the reason is javascript.

While it may not be obvious to you, browsers are pretty smart. For example, when fetching the images on a website most browsers can download more than one at a time. Older browsers (IE7, Firefox 2) usually download two items at a time and newer browsers (IE8, Firefox 3) download 6 or more at a time. Notice how there is overlap between items 8 & 9 as well as 10 & 11, those items were being downloaded in parallel. Then notice how there is no overlap between items 2-7. This is because unlike images or CSS, javascript blocks the browser from downloading anything else. While newer browsers tend to do a better job at this and download them parallel, most of the world is still running IE7 which is what was used to generate the above chart.

One way to improve this would be to take all the separate javascript files and combine them into a single larger file. This will reduce a lot of the overhead incurred by download many files. Note how most of the bar of the javascript file is actually green, that is "time to first byte." It is the time between when your browser say "Hey give me foo.js" and when it finally receives the first byte of that file. You can barely even see the blue lines at the end which is the time spent downloading the actual data. By combining those javascript files together you cut out most of the time spent waiting for the data to come back. There are many tools online and available for download which will automatically concatenate the files together for you.

Zipping it Up

The easiest way to ensure that your site loads fast is to send less data. Less data means less time required to download it which means better performance. This is where data compression comes in. Most big web servers provide the ability to compress the data it sends out into a much smaller format so that it can be transferred more quickly. Then once it reaches the browser it is smart enough to know that it is compressed and it will uncompress it and read it just the same way it normally would. The most popular compression algorithm used today is called GZIP, and it can often cut file size in half or even a third. It is important to note that this compression is only effective on text data, so you want to gzip your HTML, CSS, and Javascript files. You don't want to gzip images since it usually eats up more server CPU then its worth. The best thing about gzip is that it only requires a few extra lines in the server config file and then your done. This website has a good guide for how to enable it for Apache.

As I mentioned above it is often a good idea to combine all of your javascript files into a single file, the same holds true for CSS. So if you have your css split across many files, combine it into a single file so that it loads faster. An extra bonus to this is that compression rates tend to get better as files get larger. Thus while the amount of text is still the same you will probably end up transferring less data and skipping much of the time wasted waiting for the data to arrive.

That's where we will stop for today. Hopefully you learned a few things about web performance which you can apply to websites you are working on. Keep in mind there are a lot more tips and tricks for improving performance, this is only the tip of the iceberg.

Private clouds are transitional

2010-01-17T18:37:00.000-08:00

Given that I am going to be talking with the Eucalyptus guys in a week I figured it was a pretty opportune time to sit down and really think about private clouds. Much to the chagrin of some people, I firmly believe that private clouds are still clouds. In my mind the real question is where do private clouds fit in the cloud ecosystem?

Lets get one thing straight, private clouds are transitional. What I mean by this is that private clouds are not going to be here forever, they are instead filling in a very important gap in the current cloud landscape. While a large number of servers could be moved into the cloud today, there are still quite a few use cases which don't allow for such rapid change.

To illustrate one such example, lets say you head IT for some Fortune 500 mega-corporation. After a few months of waiting your purchase order for X racks of servers (finally) goes through. You estimated that with the servers you have ordered you should have sufficient capacity for the next year. Shortly thereafter you reassess the opportunities provided by the cloud and deem it fit for use by your company. So enthusiastic about the change you even decide to join Lew Moorman and Marc Benioff on stage at some cloud event to chant "No more servers!" and pledge never to buy another one. However, upon returning to the reality of work you recall that in addition to the servers you currently have running in your data center, the new racks you just ordered are going to provide you sufficient capacity for the next year. Assuming a 3-5 year lifespan of the average server you are looking at at least two years before you can move a majority of your servers into the cloud. Enter the private cloud. You already have the resources, there is no reason for them to sit around and gather dust. Make your own private cloud with the existing resources and expand out into the public cloud in as necessary. Once the servers in your data center have run their course, toss them and move to the cloud.

The other big elephant in the room with regards to moving to the public cloud is compliance. These days it seems like every big industry has its own compliance and regulatory constraints that must be met. Whether it's PCI for credit card processing or HIPAA in the health fields, almost none of the big cloud vendors have met the requirements for becoming compliant. In fact, its not even clear that they are trying. Unfortunately, regulation is not something that can be skirted around, it is a big time show stopper. This means that companies in industries which have regulatory requirements are going to be in a holding pattern around the public cloud for the foreseeable future. What's the next best thing? That's right, private cloud.

While private clouds are transitional, they will be around until the aforementioned issues are addressed. For some this may be on the order or months or a year, but for others it will probably be much longer. The regulatory issue in particular is not an overnight fix, I think it is going to be a big parachute that prevents the cloud from running at full sprint for the year. So while the cloud is transitional, that transition period is looking like its going to be a long one.

Rails named_scope Time Surprise

2010-01-05T10:19:00.000-08:00

I have written previously about how much I like named_scopes in Rails and I still do very much. After using them for some time I got tripped up on an issue that came up with them which surprised me a bit. I thought it would be a good idea to document it here in case others have the same issue.

To demonstrate the issue, lets say I have an app that has a User model. On the home page I want to display a list of the users which have signed up in the last hour. This is an excellent use case for a named scope. We can start by creating a named scope called "recent" which will then allow us to simply say "User.recent" to retrieve all the recently created accounts from the database. This seems simple enough so I went ahead and wrote it up as follows:

Now you will notice that I named it recent_bad, and that is because this named scope is BAD! Take a look at the queries generated when I call recent_bad three times, notice anything wrong? Its subtle. Note how the date after created_at "2010-01-05 16:55:44" never changes. For effect I made the model acts_as_paranoid so you can see what the timestamp should be. What is happening here? The named_scope is at the class level, that means that the when the User class is loaded the Time.now.utc is evaluated once and then never again. This is why the time only changes when the server is restarted. In order to avoid this issue simply put the condition within a lambda as follows:

Now you will see that the created_at time is updating as it should be. Its a subtle bug and one that caught me by surprise.

Slashdot loves Cloud Computing!

2010-01-03T11:59:00.000-08:00

One of the reasons I love reading Slashdot is because the inhabitants of their community are unlike any other. This morning I woke up to an article on the front page about a venture capitalist who was defending the incredibly popular social game Farmville. Being interested in the topic I thought I would take a gander at the comments, that is when I came across this gem (direct link):

   This needs to be the year that those of us with even the slightest degree of technical knowledge take a stand against the goddamn "Cloud".
   It sounds fantastic in theory, but once in the real world, Cloud Computing falls flat on its face. My development and ops teams wasted too much time dealing with Cloud providers over the past year. So my resolution this year is to tell anyone who proposes the use of anything Cloud to cram it. We aren't doing it any longer. It's a failed approach.
   Just last week, during the holidays, we had to scramble after one of our Cloud providers ran into some hardware problems and couldn't get our service restored in a timely manner. After the outage exceeded my threshold, I called up my best developers and had them put together a locally-hosted solution in a rush, and payed them quite a bit more than usual due to the inconvenient timing. Then I called up the Cloud provider and basically told our rep there that we are done using them and their shitty service. Then I called up the manager in our company who recommended them, and told him to basically go smoke a horse's cock.

The commenter was apparently so proud of their work that they decided to post it anonymously. Now keep in mind that the article was about social gaming, it had nothing to do with the cloud. While Farmville does run on Amazon EC2, the article does not mention or discuss that at any point. Regardless, lets take a look at this comment and pretend that it was posted in a reasonable context.

Perhaps my favorite thing about the comment is the fact that it makes these huge substantive claims yet provides absolutely zero reasoning behind them. For example, "It sounds fantastic in theory, but once in the real world, Cloud Computing falls flat on its face." Really? In what sense? You must mean Animoto scaling form 40 to 4,000 servers in 3 days. Or how about the millions of people who are using Google Apps. Both of those are certainly real world and as far as I can tell there was very little falling on faces. In fact, the cloud helped Animoto avoid falling on its face!

As if the first claim wasn't enough the end of the second paragraph provides a real doozie. It completely writes off Cloud Computing by stating that "[i]t's a failed approach." Well I'd better get on the phone and tell all the people who put 64 billion objects into Amazon S3. That might take a little while...

The last paragraph at least provides a little bit of background on why this particular individual will "tell anyone who proposes the use of anything Cloud to cram it." (Well if he is going to tell them all to cram it maybe he can make all those calls to the S3 users for me...). This is where I start to get a little empathetic. Downtime sucks, it really does. Customers and providers can both agree that downtime sucks since everyone loses when it rears its ugly head. My question is this, if this service is so critical then why wasn't it built to be fault tolerant? If you are truly concerned about availability then you need to either a) build the service such that it can withstand failure or b) have an SLA in place with the provider. But honestly, who wants to do that? Instead I would recommend following the actions of the commenter which is 1) don't take the necessary precautionary steps to avoid downtime and then 2) complain when there is downtime. Whats next, eating three Big Macs a day and then complaining when you need triple-bypass surgery?

Lastly, I would like to commend the brave commenter for being brazen enough to tell a manager at his/her company to "basically go smoke a horse's cock." I can see your career blossoming as we speak.

Numbers Everyone Should Know

2009-12-13T16:17:00.000-08:00

I was looking over a presentation titled "Designs, Lessons and Advice from Building Large Distributed Systems" By Jeff Dean from Google when I came across a really useful slide. Its the 24th slide in the presentation and is titled "Numbers Everyone Should Know". It has the latency of some common processor/network operations:

L1 cache reference............................................0.5ns
Branch mispredict................................................5ns
L2 cache reference...............................................7ns
Mutex lock/unlock.............................................25ns
Memory reference...........................................100ns
Compress 1K bytes with Zippy......................3,000ns
Send 2k bytes over 1Gbps network.............20,000ns
Read 1MB sequentially from memory.......250,000ns
Round trip within datacenter...................500,000ns
Disk seek.............................................10,000,000ns
Read 1MB sequentially from disk........20,000,000ns
Send packet CA->Netherlands->CA...150,000,000ns

Having numbers like these are really useful for just ballpark estimates. I think the biggest surprise to me was the huge disparity in a datacenter round trip to disk seek. We all know how slow the disk is, but the fact that you can make 20 round trips within a data center in the time it takes just to make a disk seek (not even reading any data!) was pretty interesting. It reminds me a lot of Jim Grays famous storage latency picture which shows that if the registers were how long it takes you to fetch data from your brain then disk is the equivalent to fetching data from pluto.

Be Wary of the Paranoid

2009-11-25T18:47:00.000-08:00

I recall distinctly when first learning about Rails and plugins that the very first one I used was acts_as_paranoid. Something about actually deleting data concerned me and so I figured adding acts_as_paranoid to some important tables in my application would save a lot of headaches. While it is tremendously useful it also has a pretty big unintended consequence that I think gets overlooked by most. Lets take an example query generated by acts_as_parnoid.

>> User.find_by_id(7)
SELECT * FROM `users` WHERE (`users`.`id` = 7) AND (users.deleted_at IS NULL OR users.deleted_at > '2009-11-26 02:17:28')

Looks harmless right? What I want to draw attention to is the "OR users.deleted_at" part. In order to ensure that the user isn't deleted it checks not only that the deleted_at field is NULL, but also that deleted_at is greater then the current time. In reality the IS NULL check is sufficient unless you are setting the deleted_at of some object to be in the future. I have yet to see anyone actually use it in that way. This is what makes the use of the current time in the query so bad, it is slowing down tons of Rails applications and most people don't even know it.

One important thing to notice about the MySQL query cache is that it is pretty dumb. Basically it caches the incoming query string exactly as written and then stores its associated result set. This becomes a problem when you use something like the current time in the query string, it functions as a cache-buster each second. So at 0 seconds you make a query and it is stored in the cache, then at 0.5 seconds you make the same query and it is read from cache, then at 1.0 seconds you make the same query but it will miss the cache since the time has increased by a second. This means that anything written to the query cache which uses acts_as_paranoid effectively has a 1 second expiration time. That's awful , and all that for the 0.005% of users who want to expire things in the future. Not to mention the fact that it completely pollutes the cache with old data which never gets touched a second after its written.

Alright, enough moaning, here's how to fix the problem. Open up paranoid.rb and in the "with_deleted_scope" function rip out "OR #{table_name}.#{deleted_attribute} > ?" along with the current_time variable after it. Similarly in has_many_through_without_deleted_assocation.rb in the construct_conditions method delete the same string where it is appended to the conditions variable. Keep in mind that if you are setting deleted_at to values in the future then you don't want to make this change. But for everyone else, enjoy the improvement in your query cache hit rate.

As a final note, for the tables which you have made paranoid you probably also want to consider adding an index which includes the deleted_at field since it will be a condition of every SQL query on that table.

Updated: There is a fork of acts_as_paranoid courtesy of mikelovesrobots that provides the fixes that I talked about previously, it is available here. I'm gonna switch out my versions of acts_as_paranoid for this one, I'd suggest you do the same.

Lacking in Persistence

2009-11-24T22:29:00.000-08:00

A few weeks back was the very first incarnation of Cloud Camp in the Cloud. While I was unable to attend I did get around to watching the screencast of it courtesy of @ruv (available here). There was certainly a bunch of good information and intelligent discussion but I found one question to be particularly interesting and insightful. The astute attendee asked "Why was Amazon EC2 designed such that instances have transient (ephemeral) storage? Rackspace has been pushing their marketing on the fact that their servers have persistent storage is this a big deal?" That is certainly a loaded question but I think I'm in a reasonable position to take a shot at it.

Lets start by talking about some of the reasoning behind why Amazon would make their instances transient. The first reason is simple, making instances transient makes life a lot easier for them, particularly given their scale of operation. If they wanted to make instances persistent they would need replicate that data at least twice, if not three times, with at least one being in another data center. Imagine the amount of traffic that would be needed to keep a write-heavy database server consistent across multiple disks. Also, all the data to that last replica has to ship over the intertubes means say hello to Mr. latency. Ever notice that Amazons EBS volumes are replicated only within a single availability zone? I would be willing to bet this is because of network traffic and latency concerns.

One thing you will notice about a lot of the cloud providers that provide persistent instances (e.g. vCloud Express, ReliaCloud) is that they break that nice little "pay-per-use" model everyone is so fond of. In order to maintain those persistent instances most providers charge you even if they are "powered off". ReliaCloud instances are roughly 1/2 price when powered off while vCloud Express charges the full price. Rackspace has a somewhat different approach. First they RAID 10 the disk which should make failure less likely. In addition, they claim that if a failure occurs they will automatically relaunch your instance for you complete with data. How long does that fail-over take? They don't say and I have a hard time believing you will get a guarantee from them.

Taking a step back, what you will find is that Rackspace is really the midpoint between EC2 and vCloud in terms of persistence. On EC2 if an instance fails your it is gone along with your data (unless your using EBS). On vCloud, if your instance fails or you power it off it still persists. Rackspace falls in between in the sense that if your instance fails it will come back (with some delay) but if you shut off your server its gone along with its data. Thus, the only real way to make your data persistent on Rackspace is to keep the server running (at full price), or dump it into CloudFiles. This points out one of the really nice benefits of EBS which is that you can have persistent data without needing an instance to store it on (read: cheaper). But why would you want to store data without an instance attached to it you might ask? Its simple, there is probably some portion of data that you would like to keep persistent (e.g. database) and while you could dump it onto CloudFiles/S3, reading that data back onto a newly launched instance can take a loooong time. This is what it was like on EC2 pre-EBS and it wasn't pretty.

Now the trickier part of the question which is whether this persistence makes Rackspace a more attractive cloud infrastructure service. Having machines come back after failing is certainly a nice feature but it you still need to have a second server for fault-tolerance if you want to attain a reasonably high availability. While it may take some time it is almost certainly faster then the time it would take you to figure out that your EC2 instance has died and having you hustle over to your laptop to fire up a replacement. The bottom line is that it probably not available enough for any reasonable sized service to rely on it without a proper backup. On the other hand if you are hosting something like your blog where a few minutes of downtime isn't a critical then it can be a handy feature.

On paper it certainly looks nice, but in my opinion its not really a huge benefit. In all the time I have used EC2 I have only ever seen a few instances fail and it was after I had accidentally set the Java heap size to be 10x the available memory and then ran a big Hadoop job. The machine was thrashing so hard I'm not surprised it died. Aside from that (very extreme and operator induced) case have never seen an instance fail. In my experience EC2 instances simply don't fail frequently enough for this to be a big deal to me.

To be honest I think the fact that Rackspace allows you to "grow" your instance size is a much more attractive feature, but that's for another post...

For an interesting and thoughtful comparison of EC2 and Rackspace I would take a look at this blog post. He does a good job hitting on many of the important points and even agrees with my thoughts on the benefits of EBS.

Boy and Girl Birth Rate Interview Question

2009-11-14T14:21:00.000-08:00

I came across an interview question which is pretty interesting and a bit of a head fake (in my opinion). The question is...

"In a country in which people only want boys, every family continues to have children until they have a boy. if they have a girl, they have another child. if they have a boy, they stop. what is the proportion of boys to girls in the country?" --courtesy of fog creek software forums

After thinking about it for a little while I though I had the answer, but of course I was wrong. After reading the right answer I still couldn't quite convince myself that it was true, so I figured I'd test it out.

Sure enough the answer generated matches up with the explanation provided in the link above. So there you have it...

Installing RubyGems faster

2009-10-25T12:05:00.000-07:00

One of the things I learned about over summer while using Rails quite a bit was that the gem installation process can be a slower one, especially when you are installing lots of gems. In many cases it is not even installing the actual gem that takes all the time, its generating the rdoc and ri information. In my experience very few developers actually use the rdoc information on their local boxes, and no one should be looking at rdocs on your production environment, so why bother installing them?
You can prevent those from being created by adding special flags to the end of the install command (e.g. "gem install rspec --no-ri --no-rdoc"). This is nice but I seem to always forget to add the flags until its too late and the gems are already installing. This can be fixed by adding the flags to your gemrc so it will happen automatically. Simply open your ~/.gemrc file and add the following line to the end "gem: --no-ri --no-rdoc". My .gemrc was created by root so I needed sudo to edit the file, but this may not be the case for you. Just for reference my .gemrc now looks like this:

---
:sources:
- http://gems.rubyforge.org/
- http://gems.github.com
:benchmark: false
:backtrace: false
:update_sources: true
:bulk_threshold: 1000
:verbose: true
gem: --no-ri --no-rdoc

As a quick test I installed the cucumber gem on my local box without the flags and it took 31 seconds. After changing my gemrc to include the flags the same installation time took 13 seconds, a pretty nice improvement. If you are deploying your app in an environment like RightScale where your machines are configured at boot-time I would certainly include that line in your gemrc, it should speed up the gem installation process a good deal.

The cloud solves a lot of problems, stupidity isn't one of them

2009-10-11T18:18:00.000-07:00

The past few weeks brought some really unfortunate news for users of T-Mobile's Sidekick phones. This started with a outage of their data service which started on Friday the 2nd and lasting four dreadfully long days. During that time users couldn't access the Internet or, more importantly, their contact information since that information is all stored remotely. News only got worse this weekend when Danger (the company that makes the phone, also a Microsoft subsidiary) announced that the data not stored on the phones had "almost certainly has been lost" and that the chances of it being recovered was "extremely low".

A big undertone to this event has been "Should we continue to trust cloud computing content providers with our personal information?" (from this Slashdot article). Many people have pointed out that Microsoft purchased Danger about a year ago, and that this catastrophic data loss has cast a cloud over the soon to be launched Microsoft cloud, Azure. This is where I take issue.

First off, since when has the simple act of storing data remotely constituted cloud computing? Regardless of what definition of the cloud you subscribe to it probably has the words "virtualization", "elasticity", and "pay-per-use" in it somewhere. I don't see any of these three things or any other cloud-like properties which would lead me to believe that Sidekick == cloud. However, let us take this ridiculous assumption of the Sidekick being cloud and continue with it.

While there has been no official announcement from Danger regarding the cause of the data loss, word has surfaced that it was a result of a botched SAN upgrade. While things certainly can go very wrong when messing with a SAN, the kicker is that no backup was made prior to attempting the stunt. As far as I can tell, no backups were made at all (or at least ones that worked). Like the title says, the cloud solves a lot of problems, but stupidity isn't one of them. With its seemingly unlimited amount of storage and minimal cost its just plain stupid to not make backups of any important data. Better yet, get it all nice and encrypted and use something like the simple cloud API to back it up to multiple storage providers. Why not? In the long run the cost of keeping tons of backups of that data is so trivial that it shouldn't warrant a second thought.

Given that Danger was purchased by Microsoft many have now brought up the question, how does this affect Azure? The answer, it doesn't. If you have ever worked for or dealt with a large company you know that it takes a long time to get anything done. It has only been 18 months since Microsoft purchased Danger so I have a hard time believing that much changed for Danger aside from the sign on the building (if that). This is particularly true about system architecture where things are so complicated that the old adage "if it ain't broke don't fix it" often holds true until the very bitter end. It is pretty clear that the Danger infrastructure wasn't running on Azure and the two have very little in common. I think the only real impact this incident will have on Microsoft is for them to receive more questions the reliability of their data storage. Their response will be that they replicate data 3 times across multiple geographically separate data centers (this is just a guess) and then everyone will report this back to their CIO's who will approve and then everyone goes home happy. End of story.

Updated: Microsoft has just confirmed that they have been able to recover most, if not all of the data lost in the sidekick outage. Thats great news for sidekick customers, while they were out of a usable phone for a few weeks, getting they contacts back is a big win. As it turns out that they did have some backups in place and were able to recover from that, although it took quite a while (and still continues). While ultimately little or no data may have been lost, the damage has been done.

Fun with named scopes in Rails

2009-09-29T11:56:00.000-07:00

One of the features that I definitely had no idea about when I first learned Rails was named_scope. I went back at look a look at some of the old projects I had worked on I frequently found myself writing a bunch of extra finder conditions or methods in order to achieve that goal.

That is a relatively simple example, but since these conditions are used frequently I can improve upon this by using named scopes. Named scopes allow you to encapsulate some finder arguments into a simple, chain-able, and efficient methods. Here is what the named scope definitions and finder looks like:

While these could be defined as standard methods, you loose a lot of the power and flexibility which named scopes provides. For example, with named scopes I can add in additional arguments or conditions to any of the above methods. For example:

If you are using Rails 2.3 they have added a feature called scoped_by which will dynamically generate a lot of the boiler-plate scopes that you would need.

If you aren't using Rails 2.3 or aren't crazy about that syntax there is also a Ruby gem which provides a lot of the same functionality in an arguably more elegant manner. Its called Pacecar and it made by the guys over at Thoughtbot. Using Pacecar the above example would look like:

That looks a lot more elegant to me. So get out there and start using some named scopes.

Ballmers iPhone stunt could hurt Microsoft long-term

2009-09-17T19:34:00.000-07:00

Just about a week ago news came out that Microsoft CEO Steve Ballmer had a strong reaction to an employee who tried to take his picture using an iPhone. At the company meeting in Safeco field, an employee was trying to take his picture when Ballmer grabbed the device, said a few remarks about it, and then pretended to stomp it into the ground.

Anybody who has ever seen Ballmer speak knows that he is a passionate guy. While most people can appreciate the passion and enthusiasm that he brings to the table, there are times when this can bite you. I think this was one of those times. Let me explain...

Prior to last week, Microsoft had actually made some inroads on the iPhone with the release of the SeaDragon and Microsoft Tag apps. While I would not consider these everyday-use type apps, they were a pretty clear step in embracing the iPhone platform and utilizing it strengths. I played with the SeaDragon app quite a bit and it really is a perfect marriage of the two technologies (e.g. pinching to zoom on the large images, etc).

In my eyes those inroads have now been left be left to rot as a result of Ballmers actions. Think about it this way, say you are a Microsoftie who has a great idea for an app which really showcases a Microsoft technology, would you be willing to stand up in front of Ballmer and pitch the idea? That idea had better be an incredibly good one with a ton of potential otherwise...

Who does this really hurt? Search. Contrary to what many people believed, Microsoft has really made a big comeback in the Search space with Bing. In fact it was this past August that Bing surpassed 10% marketshare which is a pretty big milestone in its growth along with a nice month over month improvement. Currently desktop search dominates the market, but anyone with a little bit of foresight will tell you that mobile search is and will continue to be a big growth area in the near future. Unfortunately, Bing's presence in the mobile market (particularly the iPhone) is...shall we say...weak. Yeah, they have a mobile site, but thats about it. Unlike Google and Yahoo!, Bing has no search app as far as I can tell. Furthermore you can't even set Bing to your default search engine (Google/Yahoo! are currently the only options).

Now if your Bing fighting for each point of marketshare, dumping tons of money on the "search overload" ads, why not get a few developers together and have them make a Bing iPhone app? One word, Ballmer.

When being anti-web is just not cool: Fever RSS

2009-09-05T22:13:00.001-07:00

I have been a pretty avid user of RSS for a few years now and as a technology it is awesome. Just about everybody knows that pulling technologies are much more efficient then pushing and thats what makes it possible to consume so much more information. To show you what I mean, looking at my Google Reader I am currently subscribed to 80 feeds and in the past 30 days I've read over 6,000 articles (~200/day). While I really like RSS, one of the things I really found missing was the lack of intelligence of RSS readers. Instead of just displaying me all the information in reverse chronological order, how about displaying them based on how interested I am in that article or feed. Sure that is a little more complicated, but in a world where theres tons of information it is a lot more efficient. The reality is that it is just a big text mining and ranking problem with the data right there in front of you, so get cracking. I'm genuinely surprised Google hasn't tried to tackle this problem since it is right in their wheel house, but that is a whole other story.

Over the past year or so I have been keeping my ear to the ground for new RSS readers which try to do what was described above. Then about a month a go a discovered a tool which was almost exactly what I was thinking about called Fever. Using the temperature as a nice metaphor for the things you are interested in, fever displays the information aggregated from your feeds based on how frequently an article is linked to. It is basically a personalized version of Techmeme. Looking at the screen shots it has a really nice design aesthetic which I can definitely appreciate and it even has an iPhone app. At this point I am looking for the nearest input box, chalk one up on the conversion rate I'm ready to use this thing. All jazzed up to get my RSS on and I'm greeted by the ugly credit card monster. An RSS reader that costs money, scoff. But given how nice it seems I figured I might consider spending a few bucks a month on such a nice tool. So how much does it cost? $30. That seemed like an odd price, and it is, the reason being that fever isn't a hosted service, its a desktop client. You plop down your $30 bucks for a license (you remember those things right?) get yourself the software and use away right? Not so fast, if you take a look at the answers section you will see the following:

What are the server requirements for Fever?
Fever requires a Unix-like server (no IIS) running Apache, PHP 4.2.3+ (preferably compiled with mbstring and GD with PNG support) and MySQL 3.23+.

Thats not your standard desktop software requirements list (e.g. dual-core processor, 1GB of ram) by a long shot. Instead, fever requires that you set up your own LAMP stack to host the stinking thing. While the software is nice, there is no way in hell I'm gonna set up and administer a server just to run it, thats just too much. Even with a variety of cheap VPS hosts out there, its just not worth the effort and cost.

After I gave it some thought, fever really is the perfect app for hosting. Its way to much of a pain for a single user to setup, requirements are fairly high, and you can pretty easily do database multi-tenanting. I looked around online to see if anyone was hosting fever but came up completely empty. Being a bit of an entrepreneur I figured this is something I could pretty easily setup, so I went ahead and contacted the developer to see if he was interesting in allowing others to host fever as a hosted service. The terse response I got was "I have no interest in offering Fever as a hosted application or in partnering with another party to offer Fever as a hosted application." Harsh.

Needless to say I was pretty disappointed. Not only did I loose out on the opportunity to make a cool little hosting business, but I lost the opportunity to actually use a really cool product. This is where I get back to the title of why being anti-web and why it is so uncool. There was a time when having software licenses and hardware requirements were all the norm, during that time many of todays software mega-corps (e.g. Microsoft, Oracle) were built. Unfortunately for them and much to the benefit to users everywhere, that time is rapidly coming to an end. Desktop software is simply a pain for everyone involved. Developers don't like it since its incredible difficult to make software for the different platforms plus all the complication of pushing out updates to all the users. Users don't like it because they have to fork over an arm and a leg to buy it and the onus to deal with updating it falls on them. Now I'm not going to fall of the deep end and say that all software is going to be hosted, but there is little reason that a significant portion of all the software can't live in the cloud. There are always exceptions for things like operating systems and really complex and resource heavy software but those only make up a minuscule fraction of the software in use today.

The moral of the story is that if you are making cool software that people want to use, do everything in your power to allow them to use it. If you want to charge, thats fine, but give it to users in the way that they want to use it and if you don't have the resources or desire to do that, by all means allow others to do so.

The Low Down On Cloud Brokers

2009-08-31T22:18:00.001-07:00

The notion of a Cloud Broker is a new and interesting topic and I believe it is something which will be talked about increasingly frequently in the near future. If you don't believe me, the folks over at Gartner have discussed the importance of this role in the cloud computing space:

"The future of cloud computing will be permeated with the notion of brokers negotiating relationships between providers of cloud services and the service customers. [...] Enhancement will include managing access to these services, providing greater security or even creating completely new services." -- Frank Kenney

Given that they are going to prove to be important, its probably worthwhile to take the time and discuss what it is that these cloud brokers actually do. In my eyes, cloud brokers are an abstraction layer between the end user and the many cloud services at their disposal. While this notion could be applied in many different contexts, when I say cloud services I am referring to Infrastructure-as-a-Service providers (e.g. Amazon, Rackspace, GoGrid, Joynet). With a plethora of cloud providers, each with a their own API/set of services/pricing model/etc it would be quite cumbersome for the end-user to programmatically access each service. Instead, the cloud broker creates the layer of abstraction between the user and providers so that the end users can see one cohesive view of all of the services. This way the customer doesn't have to worry about the nitty gritty like the different REST calls required to create a server, they just hit the launch button and a server appears on the desired cloud. This is roughly the state of cloud brokering today from companies like RightScale, CloudKick, and Elastra (give or take a little bit). Yes, there is some work left to do by these vendors to support more clouds but its only a matter of time before they have most of their bases covered.

While that is the current state of affairs, the future of cloud brokers is something that is a lot more murky (at least in my mind). I think one of the big ideas that many people have for cloud brokers is that they are going to become these intelligent tools which the user tells what they want and then it figures out the best way to do that for them. So if I need a server and I need it now (at whatever it costs) then it will pick the appropriate cloud. Similarly, if I just want a server at some point this week but I want it cheap then it will wait around until it finds a good deal and then starts up that server for me.

On the surface that seems all honkey dorey but in practice there are two really big issues with delivering such a vision. The first is transparency. Any time you are making an intelligent system which is making decisions on behalf of the user, it is absolutely critical that the decision making process be completely transparent to the end user. There are a lot of contexts where transparency is not terribly important (e.g. search results), but the second a users credit card is being charged the game changes in a big way. It is almost guaranteed that at some point you will get angry customers calling in asking why at some point your system made decision X where they expected decision Y. This is not the time to start having to explain the nuances of your cost-based least squares weighted estimation algorithmic ridiculousness. You might even try to be tricky and come up with some metric which is an amalgamation of a ton of different factors into one nice number scale (e.g. "Our system computed a doodle factor of 8.3 and as a result launched a new server"). While having that one number to point at is a nice trick, it generally does little to hide the formula that is used to compute it which inevitably becomes filled with unexplainable magic numbers. I believe that the result of this is going to be extremely simple systems which instead of trying to make the complex decisions are going to do their best to present the user with as much relevant information as possible and allow them to make the decision. If you imagine the case where the user wants a server very quickly one could imagine a system presenting you with the current cost and average boot time of a server across each of the clouds and then allowing the user to decide. The money vs speed trade-off is a complex decision and ultimately it is only the user who really knows how much they are willing to trade for one or the other.

The other major issue I see with these systems is that without variable pricing the utility of the intelligent system decreases greatly. As far as I know there are no cloud vendor out there changing their prices each day/hour/minute in order to account for changes in supply vs demand. While an interesting option and one any economist would be proud of, I think that vendors have moved away from this model as a result of the lack of transparency. What was once the brain-dead simple 10 cents an hour now becomes a decision requiring a lot more consideration. How much does it cost now? Well do I really need it now? If I wait will it get cheaper? And so on. Eventually I think a someone will try to do something along these lines and while I'm not a huge fan of the idea I wish them the best of luck. Getting back to the original point, until we start seeing clouds employing truly dynamic pricing the idea of having an intelligent system making decisions for me becomes a lot less necessary.

Ultimately I think the cloud broker space is one that is going to grow into many areas and service many niche markets. However as it continues to grow, don't be surprised as they all continue to keep a pole's length distance between themselves and the all knowing sentient system of doom.

Moving from SVN to Git - a users perspective

2009-08-23T19:20:00.001-07:00

Just about 6 months ago I took a grad class on Scalable Web Services, the main portion of the class was to 1) come up with a cool app idea, 2) develop it using Ruby on Rails, 3) make it scale. Very cool class, I enjoyed it quite a bit and it definitely sparked my interest in Rails and web performance both of which I have learned a lot more about since then. After the class there was some discussion as to opinions on the technologies used and one thing that most of us agreed with was that git was frustrating and difficult to use. The good news is that in the months since that experience I have used git quite a bit more and as a result I am really starting to like it. So much so that if I were to start a project today I would use git without a doubt.

Given that I pulled a full 180 on my opinion of git I took a step back and started to think about what it is that changed that caused my opinion of git to change. When I really thought about it the conclusion that I came to was that the main difference was my mental model of git. Let me explain...

I would think that I am probably similar to a lot of folks in that up to that point I had used SVN a decent amount, a few projects here and there along with some personal projects/data. As a result of this my mental model of version control was pretty simple, checkout a repo, 1) update, 2) make your changes, 3) commit, see (1. It's pretty simple and I suspect that most people who have used SVN have agree. Given this mental model, when I started using git I really thought it of as SVN with another step, you do steps 1-3 above and add 4) push to remote to the list of steps. Like I said, same as SVN just with an additional step. Anyone familiar with git will tell you that if you are using that model you are missing out on some of the best and most powerful features of git, namely branching. While SVN provides the ability to make branches I never really got the sense that they are a first-class object in the same way that they are in git. To get a sense for why branches are important here is my work flow for a standard task that I am working on:

1. create branch
2. write failing tests
3. write code to make tests pass
4. commit
5. see 2 until task complete
6. merge branch back into master (smushing commits)

Anyone who is familiar with TDD will recognize steps 2 & 3 as the standard red-green loop but there are two other important things to note. If you are working in a tight red-green loop (which you should be) then step 4) is gonna result in a lot of commits which although the tests pass are probably not something you want commited into the record books forever and ever. In the standard SVN model you could either make all of these commits and spam the commit log with bunches of commits (and merging, ugh) or alternatively just not commit. The former is definitely undesirable and so is the ladder since its important to be able to go back to the last working state. The nice thing is that git allows you to get what you want in multiple commits which you can revert to while not spamming up the commit log. What you notice is that in step 6 when we merge back into the master branch we can either rebase (moves all the commits over) or my preference which is to merge without committing (smush commits). What this does is allow you to take all the commits you have made, smushes them together and then allows you to review what you have changed before you actually commit it into the master branch. For those of you wondering what I'm actually doing in step 6, assume you had your branch called "featureX" which had all the commits on it, you would checkout the master branch (git checkout master) and then do a merge using "git merge --no-ff --no-commit". Then you should have all the changes you made on the featureX branch staged ready to be committed, after you review it of course. It also helps that Macs have GitX which is one of the nicest diff tools I have seen to date.

One other awesome feature of Git that I use all the time (read: way too much) is stash. How many times have you made a change to something that you didn't want to commit but just wanted to keep around in case you needed it later? I seem to need this all the time, whether its simple config changes or some silly snippet I want to try out. If I want to keep that code around without having to commit it I can just do "git stash save some silly snippet" or more concisely "git stash" (the former adds a label to it). Then anytime something like that comes up you can just stash it away for later use.

So if you've been scared to make the switch from SVN to git, don't be scared, it wont bite.

Linux tools that make you go zoom

2009-08-15T18:31:00.000-07:00

Over the past few years I have used Linux almost exclusively for programming and over that time I've gotten comfortable with a lot of the standard utilities. Want to find a file? Use find. Want to find a regexpn in a file? Use grep. It almost seems like a given. Until a little while ago I didn't even question whether there were alternatives since they are pretty much the standard go to tools and have been around forever. More recently I have begun working on larger source trees which I am less familiar with and as a result find myself using these tools a lot more frequently. Whereas previously waiting a bit was acceptable it quickly became a drag for me and I went looking for better alternatives.

I know it seems like a bit of heresy to replace the beloved grep but honestly ack makes it worth it. To show you the difference I'll do a quick search for the string "foobar" in a source tree that's ~500MB and ~9000 files.

Yeah. Seriously. The recursive grep on the directory took over 40 seconds to make it through while ack took under half a second. Anytime you get a two orders of magnitude speedup and its more convenient/fewer characters to type I find it hard to argue against that. The nicest thing about ack is that it automatically ignores all the standard version control files (e.g. .svn) which usually cloud up normal grep results with unwanted results. Of course all that is configurable. So how do you get this magic tool? Its simple:

If your a TextMate user you should be happy to know that there is a awesome bundle which allows you to find text in files really quickly called Ack In Project. I've been using it and found that the interface is awesome and much faster then the grep script I was previously using. The only downside to ack is that the default colors for it are pretty ugly but that can easily be changed via environment variables.

The other thing I frequently need to do is finding files in the file system. Its always something like "where is the config file for X again?" Since those things are frequently in the most bizarre of places I usually end up searching the entire file system which takes forever. Take a look...

Yep, find takes a whole 5+ minutes to get through the file system plus it spits out a bunch of permission denied messages where are really unnecessary in this context. On the other hand locate speeds through and finishes in a second and a half. Nice. The secret sauce is that locate uses a database of the file system the find things much quicker. Since its a standard linux utility and its doing it anyways, so why take advantage? It should be noted that locate searches the entire file system each time while find can search only relative paths so if you are searching a small folder find is usually faster while locate stays pretty constant.

While I'm a fan of the standard linux utilities, but anytime you can find solutions that really speed up things that you do frequently with little or no downside it seems almost foolish to not take advantage of. What are you waiting for?

How to handle mistakes

2009-07-31T21:17:00.000-07:00

We've all heard the story far to often, company X makes a major mistake and does something they shouldn't have done, customers get outraged and start making a major hoopla, and only a few weeks later do they receive any admission of the mistake or maybe a weak apology. This type of situation happens way too often to count and really no one wins. Customers are usually left upset and companies are left with a tarnished image.

Last week Amazon made a bit of a slip up when they decided to delete copies of George Orwell's 1984 and Animal Farm from all Kindle devices. Apparently the copyright holder claimed that the version of the novel that was being distributed was violating copyright law. Amazon complied with their request by remotely deleting all copies of the novels and then refunding customers they cost they paid for the book. Amazon did notify users that the novels had been deleted and their money refunded, but only after the fact which didn't help things much.

So as the story goes above, customers get up in arms, allusions to Amazon being Big Brother are being typed faster then you can imagine, things are quickly getting ugly, what does Amazon do? Have Bezo's himself posted an apology on the Kindle forums. Short, sweet, he flat out admits that they got it wrong, no ifs-ands-or buts. While I applaud them for addressing the issue in such a straight forward manner, they made quite a daring decision in posting that apology in a public forum while allowing open comments. When I first saw that there were comments I not doubt expected them to be filled with much of the same venomous Orwell references that littered much of the press on the event. Much to my surprise, the vast majority of the comments I read were from from poisonous, in fact most of them were thanking Bezos for apologizing.

Being a skeptic I though that perhaps it was some of Amazons famous "is this post helpful" ranking magic happening to get some of the nastier stuff out of there. After clicking through the first 5 of so pages of comments (include numerous which apparently weren't very helpful) and there was little change. I'll admit that I didn't go through all 27 pages of comments and I'm sure there is quite a few less then helpful posts towards the end of it, but regardless of that fact, I have to admit that posting on forum with open comments is incredibly ballsy move but it seems to have worked out. Bravo.

On being Anti-Anti-SQL

2009-07-02T20:03:00.000-07:00

There seems to be a draft forming as a result of a new movement of people who are "Anti-SQL" or "Anti-RDBMS". This article for example talks about the first meeting of what is being called the NoSQL community, and they are not alone. There are plenty of articles and blog posts online about "Thinking Beyond the Relational Database", "Ten reasons why CouchDB is better then mysql" and Beyond MySQL, a paradigm shift from RDBMS, noticing a pattern? While I'm sure there are many reasons why people are Anti-SQL the key points of contention are generally (in no particular order):

1. It doesn't scale well in terms of request rate
2. It doesn't scale well in terms of data size
3. Too many unnecessary features (e.g. joins, transactions)
4. They are slow

The overarching argument here is that since relational databases are very general-purpose and have to support just about all use cases that they suffer as a result, either in terms of performance or scalability. This observation has lead those members of the anti-SQL camp to take the opposite approach, namely start with a minimal feature set and only add that which is necessary. This approach has resulted in key-value stores like BerkeleyDB, Tokyo Cabinet, and Memcached, column-0riented databases like BigTable and its open-source counter parts HBase/HyperTable, document databases, graph databases...and so on. While each of these datastores are unique in their own right, I'm not going to spend time discussing the relative merits of one over the other and will instead point you to the awesomely titled (and informative) talk "Drop ACID and think about databases", this concise writeup which compares their relative features, and this very matter of fact review of many of them.

Having used a large number of such datastores I can confidently say that if the use case you are looking to fulfill can be met by one of the previously mentioned datastores that there are some great performance gains to be had. Performance gains are awesome, but the crucial part of the previous sentence is whether you can find a datastore that meets your use case. Like it or not, we have all become accustomed the relational database. It is an important part of just about every web framework, and is the crucial component of a large portion of web applications (chicken and egg discussion left as an exercise for the reader). While there are likely some cases where problems fit brilliantly into such systems, there are many which simply do not. My advice on this is simple, if you need the performance/scalability or it is a natural fit for your application then by all means go ahead, but the second you have to start employing any sort of trickery to get your app to fit into that model, stop right there, you are fighting a battle that you've likely already lost.

This leads to my last point and the one that inspired the title of this post which is my opinion of being Anti-Anti-SQL. That is not to say that I am against non-relational databases, I am in fact a very big believer of them and have extensive knowledge about quite a few. Instead, I have an issue with those who are trying to demonize the relational database into something that is outdated, decrepit, and woefully inept at the task it performs. Granted the relational databases is not without its issues, but realistically the number of applications which would fit into a non-relational database is fairly small, and the number which require it for performance reasons is even smaller. There are the Google's, Amazon's, and Yahoo!'s of the world but for every one of them there are an infinite number of tiny web services who could only dream of hitting the scalability bottleneck of their database.
If the Anti-SQL community truly feels like the relational database needs to be sent out to pasture then what they need to do is simple, change the way we (application programmers) think. Almost all web applications being developed today assume the existence of a relational database and code accordingly. Non-relational databases are generally not considered until 1) it is a necessity as a result of scale/size or 2) the developers previously experienced 1). Instead the goal should be to make the non-relational database into a first class citizen, the default unless a relational database is absolutely necessary.

This is clearly not an easy task, but in order to have a chance it is crucial to focus on the things that can make a difference. If the adoption of high level scripting languages has taught us anything, it is that people are willing to sacrifice performance for ease of use. While you could spend time bit-twiddling your C code trying to eek out a .5% improvement, instead spend time writing a good Ruby/Python/PHP plugin to interact with your datastore. In essence it all boils down to this:

While performance gets your foot in the door, usability makes the sale.

Got Infinite Scalability?

2009-06-30T21:38:00.000-07:00

I was perusing the usual RSS feeds today when I ran across a link to New Yorker article written by Malcolm Gladwell. The article is apparently a pretty lashing review of a book titled Free by an author named Chris Anderson. Since I'm a pretty big fan of Gladwell's work I thought I would head on over and take a gander. Only one problem, the website has been timing out for the past 30 minutes. Thats right, the New Yorker is likely getting more traffic right now then it received all of last month, yet its servers are seemingly balled up in the corner weeping and asking it to stop. Considering the fact that the article is currently is linked to by Slashdot, Techmeme, and Silicon Valley Insider (and probably a ton of literature focused blogs) I can't say I am surprised that they cant handle the traffic, thats a flash crowd and a half.
Its situations like these where having the ability to scale out your website is quite critical. It's not like they didn't know it was coming...they must have seen a decent uptick in traffic when the Silicon Valley Insider article and Techmeme link came out yesterday morning. I'd be very surprised if there wasn't some inclination that things needed to be ramped up.

The cynic would note that flash crowds from Slashdot are notoriously difficult to deal with and that one gets almost no warning. While this is true, the Slashdot article came out over six hours ago. By now they could have gone down to the local Fry's, bought some cheap servers, formatted the hard drives, and had their entire stack up and running if they wanted to. From the looks of a quick whois on their domain it seems like they are hosting their own servers, whoops.

Since it is apparently my new favorite website, I ran a pagetest on their site (sorry servers) and the results go to show that what I was talking about in my last post is pretty damn important. If you take a look at those results you will see a waterfall so big you'll want to swim in it. After 11 seconds and an impressive 123 requests the website should be nice and loaded for everyone to enjoy. Alright, first load performance is hard but at least they are using expire headers right? Ooops. Yep, thats right, the second page load takes a whole 116 requests, and just about all of them are 304's (Not Modified). The fact that its not actually fetching the data makes the second load a little faster (~7 second) but its still pretty slow, and when you are getting tons of traffic having to serve over 100 requests per page load makes a server quit on you much faster then it should. It should be clear that they aren't using multiple asset hosts or any thing fancy like that, but what is the kicker of the whole thing? Not only are they loading a zillion objects, but take a guess where their server is located. Dallas? San Fran? Virginia? Nope, how about the Netherlands. And there is my queue to stop trying to explain what on earth they are thinking...

Optimizing Blog Performance For Fun and Profit

2009-06-23T22:12:00.000-07:00

At some point over the last year I have developed into a speed freak. No, not the kind who hang out in back alleys doing questionable things, the kind that wants to make websites fast. Just in the past year or so I have picked up a ton of relatively simple techniques for 1)figuring it out why a website is slow and 2)improving the performance. I was taking a look at my blog this evening and while I did not feel like it was particularly slow, I knew that there were a lot of things which could be improved so I thought it would be a good exercise. Since it is a relatively simple and closed experiment, it seemed worth while to document here so that others could use it to guide them, so enjoy :)

Lets first start by determining why my blog is slow so that we can focus on what could yield the most improvement. If you have not heard of or read Yahoo!'s "Best Practices For Optimizing Websites" now would be the opportune time to do that. That document has become a Bible of sorts for website optimization, they do a very good job of thoroughly covering all different aspects. One thing to note is that in many cases there are practices which simply don't apply or are not an option in many contexts (e.g. use a CDN), don't worry, that's fine. Its definitely not an all or nothing system, so do as many as are feasible.

Now that we've read all of these best practices, we want to see where our site does not follow them. For the rookie optimizer, you may be surprised to learn that there are actually quite a few utilities available for studying website performance. Not surprisingly, the most popular is a Firefox plugin named YSlow which was developed by the folks at Yahoo!. Not to be out done, Google has created their own Firefox plugin called Page Speed, while the interface and rules are slightly different it is similar to YSlow in more ways then not. While both of these plugins are great for quick checks and such, what you will often notice is that performance can vary greatly from refresh to refresh which can be quite frustrating when you want to find out if change X actually helped or hurt your website. As a result, I use the terribly-named Pagetest, it follows all the same rules as YSlow it simply provides additional useful functionality like running trials multiple times, along with visiting multiple urls, and my favorite the waterfall chart.

So I went ahead and put my website on Pagetest, had it do three runs and then waited for the results. Instead of putting images on my site of the test results, here is a link to them so you can explore along with me, I'll be discussing run #2 since its the closest to the average. To start lets look at the waterfall view. On top it provides a nice little summary of the most important information, so from there we see that the load time is 4.5 seconds. Just for a point of reference we check Alexa to see that Google has an average load time of 0.7 seconds and Yahoo!'s average is 2.6. While my site is quite a bit slower, it is somewhat expected when you consider the amount of time and man power both companies are able to dedicate to optimizing even the smallest of things.

In looking at the waterfall view there are a few important thing to point out, 28 requests are made most of which are images from cs.ucsb.edu (which is where I was hosting the images for my blog). Since that was the first time the page was being loaded all of the images/css/scripts needed to be loaded so this is in effect paying the full penalty since you have to get everything. Ideally, since most of that content is static we could utilize the browser cache such that those objects don't need to be fetched again. We can then take a look at the Repeat View which is the second time the page is loaded, if caching is properly utilized we should see that a lot of the static content is not being reloaded since it is in the browsers cache. In the repeat view we notice that the load time has gone down to 2.4 seconds which is a nice improvement, but that there are yellow lines on most of the requests to cs.ucsb.edu for images. Below in the request details it shows that all of those requests are returning a 304 status which is basically the web server saying that we already have the most up-to-date version of the object, so use the version we already have. While that is definitely faster then before it is important to notice that over 1/2 of the time is wasted making those requests. If you head over the Performance Overview you notice that all of those images have little x's marked in them under Cache Static. What this means is that the server is not setting expire headers on those images. Without an expire header the browser can't tell if the images are still valid when reloading the page, thus it still makes those requests just to check if they are up-to-date only to get 304'd.

Another observation to be made is that even though there are a lot of requests being made, they are not all happening in parallel. Ideally, once the CSS was fetched the browser would know that there are 10 more images to get and start downloading them in parallel. Unfortunately that doesn't happen. The reason is that most browsers are configured to only allow two parallel connections to a given hostname, so even after the css has been fetched and the browser knows it needs to fetch 10 more images from cs.ucsb.edu, it can only grab two at a time. That is a bummer since the images have already been optimized (using smush.it) so they can't get much smaller. There are solutions for this problem which we will discuss below.

Now that we know all of this information it is time to move to phase two which is actually improving the performance. Since I have no control over the server which was hosting the images, I could not configure it to use expire headers and thus a more appropriate solution was needed. While I could host the images on a CDN, they are generally pricey and its not something I'm interested in paying for on a relatively low traffic blog. Instead, I chose to use Picasa, Google's image hosting service. A few of the reasons for choosing Picasa 1) it's free, 2) its fast, 3) I could just login with my Google account instead of needing to create another account somewhere (e.g. Flickr). I grabbed all of the images my site used and then uploaded them to Picasa in a snap. After that I just had to switch all of the images references from the old urls to the new Picasa url. After doing that I reran the performance test and here are the results . Right off the bat we see the load time has gone down to 3.35 seconds on the first load, which is over a 1 second improvement from the original 4.5 second load time. Its important to note that this is not a result of expire headers since that only comes into play on the second page load, something else is happening here. Unlike in the previous case where images were being downloaded from cs.ucsb.edu two at a time, we now see that many more images are being downloaded in parallel as many as 10. If you read Yahoo!'s Best Practices site you will remember the "split components across domains" recommendation which is what is improving the performance. Previously we found that only two connections can be made to a hostname at any given time, but its important to notice that host1.website.com is a different hostname then host2.website.com. It fact can be exploited to allow more then two parallel downloads to occur for a given site. Taking a look at the waterfall chart we see that now the images are spread across lh3.ggpht.com,lh4.ggpht.com, and lh5.ggpht.com (the urls for different Picasa servers). Since the images are spread across these servers the browser can make two connections to each of the three hostnames (six total) at any point in time. A relatively simple and harmless change, but it shaves a whole second off the page load time.

Now we can shift over to the results of the Repeat View to see how well the page performs with a hot cache. The summary shows that the load time dropped to 1.46 seconds, down from 2.4 seconds previously. This is where the expire headers are kicking in. Looking at the waterfall chart we see that only 4 pages are loaded, and this is because the rest of them have been cached, in particular the images. Since the expire header is set there is no longer any need to make the image requests (two at a time) only to get a 304 back from the server which wastes both time and resources. Instead those images are taken from the browser cache which is much faster and as a result the load time is cut down quite a bit.

While the above description was quite long, it was two simple changes that cut down of page load time by 25% and 40% in the first and second load cases. I didn't have to shovel out a bunch of money on a CDN or pay some expensive consultant to do it for me, it was all free and didn't take much time at all. While these to optimizations are by no means the only ones available, they are usually easy to implement and can give you big gains. While I could continue to improve things via more advanced or cumbersome improvements, there is likely going to diminishing returns either in terms of performance/maintainability/cost which makes me less inclined to do it.

Whats the moral of the story? There are easy ways to make your website faster, do it and make your users/viewers happier (one second at a time).

Browser Knowledge

2009-06-20T22:24:00.000-07:00

Just when I start to think that our culture has finally become as tech-obsessed as I am, I get a reminder that they are not and I can breathe a little easier. What the hell am I talking about? This interesting little video put together by a Google intern asking people what a "browser" is.

Alright, so clearly some of them are confusing a web browser (e.g. IE/Firefox) with a file browser (e.g. Explorer on Windows, Nautilus on Linux), and admirable mistake. The fact that on Windows there is Explorer and Internet Explorer both of which do very different things doesn't help the cause very much either. Why exactly so many people think that Google/Yahoo is their browser, I don't really know what to make of that. I guess since that is the first thing they see when they open their browser, they assume that is what their browser is as opposed to the icon/name that they clicked on to open it....maybe...who knows.

Perhaps my favorite part of the video is the guy who proudly says he uses Firefox, and then when asked why he admits that a friend removed (probably from the desktop) all of his other search engines and said to use Firefox. What a good friend.

Why Bing?

2009-05-28T11:10:00.001-07:00

At D7 today Steve Ballmer announced that Microsoft is going to be releasing a new search engine called Bing!. Apparently when asked why the name Bing! Ballmer gave a few pretty generic reasons like its short, easy to say, and works globally. Sure, thats all good and nice, but that really doesn't explain why they picked that name.

Had you asked me I would have said, "Because Bing! is the sound you make when you know that answer to the question, and thats exactly what Bing! does."

Arrogance

2009-05-14T23:07:00.001-07:00

Typical IE arrogance, they don't see Firefox as a threat? Sure.

Is reporting outages really news worthy?

2009-05-14T22:20:00.000-07:00

If you are foolish enough to read as much tech news as I do you probably heard about Google having some issues today. Reports said that search, GMail, blogger, and a few other services were hit for about an hour [official announcement here]. Thats right, reports said. As is people reported on this. If you didn't notice, every single word in that sentence was actually a link to a different article, reporting on the down time. If I was really determined I would have done that with this entire article, I'm sure it possible I am simply unwilling to copy paste that much to make my point. My favorite part of most of these articles is that they all seem to contain a phrase similar to "We can access Google just fine, but people on Twitter are reporting issues." They are absolutely right, when I first heard about this I did a twitter search for everyones new favorite #googlefail and sure enough tweets were popping up faster then I would glaze my eyes over them. Literally within 5 seconds of doing a search, it already said that 80 more people had tweeted with #googlefail in it. Here is my question, why is it that no bloggers seemed to be affected, but everyone and their grandmothers on twitter was going bezerk, coincidence?

Not being able to access websites is a tricky issue, its often not very clear what the issue is. Is it my computer? Is it my router? Is it [favorite website of choice]? Or is it somewhere is the middle? Thats why I actually see value in websites like http://downforeveryoneorjustme.com/, there are simply lots of times when you have no idea. There is a great quote by a brilliant man name Leslie Lamport which says:

A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.

While I agree with the quote, I think I am going to have to modify it slightly for this occasion:

A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unable for doing any real work, and as a result your only option is to complain about it on twitter.

Honestly, they put solitaire on your computer for a reason.

Kindle For Kids

2009-05-04T22:45:00.000-07:00

Word is out that Amazon is planning on releasing a larger version of the Kindle on Wednesday. The new device is supposed to be geared towards magazines and textbooks which require a much larger screen then regular novels. If you've been here for a while you would know that I have high hopes for bringing the Kindle into the textbook market. That industry (empire?) is desperately lacking in competition and could use someone to ruffle their feathers a bit.

Unfortunately, the more I see of Amazons strategy with the Kindle, the more concerned I become. The current Kindle sells for $359 on Amazon and has a 6" screen, a few ticks on the abacus would tell you thats just under $60/inch. Now they are coming out with one with a bigger screen? Don't they know people can't get bank loans these days? In all reality I'm sure they don't charge per inch of screen real-estate but they have held on to that price point like it was the only one in town. My uneducated guess is that they are going to actually bring in the new Kindle above the $359 price-point and keep the Kindle 2 where it is. Lets just throw a number out there for good measure and say $429.

Once you get past its inflated price it will be interesting to see how they deliver on features. People who intend to use this as a replacement for textbook are much more demanding then those who are using it to read the latest Daniel Steel novel. As I previously harped on, search is critical since people are way too used to finding what they want very quickly. Beyond that, considering all the crazy sidebar notes/sticky notes/highlighting schemes people employ on their textbooks having a replacement or improvement on that would be important.

For a good opinion on the priorities in Kindle features I think this fake conversation between Jeff Bezos and Steve Jobs captured an important point:

Jobs (laughing): Surf the Web? On an Etch-a-Sketch?

And then proceeded to sum up my opinion on "value added" features:

Bezos: The Web is a value-added feature.

Jobs: No features are value added. They're either features or they're not.

Update: Wow its actually more expensive then I had guessed, it rings it at a hefty $489. At least there is free shipping!

Design Question

2009-04-22T19:21:00.001-07:00

Why is it that for blogs its perfectly natural (and visually comfortable) to have the posts on the left or center of the page with the archive, tags, etc on the right. Whereas with RSS readers its perfectly natural (and also visually comfortable) to have the list of feeds on the left with posts on the right.

Or the more pressing question, why is it that if you would consider switching either of the two around it would seem ugly and uncomfortable?

Amazon-as-a-Platform(-Provider), Making Hadoop A Bunch Cheaper

2009-04-02T11:12:00.000-07:00

I have to say that saw Amazons announcement today for Amazons Elastic MapReduce caught me off guard quite a bit. Having previously worked on a project which used a reasonably sized (100 nodes) Hadoop cluster running on EC2 I am familiar with many of the pains of setting up and running Hadoop on EC2. The reason why I found this announcement so surprising is that it demonstrates Amazons willingness provide even more middleware services. Many of the other AWS services like S3, SQS, and SDB provide very fundamental services, namely storage and queuing. I'm not saying that everyone on AWS uses all three services, but instead that just about everything running on EC2 requires some form of storage and some form of queuing. Thus if Amazon can provide the services that just about everyone needs, there is a decent chance that Amazon can get some of those people to use their services either as a result of convenience or cost. MapReduce is definately quite popular right now, and for good reason, but it is a far-cry away from being as fundamental as something like storage.

The point of that whole discussion is this, Amazon just made their first true Platform-as-a-Service (PaaS) offering. I know Amazon is the Infrastructure-as-a-Service (IaaS) provider, but Hadoop is absolutely a platform in the same way that Rails or Django is a platform. Sure it doesnt serve out websites (unless you count the JobTracker) but that is in no way part of the definition of what makes a platform. This is a particularly interesting move becuase it opens up the possibility of Amazon providing more Platform layer services (Rails anyone?) and encroaching on the space currently occupied by the Google AppEngines and Herokus of the world. One might ask why would Amazon venture into already occupied territory, why compete with people already providing those services. Its simple, what is likley the most common use of Amazon EC2? If you guessed hosting websites you are correct, and if you guess hosting Rails websites you get a bonus point. Since that is the case, it is pretty clear that EC2 is providing things that Heroku is not, wether it be flexibility, cost, or otherwise. So why not exploit that fact, make your customers happy and make money from it as well?

The other thing that I found absolutely stunning about the Amazon announcement is the pricing. Take a look at this:

Go ahead, double take if you have to, make sure you got those decimal places right. Yeah so running a job using Elastic MapReduce (EMR?) is effectivly 15% of what it would cost to run it yourself on EC2. Ridiculous. To be honest it doesnt not make any sense to me that they are able to offer such a discounted price for a service that gives you the same exact machine as what you would get for $0.10/hour. I am going to have to think about that one for a while.

Either way, that made something that was already dirt cheap ($0.10/hour) into something even cheaper than dirt ($0.015/hour), and I am very excited about the prospects and implications. As stated on CNET "Bring your datamining to us".

Update: Dave has correctly pointed out that I missed a very big sentence in the Elastic MapReduce description which is "Amazon Elastic MapReduce pricing is in addition to normal Amazon EC2 and Amazon S3 pricing." That makes quite a bit more sense.

Even still, the 15% premium is a tiny price to pay to not have to deal with bringing up and tearing down servers all the time, along with the headache of actually getting the thing setup.

Are Macs Really More Expensive?A Practical Study

2009-03-31T14:51:00.000-07:00

Microsoft recently started a new advertising campaign which truly speaks to the consumers wallet in these troubled economic times. Simply put, the advertisement (can be seen here) states that PCs are cheaper than Macs, and what people are really concerned about these days is price. This ad is very much in line with the recent comments made by Microsoft CEO Steve Ballmer who stated that Mac users are "paying $500 more to get a[n Apple] logo on it". This however is not ground breaking news; the fact that Macs are more expensive (commonly known as the “Apple Tax”) is one of the most common retorts for the PC people in the Mac vs. PC debate. This new advertisement has definitely sparked up this debate quite a bit with articles being featured on Slashdot and Gizmodo, among many other tech sites.

I have found myself in the Mac vs. PC debate quite a few times, generally when a friend asks for advice on buying a new laptop. In general the argument goes as follows, “PCs are cheaper, and the baseline specs are going to be higher, but Macs are known to be more resource efficient so the higher specs don’t necessarily mean a faster machine.” The conversation then generally returns to the issue of price where it is pointed out that a decent low-end Dell or HP laptop can be purchased for $700-$800 whereas with Macs the low-end is $1000-1100. The fun really beings when it is pointed out that while Macs are more expensive they generally have much higher resale value. While this is generally true, it was never clear to me how much higher the resale value actually was, and if that higher resale value actually covers the higher initial cost. That is why I chose to do a little research on the topic to answer the question, are Macs really more expensive than PCs?

The Experiment

In order to determine which laptops are more expensive I would first find popular laptops which were released approximately three years ago, determine the resale value of these laptops today, then determine the “adjusted retail price” which is its initial sale price minus its resale value three years later. The decision to pick laptops which are three years old is twofold, first three years is generally the time frame which many people begin selling their old laptops and looking for new ones. Also, it was about three years ago that Apple switched from PowerPC to Intel. Based on these two factors three years seemed to be the ideal time frame for how old the laptops should be.

Once that time frame was decided upon, I the next task was to find popular laptops which were released around that time. After a little bit of hunting around I found the following laptops:

• Dell Inspiron 1501 ($649), Released 11/2006
• Dell Inspiron e1405 ($779), Released 9/2006
• Apple MacBook Early 2006 ($1099), Released 5/2006
• Apple MacBook Late 2006 ($1099), Released 11/2006
• HP Compaq nc6400 ($1119), Released 9/2006
• Lenovo Thinkpad T60 ($1399), Released 11/2006

These laptops were all released between two and a half and three years ago and are some of the most popular models offered by the given manufacturer at that time.

Now that the laptops were picked their approximate resale value needed to be determined. While there are services which will tell you an approximate value of a laptop I’m not particularly confident in their accuracy. Instead, I wanted to see how much money these laptops would fetch if they were being sold by your average Joe. What better place to look then everyone’s favorite auction site, eBay. Using eBay I then located successfully completed auctions for each model listed above. When picking completed auctions I set aside a few ground rules, it had to be in working order and no large defects (as described by seller), it could not be refurbished, and it could not have a warranty. Considering laptops which violated any of these conditions would have significantly increased the variance in the prices (consider a “for parts” laptop vs. a working laptop with a 2 year warranty).

I chose to sample three auctions for each of the laptop models to try to avoid any oddities in the sale price of a particular laptop. While this is a tiny sample size it was sufficient to get a reasonable estimate for prices, particularly since the sample pool was fairly constrained. It should be noted that some of the auctions for a given laptop model had slightly varying specs (e.g. 60GB vs. 80GB hard drive or 1GB vs. 2GB of RAM) but most were minor modifications likely done by the manufacturer, I avoided any laptops with significantly improved specs (e.g. 4GB RAM). While these improvements were likely paid for by the original buyer, determining how much these improvements cost at the time of purchase is an exercise I am unwilling to subject myself to. As a result, each laptop is considered to cost its base price, and thus the percentage of cost recouped upon resale is an upper bound.

Results

The full results of this experiment have been recorded in this Excel Spreadsheet, they have been summarized in the figure below.

Figure 1

Figure 1 displays the initial sale price of each laptop along with its adjusted price which is simply its initial price minus the resale price. The adjusted price is in effect its “actual” cost assuming the laptop is resold in a working condition three years after it was purchased. From this figure it is clear that while the Dell Inspirons are sold at a considerably lower price point than the Macs, the adjusted price of the two models are very similar, particularly in the case of the e1405 which has a Intel Core 2 Duo processor (as opposed to the AMD Turion in the 1501). This indicates that while the Macs are in fact $200-300 more expensive than Dells, that extra cost is recouped upon resale as a result of Macs increased resale value.

Next is the comparison between the Macs and the pricier PCs, namely the HP and Lenovo models. Both these models had price points above that of the baseline MacBook, with the Lenovo fetching a pricey $1,399. From these results it is clear that the PCs which do cost more than their Mac counter parts do not recoup nearly as much of their value upon resale and result in adjusted prices significantly higher than Macs.

Figure 2

This is illustrated more clearly in Figure 2 which displays what percentage of the initial sale price which was recouped upon resale (higher is better). Three years after being purchased Macs recoup an impressive 51-52% of their initial sale price while Dells net around 36%. The more expensive HP and Lenovo recoup a measly 26-28% of their high initial prices. In the case of the Lenovo, its $1006 adjust price is nearly twice that of the Macs and Dells. In fact, while the HP and Lenovo are nearly double the price of the Dell e1405, they only receive an additional $20 and $120 in resale value.

Conclusions

After seeing these results I am pretty clearly convinced that Macs are not more expensive than PCs. In fact, Macs are very much comparable pricewise to much cheaper Dells when resale price is factored in. Macs are much cheaper than more expensive models sold by HP and Lenovo which do not recoup nearly as much of their high cost upon being resold.

While I’m sure this will not put to an end the Mac vs. PC argument, the old argument about Macs being more expensive is simply not the case, especially when considering higher end PCs.

A special thank you goes out to Bryce Boe who proof read this post for me.

Blue Steel And Fun With Colors

2009-03-30T23:44:00.000-07:00

Welcome to my new and improved blog now known as Regular Expression. It is now being hosted on the domain http://regexprn.com but http://jmkupferman.blogspot.com should continue to work (including RSS).

After short amount of time coming up with a new name, a long time looking for domain names, a relatively short time making the design, and a really long time trying to make it into a Blogger theme, its finally made it. Its incredible how the "easy" parts take the longest time while the more difficult parts just seem to work themselves out.

When I was working on the design I ended up getting myself into a bit of a snafu which lead to a somewhat interesting journey that result of which is the design you are looking at. I liked the name Regular Expression because like the pun, its pretty self-explanatory, and was able to snatch up a nice short domain name to host it on. Once I had the domain name it was time to get going on the design. I had some ideas floating around in my head, in particular I wanted a large text logo with regular expressions in the background. I recalled seeing a tutorial that had background text and a logo which was made up of the background text (available here). After a bunch of tweaking and changing I ended up with this:
Unfortunately I was unable to get reasonably smooth looking text, so it ended up looking more like I had drawn the logo on a fabric softener sheet, and thrown it in the wash. I decided to abandon trying to write the logo out of the background text and just wrote it out normally and that looked much cleaner, and easier to read. I then went ahead and added a little navigation bar, in this case it is more of a link bar, but it looked nice and I was happy with the little bit transparency.
Since its just a blog all you really need is a header and color theme and your all done right? Sure, its just a matter of adjusting the template to fit the color scheme and you should be on your way. This is where things got a bit tricky. First, when trying to modify the template I couldn't get the links to work. It was incredibly bizarre since they worked in the simple HTML site I made and as far as I could tell Bloggers template engine wasn't doing anything funny. I just played around with the HTML using Bloggers template preview feature to try to nail down the problem. (insert a few hours frustration and relearning of the loveliness that is floats). Long story short, Bloggers template preview has a bit of an issue. It would render the links correctly, but they simply weren't clickable. It wasnt until I actually submitted a template (and made my blog ugly for a few minutes) that I figured out that it was preview that was the issue. How frustrating.

I finally got the template looking alright so next it was time to figure out colors. What I very quickly figured out is that the black-grey-red combination I had was going to be difficult to work with. The accent color (red) was particularly tricky since its very difficult to find a shade of red that is readable for any decent sized line of text, say a title. While I could have used it in small spots like an underline or the numbers in the sidebar, it was gonna make finding other colors for text, titles, and links quite a bit more difficult. After looking around for a while to see if I could find a few ideas on what would work, I figured I would switch things up and try a lighter design.

I had pretty quickly arrived at something which I thought looked pretty nice, still kept the theme, and was able to use red in a readable way. I was pretty surprised how quickly it came together and how nice it looked, and then I remembered why. It was a design I definitely had already seen before. Apparently my subconscious remembers websites incredibly well, even down to changing the red links to black with an underline on hover actions. That's no fun. So back to the drawingboard I go.

I was playing around with templates and ended up with my light header on top, and old (dark) blog design on the bottom, and said hmmm. I liked the blue that I was previously using so I went ahead and changed the red in the logo to a nice cool blue, switched up the link colors and was on my way. The only problem left was the header, it was dark while the main background was white. I tried changing it back to the white one, but it wasnt working. Instead I kept the black, and just eased the transition between the two a little more. The first way was to just add a little fade below the nav bar, pretty simple. The second way is a little bit tricker and I actually stumbled upon it by accident, but I liked it. I'll give you a hint, while the nav bar is somewhat transparent it darkens the background as opposed to lightens it.

After all of the messing around I finally ended up with the page you are looking at.
Whats the moral of the story? Start with a diverse color pallete (kuler is a nice tool for that, thanks Mike) and then let the design come from that. Its a lot easier to not use a color in a pallete than it is to add an additional one. It will also likley stear you away from having a very plain black and white website, but dont go overboard.

iPhone OS 3.0 Quibbles

2009-03-12T21:13:00.000-07:00

The announcement a few days ago that the iPhone OS 3.0 will be previewed in the coming week generated quite a bit of buzz among tech news sites. With all sorts of excitement and nothing to really talk about, it seems like the conversations on these matters inevitably turn into "Comment on what you think it will be" type of affairs. Interested in what others think might be on the horizon for next weeks preview I took a look at some of those comments and what did I see....

"Cut and Paste, please?"
"come on MMS and copy/paste..."
"unless this adds copy and paste, mms etc this is going..."

Really? That's what people are looking for? MMS...and copy & paste...really.
MMS just seems like a huge waste to me, if you though texting was a ripoff, you haven't seen MMS charges. Furthermore, why not just email? iPhones can open email attachments, particularly images. I'm not quite sure what people find so incredibly attractive about this, but perhaps that just me.
The title of this post is iPhone OS 3.0 Quibbles, emphasis on quibbles, because that is exactly what copy and paste is. I have tried my hardest to think of cases where I would have used copy & paste, and I would have to say that the number of instances of that occurring is less than a handful. In particular I remember one time when someone sent me an email which contained their phone #, unfortunately the iPhone did not recognize that it was in fact a phone number (it does, but that feature is somewhat spotty) and didn't provide the option of clicking on a link to dial it. So I had to remember a phone number for a whole three seconds while I entered it into the phone, no biggie. I have a sneaking suspicion that the people who are really excited about copy & paste are frequently typing "OMGOMGOMGOMGOMGOMGOMGOMGOMGOMG!". If that is the case, then please no copy & paste, there is plenty of that already.

So the natural question to then ask is, if these features are blaze, which ones should I be hoping for/expecting? Landscape keyboard is a popular choice, I don't think you can really argue against this, who knows why it isn't done yet.
Ultimately, one of the most important features that people don't mention enough is lovingly known as Push. It was supposed to launch in September 2008 (Apples words not mine), and yet there has been hardly a peep about this. No mention of it, no explanation, nothing. This is somewhat unfortunate since the addition of Push is likely one of the biggest changes we will see in the iPhone. The classic example is apps like AIM which could actually use push to notify users of new messages without having to open the app itself. AOL has gotten so impatient that they chose to hack their way around this by offering to send users texts when they receive a new message, also allowing them to remain logged into AIM for up to 24 hours after then exit the app. Email is obviously another no-brainer. No reason to waste battery life polling every X minutes, push is clearly the right way to do it.
There are tons of other apps out there which could use Push to improve user experience. How about RSS feed readers? Sports scores? I would mind being notified every time the Lakers were playing. I'm sure twitter'ers wouldn't mind being notified upon getting tweeted to (that cant be right).

So while people continue to hope for the improvements on the minor quibbles they have, I'm holding out for the big improvements which can change the game.

Quality In Error Messages

2009-03-10T22:57:00.001-07:00

You know your running Linux when....

I mean at least Microsoft has the decency to make their error messages so cryptic and obscure that you have no choice but to give up.
But this...I'm just so torn. Was it success? Was it an error? Or perhaps it was a successful error? Well in that case, congratulations! You've done it!

Design and interface decisions part 2

2009-03-04T23:22:00.001-08:00

Can you spot whats wrong in this image?

Take a minute, really look if you haven't spotted it yet.

Somehow I find this more entertaining then Wheres Waldo...

In case your lost, the image is a screen shot of a GMail message which is being composed. The word highlighted in yellow is the GMail spell check feature pointing out that "ignorre" is likely a misspelled word (at least in English). The interesting part comes when you notice that the of all the words listed, the last one ("Ignore") is not a suggested spelling but in fact is the action to ignore the misspelled word. Tricky isn't it?

Ok, so this example is somewhat contrived. If you were to misspell say, "spaghetti" as opposed to "ignore" the list of possible corrections would likely not include the word ignore so it would be somewhat more apparent that is in fact an action and not a suggestion. Even still, what indicates the fact that it is different then the other entries. Yep, its capitalized. Does that help, nope.

What if you were unluckly enough to have started your sentence with with a misspelled version of ignore. Then your out of luck, no clues here except for the fact that the word ignore is duplicated, and even still which one is the suggested spelling and which is the action?

You might be wondering why I am so concerned with the way in which GMail's spellcheck handles the word ignore. Do I have a compulsion to spell ignore with two r's which I can only fix via spell check? Not quite.

Yet this illustrates a very simple point which is make clear distinctions or separations. No capitalization does not qualify as a clear distinction. A subtle separator probably could have done the trick, but still be considered somewhat ambiguous. What about the text "Ignore this word" or "Ignore 'Ignore'"? The first more clearly labels it as an action that would be complete which I am fond of. The second one might seem somewhat weird in this context, it would still be clear what you are doing, you are ignoring the word being displayed in quotes.

Alright, time to get this high horse back to the stable.

I guess I should probably note that I have absolutely nothing against GMail, I use it and like it very much. I actually was somewhat surprised since GMail is (or at least seems) very meticulously designed, in a good way. And..I picked on Microsoft earlier, so I'm an equal opportunity nit.

Computers Stunk Back In The Olden Days...

2009-02-09T22:01:00.000-08:00

I was just looking at a manual for some older software and I stumbled across this

- Insert the 27th disk in your floppy drive and
execute SETUP.EXE

I can tell you this, if I ever install anything that required 27 floppys/cds/dvds there better be a pot of gold that this application is going to magically generate in order for the install to be worth it.

Why not make it more realistic...

-Insert disk 1 and execute SETUP.exe
-Insert disks 2-10 when prompted by the installer
-Go get a drink of coffee, your gonna need it
-Insert disks 10-20 when prompted by the installer
-Catch some zzz's, its been a long day
-Wake up, shower, etc
-Insert disks 20-26 when prompted by the installer
-Search aimlessly for disk 27, seems to have gone missing
-Accuse the house pet of stealing it
-Continue searching for a few days to no avail
-Give up, the disk seems to have been eaten by 3.25" leprechauns

-Disk 27 reappears under the kitchen table and appears to have been attacked by a veloceraptor
-See step 1.

Then again, there is plenty of software that is multi-dvd installs these days. I hope for all of us that we get to BluRay before we get to a 27 dvd install.

Is Vendor Lock-In Really A Concern?

2009-01-30T17:07:00.000-08:00

With tons of new web services popping up all the time one has to wonder why aren't people switching over in droves? These services offer tons of advantages over their desktop couterparts: cross-platform, secure, and accessible anywhere (with a browser). I'm not talking about any particular services but in case you are lost, here are a few popular ones, Evernote for note-taking, Flickr for hosting photos, or Remember The Milk for to do lists. Search around an you will surely find hundreds of other online services which aim to make those things that we do in a sloppy (storing photos) and unreadable (taking notes) fashion and make it easier to manage. Some of these services even have desktop clients which can be installed so that you can get a richer experience when you are at home, but still have access to it everywhere via the website.

So why are people not switching over as fast as they started wearing uggs? I am confident that many people simply aren't interested in the particular service being offered, or aren't even aware that it exists. Realistically, unless you search for that type of thing, it probably wont fall into your lap.

Now when you start asking early adopters why it is that they don't use or like these services, one of the reasons is inevitably they are concerned about "vendor lock-in". The canonical example is that for some reason service X decides to close its door and with the snap of my finger the service which is storing all of my data disappears without a moments notice. Or what if they simply decide that they are going to hike up prices, or allow only paid users to access their service, what will happen then?

Well I think much of the concern over this issue can be lessened by the announcement Google was going to stop developing its Notebook service, and that Yahoo was dropping its Briefcase storage service. While I'm not happy that either of these services are hitting the dirt (I liked Google Notebook) there is a silver lining here, and that is the way that these services dealt with it. Given that both of these services occupy what are crowded marketplaces there was no shortage of competitors chomping at the bit to corral in their users. Those services went ahead and built tools or plug-ins which allowed users to import their data from their defunct services to their new shiny home.

In the case of Google Notebook for example, CNET made a list of four services which have tools which allow you to simply and easily copy over all of your data from Notebook to their service. While Breifcase only recently (today) announced the news, there are already a few services waiting with open arms.

The fact that these services are behaving honorably towards their customers by providing them early notice of their service's termination and then making it easy for users to get their data out of the service is somethat that definately should be done, but even still I'd like to applaud their work.

While I dont know that every single service that closes its doors is going to do so in such a graceful manner, its good to see these services setting the example for other services which will likely be weeded out in such crowded marketplaces.

Design and User Interface Decisions

2009-01-29T23:22:00.001-08:00

    It is probably a result of reading Donald Normans The Design of Everyday Things that I have spent more time recently thinking about design, and specifically UI design. While I don't think I am, or will ever become, a great (or even good) designer I find it to be a good exercise. For me, trying to find what I believe to be is a good design is somewhat akin to the "which is better A or B?" question which you get asked at the optometrist. Except repeated, over and over, and then over again. I have a lot of respect for those who work in design, it can be a very difficult and frustrating thing to do, and that's before you even get to a good design.

    After reading Normans book, one thing that I think most people start doing (myself included) is taking a closer look at some of the common things around us to see if they really were designed intelligently. One of the things I found striking is seen in the picture to the right. Whats wrong with the Eject option for removable devices? Nothing..by itself.
    What I find odd, is its placement, namely directly below the Format option. I know that in my use that those two things are the most (eject) and least(format) used of all of the options in that dialogue. The thing that exacerbates this issue even more is the fact that the Format option is the most "aggressive" or drastic options available, in terms of consequence. If you mistakenly scan your flash drive for viruses, or click the explore option, no harm done. Format on the other hand, eeek. Yes theres a pop up dialogue confirming that you want to format the device, but really it wouldnt be an issue if it were placed more out of the way.

A Physical Keyboard? Your Insane

2008-12-29T12:28:00.000-08:00

    This morning I woke up to find some articles posted online (here and here) more or less asking Apple to add some functionality to the iPhone. The first article was written by someone looking to ditch their Nokia N95 for the iPhone, but with the snag being that they require it has a better camera. While that is technically the only request, he went out of his way to include a mockup of the phone, which also includes 60GB of storage, a slide out keyboard, a front-facing camera for video chatting, and a camera with optical zoom. The folks over at Gizmodo took the idea and tweaked it some to end up with this.
    As I read over these things I can help but wonder, have they not completely missed the boat here? The decision by Apple to not include a physical keyboard, but only a virtual one was without a doubt a huge, nay epic, decision. While the folks over at RIM were scratching their head on how they could possibly make QWERTY keyboard keys smaller, and even resorting to putting two letters on a single key, Apple saw the issue and sidestepped it like the plague. Of course, this decision was very much a gamble, are people ready for this? Are they willing to give up a physical keyboard? The answer is yes. While I can hear the hardcore Blackberry users yelling, I believe this is a case where they need to be ignored (end users don't like to hear this, but there are occasions when its true).
    Want proof? Take a gander at the Blackberry Storm and the Android G1.
    Like the iPhone, the Storm chose to skip over the physical keyboard and in its place put a virtual one with "haptic-feedback" (the much discussed 'clicking'). I had the chance to play with a Storm briefly this past week and found it to be a nice phone, and while it did take me some time to get used to the keyboard I could see myself becoming quite good with it after a few days of use. Oddly enough, one of the common questions/complaints with the Storm is whether or not the 'click' can be turned off and the keyboard can just be used as a touchscreen (the answer is no). While were on this tour d'phone, it makes sense to discuss the     Android G1 which did choose to include a physical keyboard in its design. What feature are the Android developers feverishly working on? That's right, a virtual keyboard. While I have not heard many complaints about the G1's keyboard (granted I don't know anyone with one) people still have a desire for a virtual keyboard, and how convenient it is to be able to immediately type away without having to flip up a screen.
    The transition to a virtual keyboard can no doubt be a unpleasant one, but realistically a keyboard the size of a business card (virtual or not) is going take some getting used to. Before you start pondering how you are going to text while driving on a virtual keyboard, you wont have the problem soon, it becomes illegal in California in two days. Problem solved.
    After having used the iPhone keyboard for a while now, I can say that the only real issue with it is the lack of landscape typing in crucial applications like Mail and SMS. While I do fine with the vertical keyboard, I think the addition of sideways typing could really make it easier to type away on a long email. I am in no way the only person who feels this way, and I know Apple has received may-a-request for this.
    Getting back to the original point of the article, the decision to omit a physical keyboard was no doubt a large one, but it is not something I see being reversed. Virtual keyboards are going to continue to improve to the point that we will look back and laugh at the Blackberries with rice-grain sized keys. So those of you holding out for an iPhone with a full QWERTY, don't hold your breath, or better yet keep dreaming.

Just a few quick note on the other additional features added to the phone:
60GB of space - Sure, who doesn't want more space? That's just a matter of price, SSDs aren't getting cheaper as fast as anyone would like
Front-facing camera - Word is the G2 is going to have this, well see. The type of video you would see (and transmit) on a 3G network in real-time seems pretty awful. While there are physical limitations which constrain this, I think the network is another big consideration.
Optical-zoom camera - No doubt the iPhone camera is beyond pathetic. The lack of even primitive video capture abilities is quite perplexing considering that every cheapish cell today can do this. I think the full zoom is a bit much. Realistically I see improvements to the camera in the future, but its still going to be second-class in the camera department, as are all cellphone cameras.

Blackout

2008-12-17T16:11:00.001-08:00

So I finally had some time and decided to update the styling on my blog. The old one was nice, but I definitely prefer this one much more. Its somewhat different from what I originally had in mind, but I think it's for the better.

Bittorrent using UDP

2008-12-01T18:02:00.000-08:00

BitTorrent the creators of the very popular uTorrent client have stated that in the next version (currently alpha) they are going to make a proprietary UDP-based protocol called uTP the default protocol used by uTorrent (see announcement). This prompted a fellow over at The Register to predict that this move will cause the internet to meltdown. Of course such a drastic prediction will bring about all sorts of discussion of the issue, see Slashdot posts.

The first thing to understand about BitTorrent and TCP (what it currently uses/defaults to) is that the two are a poor fit. If you think about the way BitTorrent works, it generally creates lots of connections for short lived high throughput transfers. Unlike previous P2P applications (e.g. Kazaa/Napster), files on BitTorrent are split into small chunks, this way the chunks can be downloaded from different hosts simultaneously and likley signficantly faster. Chunks themselves are generally quite small (1MB by default), furthermore, most people tend to have high (broadband) speed connections, thus the duration of the connections to individual hosts is short. These factors make TCP a terrible fit for use in BitTorrent. Basically, when TCP starts a connection it begins sending very slowly (1 in-flight packet) and (exponentially) builds up until it a packet is lost which signals the sender that they may be going too fast. While the exponential increase should lead the sender to reach the maximum send rate quickly it is not quick enough. With the high speed broadband connections available today, it actually takes a reasonable number of round-trips for TCP to reach its maximum send rate. Based on some back of the napkin calculations a 1Mbps link requires 93 in-flight packets to be sent (assuming 1500 byte packet size and 100ms RTT, Throughput=.75*(window size/RTT)). Starting at 1 packet, it will take slow-start 7 round trips (2^7=128) to get up to that rate, and 1Mbps isnt even that fast.... Since the connections made are generally short-lived they will likley spend most of their time in slow start sending much slower then it can actually be.

With all of these factors it comes as no surprise that the folks over at BitTorrent want to make their own protocol which they can optimize for torrenting. Especially since torrent traffic is completely different from the types of traffic TCP was designed for and performs well with. The protocol and the client itself is proprietary which is going to be a point of contention with quite a few people I am sure.

The reason why some have predicted the meltdown of the internet is because UDP does not employ any congestion-control or congestion-avoidance schemes like TCP does, it instead continues to blast away packets as fast as it can. This is the basic version of UDP, the nice thing about UDP is that one add to it the services that they require, and omit those that they dont. TCP already has so many things built it (reliability, etc) that result in poor performance that they simply chose to build it on their own, only including those things that they need while not having to navigate around those things which TCP has already included.

As for now, the UDP based protocol will likley avoid many of the throttling mechanisms put into place by ISPs (e.g. Comcast) to help curb downloading. I dont believe that this is an attempt to avoid these mechanisms since making the traffic UDP allows it to be much easier to distinguish between torrent traffic and other traffic whereas with TCP it is a somewhat more convoluted process. I suspect that this will make it easier for ISPs to classify torrent traffic, the question is what will they do with it.

Update:
I just read this article over at networkperformancedaily.com where the author actually talks to the VP over at BitTorrent Inc. about why they chose to make a protocol, based on what they said it does in fact seem like they are doing the responsible thing, even stating that the protocol needs to be "MORE sensitive to latency then TCP." He also says that it is a "scavenger protocol. It scavanges and uses bandwidth that is not being used by other applications at all.." I think this is the absolutely correct approach to be taking. Just as a kicker to the above article, the first comment posted was by Richard Bennet, the author of the Register's "The Internet is going to meltdown" article.

How about that Facevertising?

2008-11-11T13:12:00.000-08:00

    There has been quite a bit of publicity lately regarding Facebooks (relatively) low ad revenue and how much better MySpace is doing in terms of advertising. Well, hows this for advertising? I never thought I would see the day when a company would pay you (via gift card) to become a fan of theirs on Facebook. Initially I scoffed at the idea, I mean paying people to become their fan on Facebook? Seems bizarre to say the least, I mean why does Sears care about having fans on Facebook?
    The more I think about it, the more clever the idea is. Think about this, although Sears is a tried and true name in their business, their reputation with younger audiences is less...inviting. Yes, if my dad is looking to buy a new washer and drier he'd probably look at Sears, but what do I need there? Sears is about to tell us (the younger audience).
    This advertising methodology relies on two main components. The first part is very similar to rebates, sure it all sounds good on paper but how many people will actually go through with it? Sears has eased the process by shipping you the card, but in order to give them your information you need to register on a website within 24 hours of becoming a fan. I'm quite sure that many of the people who do become a fan will miss this little tid-bit and end up giftcard-less.
    Now that Sears has you as a registered "fan" of their store, they have a direct communication channel with you. They are now prime to "update their fans" on all of the new and exciting offerings from Sears, think about how many eyes will see these updates relative to advertising in the newspaper, and at what cost. Furthermore, updates on their site will be displayed on the fans news feed (you know, this one) Thats a nice piece of real-estate that Sears will get to inhabit when posting updates. That is the first page the 125 million Facebook users see when they login to their account, which apparently over 50% do daily (oft quoted but apparently never cited, sorry).
    All in all, although the kneejerk reaction isn't a goood one, I think Sears may be onto something. It is definately an appropriate campaign for them, and I suspect you will start to see quite a few like it pop up quite soon, especially with holiday season coming up.

Read a random line in a (large) file in Python

2008-11-04T21:16:00.001-08:00

I've been working on a project which requires that I read random lines from a set of files. Unfortunately these files range between 200MB and 2GB so reading the entire file into memory is incredibly slow if possible at all. I looked around for a simple way to do this with large files and I was unable to find a good one, so here you go:

#!/usr/bin/python

import os,random

filename="averylargefile"
file = open(filename,'r')

#Get the total file size
file_size = os.stat(filename)[6]

while 1:
      #Seek to a place in the file which is a random distance away
      #Mod by file size so that it wraps around to the beginning
      file.seek((file.tell()+random.randint(0,file_size-1))%file_size)

      #dont use the first readline since it may fall in the middle of a line
      file.readline()
      #this will return the next (complete) line from the file
      line = file.readline()

      #here is your random line in the file
      print line

Using this method has shown to be MUCH faster then doing something along the lines of:
for line in file:
     #do something
and furthermore, this method does not make it easy to get a random entry in the file.

Enjoy!

Update: Just a note on the "file.seek" line above, the "file.tell()" call is not strictly necessary since what line you are on currently has no bearing on the next line you will be going to. That would also remove the need for the mod operator since the randint would never be larger than the file. Furthermore, since you are advancing the file pointer to a random spot in the file, if all lines are not an equal length then you will not get true randomness, longer lines will in fact be more likely to be hit. Since, however, the algorithm takes the line after random spot seeked to in the file, this issue is lessened but not resolved. I'm looking for a better and more efficient solution in the meantime.

I (We?) still dont know how to read RSS feeds

2008-11-02T13:34:00.001-08:00

      From the time I discovered what exactly that little orange square with the three lines in it was I have been using RSS feeds to keep track of all of the sites I frequent. I think most readers agree that RSS feeds are in fact far superior to keeping up-to-date on all of the websites and news that we want to read. It took the old model of us going out and getting what we want to the new model of what I want comes to me. Its absolutely a huge step and although you wouldn't know it by the numbers, RSS feeds are going to quickly become the norm.
With that in mind...
     One thing that is infinitely entertaining to study and see is the techniques that we all employ dealing with the different parts of our daily lives. Things like email, that we all use, and all have our own ways of managing. For example, when I check email I do my best to ensure that when I am done that my inbox has 0 unread messages. Of course I don't read every email, I tend to do the "mark as read" bit quite often and it seems as if many other people do the same given how prominently placed it is on just about every email site. I use labels/folders and filters to separate out emails into different groups, nothing wild there. I often catch glimpses of other peoples mailboxes and it is interesting to see how they manage theirs. I'm not going to go into details on all the interesting and bizarre things I've seen, but it seems to work for them (at least for the time being).
     So now we arrive at RSS feeds, which similar to email, requires a some time to figure out how to manage. I know that I have 22 RSS feeds which I subscribe to and the range of post frequency between them varies quite widely. On one end I subscribe the the feeds of a few authors books who I've enjoyed and generally post a few times a week if that, and on the other end of the spectrum there are things like "deal" websites which have a few 100 posts a day. In the middle I have the gamut of tech sites (a la Slashdot, Gizmodo, etc) which tend to have roughly 10-20 posts per day.
     Obviously the sites I subscribe to contain a wide range of things I want to keep track of, but at the same time is brings all of those things together, for better or worse. Before, if I wanted to get my news I would browse through a few sites with how many depending on how much time or interest I had at that particular time. In general I tend to look at things in groups, so may start with regular news, then tech news, then a few blogs, etc. So I would get all the information from a type of site that I was interested in, then move on to another. With RSS feeds everything is all in one location, so I can quickly see posts on all of the sites, and it tells me when there are new articles to read. Unfortunately that means I frequently find myself switching from Techbargains to xkcd a bit too quick and I dont properly "context switch" between them. As a result I often lacklusterly browse the posts in the feed when I really wasnt in the mood to read that particular information at that time, and if I had read it later I would likely have done a much better or more thorough job. Given that most people, including myself employ the "only show me unread messages/posts" feature (if I already saw it, why see it again?), we will likely not see that post again.
     All this leads me to the fact that I simply dont know how to read RSS feeds. I am working on my own technique, but it simply takes time to develop. I thought that after already having one for email the rest would be easy, but thats not quite the case.
     I think RSS is one of those technologies that is going to be hear for quite a while, its really a viable way for information transfer (once we figure out how to use it). For a testament to it, and how acceptable it is becoming, open up your Facebook. What is the first thing you see, yep, your "News Feed". Surprised?

An Array of Hope

2008-10-31T19:52:00.003-07:00

The very elusive nerd & politics reference, not easy to do....

Thank you Google

2008-10-30T21:25:00.000-07:00

For those of us who use Gmail like its the antidote I was happy to see this announcement which is that in Google Labs there is a new feature to add a Calendar and Docs window to the main Gmail screen. Brilliant! After reading one of their older posts about right-side labels and chat I got the sense that some work was going to get done which would allow users to better utilize the space on the screen that many of us look at quite often per day. There tends to be quite a bit of hoopla about some of the more "interesting" features that Google releases (e.g. Google Goggles), but its features like these that really make people appreciate the service that you provide.

I am also glad to see that Google is taking a hard stance on the cloud downtime complaints and fears. This is one of those fears that is continuously bring up when you talk about cloud services. It will be interesting to see what sort of response this gets, I suspect plenty of people out there are going to be investigating.

Why is a cloud OS necessary?

2008-10-27T17:29:00.000-07:00

      Ever since Ballmer announced that Microsoft is going to release a cloud operating system I have seen the same question posted over and over again, “Why do we need another operating system for the cloud?” The answer is simple, you dont, Microsoft does. The “Azure” operating system was designed by Microsoft specifically for running in the cloud. The OS is not made for people accessing the cloud, instead it is made specifically for running on servers inside of Microsoft datacenters. Will the OS be made available outside of the data center? For all intensive purposes, no. If it is made available it will likely be limited to use by developers and companies specifically for creating and testing applications which will run in the cloud, and for installing on some corporate servers so they can have seamless integration between their in-house servers and the Microsoft cloud. For example, say a company has a small server farm which they run to host their web application. If they were to run that application on servers running Azure, in the case that their application gets very popular and their servers become overloaded they could send some of the excess traffic to the Azure platform as opposed to become completely Slashdotted into submission. By keeping the operating system consistent it will ensure completely seamless transition between the two as well as an actual test-bed to run applications on before putting them on the platform.
      Seeing as how questions come in packs, the immediate follow up to this is, “isn't there already Windows Server 2008? What wrong with that?” Yes, both operating systems are designed to run on servers, but by making an OS specific to the cloud there are quite a few optimizations which can be made to get even more performance out of the machines. In the case of Azure it is definitely going to be run as a virtual machine inside the data center much in the same way that Linux/Windows images are virtualized on EC2. Virtualization is really the key to making the cloud possible. It allows multiple OS instances to run on the same physical machine while encapsulating them from each other. That way one VM cant mess with anothers data or processes. Also, when someone is done with their VM, they simply replace it with a clean VM image and its as if they were never there. On EC2 they run up to four VMs per machine, I highly suspect that by trimming down their server OS they will be able to get a lot more VMs running reasonably on a single physical machine (I wont speculate as to the exact number, but I think that number will be at least double digits, maybe more?). Keep in mind that as opposed to EC2 where users have direct access to the OS, on the Azure platform users wont have direct access to the operating system. Therefore Microsoft can toy with the number of VMs to figure out the optimal number to run on their servers.
      So in summary, the new OS is not absolutely necessary, they probably could have done it on Server 2008. But, given the scale they are shooting for, and the amount of optimization they can achieve by making a specific OS it is no real surprise they made a new version specifically for this function.

What Microsoft "Azure" really is...

2008-10-27T16:13:00.000-07:00

      This morning I awoke to some very interesting news, straight from the get-go of PDC, Microsoft announced "Azure." I was very much interested in what exactly this service is, what it does, etc. so started reading some of the news stories about it, wow, people are confused. I partially blame Microsoft for clearly not giving people a good enough idea about what exactly Azure does and does not do. At the same time I also blame journalists who dont seem to really know whats going on (yet dont seem to mind). What is really going on? Lets talk about it.

      Today, at PDC, Microsoft announced Azure, its cloud computing platform. Just so it is clear, Azure is also the name of the operating system that the Azure platform will run on (previously dubbed Windows Cloud by Ballmer). I will address the "Why does the cloud need another OS" question in the next post, for now lets just stick to the platform. For those of you unfamiliar with the cloud, it effectively breaks down into three different levels of services which can be provided (see this Tim O'Rielly article for a full definition of the three layers). Azure falls into the platform layer, which is the middle of the three layer hierarchy. As the title suggests, this layer provides a “platform” which developers can utilize to write and run their application. For example, lets say that I want to create a web application which allows users to input their contacts and access them from anywhere. If I were to do this using a framework like Rails (with Ruby) or Django (with Python) then after designing the application itself, I would also have to set up a database which would be used by the application to store the data like user login information and the actual contacts themselves. Using the platform layer, instead of running a database myself, they (the service provider) run the database for me, and I just access it though the provided interface. I still determine what tables I want, how I want them to be laid out, its just that instead of actually writing the SELECT statement, I call a SELECT method. It is up to the provider to figure out how they should host the database and make it scale. Similarly, I as the user dont have to set up the platform and the underlying operating system, this is handled by the provider, note how this is different from the infrastructure services (a la Amazon EC2). Basically, I would write the contact application, making sure that the database interaction would go through their API, once I'm done (with this release) I then hand it over to the provider. It is up to them to figure out how to deploy it and scale it up and down as necessary. Again, very different from EC2 which requires the user to determine how to deploy it and manually scale up and down the number of machines (or use a third-party service like RightScale).
      Ok, now that you have a better understanding of what a platform and platform service is, it should be clear that Azure's main competitor is Google App Engine. Many people will lump Azure and Google App Engine together with EC2 but in fact they provide very different levels of service. Think about it in terms of a car. Amazon EC2 is really the infrastructure, its as if someone gave you the frame of a car, it is up to you as the car-builder to find the appropriate engine, tires, and seats depending on what you intend on doing with it (drive on the freeway vs entering in a monster truck rally). With platform services its as if you were given the frame of the car, with the engine, and tires already installed (and you cant remove them). The key parts of the car are there, its up to you to put in things like the seats and gas in. Notice however that you dont have a choice on what engine and tires were provided, so they may fit your needs, but they may not. You have less control of the infrastructure in the sense that it is provided for you and you cant change it, but you also save yourself the complexity of having to figure out what engine would fit in your car, how many cylinders etc.

Got it?

Shameless Windows Vista bashing

2008-10-15T16:33:00.001-07:00

      So for the millionth time I somehow find myself reading a Vista bash article (this one) and once again THEY GOT IT WRONG. Just to be clear, it is supposedly an article about Windows 7, which apparently implies that it is a forum to bash Vista. The more articles I read the less I respect online tech journalism, but I digress. So a quick summary of the arguments presented against Vista:
      1.Incompatibility was and still is an issue, albeit the tip of the iceberg
      2.UAC is annoying
      3.Vista is bloated
      4.Vista tried to hard to be pretty
      I suspect that if have read other articles about Vista you have probably seen some combination of this complaints in the various other articles. This is by no means groundbreaking stuff, in fact I'd bet I could probably find a few articles from 6 months or even a year ago that present just about the same argument. Even still, that doesn't prevent you from getting it wrong.
      Incompatibility was definitely a big issue when it came out, there is no getting around that. Basically, the second hardware vendors saw that Vista was getting a lukewarm reception they put Vista drivers a few notches lower on the list (aka they weren't going to do it, or at least quickly). Then, when some user goes to install their Brothers printer and it doesn't work they get mad. Is that Vistas fault, or the hardware vendors fault? No reason to point fingers, it just stinks for the end user. In either case, Vista post SP1 incompatibility is unlikely to be the issue. Sufficient time has elapsed for even the slowest of vendors to get Vista drivers out. If they have no done so already, I definitely put the blame on vendors, they have had long enough.
Good ol' UAC seems to be pissing a lot of people off, does this not strike you as odd? OS X and Linux have had the equivalent of UAC since their inception, but no one seems to complain about it there. Yes, UAC likely pops up a lot more often then you are used to in OS X and Linux, but in all reality, if it is popping up, it likely should be. People have become so accustomed to the pre-Vista Windows where nothing pops up to notify you that some terrible website is trying to put a Trojan on your system. Realistically that is just a poor model for security and is likely to result in a lot worse results (keeping a clean system) then the UAC model. It is a lot easier to prevent malware from installing, then it is trying to remove it once it has already been.
      I think when these writers think of Vista they get the image of the Micheline tire man running in their head. How exactly people determine that Vista is “bloated” is a very subjective process and in all likelihood does not imply that they have any actual knowledge of the underlying operating system. Much of this perception comes from people seeing the amount of RAM Vista uses by default compared to other OS's. Its hard not to make that association, but it simply is not valid in this case, not by default at least. Vista has a feature called SuperFetch which proactively puts often and recently used programs in RAM even before it is started by the user. Why? Its simple, if they guess what you are going to start and put it in RAM before you start it, start up time is drastically faster then it would be otherwise. So when someone looks at Vistas RAM usage and sees that it is using 2GB on start they are wondering “How is it using so much RAM? I haven't even started anything!” Chalk a lot of that usage up to SuperFetch, and hope that SuperFetch uses as much RAM as possible. Using RAM is GOOD, thats what its there for, and as long as SuperFetch gives it up when necessary all is well in Windows world.
      Even still, the Vista bloat argument continues. Consider this, when Macs changed from PPC to Intel hardware it was a HUGE change. It required a new OS, programs to be rewritten, and just about anything Apple related that was PPC became outdated. They effectively drew a line in the sand and said it stops here, anything older then this we are done with. What happened? People bought new Macs. Those left with PPC's were left to wither away out in the cold. This wasn't too terrible for those users since Mac users (and Apple users) in generally tend to be a lot more dedicated to the brand then most people are to just about anything. Lets say Microsoft employed this strategy with their next version of Windows (clearly they arent with Windows 7, but lets just say for the sake of argument). I don't think that any of us could imagine the amount of uproar such a decision would make. Even though Microsoft wouldn't be changing the hardware (they don't make their own hardware, Apple does), the software changes alone would bring about so much complaining we wont be able to hear ourselves think. The sheer amount of labor required to rewrite software for the new Windows would be enormous and an absolute nightmare. You think people are pissed about Vista, the amount of headache this would cause would make Vista seem like a blip on the radar. Based on the amount of people who use Windows, and the amount of software written for Windows it is going to be much more difficult for Microsoft to draw that line in the sand.
      Finally, there is always the claim that Vista is trying to be too much like OS X and be too pretty. Its no secret that Microsoft significantly dressed up Vista compared to it predecessors, and its hard to argue that Vista doesn't look better. Whether or not it is worth the resources it draws is a matter of preference. To me, it is. So Vista is trying to be too much like OS X? Really? Are we talking about specifics in the UI, because I don't see many. Your not going to see Windows with a “dock” any time soon (especially now since its patented), I cant see Windows using left side close and minimize buttons, or adopting the silver theme by default, so where is the copying? If you are simply making the case that they are trying to make Vista pretty, then you cant fault them for that, part of being a successful operating system is being pretty. Dont believe me, ask Linux? One of the big pushes in Linux over the past few years has been beautification, and it has paid off. Compare your distro of choice now to how it looked 2-3 years ago and you will see that it is likely a lot prettier, shinier, and slick overall. Linux demonstrates this case so beatifically, pretty is a very important part of being a good OS, like it or not.
      I suspect with Windows 7 hype continuing to build, more and more of these articles will be resurrected from the murky puddles they belonged in. Please, if you are going to make an argument about Vista being “sucky” think about it before you just regurgitate the arguments of others.

Is Google really a threat? Part 1:The Browser

2008-09-28T10:51:00.000-07:00

Lets start with Google Chrome, when word hit that Google was creating a browser it seemed as if everyone and their parents were downloading Chrome to give it a try. Market share for Chrome quickly jumped up to 1 or 2% in the days following the release, an impressive feat, but in weeks since has slowly fallen. Unlike most of the products which Google labels beta, Chrome is very much still a beta. There are still quite a few issues which remain unsolved with Chrome along with quite a few glaring issues. Even still, I have no doubt that there are Googlers working away at right this moment to fix these issues and improvements will be made. Getting back to the original question, is Google really a threat in the browser market? The short term answer, not really. Lets categorize people for a moment to see why.

"Techies" were very likely to try out Chrome and I would assume they accounted for a significant proportion of that initial 1% as evidenced by many tech sites receiving significantly higher percentages (ReadWriteWeb saw 7%, Silicon Valley Insider saw 6.6%). Unfortunately for Google, they did not release an add-on framework at the time of release and many geeks are so hooked on their Firefox plug-ins of choice that they couldn't imagine switching to Chrome permanently (AdBlockPlus and Firebug in my case). I'm sure there are many people typing away at these, but so far nothing.

But techies are only a collective few, what about other people? People browsing at work whom are employed by large companies is a large contingent of people (and not mutually exclusive with techies), how is their Chrome usage? In short, not good. Upon its release Chrome had quite a nasty looking EULA which may people believed was akin to signing over a first born to Google. While not true, it was a firmly worded EULA which had a lot of people concerned which Google has since rectified. Even still, so many large businesses were concerned about that EULA and as a result told employees not to download Chrome until it had been reviewed. Considering how strict many companies are about using non-IE browsers, it will be difficult for Chrome to gain significantly large market shares in this slice of the world. Finally, there is the general population. "Google what?" is the most likely response you will hear when asking the general population what their opinion is on Google Chrome. As ubiquitous has the Google name has become, the reality is that to many people the internet is still that e with a halo on it on their desktop, it is just that simple. Simply consider the fact that last year over 30% market-share belonged to IE6, yes the same IE6 that came by default with Windows XP circa 2001. The same IE6 which is mired with security flaws and whose rendering abilities are the bane of most web developers existence. While that was last years numbers, many sources have seen IE6 holding fairly strong with a 25% market-share this year. If this is not evidence of the "Internet=Desktop e" theory then I don't know what is.

So then, what does this all mean? It means the while Chrome has quite a few innovative features (process per tab to name one), it is still lacking some of the features to draw the techies over. Employees at large companies are unlikely to use Chrome since either their legal department hasn't given it the OK, and may never. For the general public Chrome is still what lined their favorite Ford Thunderbird back in high school. By Christmas time this year I can see Chrome getting to about 2-3% market-share without really breaking a sweat. Following that I think Google is going to have to make some big announcements to remind us all what Chrome is, and why I should switch over. So to answer the question, no, Chrome is not a threat.

Is Google really a threat?

2008-09-28T10:02:00.001-07:00

In the past month or so Google has been involved in the release of two very highly touted products in Google Chrome and Google Android being released on the TMobile G1. Both of the announcements have generated enormous buzz and having the Google name attached to both of these products have really fueled the fire. But with the release of these two products I ask, is Google really a threat?

Google Gaming?

2008-09-17T23:51:00.001-07:00

Forbes recently made a wildly speculative article about whether or not Google should and will enter the game arena. I'm not sure if the Forbes folks have been watching too much of the crazy news with Keith Oberman, but Google, making games, what are you on? Lets get one thing straight, the only games that Google with be making will be the ones that would come with Android by default or the ones of gOS. Why on this green earth would Google decide to go after gaming? Yes, gaming is a booming market, and yes there is advertising revenue that can be generated from games, but lets be honest, Google making games is about as likely as Toyota making MP3 players. Now it has become clear that Google is willing to spend money developing technologies that help to feed their search (read: advertising) business. Google Docs is the perfect example of this, by having you store your documents in their cloud, they have the opportunity to take a look see and make there add recommendation just that much better. I assume the belief is that they would enter the game market by making games and not making actual consoles since the console market is only ready for the suicidal. So Google makes a game, and then they make some sort of advertise bar which displays in it (a la Net Zero), seems logical enough right? Wrong. Assuming they were to make an immersing game, and not just a pass the time waster (hi Freecell) ads are a terrible idea. Back in my hay day I used to play a few games, things like Counter-Strike, Command and Conquer, and the likes, and if there is one thing I learned from games like that is that once you are in games like that there is very little chance you are going to click an ad which escapes you from your full screen game, much less even look at them. Microsoft recently implemented ads in Xbox live, so when browsing around the demo section you will probably catch an add or two, but not if they are dumb enough to attempt to put ads inside of a game like Halo or Tiger Woods Golf.
The only real advertising I could really see happening in games is product placement like they do in TV shows where someone like Pepsi will pay to get some member of a TV show to take a sip of their soda on camera. Cant you just see Tiger taking a nice big swig of Pepsi after blasting a 350 yard drive down the 18th of St. Andrews?

Oversight of the century?

2008-09-17T23:51:00.000-07:00

Its really unfortunate, I was walking into the bathroom of the lovely Chicago O'hare airport and I saw one of the greatest photo-ops I have seen in a while. There was a woman dressed in a very serious dark suit sitting on the floor with her laptop on her lap, funny but nothing special. Heres the kicker, shes halfway cramped in between a newspaper stand and a pole, likely one of the most uncomfortable sitting positions I have seen someone in such formal wear in. So why would a woman like this put her self in such an uncomfortable position? Simple, there was an outlet nearby and she was charging her laptop. Unfortunately she was packing up by the time I got out of the bathroom so I didnt get a good chance to snap a picture of it, but I think you get the idea. Now why the story about me using the restroom and people sitting uncomfortably? Because it points out one glaring defect of our nations airports, and many other sites (including universities) which lack power outlets. People, in case the pigeon didn't make it to your neck of the woods with the message, we are in the 21st century, the information age, remember those tube things?
In the past few years the amount of electronic equipment that the average traveler carries with them has gone up significantly (% increase from 0->1 is infinity). So when people are traveling in the airport between flights aside from eating, what are people looking to do? Thats right, charge their , whether its laptops, iPods, phones, people always need to charge something. No one wants to be on the flight where there battery dies halfway through because they were watching Californication in the airport before their flight. The only problem is that since most airports are fairly old, there is no way they could have accounted for the vast amount of electronic outlets that would be required by the present days business traveler (Come on, try to find somewhere to charge your blackberry, I dare you). So what are we left with? Airports which have very few outlets, none of which are marked, and often in very awkward places (I know we've all considered using the outlets which are under the check-in counter). The situation is so bad, that even companies have started to take advantage of the glaring oversight. In O'Hare airport Verizon has setup a counter which is simply 10 stools and divided little work areas each with two power outlets solely for the purpose of allowing people to charge their stuff and if they happen to get bored they can stare up at there big advertising board directly over it. Thanks Verizon, appreciate the consideration, but thats just the tip of the iceberg, and those stools are incredibly uncomfortable. I'm not electrician, but it seems to be that doing some wiring to get a whole bunch more outlets would be fairly trivial and would not take long at all. But theres a lot of airport, and a lot of gates to cover. So lets make it easy on you. How about just labeling the walls or posts where there are outlets so we don't have to hunt around the airport like a bunch of crack addicts looking for the next fix (crackberry anyone?).
Considering how little traveling I do compared to the very sharply dressed gentleman next to me who is also clamoring for an outlet it is clear that I am not the only one who has these thoughts? Considering the burlesque show that we have to put on going through airport security can we at least get some power outlets? I mean if thats not the oversight of the century, I don't know what is.

Why buy the cow

2008-09-14T22:00:00.000-07:00

In case you haven’t heard, the excellent website Hulu will be premiering many primetime TV shows either on or before their on-air premier dates. In case you weren’t aware, Hulu is a very good online video site which has a good selection of TV shows and movies which can be streamed and is shown with minimal advertising. For a 30 minute TV show there are generally 3 commercials which are 15-30 seconds long. I was reading this article on Wired which brought up the interesting point, with Hulu airing the premier of Prison Break on the same day it will premier on TV, why have over a million people downloaded it (illegally) via torrent? This situation is not unique. On October 10th of 2007 Radiohead released their album “In Rainbows” only on their website. Fans were allowed to pay as much as they wanted (including $0, free) for the album, which was available by download. Well, no surprise when shortly thereafter the album appeared on torrent sites. Again, if it is available for free why would an estimated 2.3 million people download it (illegally) via torrent?

Note: while I would never encroach on either of these actions, I may know a few that do, and so it is their coalesced opinions and my own spin that I include here.

The answer is not made of up a few different sets of people.
First, there are the people that didn’t know about that availability, and therefore went to their main method of acquiring that content, torrent. The same way they have done for all the other episodes of Prison Break or Radiohead albums, fire up the tracker of choice and download away. These people cant be blamed, although publicized, the number of people who knew about them being available was still a very small set of all of those whom would otherwise be downloading. They might be on the front of the tech-section of every online news/blog site, but you have to read it to know.
Next, there are people who heard about it, but were too late. They had already downloaded the show/album before they heard of the promotion. At that point they could go back and watch it on Hulu or download it from the Radiohead site, but much of the allure is lost, although both sites have their revenue opportunities intact which I’m sure they are pleased about.
Finally, there are those people who did not agree with the terms under which they would get the media. Ads in the case of Hulu and a required email address and an ask for donations on the part of Radiohead. To be honest, I really don’t find either to bad at all. Ads on Hulu are significantly shorter then those on primetime TV, you generally only see one per break, and you have all the abilities of a DVR (pause, rewind, etc) without needing one. Radiohead simply asked for an email address (no spam to date) and ask for donations (which they made clear could be $0, no CC# required). What’s not to like? Some people just wont give in…what can I say?

As for these two occurrences, I think they are incredible. They demonstrate an embrace of technology by the two industries which have classically fought it off like Ugg boots. I just have two comments about these:
1. Hulu, why on earth do you not have an iPhone app out yet? Are you waiting for me to make it?
2. Radiohead, why not make the album available via torrent, you save on the hosting cost and make it more widely distributed? Seeing as 2/3 of the downloads came from torrent, why not focus on those people, and hitting them up for donations instead?

A Promise

2008-09-14T21:14:00.000-07:00

It has become really clear to me that in terms of Vista advertising bloggers are just going to bash, bash, bash, and then bash again. Much the same way they treat the operating system itself. As opposed to posing responses every time a new commercial drops, my official stance is "see message below." Everything that was applicable before, is still applicable, so no reason to needlessly post more updates. With that, I promise no more blabbering about the new ad campaign, just see the post below and you will get my opinions.

The new Microsoft ad

2008-09-04T20:13:00.000-07:00

Well, it’s out. And if there isn’t going to be enough gasoline in this fire I figure I might as well throw my two cents (worth of gasoline) in.

I’m not really sure what people were expecting this ad to people. Based on my very empirical study this ad was expected to be the hippest, coolest, most ultimate Vista rocks my world ad…featuring Bill Gates and Jerry Seinfeld. In case you were disappointed by the distinct lack of heavy rock music (a la video game commercials) or some sort of play on the Apple ads, you clearly haven’t been paying attention. If you had read my previous article entitled “Don’t be surprised by Seinfeld”, you would have been much more in tune. Is it going to make you fall off of your chair laughing? No. Was it direct shot across the bow at Apple? No. Was it edgy/risky/crazy? No. It was Microsoft.

It was mildly entertaining, but only since it relies two of the most famous faces in the world (t-minus 10 years ago of course). The premise is a play on BillG’s well-known…thriftiness…. And has some shout-outs to those nerds who have been paying attention. In terms of Seinfeld, its roughly the Visa ads, except without Superman.

I think the real surprise of it all is the lack of heavy branding or slogans. Spanning the entire minute and thirty seconds of commercial the word Microsoft was mentioned once, in passing, by Seinfeld. Then there is the slogan “The future. Delicious.” Followed by the Windows logo. No Microsoft logo, no mentions of the word Windows, or dare I say…Vista.

I’m sure pundits are savagely pounding at their keyboards about this one, “$300 million for this!” or “that’s going to beat Apple?!” Stop it. Please. Just, stop it. First, I’m very sure that this is the first of many (random guess says 5) commercials to come out of this campaign. Second, like I have said you will not see direct shots at Apple. Why? By “competing” with Apple, Microsoft would be admitting that they are on their level. Believe what you will, Microsoft won’t be admitting that any time soon.

Kindle the college demographic

2008-08-25T22:48:00.000-07:00

There have been some recent news articles about how Amazon is planning on entering into the textbook market by making a textbook version of their Kindle eBook reader. I think I speak for everyone when I say, finally, what took so long? Some of us (me) have had had this idea for some time now (three years) and was hoping someone would do it (I have the design to prove it), and do it right. The college textbook market has been almost comically redicilous, but the current options are so poor that they can still dominate. So many of us have tried buying books online and although prices are much better, there is a significant hassle associated with it. In most cases you have to wait until the first day of class to get the syllabus, only then can you figure out exactly what books you need (especially editions) and order them. Media mail is awesomely cheap, but coupled with slow processing times can make ordering online an extremely slow process. The alternative is to download textbooks via torrent. All I can say is having three huge textbooks on a flash drive is absolutely awesome. Furthermore, being able to search the text within the books is almost priceless.
Clearly the Kindle is a very intriguing idea, especially in the textbook market. For a long time people have said that eBook readers were just flat out sub par. Hard to read, poor battery life, small memory were the most common complaints, and the Kindle seems to have solved some of these (battery life is mediocre).
In my reincarnation of the eBook reader I took a different route then the Kindle. I really didn’t see the need for a keyboard, all you really need is a power button, two buttons to flip the pages. In the case of textbooks I though it would be very cool to have some highlight functionality, so add a stylus and you could highlight portions of the text.
When I look at the Kindle there are a few things that I can only hope for which would make the Kindle a useful tool. First, some form of highlight/tagging feature which allows you to select parts of text. Then once selected you can iterate through all of the highlighted items for a quick review of the most important parts. Next, if Google’s $700 stock price has taught us anything, search is key. There is a keyboard on the thing, please make use of it and allow people to search through text books. Sometimes you need a quick reminder of the truth table for a clocked d-latch, and if I cant search in the textbook everyone will just resort to searching for it online, or spending a while finding it in the book.
I read an article which touted the Kindle as the next iPod. Please. As innovative and interesting as the Kindle if it was 1/10th as popular as iPods I think it would be a success in most people’s books. Its price point has to drop significantly from $350 for it to even have a chance.

Dont be so surprised by Seinfeld

2008-08-21T21:56:00.000-07:00

I’ve read so many articles about Jerry Seinfeld apparently becoming part of the new Windows Vista ad campaign that I am about ready to explode. Most articles are quick to criticize how Seinfeld as a long way from his popularity 10 years back or how on his show there was always a Mac in his apartment (apparently that makes him the biggest fan boy of all). To address the ludicrous first, I have yet to see any article or interview that actually have Seinfeld saying he owns or even likes Macs. I have no doubt that the computer in the show wasn’t an accident; if you have ever seen advertisements and more recently commercials you will notice that all computers are mysteriously white and curvaceous. Macs are pretty, there is not question about that so why would they put anything less in their commercial? Yes, it was mostly an Apple IIe which may not be pretty now but you don’t even want to see the PC’s back then. Show me an interview, article, picture, then we can talk…

People seem perplexed by them picking Seinfeld, I mean weren’t they supposed to pick someone hip, cool, and just flat out AWESOME?! Not to be brief, but no, absolutely not. If there is one thing that Microsoft surely is not going to do it, its try to out-cool Apple. Criticize as you please, but Microsoft is not dumb, and they know that trying to out-cool Apple is a loosing battle. Now I was surprised as any when I heard the news, I didn’t even think Microsoft would have a “spokesperson” for the campaign since that is an extremely risky venture. But, if you are going to pick someone, Seinfeld is an excellent candidate. When you think about the target market, as much as they would love to go after the hipsters, Microsoft has a somewhat older market to cater too. So although Seinfeld has been out of the limelight for a little while now, those of us who are old enough to remember the show like Jerry and don’t have one bad thing to say about him. You wont find pictures of him on TMZ smoking outside a club or running over paparazzi, in fact, aside from his show, Seinfeld lives a quiet, unpublicized life.

As George said:
George: Well, I got just the thing to cheer you up. A computer!
Huh? We can check porn, and stock quotes.
I probably should bring up the episode which involves Jerry and computers, the white laptop crowd will have a field day..

Laptops in Corp Land?

2008-08-12T23:32:00.000-07:00

I started reading this article over on Arstechnica and the more I thought about it I still see a fundamental flaw in their logic. To quickly summarize, the article makes a case for why very soon desktops will only “appeal to a niche market” even in the corporate world. Since employees are traveling significantly more and mobile broadband provides reasonable coverage laptops will soon dominate the corporate market. The article continues about how laptops serve as a viable extension for ones personality, and if you’ve been paying attention, you will know how annoying I find that.

Even still, I will concede the home market. If you have been paying attention you would have heard that notebooks now outnumber desktops in terms of sales and I don’t see much reversing this trend, in the home market.

The corporate market, are you kidding me? Clearly the author sees the corporate employees as a bunch of busy bees just buzzing their way around the country for months on end. Only returning home to the hive for the occasional meeting and suckle of honey (take that as you will). Perhaps Mr. Reisinger has been eating too may dinners out with his journalist cohorts because although people are traveling more, not nearly on the scale that he is implying. Without sounding too green, I might venture to say that most non-sales people probably spend about the same time on vacation then they do traveling for work. There are, however, practical uses for laptops which don’t require an airport or gas stations. Having a laptop to take around the office, into meetings, even home can be quite convenient and is probably where most laptops get their use. When you think about these three cases you will mention the fundamental struggle with laptops: Size/Portability vs. Usability. For the most part, the bigger the laptop the most usable it is (bigger screen, full sized keyboard, track pad) and the less portable it is (larger, heavier). Yes, some laptops make better use of space then others, but it’s rarely by much. Based on my empirical (and utterly unscientific) study I’ve found that the median home laptop is 15” while business laptop would be 14”. Like I said before, this is because business users travel more, albeit still not very much.

One thing that becomes apparent when looking at laptops however is that they are not even close to desktops in terms of usability. Clearly a smaller monitor has a pretty large effect on usability, but even worse is mice. Trackpads are the most popular “pointing device” in laptops and basically the only improvement we have seen in them over the past many years are the scroll bars. Multi-touch you say? Meh, wake me up when you can do something interesting. And if you have the ThinkPad “TrackPoint”? Ouch.

Wait a minute, why not just buy a docking station and you get the best of both worlds. Well you smart cookie, that’s a good point. You get the portability of a laptop since you can undock it and take it with you, and when its at your desk you can use full size peripherals. Unfortunately one other thing you will get with a dock is a lighter wallet, they do not come cheap at all. Why pay $300 for a docking station when you can get a whole new desktop for $500?

If you aren’t yet convinced that laptops wont be taking over the corporate landscape any time soon you should take a look back over this article, because you must be missing something.

Why waste your money?

2008-08-07T22:41:00.000-07:00

Throughout the past few years quite a few people have asked me for advice on buying computers, whether it’s because I’m a Computer Science guy or Jewish is not necessarily clear, but I sense (hope) it’s probably both. In most cases I get the standard “I just want to surf the web, check some email, and the occasional X” where X is some software that requires a somewhat more powerful machine (e.g. photo/video editing, gaming, etc.). Interestingly enough, my answer over the past few years has been the same and it is this; “You have two choices, A) you can bargain hunt for a few weeks and find a Acer/Compaq/HP deal with decent specs that will run you $500-600 depending how hard you look or B) Get a coupon for a Dell laptop and spend $1000 and only pay ~$800.” Of course, since I also fall into that category, I practice what I preach and have ordered combo B with a large soda a few times. To be honest, as long as you do your homework you will find a good machine which can handle all of the WOW you throw at it.

With that in mind, I am baffled when I talk to people who have and buy $1500+ laptops. Before you get offended, if you require a quad-core, 13”, 2 lb. laptop (or an Apple) then by all means, close your eyes and hand over your plastic. Those crazies aside, (I knew was going to offend people) I have seen a lot of expensive laptops being purchased and they all tend to share two key things, they look sexy, and they have very mediocre spec’s. Clearly a sexy looking laptop is a very important feature, I know that when I am out about on the town that I cannot be seen without a sexy laptop to match my handbag. Now I am not going to lie to you and say that Macs/Dell XPS’s/Sony Vaio’s are ugly because they clearly not. Yet, I find it difficult to justify paying so much more for a nice looking laptop when realistically most laptops look pretty nice. Before you boil over about my mediocre spec comment I’ll explain. One would think that if you were shelling out almost twice the amount of money that you would get a beast of a machine. Not so fast, the low end models are generally WORSE then the low end of less sexy laptops. So…you pay more to get less? That is correct. But…if you want to pay MORE money then you can actually get a sexy laptop with decent specs.

Note: For those of you who are squirming out of their seats about how Macs don’t require as good specs to run decently, hold onto that thought, I’ll be there soon.

Workspaces in Windows: Redux

2008-06-25T20:24:00.000-07:00

I was talking with some coworkers about random technical stuff the other day and I had a good shot to bring up the workspaces question. Given that both guys are very intelligent, articulate, and considered "power users", I was interested in their take on it. The first point they made was that if they wanted to do workspaces in Windows, they would have done them. It is not a technical limitation or a legal issue, it was a matter of choice. That means they didn't believe that enough people would use it, and too many would be negatively impacted (see prev. article) to have it. Yeah, that sounded about right, but now "aren't there still enough people out there who want it to have it available, but disabled by default, or only in a professional/ultimate type edition?" I asked naively. To my surprise I found that even within the "technical" community or "power users" a large portion dont utilize workspaces and or dont find them useful enough to install them.

Realistically, if they cant convince a significant portion of more technical users to ask for it, or use it, then its pretty clear why they wouldn't include it by default. I guess those of us that want it will have to rely on third party tools to implement them for us, although so far I have not really been happy with what I have seen. Guess I will have to continue looking harder.

Workspaces in Windows

2008-06-22T21:37:00.000-07:00

For some time now I have been left to wonder why there are not "workspaces" in Windows. In the time that I have been using Linux I have found multiple workspaces to be incredibly useful. Being able to have windows open on different workspaces is a really great way to have quick access to information, but not have it constantly distracting you away from work. I usually ended up with a layout something like this:
1. Browser w/ email, sports page, slashdot
2. gEdit, terminal, browser for programming reference
3. Either a different programming window (like #2) or a paper I'm writing or reading
4. Usually empty, otherwise it could be another space like 1-3

Now initially it doesn't seem like much, you could easily navigate around a Windows desktop with those Windows open. So what the big deal? Separation. Having email/sports/slashdot open, refreshing in the background, is just too tempting. All too often I find myself flipping over to that window just distract myself with Slashdot for a while.
So Linux has workspaces, and as of recently, OSX now has them as well, that only leaves Windows.

There are mainly a few arguments for why multiple workspaces are worthwhile. First, many users (myself included) find that workspaces improve productivity greatly. In the jobs I have worked at I often have seen co-workers with TONS of windows open and they have quite a time managing them. For those of us who don't have multi-monitor setups (or even if we do), workspaces provide a lot of extra room to work with.

Now if they are all I have cracked them up to be, why wouldn't Microsoft jump through rings of fire to make them? Well there are a few reasons why I would see them not doing this. First, there is probably a large population of Windows users who would likely not use them. Think about the grandmas and moms who use their computers to check email and look at pictures of their kids online, they either wont know about them, or not seen any use for them. This doesn't seem that bad since they would effectively be in the same position as they are now since they would not use the workspaces. It turns out not to be so simple, one could imagine how many times windows would magically "disappear" to other workspaces and they would have no idea how to get them back. Given that number of users who use Windows on a daily basis, the number of times this would happen would be astronomical.

I think a reasonable solution to this would be to only include workspaces in upgraded Windows editions, so leave it out of "Home" editions but include it in "Professional" or "Ultimate" editions. In doing this, workspaces would be provided to business and power users, while home users wouldn't have to deal with them, everyone is happy.

Originally I wrote about some of the patent issues that have come up with Red Hat and Apple from IP Solutions, and how that would affect Microsoft's decision to include that feature. In looking into it there seems there is more going then I originally noticed, so I'm going to do some homework on the topic before I comment here.

Tune in next time when I talk about how file sharing and BitTorrent help Web 2.0 technologies

To touch or not touch

2008-06-22T20:48:00.000-07:00

I've had this opinion since the 3G iPhone release, but am only getting to write about it now.

I think I am part of a very large contingent of people who where (and still are) awaiting the release of the iPod touch V2. Since it was first released I knew that the touch was an incredible product and enviously played with a few of them. Music, of course, video, excellent, and just to top it off Wifi. Being able to connect to Wifi hot spots at great speeds and not having to pay a cell provider a bunch of money for a limited service are huge plusses for the touch. Yet there were a few things that held me back from jumping on one from when it first came out was space, and cost. Upon its release the touch came out in 8 and 16GB sizes which carried the price tags of 300 and 400 respectively. In talking to my compatriot Phil we both came upon the same conclusion, its simply not enough space. 16 gigs on a device which can store video, including full length movies, is insanely small. This was coming from the guy who was having trouble squeezing just his music onto a 20GB iPod photo. I decided that I would wait till the second round of iPod touches since they would increase capacities to 32 and hopefully 64 GB and make some software improvements.
I was really disapointed when I found out they were going to release a bigger 32GB touch, but it would still be the first gen. Mainly, they added this on top of the other two sizes, without a price drop, which left the largest model with a hefty price tag of $500. This also meant that they would not be releasing a touch V2 anytime soon.
Fast forward to today, and I am still torn about what to do about it. Surprisingly, Apple has yet to drop the 8GB touch which means prices have stayed the same. To pour a little fire on it, Apple has continued to stick it to touch users by charging them "nominal" fees for software updates. There have been (I believe) three updates so far which have been free for iPhone users, which touch users have had to pay for. Why Apple? I don't see how Apple can justify charging touch users but not iPhone users, either charge both or none.
I guess for now I'll continue to wait and hang on to my old school iPod. Perhaps soon enough Apple will decide to lower prices on touches and toss in those updates for free and then perhaps ill change my tune and put my money where my mouth is.

I'm back, with content

2008-06-22T20:37:00.000-07:00

Today I decided to take a stroll and get some dinner. From this came two things, first, I made the unfortunate decision to eat Arbys, which I have not had in ages. Second, I had a chance to think about a few ideas which had been bouncing around in my head. So here are the fruits of that labor.

About Me

2008-03-26T18:03:00.000-07:00

Hello and welcome to everyone visiting, glad to have you here. My name is Jonathan Kupferman, I recently graduated from UC Santa Barbara with a degree in Computer Science. This fall I will be returning to UCSB to get a Masters in Computer Science. I would like to do some research or a project relating to cloud computing or services offered in the cloud, but that is TBD for now.

The main purpose for this blog is simply a place to write my own take on many of the things I see and read about in the tech world. Whether its ideas I have been thinking about or comments on things others have written it ends up here on my blog. I welcome comments or opinions on any posts I have made since these ideas are always up for debate.

For anyone interested, here is a copy of my resume. More information is available on my CS website.
Feel free to contact me by email: jkupferman (AT) cs .. ucsb .. edu.