Requirement: Artist portfolio site with content management capabilities. The site wasn’t not to be a blog, and therefore solutions such as wordpress were out of the question.
In addition, wordpress doesn’t really work very well for items of the gallery type.
On the other hand, a flickr/picasaweb album link isn’t suitable for printing on your business card.
Thus I created “Quickfolio”.
The source is shared on github.
I made it Tornado based, simply because I love the Tornado’s ease of use and the fact that it is in Python.

The nginx config file sample provided in the repo, enables nginx to serve the static files directly as well as refer all dynamic request back to Tornado.

I used Google’s developer guide for Picasa Web Albums extensively.

So, no database, no interface, no nothing. Just a quick setup and you have a full blown gallery with Google acting as the CMS and CDN!

I even added two examples of sites I built using QuickFolio in the readme on github.

TODO: make init script for Tornado (I’ll probably base it on my unicorn solution)

The problem is stated here.

I will first attempt explain the challenge simply, since from my recent experience, it seems that a lot of software engineering exercises have as much to do with reading comprehension as with actual programming skills.

Then I will describe my approach and also present an alternative quick solution.

Finally I will paste the annotated code and make the script available for download.

Continue Reading »

Background:

I inherited a legacy version of the hadoop stack (0.20.2) running hbase v. 0.20.3 on top.
As this is quite an old and unsupported version, we obviously wanted to upgrade.
The main problem with this is that this system is already in use in production, thus increasing the upgrade risks. In addition, the instance installed was compiled and patched manually, so an in-place upgrade was out of the question.

For the new cluster, installed side by side the old one, I chose the “Cloudera Distribution including Apache Hadoop (CDH3)” which is very easy to install and set up via aptitude, and prepares all the standardized stuff like config file locations and init scripts.
This should also hopefully enable relatively painless updates in the future.

Sematext already has a nice blog post on the subject of various HBase backup options, so I won’t go over them in too much detail.

All I will say is that the built-in CopyTable MR job is out of the question since it only works with two clusters of the same major version of HBase. Otherwise life would be much easier…
Continue Reading »

This is just a short guide on how to find the main offenders in case of web server hammering.

Sample of eventual output:

Continue Reading »

Holy Crap

Nothing technically significant there – but since this is the first time I’ve been exposed to such traffic – here is a shout out to people who referred here.

Thanks Ruby Weekly and thanks to the Ruby Show.

Another special thanks to the commenters here and on reddit – provided some very interesting technical insights. It is always humbling and informative to be peer reviewed by such experienced people.

The funny thing is that I’m not even a Ruby guy – I guess several people found the system side of things interesting as well.

Glad to be of help in any case, and will do my best to add interesting content to the site.

Ruby, the ever-so-popular scripting language and Rails – one of the leading web application frameworks are prone to serious performance issues if not served efficiently.
Enter unicorn, a high performance replacement for mongrels and such.
And of course, my goto tool for all things http: nginx.

This is a short step-by-step detailed guide to achieving the following:

• RoR application listening on 2007 via unicorn
• nginx listening on 80 forwarding all dynamic requests to the unicorn, and serving static files.

The nginx and the ruby are compiled from source, to provide the latest versions (1.9.2-p136 and 0.8.54 respectively) rather than the ones available via apt (for Ubuntu 10.10 that means: nginx 0.7.67 and ruby 1.8).
Continue Reading »

Ok so it all started like this:

Bunch of web servers, all the same model (Dell R710), all functioning properly – however two of them are faster then the others (40% lower response time AND lower load average).
Average connections to all the machines was identical.

• Hardware: Identical
• Software: Identical (to eliminate this as an issue, a disk was removed from the “fast” server RAID and placed in the “slow” server as a primary drive, and the array was rebuilt with the exact same data)

So it wasn’t a hardware issue (as far as the specs and components could show) and it wasn’t a software issue (same applications and code were the same on both machines).

Just a 40% performance discrepancy…
Continue Reading »

Image the following scenario:

OOPS
You have just destroyed the server.
This, and other similar mishaps (i.e. chmod 777 / -R) occur more often than one would imagine.

If you are in luck like me, you have access to more than one linux server, and in most cases these servers are similar enough in order to enable a quick restoration of the many file permissions you just destroyed.
Continue Reading »

It has become quite commonplace today, for high traffic sites to require more than one web server.
Here is a small script in charge of synchronizing the server configuration files (/etc/httpd in this example).

In this case, I will be using apache as the web server of choice, simply because of its prevalence and popularity, however the idea is the same, and the script could probably be easily edited to fit your web server of choice (nginx, lighttpd or what have you).
Continue Reading »

As promised, here is short example of the Daemon in operation.
Continue Reading »