tom does things » scripts

HBase migration across major versions of HBase (0.2x -> 0.9x)

tom — Thu, 29 Sep 2011 23:28:22 +0000

Background: I inherited a legacy version of the hadoop stack (0.20.2) running hbase v. 0.20.3 on top. As this is quite an old and unsupported version, we obviously wanted to upgrade. The main problem with this is that this system is already in use in production, thus increasing the upgrade risks. In addition, the instance [...]

Ruby on Rails served by unicorn with nginx for static files and reverse proxy on Ubuntu – detailed guide.

tom — Thu, 10 Mar 2011 15:39:01 +0000

Ruby, the ever-so-popular scripting language and Rails – one of the leading web application frameworks are prone to serious performance issues if not served efficiently. Enter unicorn, a high performance replacement for mongrels and such. And of course, my goto tool for all things http: nginx. This is a short step-by-step detailed guide to achieving [...]

Owner/Group/Permissions Synchronization

tom — Thu, 11 Feb 2010 09:46:13 +0000

Image the following scenario: cd /etc chown root.root * -R OOPS You have just destroyed the server. This, and other similar mishaps (i.e. chmod 777 / -R) occur more often than one would imagine. If you are in luck like me, you have access to more than one linux server, and in most cases these [...]

Apache Cluster Config Sync

tom — Mon, 11 Jan 2010 19:07:21 +0000

It has become quite commonplace today, for high traffic sites to require more than one web server. Here is a small script in charge of synchronizing the server configuration files (/etc/httpd in this example). In this case, I will be using apache as the web server of choice, simply because of its prevalence and popularity, [...]

MySQL Static Cache Daemon usage example

tom — Sat, 09 Jan 2010 16:04:05 +0000

As promised, here is short example of the Daemon in operation. First, in order to generate some data for us to work with, I did the following: This nice page shows us how to get various stock data from Yahoo, who are kind enough to provide an easy-to-access API to their data, and even retrieve [...]

MySQL Static Cache Daemon

tom — Tue, 05 Jan 2010 09:52:23 +0000

The problem that this program was designed to solve is a rapidly changing MySQL table in use by a high-traffic website. For instance, stock quotes on the front page of a bank’s or investments firm’s site. The data is constantly updated by a service on the backend, and is referred to by some ajax widget [...]

MySQL on Windows Batch Backup Script

tom — Mon, 30 Nov 2009 19:25:46 +0000

Don’t ask me why scripting in batch is still necessary on the verge of the year 2010, however, what can you do when some clients still use Windows to host their MySQL servers… REM Simple MySQL backup script per database @echo off REM Set some variables set mysqlcmd="C:\Program Files\MySQL\MySQL Server 5.0\bin\mysql.exe" set mysqlpwd=amazingsecretsamplepassword set mysqlconnect=%mysqlcmd% [...]

MySQL replication monitoring with Python

tom — Mon, 27 Jul 2009 23:21:56 +0000

Setting up a MySQL fail over Master-Master replication system is something one can find many guides for. However, monitoring and recovering from a failure of the system is not as well documented, according to a quick Google search. The top items from that search were: Something called: MYSQL master-master replication monitor shell script that mainly [...]

RAID controller *NIX agent

tom — Tue, 21 Jul 2009 14:02:10 +0000

When managing a large number of servers, it is most likely that you will come across a server that makes use of a RAID controller, and as such you will need to know the status of the disks and the disk array. As these are usually proprietary controllers, normal tools like ipmiutil, lshw, lspci, etc. [...]

Convert MySQL Tables: MyISAM to InnoDB

tom — Mon, 20 Jul 2009 16:47:24 +0000

Not getting into the whole which engine is the best debacle, since obviously, like most important questions in life the answer is: "It depends." In addition, most of what is written here can be used to convert from any engine to any engine, and thus answering any specific needs your might have. I needed to convert MyISAM to InnoDB so here it is. Without further ado... As far as I know, there are two basic ways to accomplish this:

 ALTER TABLE 'tablename' ENGINE = InnoDB

For each table in the database.

Dump the table, edit the dump file where it says "CREATE TABLE" and fix to the preferred engine, and then reload the data back into the DB

Obviously, these methods are ok when you are dealing with a few tables, however a while ago we had a client with a huge database with over 50 tables, so of course, a script was in place.