MySQL rants

For the last year my primary production database is MySQL. We are using community version that comes precompiled with Ubuntu. It is a very good and stable product suitable for production. We buy support directly from MySQL and due to the nature of their business model, the support is excellent.

I have two issues with supporting MySQL in production: profiling and backup/restore.

Profiling MySQL is very primitive and virtually impossible. MS SQL for example, has built-in instrumentation and can provide a lot of useful real time, and with appropriate tools like Symantec I3 for databases, historical information. In the past I successfully used I3 to improve application performance of a real time ASP application for VoIP tellecomm. With MySQL all you have is show processlist, show status and show innodb status (if you are using innodb engine). There are some tools that can help analyzing the output of the commands and provide some trending information. Since we are MySQL paying customers we have access to the MySQL Enterprise Dashboard that monitors the database and the server. With this tool, I  can see that the server has I/O bottle neck or had a CPU load 2 hours ago, but I have no idea which query caused it. It looks like for really deep profiling there are two options, try to write custom solutions, my colleague Chetan, did that or use DTrace with Solaris. Since Chetan as developer took the first path, I will take the second.

The second operational issue for me is backup/restore. MySQL has no API for both in versions 5.0 and 5.1. API was introduces in version 6.0 which is in alpha stage right now. There are serveral ways to hot backup MySQL with innodb database. I do not like the tool provided by innodb it self since it is slow, expensive and the license is tight to the server name. The other two options are to do sql dump of the server, not a recommended for databases bigger than 5Gb, or do LVM snapshot. LVM option doesn’t allow backup/restore of individual databases and because of that I only use it to create backup for setting up replication. So yes, right now I’m using the dump option with understanding that the restore process will take time.

Sun server x4440

I have worked with many x86 servers including the major vendors like HP/Compaq, IBM and Dell. I like the HP servers mostly for the best management tools and the ease of drivers installation (especially on windows). When Sun just started to ship AMD based servers I bought one 1U server for test and as a cheap x64 test machine. It was a very bad design and I had a lot of HW problems. I discovered, the hard way, that   Sun was not ready with people to support the servers.

Recently I had a chance to see the servers on MySQL conference and was very impressed. So when I needed couple of  4 sockets servers I decided to buy x4440. Those are 2U machines with up to 4  quad or dual AMD CPUs, 128Gb RAM max,8 disk drives and 4 on board NICs. HP has DL585 G5 but it is 4U and has only 2 onboard NIC. All the “main stream” operating systems are supported. I’m planning to run Ubuntu 8.04 on them. The installation of Ubuntu was smooth. Unfortunately none of the hardware vendors support Ubuntu and I had to compile Adaptec management services and tools. I think it is important since this is the only way to know if any of the disks in the raid failed.

Since I’m planning to run MySQL on the servers, I did some sysbench tests and the results are impressive.

Overall I think Sun has has a good product.

My MacBook Pro died

I’m a long time Windows user and used Windows on servers and clients. At some point I even had MCSE certification. I consider my self a very powerfull user. When I started my new job a year ago, I had an option to take MacBook Pro and switch to OS X. After a short learning curve and getting used to do things OS X way I’m totaly converted. I do run Windows XP in virtual machine using parallels for all those application that require Windows or Internet Explorer. VMware VI is one of them.

This week my Mac starteed to freeze or give me the Gray Screen of Death. I do not know if this is the official term for it. I have MozyPro backup for documents and files and I backup media files on FireWire disk.  I was not much concerned about loosing any data.  By the way, Mozy client doesn’t backup hidden directories and files so it didn’t backup my .ssh directory and I lost my private key. I work in midtown New York, so I took the notebook to TekServe. After waiting for more than an hour I was told that the logic board is faulty and it will take at least a week to fix it. Until I will get my Mac back I’m using Lenovo X61 tablet running Windows Vista. The only advantage this machine has – it is really small. The disadvantage – it runs Vista and IE 7. Simple operation as opening new tab in IE7 takes couple of seconds. Luckily FireFox 3 is a great alternative. I hope TekServe will call me soon.

Hello world!

First entry, probably need to tell about my self.

I’m a geek and IT professional. Worked for different companies, including Fortune 100 corporate and startups in different stages. Started as VAX/VMS operator working shifts. Developed my career up to beeing VP of IT, managing people, technology and budget.  A year ago I decided to check the option of beeing individual contributor and became Systems Architect.  From time to time a have a privilege to help other IT organizations as a freelance consultant.

I’m very hands on and will probably write about the things I’m working on but I’m also interested in how IT is integrated in the business and helps the business to achieve the goals (my corporate roots are deep).  There are many people who write abut technical stuff but almost none who write as IT middle management and the challenges these managers are facing.