Hillel's Weblog

Shazam Magic

hantonov — Wed, 06 Jan 2010 17:26:35 +0000

On my short drive to the computer parking this morning, I tuned in to the WBGO radio station. There was a song I liked and I wanted to know what it is. Usually when this happens, I check wbgo’s web site where they have the play list, but for this I need to remember the time and I need to remember to check the web site. This time I decided to use Shazam app on my iPhone. The app managed to tag it as one of Michel Camilo songs (why not!). This made me wonder what is the technology behind this. My initial thought was FFT on the song sample but then comparing this to every second of every song would be impractical. Quick google search reviled the academic paper written by someone from Shazam and a good explanation by this blogger. Very impressive technology.

Random password generator.

hantonov — Thu, 17 Sep 2009 15:38:01 +0000

Very simple. I checked it on OS X and Ubuntu…

egrep -ioam1 '[a-z0-9]{8}' /dev/urandom

Exchange 2007 Management Shell

hantonov — Mon, 20 Apr 2009 18:59:16 +0000

When activating Exchange Management Shell directly (for example from scheduler or monitoring application) and not from the link that installation is crating, this is the command line

PowerShell.exe -PSConsoleFile “C:\Program Files\Microsoft\Exchange Server\Bin\ExShell.psc1″ -Command “./myscript.ps1″

Update

As an alternative, the first line in the script can be

Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Admin

Autocomplete for ssh

hantonov — Thu, 16 Apr 2009 18:01:37 +0000

This is a handy autocomplete for hostnames for the ssh command. Works with bash on OS X

Add this to .bash_profile

complete -W “$(echo `cat ~/.ssh/known_hosts | cut -f 1 -d ‘ ‘ | sed -e s/,.*//g | uniq | grep -v “\[“`;)” ssh

Windows performance counters and WMI

hantonov — Tue, 14 Apr 2009 00:43:34 +0000

I wanted to monitor performance of HP EVA storage with cacti. Installing EVAPerf on a server connected to the SAN installs performance counters for the EVA. Using WMI queries it should be possible to gain access to the counters. WQL query similar to “SELECT * FROM Win32_PerfFormattedData_EVAPMEXT_HPEVAPhysicalDiskGroup” should return objects for each physical disk group. Each object contains counters like read/write latency. Unfortunately my queries didn’t return anything. I have used similar techniqe to access terminal service gateway counters on a different server so the problem was not with the way my perl script was accessing the wmi. I also could see the EVA counters in perfmon mmc so it looked like the wmi service didn’t have access to the counters. Using wbemtest.exe I confirmed that the definitions for the counters were installed but there were no instances. Apparently it is possible for wmi not to be in sync with the installed counters and Microsoft provides a way to sync it. Here are the steps.

Stop the Windows Management Instrumentation service
From the command prompt, run winmgmt /resyncperf
From the command prompt, type wmiadap.exe /f
Start the Windows Management Instrumentation service

After synchonizing wmi I started to get results and some nice graphs in cacti.

Sending test snmp traps from command line

hantonov — Thu, 19 Mar 2009 18:00:37 +0000

Example of a command to send snmp traps from Linux shell.

snmptrap -v 1 -c public host UCD-SNMP-MIB::ucdStart “” 6 17 “” SNMPv2-MIB::sysLocation.0 s “Hello World”

Linux High Availability Cluster

hantonov — Sun, 21 Sep 2008 17:20:04 +0000

Our application is using NFS server, since we use it just for serving static pages, we keep it on a virtual server running on VMWare ESX 3.5. This week the server froze and after cold restart didn’t boot. Luckily, the file systems were intact and I mounted them to a temporary server to export as an NFS service. The downtime was short but noticeable. After restoring the service I started to investigate the options to provide High Availability (HA) NFS service.
Looking from pure OSS perspective the only viable option with Ubuntu distribution is HA/DRBD configuration. It is not unusual for OSS solution to be a combination of several application to compliment functionality. I assume that for historical reasons, cluster solutions for Linux are mostly share nothing configurations. In the past, when shared storage was expensive, this was a great advantage that overweighted the complexity of the setup and the duplication of the data.
DRBD is used to replicate in realtime block devices. This is Active/Passive configuration where changes to the file system are replicated in realtime to the passive node. HA is responsible for the heartbeat and failover of the resources from Active to Passive node. For NFS, there are 3 resources. The IP address, the files system (replicated by DRBD) and the NFS service that needs to be activated after the file system is mounted after the failover.
After completing the setup and deploying to production, here some thoughts.

The products are not integrated properly and although HA has a script to mount the DRBD device after the failover this is where the integration stops. HA has no knowledge about the status of DRBD and if there is a problem with the replication, the failover will still occur.
Both products require monitoring of their own, especially DRBD, since if there is a problem with the replication, there is no way to know about it without directly checking DRBD.
The replication is one way. Although HA supports fallback, the DRBD doesn’t out of the box and needs some tweaking with the scripts. If the Master server needs rebuilding, full replication must be performed before fallback is possible. This takes time depending on the size of the block device. During this time the cluster is not redundant.
There are some concerns about latency that DRBD introduces into the I/O path since the replication is synchronous. Not that important in my NFS implementation, but can be a problem for MySql for example.
HA is relatively primitive, and has no knowledge of the resources. For example, if NFS service will fail, HA will not initiate fail over. There is a need to write scripts to monitor the services (NFS, MySQL, etc…) or use applications like Munin.

I would really like to see a cluster product that is more like Windows cluster that after the initial shock of the price is very easy to install and support and has an API that if used can make any application cluster aware.
I saw that RedHat has a cluster offering that works with a shared storage/GFS. I think I will give it a try.

MySQL rants

hantonov — Wed, 30 Jul 2008 20:52:31 +0000

For the last year my primary production database is MySQL. We are using community version that comes precompiled with Ubuntu. It is a very good and stable product suitable for production. We buy support directly from MySQL and due to the nature of their business model, the support is excellent.

I have two issues with supporting MySQL in production: profiling and backup/restore.

Profiling MySQL is very primitive and virtually impossible. MS SQL for example, has built-in instrumentation and can provide a lot of useful real time, and with appropriate tools like Symantec I3 for databases, historical information. In the past I successfully used I3 to improve application performance of a real time ASP application for VoIP tellecomm. With MySQL all you have is show processlist, show status and show innodb status (if you are using innodb engine). There are some tools that can help analyzing the output of the commands and provide some trending information. Since we are MySQL paying customers we have access to the MySQL Enterprise Dashboard that monitors the database and the server. With this tool, I can see that the server has I/O bottle neck or had a CPU load 2 hours ago, but I have no idea which query caused it. It looks like for really deep profiling there are two options, try to write custom solutions, my colleague Chetan, did that or use DTrace with Solaris. Since Chetan as developer took the first path, I will take the second.

The second operational issue for me is backup/restore. MySQL has no API for both in versions 5.0 and 5.1. API was introduces in version 6.0 which is in alpha stage right now. There are serveral ways to hot backup MySQL with innodb database. I do not like the tool provided by innodb it self since it is slow, expensive and the license is tight to the server name. The other two options are to do sql dump of the server, not a recommended for databases bigger than 5Gb, or do LVM snapshot. LVM option doesn’t allow backup/restore of individual databases and because of that I only use it to create backup for setting up replication. So yes, right now I’m using the dump option with understanding that the restore process will take time.

Sun server x4440

hantonov — Tue, 08 Jul 2008 13:18:41 +0000

I have worked with many x86 servers including the major vendors like HP/Compaq, IBM and Dell. I like the HP servers mostly for the best management tools and the ease of drivers installation (especially on windows). When Sun just started to ship AMD based servers I bought one 1U server for test and as a cheap x64 test machine. It was a very bad design and I had a lot of HW problems. I discovered, the hard way, that Sun was not ready with people to support the servers.

Recently I had a chance to see the servers on MySQL conference and was very impressed. So when I needed couple of 4 sockets servers I decided to buy x4440. Those are 2U machines with up to 4 quad or dual AMD CPUs, 128Gb RAM max,8 disk drives and 4 on board NICs. HP has DL585 G5 but it is 4U and has only 2 onboard NIC. All the “main stream” operating systems are supported. I’m planning to run Ubuntu 8.04 on them. The installation of Ubuntu was smooth. Unfortunately none of the hardware vendors support Ubuntu and I had to compile Adaptec management services and tools. I think it is important since this is the only way to know if any of the disks in the raid failed.

Since I’m planning to run MySQL on the servers, I did some sysbench tests and the results are impressive.

Overall I think Sun has has a good product.

My MacBook Pro died

hantonov — Sat, 28 Jun 2008 16:46:15 +0000

I’m a long time Windows user and used Windows on servers and clients. At some point I even had MCSE certification. I consider my self a very powerfull user. When I started my new job a year ago, I had an option to take MacBook Pro and switch to OS X. After a short learning curve and getting used to do things OS X way I’m totaly converted. I do run Windows XP in virtual machine using parallels for all those application that require Windows or Internet Explorer. VMware VI is one of them.

This week my Mac starteed to freeze or give me the Gray Screen of Death. I do not know if this is the official term for it. I have MozyPro backup for documents and files and I backup media files on FireWire disk. I was not much concerned about loosing any data. By the way, Mozy client doesn’t backup hidden directories and files so it didn’t backup my .ssh directory and I lost my private key. I work in midtown New York, so I took the notebook to TekServe. After waiting for more than an hour I was told that the logic board is faulty and it will take at least a week to fix it. Until I will get my Mac back I’m using Lenovo X61 tablet running Windows Vista. The only advantage this machine has – it is really small. The disadvantage – it runs Vista and IE 7. Simple operation as opening new tab in IE7 takes couple of seconds. Luckily FireFox 3 is a great alternative. I hope TekServe will call me soon.