Slony 1.1.5 subscribe set + autovacuum == deadlock

Slony has been pretty helpful in our migration to postgresql 8.1. Since you can run different versions of postgres on different nodes of the same cluster, we were able to bring up 8.1 on the primary and vet it for a few days while leaving the secondary on 7.4. That way if we ran into any showstoppers (hey, even with lots of QA, production can often throw you a curve), we could do a switchover and be back on 7.4 in a few minutes.

Well, that scenario never came to pass, and now I am upgrading the secondary to 8.1 as well. After a rather slow experience reloading the primary after its upgrade I opted to rebuild the secondary rather than dumping it, reloading it and letting it sync.

Everything was going swimmingly and the largest table was copied over when suddenly it caught a deadlock and aborted the load. Bummer. Troubleshooting deadlock errors are a bit like coming on an accident scene and trying to figure out what the drivers were thinking before the crash. Only in this case the crash has been rolled back, at least one of the vehicles has disappeared and all you have is a couple of terse notes to go on.

Looking at the logs, I did see connections coming in from some other servers that I didn't expect. I decided to block all connections except local ones and those from the primary (via pg_hba.conf). Things were still trying to connect at regular intervals -- exactly what is a bit of a mystery to me, the secondary should only receive periodic connections at predictable times -- but now failing. This is when a bit of good luck came along: ironically in the form of another deadlock, this time much faster than before. I say this was good luck because I might well have waited another hour (yeah there's a lot of data) before it blew up.

I was scratching my head trying to figure out what was connecting. The pg logs were not particularly helpful even though I log a good bit of detail about the clients for this purpose. It dawned on me that it was probably the autovacuum daemon making the rounds. Now, I had just recreated this same secondary a few days ago under 7.4 with autovacuum (same version of slony) and there was no deadlock. However I know for a fact that the old autovac config was much less aggressive than I have it configured in 8.1 (especially given the key vacuum throttling available now), so I imagine it just never found anything to do, or there was some other behavioral difference in the new version that bit me.

So, long story short, I turned off autovac during the the secondary subscribe and so far so good, it hasn't fallen over. It's still chugging away on one of the big-ass tables, but here's hoping it comes around.

Moral: Make sure nothing else can mess with the secondary while it subscribes. Disable outside connections and turn off autovacuum.

In truth this whole problem is really a known bug in slony 1.1.5. I'm told that slony 1.2 grabs its locks early to block out everyone else at the get-go. Here's hoping that 1.2 hits prime-time soon, though I suspect being a feature release might make our integration longer to sort out. Nevertheless I'm happy to see so much effort going into Slony, it's a good reliable system, not without a gotcha or three, but well suited to its purpose.

vim 7.0 spellcheck

Here's a primer on using the new integrated spellcheck feature in vim 7.0:

To turn on spell checking add this to your .vimrc:

setlocal spell spelllang=en_us

or simply:

set spell

which seems sufficient for me. I don't want it for all files, so I added it conditionally for text, html and README files:

autocmd BufNewFile,BufRead *.txt,*.html,README set spell

Then create a directory ~/.vim/spell for vim to hold personal word lists.

Once turned on here's a quick rundown of the basics (of course :help spell is your friend):

use ]s and [s to navigate to the next and previous misspelled words.

use z= to suggest spelling for a misspelled word

use zg to add words permanently to your personal word list. This marks it as a "good" word.

use zG to temporarily add words for just this editing session.

use zw to mark a word as misspelled.

There are other shortcuts as well:

While in insert mode, use CTRL-x s to suggest spelling for a word to the left of the cursor. You can then select a suggestion from the popup menu.

If you enter a common misspelling or mis-capitalization, try the 1z= command which replaces the word with the first spelling suggestion straight away. You can always undo it if it's not the right one.

MacBook Pro, Yo

I've pretty much gotten the new MacBook setup to my liking. I must say, Apple really has it down in transferring from old to new machine. All of my stuff was seemlessly moved over from my "old" G4, barring some intel-incompatible apps. I had to install a new version of vim, but I found the 6.4.000 version would run but crash on save (ouch). So I installed the 7.0c2 beta. It doesn't crash, but there are a few bumps, specifically:

It doesn't seem to draw text quite right. Sometimes the text gets shifted over, when you cursor over it, the character it looks like you are on isn't the right one. The right one appears for a second, but then it shifts back again. Even refreshing the screen doesn't help. I guess I'll try another font, but if I have to give up ProFont I'm gonna be pissed ;^)

It doesn't seem to draw text quite the same way. I was using an anti-aliased ProFont before, but it draws too small now (though the line spacing looks right). In fairness, this may be an issue with Tiger and not vim (I was running 10.3 before)

When you use "change" commands (cw, ct, cf, etc), and you use backspace to delete from the right, the characters don't go away until you switch back to command mode. I've seen vim act like this in the past, so I suspect a change in a config option's default value

On the bright side, 7.0 adds built-in spellcheck, and the new virtualedit=onemore option removes an annoying vi artifact. onemore allows you to place the cursor one character beyond the end of the line without entering insert mode using A for example. The undo branch functionality sounds amazing, if I understand it correctly, it will mean everything you enter will stick around somewhere, regardless of intervening redos/undos. I just installed it today, so there are bound to be other handy options to figure out tomorrow. If I can get the text to draw correctly, I'll be in business.

There is still one thing that annoys me to no end with vim. Try the following: In command mode, place the cursor in the middle of a line of text. Then alternately enter insert and command mode using "i" and [esc]/[CTRL+C]. Notice how the cursor moves to the left each time. Now this doesn't happen if you enter insert mode using "a". Perhaps I'm just assbackward, but what I find is that I'm always one character off to the left from where I want to be when I leave insert mode. The onemore option does help since it allows you to place the cursor "past" the end of the line. I'm sure there's a hack to take care of this and someday I'll find it (and post it here for posterity).

update: I figured out how to get characters to go away right away when using change commands and backspace. I added this to my .vimrc and all was well again:

" Redraw immediately when replacing text within a line
set cpoptions-=$
" Erase backspaced chars immediately
set cpoptions-=v


hacman

A tribute to this blog's namesake:

http://users.erols.com/mowerman/pacfile.htm

I especially dig the part about "Crazy Otto", the original (and bootleg) game that became Ms. Pacman

what's new in python 2.5

Andrew Kuchling has provided us with an overview of what to expect in python 2.5. It's not done yet, but there's some good stuff there anyhow:

http://docs.python.org/dev/whatsnew/