Uh... stupid mistake! I still had the pull-down resistor on the slow jumper pin from my first attempts... works perfectly now that I removed it. Sorry for wasting you time!
what programming times can I expect from AVR Doper? Programming and verifying a ATmega324P takes about 130s for me. Faster programming would be nice... I am using HID mode with avrdude in Mac OS X.