You can make a karma threshold, to prevent/reduce abuse. Also you can let users have the option to enable/disable this feature.