Either keep rubbing longer, or you'll need something stronger. I'd try rubbing more myself, check the instructions on the tube. It'll take out all of those little scratches around a cage door handle quite easily, so shouldn't have any problems with swirls on your bike.