Multiple parts moving when I drag one part... :(

Hi - I'm new to mechanisms contstraints, but getting there.

I have a problem where I have a door handle that can be depressed through a limited arc.  There is also a button on the handle which can be depressed.  I have set up the constraints for both of these pieces, but the problem is when I drag the button to the different positions on screen, the whole handle moves (through the arc) on the screen too.  How do I set up so that I can drag the button to different on/off positions but I don't change the angle of the handle?  Each piece should be independently movable without moving any other part.

Counter to that, I can move the handle through the arc and the button stays in the position and doesn't move, so i know it can be done

Thanks..