You’re on the right track. Once you’ve assign a Marker to it, set it to Animated and it’ll follow your animation and interact with the rest of your simulation.
You can either remove the Marker from the Outliner, or select what you want removed and instead of running Delete All you pick Delete Selected.
The keys inside of the Manipulator, unfortunately not. But see here for a workaround.