I found it quite easy to remove by twisting it, almost as if designed to be done so. If it is not designed that way, then at the very least the hard rubber flange extending above and below the connector will naturally force the connector to move forward or back if twisted in the rectangular slot where it resides.
This same physical feature made it somewhat more difficult to push back into the fully-assembled slot, but I found that starting at enough of an angle that allowed the hard rubber flange to bend backward, and then twisting into the proper orientation as you push inward, the result was a complete and solid connection.