Note the shape of the hole under the replacement analog stick cap and how it matches the shape of the analog stick shaft. Orient them correctly for a proper fit.

Take your new caps and push them into the shaft.

Push the cap all the way down until you hear a ‘click’, which is the analog button being pressed.