Most moslems believe the ka'bah can grant favours or cleanse sins. That's why they scramble and jostle to touch it and kiss it. You don't speak on behalf of mainstream moslems, only for your weird version of islam.
"Ibn Umar was clinging on the two corners (in a manner that I had not seen any of the Companions of the Prophet doing) so I said: 'O Abu Abdur-Rahman! You are clinging on the two corners in a manner that I have not seen any of the Companions of the Prophet clining.' So he said: 'I do it because I heard the Messenger of Allah saying: "Touching them atones for sins."
Reference : Jami` at-Tirmidhi 959