Why is Rugby Union Posh?

In many countries, rugby union is considered to be a sport for the middle classes. Stereotypically played in private schools by the sons of professionals, the sport often struggles to overcome this perception. However, how true is this reputation and why did the sport gain it in the first place?