What you really must strive for is to cause minimum amount of pain to other living beings. You cannot totally avoid it. So wearing silk is okay.
There are different types of silk. There is a certain type of silk that is made without causing harm to the silk worm. Ahimsa silk is also available (silk obtained without harming or killing insects or animals) in which they do not boil the silk cocoons, but rather extract the silk without disturbing the worms.