Your comment on this discussion reminds me of Paul's letter to the Corinthians. The part about eating meat from a meat market that may or may not have been offered to a pagan god before being put out for sale. Paul counsels to avoid being a cause for stumbling to those that are weak, but concludes by asking, why is my liberty judged by another man's conscience? Pointing out that while the strong have an obligation to not stumble the weak, those weak ones are equally obliged to become strong enough that they are no longer easily stumbled.

Kind of like what you said.