Your sex drive can be fragile when there’s too much stress in your life. Lack of interest in sex can destroy relationships and cut your self-confidence to zero. There are many things, including hormones, that can affect your sex drive, but stress is one of the main culprits.