What do you call the boyfriend of a married woman?

Adulterer (or adulteress) can be used for anyone who has a sexual relationship with a person who is married to another, but again does not always imply the specific social situational that mistress does.