If when you lived away from your home town you would fantasize about having sex with your brother-in-law from time to time, just as a way to make yourself aroused, I would say that was OK, though I recommend that people pick fantasy lovers who are 100% unavailable, like a movie star, not someone who might be available. And the reason I say this is to avoid the problem you are encountering now: suddenly the possibility of making those fantasies real has entered your life.

The first thing you have to do is decide whether after 23 years of marriage, the marriage is no longer worth continuing. You didn’t indicate what, if any problems you had with your husband. If there are none and if you love him and if you want to stay married, then you have to put all thoughts about having an affair with your brother-in-law out of your mind. On the other hand, if you want out of your marriage, then you have to separate and then see what your brother-in-law might want to do. (If he’s also married, that complicates things.) But these two decisions have to be separate, otherwise if you go ahead and have an affair with your brother-in-law (who may or may not be interested after all these years) you’re going to ruin your marriage.