Look, Hon, let me put it to you this way.
If you wouldn't have his baby, then just don't have sex with him. Eliminate the risk factor. That should simplify things considerably.
It saves the wonder whether you should, the worry on the nerves about what if, and the wear and tear mentally, physically and spiritually of any and all results that/those overnight/evening/hour/2 minutes of thrill might bring.
It's okay to remain chaste. Just don't go back to either apartment together where you'd get tempted.