Should I leave him or just talk it out with him?

Run,baby run! You got it right and so you are smart. Not everyone can pick up on those clues and draw the right conclusion. He must be much older than you and be guilty of statutory . (If you have been ual). His ex would probably have the police onto him if she knew about you, so that's why he cant see you when his son is there.