It's best that you sit down and have a ncie talk to him... Work things out calmly first... if U are also Christian, my advise is "Pray" first before you approach him... Or try getting a Pastor to assist you in this area....
I understand your mixed and confused feelings now... Settle it fast... There should be mutual trust between both married couple... One shouldn't do things in the dark, behind his wife or her husband...