I think you guys should go to couples counseling before this becomes a bigger issue. Sex is an important part of the relationship and definitely a deal breaker. If you are considering this on a real level then do the right thing. Either work on it together or move on.