Ok I think I found the key. Find and read the obesity code by Jason Fung. (you can find him on you tube as well) This is it. I have already lost 3 lbs in two weeks which is much better than my typical 1 lb per week and, I don't feel like i am struggling.

He suggests that the reason we all struggling is that we have too much insulin. We are insulin resistant. Thus, your body stores most of the food you get in a day away as fat and doesn't burn it off.

Once you fix your insulin resistance, via intermittent fasting, your body will again work properly push your food into the cells to be burned off and your RMR will go up because it will sense you are getting more energy thus it needs to increase its calorie consumption.