IMO just because he wants to spend time with you does not mean he wants to move too fast. If he is a recent widower he is probably mostly lonely and just wants someone to do things with. (I could be wrong and if I am do not hesitate to let him know how you feel.)
And don't hesitate to tell him if you don't want him visiting before work. Maybe you could give him a better time?