Hi,
I would like to preface my comments in that I am very new to metal detecting.
I think we all have interests that can be both fascinating and time consuming but unless you committed crimes so you could get a metal detector or assaulted a family member to go out detecting don't beat yourself up too badly. Being human means we will make mistakes along the way. Unless those mistakes were based upon actual malicious intent the decisions that trouble you are no more than a bad judgment call.
My wife likes golfing. I have no use for it. If I divorced her because she spends time doing so with other people then chances are I was already looking for an excuse to get divorced. Because I still love her I bought her a new set of clubs and encourage her to enjoy her activity. Not my thing but if it makes her happy then all is well.
I went and bought my first detector only a short while ago and I did not tell her what I was buying. When I arrived home and showed her the machine she remarked that since she wanted me to walk more she was glad I bought it. She didn't call our lawyer or move out.
The point is that having interests in life, even if different than your spouse, is healthy in my opinion.
Another thought is since this hobby seems to be slewed towards us slightly older people I am really happy that I can look forward to yet another interesting activity when I do retire. I used to have a small business and the people I felt sorriest for had no interests after they left the workforce. Once they retired they fell apart or descended into a sea of anger and self pity. I don't think that will happen to people reading this post or on this forum.
Can you imagine living in a world devoid of mysteries, miracles and magic? I can't think of anyone who metal detects or partakes of some of my other hobbies that suffers from a complete disinterest in the world. Those that are afflicted with such apathy towards life not only are miserable but usually make everyone around them equally miserable.
One last thought. Dick Stout mentioned moving his family from NJ to TX. I wouldn't see that as horrible. After all, maybe staying in NJ would have been disastrous for them. No one knows how things might have been but for the move. The fact that you feel a bit guilty says you are a stand up guy with morals who cares very much for his family.
I apologize for my lack of eloquence to make my points better but in summary don't feel bad about having interests such as metal detecting. There are far worse things you could have done.