Generally - no, not at all. But it does depend on lots of other factors.
If there is something more important that I should be doing - if I'm indulging a hobby as a form of procrastination when I should be working or engaged in something of more tangible "benefit" then obviously I would feel some guilt.
If it's resulting in me neglecting other aspects of my life to a certain extent - family, friendships, personal development etc. then again I would start to feel a bit guilty.
If I've stayed up far in to the night playing a game to the point where I know it is going to have a negative impact on my next day - being tired or moody as a result for example, then again I would feel some guilt about it.
In none of these situations would I say that stops me having enjoyed the activity, and it is in no way an overwhelming feeling of guilt. But I don't think feeling a pang of regret at these sorts of things is "wrong" as such. There is generally a valid reason why our bodies/minds make us feel that way and we should probably pay it a certain amount of heed!
Reminds me of this bit from Peep Show. Gaming Guilt by Mark Corrigan, a tale in 9 GIFs: