Is infidelity ever right?

I have been involved with a married man for about a year and a half. We met in the workplace and although he is quite older than me, it progressed into this lusty secret affair that has spun out of control on many levels. I am not married and I am 10 years younger than him. He has children and many investments into his marriage that prevent him from leaving. But when we are alone it feels like all that doesnt even exist. Each day I feel myself becoming more infatuated with this man and I am at the point of feeling selfish when I think of the things I want and the double life he is living. I am left forgotten on holidays, vacations, and things that he must be present for. The other day I got totally hurt when I found out that he has been telling people at work he hasnt talked to me in over a year. We talk everyday. We share everything with each other from fantasies to life goals. My question is, is it wrong to want more from him? Im starting to feel like a playtoy.