You would be surprised at his attention span at that age,I took my step son when he was 4,and he was absolutely engrossed in the game,loved looking around to see what was happening around him and as soon as we left,kept asking when we were going back again.
Is there a development league game or something coming up,or even a junior game/other local team that you could take him to and test the water,see how he got in there,least it would be something you wouldn't mind missing if he wanted to leave.