It certainly sounds infuriating and annoying, and I can sympathise because I got a lot of the same during my schooldays. But ultimately I think standing your ground and finding some way of genuinely not letting it bother you is exactly what you will need to do. Because if you keep avoiding him and he works out why he will smell blood, press home his advantage and it will just get worse.
It sounds like he's trying to provoke you into kicking something off yourself, so he can claim he didn't start it. Don't give him what he wants. But consider this: if he's too scare to physically have a go again, but still can't bring himself to just leave it alone and so has to resort to such pathetic provocation he can't have much in the way of self-esteem, and as such is worthy of your pity. Not sympathy, just pity.
You might also be able to derive strength from the fact that he so desperately wants to annoy you, and this is the only way he feels he can still get at you. Basically, he feels powerless.