I would refute that, especially if you are talking motorcycle racing. In auto, the driver must have stamina to keep up his mental focus for the entire 3ish hour race, and must also endure intense heat.
In motorcycle racing, the rider must slide back and forth and side to side, to set up for the corners and such. This takes quite a lot of physical strength. And, if you read 'sliding side to side, etc' as a pretty easy feat, I can assure you, it is not. On a standard days ride, going through the twisties, my shoulders, arms, and back will feel the action from the muscles. That doesn't even take into consideration the physical aspects of MX racing, or any off road motorcycle racing. In that regard, I'd say it is more 'physically intensive' than baseball is.
So, I guess it all depends on what you think of 'sports'. I know that I have seen motorcycle racers win, switch brands, and win again. In that aspect, I'd say that it is the rider that makes the difference, no matter what he is controlling.