Because you just don't do it in baseball.
You lose the entire team as a manager. You become a clown.
If a player starts a game and prepares to start the game mentally, you don't yank him because the other team now has a lefty starter.
The fourth inning is too early to make substitutions like that.
It's like Kotsay and the team would say, "WTF, what did I do??"