When the reading flow allows for multiple visual paths, it becomes difficult to follow the action. As a result, the reader gets tired and has a tough time immersing themself in the story. Plan the reading flow of every page carefully and make it as clear and linear as possible.
Communicating your story clearly to the reader is one of the basics of a professional manga. It's one of those things that you don't notice when it's done right, but becomes really annoying when done wrong.
PRO TIP 2: Plan your reading flow from the start and don't experiment with it.