Do I have to tap on the directional arrows on the Captions screen to go to the next Caption?

No. You can also swipe to the left or right on the caption itself.