Typically physicists refer to the "arrow if time" in reference to the fact that we only observe time to flow in one direction (forward). It never goes in the other direction (backward), as far as we can tell. So time has an arrow, so to speak.

RJ had a background in physics, so I believe he was aware of what "the arrow of time" generally refers to. As such, I've never found a very satisfying explanation as to what he would mean by "thought is the arrow of time".