I think the key things were:

Amy still being the girl who waits. The doctor knows that he is inevitably going to outlive her and that the most likely thing to end her life is being near him, but he can't stay away. He wants her to have a life but his always being out there somewhere prevents that.

The doctor tries to mix it up - I have a gang now. Maybe Queen Neffy insisted on comming and so the Doctor recruited Riddle to divert her interest in him, but it seems like a deliberate attemp to not get close to people.

Rory's Dad - being out here changes people - rory has always had issues with the dangerous side of being with the doctor but most of all with the effect it has on people's mindset. His dad is effected greatly by these events and it seems Rory is ok with it because it seems to have made them closer. In the end as Bryan sits on the Tardis threshold it's Amy who gives a look of concern. Is she starting to see what Rory does?