Visual scene understanding is a fundamental building block for autonomous agents operating in dynamic environments such as transportation systems. This thesis explores the progression of visual perception from object detection to scene-level relationship understanding, with applications in transportation. We first address the challenge of re-identifying agents across non-overlapping camera views, proposing a confidence-based approach for visual re-identification. We then extend perception to aerial imagery for traffic monitoring, developing methods that adapt to the wide range of object scales in drone-captured images. Finally, we move beyond individual object detection to relationship modeling between entities in a scene, proposing Composite Relationship Fields (CoRF) for scene graph generation. The thesis presents a cohesive vision for how autonomous systems can understand visual scenes at multiple levels of abstraction, from detecting individual objects to reasoning about their relationships.