The Argo star plays the lead role in drama To The Wonder, the new project from the acclaimed director known for his unusual approach to filmmaking.
"The experience of it seemed half-crazy in that we didn't really have dialogue, so I didn't really know what was happening," Ben told GQ.
"Then I realised that he was accumulating colours that he would use to paint with later in the editing room. My character doesn't really do that much."
Ben added: "Terry uses actors in a different way - he'll [have the camera] on you and then tilt up and go up to a tree, so you think, 'Who's more important in this - me or the tree?'
"But you don't ask him, because you don't want to know the answer."
The film received a mixed response and even some boos when it premiered at the Venice Film Festival, but Ben defended its style.
"Honestly, you have to want to see that movie - there's hardly any talking. It's a tone poem. If you don't want to see that, you should not go."