"I could just leave you like this," Jim said, not intending to do that, "and see how you react."

Well, he did actually pull his hand away but that was only because she was being so slow with taking off his shirt and he was just going to do it for her. Not immediately, just to make her think he was actually following through.