Annan earlier said the strike in South Lebanon on July 25 of a UN observer post was "apparently deliberate" and called on Israel to investigate the incident.
Olmert said it was an accident and expressed dismay over Annan's characterization of the incident. Israel earlier said it "regrets" the deaths.
Israeli Ambassador to the UN Dan Gillerman said he was "shocked" by Annan's accusations. But speaking to CNN on July 25, Gillerman said Israel has launched an inquiry.
"Israel is investigating this," Gillerman said. "We're investigating this very, very thoroughly. We don't unfortunately at this very moment know what happened, but we're very sorry, as I said, about this tragic incident and we will investigate it thoroughly, fully, swiftly, and will share the findings of this investigation with the United Nations."
Among the four dead was a Chinese national, Du Zhaoyu. The names or the nationalities of the other observers are not known.
(compiled from agency reports)