The Federation Council voted by 140 votes to zero to accept his request after the issue was unexpectedly put on the agenda of the day.
Ustinov had held the post since 2000, after being nominated by President Vladimir Putin. However, his departure follows criticism from Putin that Russia is failing to combat corruption.
Ustinov led the legal case against the oil giant Yukos and its former owner, Mikhail Khodorkovsky -- critic of Vladimir Putin and once the richest man in Russia.
Inevitably, though, speculation in Moscow is rife as to why Ustinov stepped down. Rumors had long been circulating that his days were numbered but Mironov maintained that there is no subtext -- Ustinov had simply had enough.
"The president called me to his office yesterday and handed me a note on the dismissal of Vladimir Vasilyevich Ustinov. From our conversation I understood that it was a technical decision, if one could say so. There is no politics here," Federation Council Chairman Sergei Mironov said.
But in Moscow today not many seem to be buying that. The politologists and other Kremlin watchers are linking his departure to President Putin's state-of-the-nation address last month in which he singled out the failure of the law enforcement agencies to root out corruption.
Putin criticized the work of law enforcement agencies during his state-of-the-nation address in May
"The situation here regarding the oversight of the law enforcement agencies and with crime, for which the prosecutor-general is responsible, leaves much to be desired. And under Ustinov things have most certainly not changed for the better," said Vladimir Pribylovsky, the director of the Moscow-based Panorama think tank.
But while Pribylovsky concedes that Ustinov's failings in the struggle against crime and corruption may have been a factor in his apparent fall from grace, he believes the real reason lies in the endless struggle for influence within the Kremlin.
"It would appear the balance of power within the coalition of groups around Putin has changed -- the so-called Putin team. Something must have tipped the balance and the president finally decided to tell Ustinov he had to go. There are a lot of people after that job," Pribylovsky said.
In Ustinov's absence, the position of prosecutor-general will be filled temporarily by his deputy, Yury Biryukov, but the battle for the succession is already under way.
Top contenders for one of the most powerful jobs in Russia are Dmitry Kozak, the president's special envoy to the Southern Federal District, and Aleksandr Konovalov, the president's special envoy to the Volga region.