The no-confidence vote was proposed by the opposition Communist Party amid allegations of corruption in the government.
Thirty-three Communist deputies, 15 from the Democratic Party, four from the Socialist Party, and two independent deputies voted in favor of the motion. In all, 54 lawmakers voted in favor, three more than the 51 needed to pass the measure.
Communist Party leader and former Moldovan President Vladimir Voronin told deputies before the vote that the ruling Alliance for European Integration (AIE) is "responsible for impoverishing the people."
"Today, it’s impossible to imagine even one logical argument in favor of keeping the Alliance for European Integration or the government. Such arguments simply do not exist," Voronin said. "The alliance, through its government, is not able to demonstrate to the people one single argument in favor of keeping it."
Outside the parliament, hundreds of locals gathered in support of the government, booing opposition lawmakers. Many held posters saying, "We Want European Integration."
Filat, speaking to journalists in the capital, Chisinau, called the vote irresponsible.
"I do not want to judge [this vote]. The people will do it," he said. "But I do believe that above all it is an act of maximum irresponsibility."
President Nicolae Timofti now has to propose a new prime minister, who will attempt to form a government.
If this process fails to produce a cabinet with majority support in parliament three times within 45 days, Timofti will dismiss parliament and call early legislative elections.
Filat's cabinet will continue to work in an acting capacity.
In February, the AIE collapsed after Filat announced his Liberal-Democratic Party's withdrawal from the coalition.
The Democratic Party, which is part of the AIE, supported the no-confidence resolution, saying that the Liberal-Democrats had nullified the coalition agreement.
Last week, Timofti urged Filat's party to reconsider its withdrawal from the governing coalition. Filat has said he thinks a new alliance agreement can be reached and a new AIE cabinet installed.
Timofti said early legislative elections could affect Moldova's integration with the European Union.
Maja Kocijancic, a spokeswoman for EU foreign policy chief Catherine Ashton, said all political forces must cooperate to ensure stability in Moldova.
"[Ashton] and [EU Enlargement Commissioner Stefan Fuele] are calling on all the political groups and also nonaffiliated members of the Moldovan parliament to engage in a genuine political dialogue, without delay, with a view of having a strong and stable majority in parliament that will be committed to promoting democratic values, the rule of law, but also that will pursue the path toward political association and economic integration with the European Union," Kocijancic said.
Several ministers in the outgoing government are the subject of corruption probes.
Finance Minister Veaceslav Negruta, Culture Minister Boris Focsa, and Health Minister Andrei Usatii are all currently under investigation and could face criminal charges.