Rumsfeld appeared before the committee after U.S. media last week released photographs showing U.S. personnel abusing Iraqi prisoners, causing outrage.
"I feel terrible about what happened to these Iraqi detainees," Rumsfeld said. "They are human beings, they were in U.S. custody, our country had an obligation to treat them right, we didn't, and that was wrong. So, to those Iraqis who were mistreated by members of the U.S. armed forces, I offer my deepest apology."
Rumsfeld said he had "failed to recognize" the need to bring the abuse of Iraqi prisoners to those "at the highest levels," including President George W. Bush and Congress.
He said more offending photographs are likely to become public.
Rumsfeld told the Senate committee that he is establishing a special panel to look into the abuses. He said he is seeking a way to financially compensate those who were abused.
Several lawmakers have called for Rumsfeld's resignation. The defense secretary said he would resign "in a minute" if he thought he could not be effective, but not if the Iraq prison scandal is turned into a "political issue."
As Rumsfeld spoke, a small group of protesters interrupted him briefly by shouting "Fire Rumsfeld" before they were led away.