The meetings on 14 November were held even though Chalabi remains under investigation by the U.S. FBI, the Federal Bureau of Investigation, for allegedly passing U.S. intelligence information to Iran.
U.S. forces raided Chalabi's offices in Iraq last year in connection with the allegations, which Chalabi has denied.
U.S. defense officials did not allow cameras to film Chalabi's arrival or departure from the Pentagon headquarters.
A U.S. defense official said Chalabi and Rumsfeld discussed the need to improve intelligence-gathering capabilities in Iraq to enhance the ability of Iraqi security forces to defeat insurgents.
Chalabi, who has also been accused by critics of passing on inaccurate claims about Saddam Hussein's alleged weapons of mass destruction programs, met last week with Secretary of State Condoleezza Rice.