MIAMI >> President Obama has edged closer to closing the Guantanamo Bay prison by swapping five Taliban members for a captured American soldier, but the criticism he faced Sunday for the deal underscores the challenges he faces in emptying out the cells at the U.S. base in Cuba.

Several members of Congress condemned the exchange, saying among other things that it could put troops in danger if the freed Taliban resume hostilities against the United States.

That fear prompted Congress to impose restrictions on the transfer of prisoners out of Guantanamo that have thwarted Obama’s pledge to close the detention center within the first year of taking office.

Congress eased the restrictions on transfers somewhat last year and a Senate committee has approved lifting a ban on sending prisoners to the U.S. But significant challenges remain, including any blowback over the exchange for Sgt. Bowe Bergdahl, and time is running short.

"It’s possible that President Obama could make a big move in his last two years, presumably after the midterm elections," said John Bellinger, a former legal adviser to the State Department and National Security Council. "But it’s equally possible he will decide he can’t close it or Congress will make it impossible to close and it will get kicked down the road to his successor."

The remaining 149 prisoners fall into several, somewhat fluid categories. Nearly 80 of them have been approved for transfer to their homelands or a third country. Those moves have gradually resumed after coming to a halt due to security requirements imposed by Congress that were altered last year.

Some who have criticized the government for moving too slowly on transfers say the exchange for Bergdahl, which took place without a required 30-day notice to Congress, shows the Obama administration can act more forcefully to close Guantanamo.

"This illustrates that the U.S. government can transfer prisoners when it is motivated to do so," said Wells Dixon, an attorney with the New York-based Center for Constitutional Rights who has represented prisoners at Guantanamo. "There are four more Afghans cleared for transfer and they need to be transferred. He can start with them."

Many legal experts have argued that the U.S. will lose its authority under international law to hold members of the Taliban without charge when combat comes to an end in Afghanistan. That thinking, which wouldn’t necessarily apply to Guantanamo prisoners who are affiliated with al-Qaida, may have been a factor in the exchange for Bergdahl, said Bellinger, who served during the administration of President George W. Bush.

Their release, he said, could be seen as a "reasonable compromise" to a difficult dilemma. "Frankly, they are only being released seven months earlier than they might otherwise have to have been released," Bellinger said.

The U.S. has said that nearly 40 of the prisoners at Guantanamo are too dangerous to release, but can’t be charged for a number of reasons, often because there isn’t sufficient evidence against them. Officials have been slowly trying to whittle this number down in recent months with a Periodic Review Board that has been taking a new look at their cases, and several have now been moved to the transfer category.

The rest are prisoners who could be prosecuted if they could be moved to civilian court in the United States or face trial by military commission at Guantanamo. A Senate committee has approved legislation that would allow transfer of prisoners to the U.S. for detention or medical treatment but the measure faces uncertain prospects in the House of Representatives.