Washington branded the beheading of a US journalist a "terrorist attack", upping the stakes in its confrontation with jihadists, as Shiite militiamen killed 70 people at a Sunni mosque in Iraq.

The apparent revenge attack at the mosque in Diyala province Friday will increase already significant anger among Iraq's Sunni Arab minority with the Shiite-led government, undermining an anti-militant drive that requires Sunni cooperation.

It came as the US, which is carrying out air strikes in Iraq against Islamic State (IS) jihadists, ramped up its rhetoric over the grisly killing of journalist James Foley, carried out by the group and shown in a video posted online.

In Washington, Deputy National Security Advisor Ben Rhodes said the beheading of Foley "represents a terrorist attack against our country".

Rhodes also said that paying ransoms to free hostages is "not the right policy", confirming Washington's long-standing position amid claims from IS that other countries had paid to have their nationals freed.

In an unanimous statement Friday, the UN Security Council condemned Foley's murder as "heinous and cowardly".

- 'We found a massacre' -

In Iraq, army and police officers said the attack on the Musab bin Omair Mosque in Diyala came after Shiite militiamen were killed in clashes, while other sources said it followed a roadside bomb near one of their patrols.

Doctors and the officers put the toll from the attack, in which worshippers were sprayed with machine-gun fire, at 70 dead and 20 wounded.

Two officers had earlier blamed IS for the attack, saying it had included a suicide bombing, a hallmark of the group, but most accounts pointed to Shiite militiamen.

The government turned to militiamen to bolster its flagging forces during the IS offensive, sparking a resurgence of groups involved in brutal sectarian killings in past years.

Ibrahim Aziz Ali, whose 25-year-old nephew was among those killed, told AFP he and other residents heard gunfire and rushed to the mosque, where they were fired on by snipers.

Five vehicles with images of Imam Hussein, one of the most revered figures in Shiite Islam, were parked at the mosque, Ali said.

When they could finally enter, "we found a massacre," he said.

Officers said that angry residents exchanged fire with security forces and militiamen in the area on Friday, but reported no casualties.

Iraqi premier-designate Haidar al-Abadi issued a statement calling for unity and condemning the killings, which may complicate the already-contentious process of forming a new government.

US Vice President Joe Biden, writing in The Washington Post, said the US would back a system of "functioning federalism" in Iraq as a means to breach the divisions in the country.

Biden, a longtime advocate of a plan under which Iraq would be divided into three autonomous regions for Shiites, Sunnis and Kurds, also said the US was prepared to "further enhance" its support for Iraq's fight against IS.

The United States began an air campaign against IS in Iraq on April 8, and has since carried out more than 90 strikes, including three against militants in the area of the Mosul dam, the country's largest, on Friday.

- 'Beyond anything we have seen'

Pentagon chiefs warned of the dangers of IS and said operations against it in Syria may also be needed.

"They marry ideology and a sophistication of strategic and tactical military prowess," US Defence Secretary Chuck Hagel. "This is beyond anything we have seen."

General Martin Dempsey, chairman of the Joint Chiefs of Staff, spoke of a "very long contest" that could be won only with regional support and that of "the 20 million disenfranchised Sunnis that happen to reside between Damascus and Baghdad".

He was referring to the alienation of many Sunni Arab from Iraq's government and the Alawite-dominated regime in Syria.

Foley's killing has stoked Western fears that territory seized by the militants in Syria and Iraq could become a launchpad for a new round of global terror attacks.

Foley, a 40-year-old freelance journalist, was kidnapped in northern Syria in November 2012. His employer GlobalPost said his captors had demanded a 100-million-euro ($132-million) ransom.

GlobalPost CEO Philip Balboni said his team had never taken the demand seriously.

In the execution video, released online, a black-clad militant said Foley was killed to avenge US air strikes against IS.

The man, speaking with a clear south London accent, paraded a second US reporter, Steven Sotloff, in front of the camera and said he too would die if Washington kept up its attacks.

burs-wd/hc