Beijing's top foreign policy official on Wednesday slapped down Vietnamese claims to disputed waters, in talks aimed at pulling relations back from their lowest point in decades.

State Councillor Yang Jiechi met Vietnam's Foreign Minister and Prime Minister Nguyen Tan Dung in Hanoi for the first high-level talks between the neighbours since early May, when vessels from both sides collided near a Chinese oil rig anchored in contested seas.

The incident prompted deadly anti-China riots in Vietnam and an evacuation of nationals by Beijing.

But the talks ended in deadlock, with Yang telling Vietnam that it had to "stop its disturbances against China's operations, stop hyping up the relevant issue", and deal with the fallout from riots targeting foreign businesses.

During the talks Yang "stressed that the Xisha (Paracel) Islands are China's inherent territory" and said the current difficulties in the relationship were due to Vietnamese "illegal disturbances", according to Chinese Foreign Ministry spokeswoman Hua Chunying.

Vietnam claims the Paracel Islands, known as Hoang Sa in Vietnamese, which China seized from then-South Vietnam in 1974 in a period of turmoil shortly before the end of the Vietnam war.

As Yang flew out of Hanoi Wednesday evening, Vietnam issued a statement reaffirming its claims to the Paracels, calling the Chinese rig illegal and demanding it be removed.

Prime Minister Nguyen Tan Dung said China's actions "violated Vietnam's sovereignty, caused discontent among the population... and damaged bilateral ties," according to the statement released by the Ministry of Foreign Affairs.

He called for a peaceful resolution to the standoff and said Vietnam would defend its rights in accordance with international law.

The two sides have spent the last month trading accusations in the increasingly heated territorial dispute, with each side claiming the other has engaged in aggressive behaviour against its ships, including by ramming them.

The dispute has brought relations to their lowest point since a border war in 1979.

Yang was previously China's foreign minister. But he moved up to the State Council, the country's cabinet, last year, making him more powerful than the current foreign minister.

- 'China is not sincere' -

Vietnam's communist leaders have struggled to balance strong domestic opposition to China's unilateral moves in the South China Sea with their traditionally friendly ties with a fellow communist country.

Nguyen Trong Vinh, a former Vietnamese ambassador to Beijing said he was surprised by Yang's visit as previous Vietnamese attempts to initiate high-level bilateral dialogue had been rebuffed.

Even so, "China is not sincere in wanting to solve the maritime dispute", he said, adding Beijing likely just wanted to do some damage repair to its international reputation.

"They want to reduce international pressure and stop the bad press (but) they will not stop their policy of encroachment in the East Sea," he said, referring to the South China Sea.

Anti-Chinese riots sparked by Beijing's dispatch of the deep-sea rig claimed three Chinese lives in Vietnam last month, according to Hanoi. Beijing says four Chinese citizens died.

Hanoi has since moved to muzzle public protest as it seeks to reassure nervous foreign investors that the country is a safe place to do business.

The United States called the deployment of the rig in contested waters "provocative" and Dung, Vietnam's prime minister, has announced Hanoi is considering legal action against China.

Hanoi-based diplomats say privately it is unlikely Vietnam would follow through on this threat as any legal action could lead to economic retaliation by Beijing, which would hit Vietnam hard.

China is Vietnam's largest trading partner and the standoff over the rig has already affected some local companies, which rely on cheap Chinese imports, forcing them to diversify their supply chains.

"China has the upper hand and Councillor Yang will certainly hint at what might happen economically if bilateral relations are not patched up," said Vietnam expert Carl Thayer.

But the broader issue at stake is "China's unanswered challenge to the Obama Administration", which has repeatedly called on Beijing to end incursions into disputed waters, he added.

The Philippines has recently protested what China describes as reclamations in reefs within its exclusive economic zone, including in one it fears Beijing intends to convert into an airstrip.