Republican donor Meg Whitman has endorsed his rival Hillary Clinton, saying Mr Trump's "demagoguery" had undermined the national fabric.
Senior party activist Jan Halper-Hayes told the BBC she thought Mr Trump was "psychologically unbalanced".
In the latest controversy, Mr Trump has refused to support two senior figures in his own party.
In an interview for the Washington Post, he said he was "just not quite there yet" when asked if he would endorse House Speaker Paul Ryan and Senator John McCain, who are up for re-election in November. Both men have publicly criticised him.
Mr Trump's campaign has been marked by a series of controversial statements.
Ms Whitman, writing on Facebook, said that to vote Republican out of party loyalty alone "would be to endorse a candidacy that I believe has exploited anger, grievance, xenophobia and racial division".
"Trump's unsteady hand would endanger our prosperity and national security. His authoritarian character could threaten much more," the Hewlett-Packard executive said.
Dr Halper-Hayes, vice-president of Republicans Overseas Worldwide, told the BBC's Today programme that she was "very concerned" about Mr Trump's behaviour, although she would not go so far as to endorse Mrs Clinton.
"I think there is an element of him that truly is psychologically unbalanced, and I feel very guilty for saying this because I'm a Republican and I want the Republican ticket to win," she said.
"But Donald is out of control right now and he's not listening to anyone."
Dozens of senior Republican Party figures have already said they would not vote for Mr Trump, including the party's 2012 nominee Mitt Romney and former Florida Governor Jeb Bush.
Muslim soldier row
New York Representative Richard Hanna became the first Republican member of Congress to publicly say he would vote for Mrs Clinton. He described Mr Trump as "deeply flawed in endless ways".
On Monday, Sally Bradshaw, a top adviser to Jeb Bush, said that Mr Trump's candidacy had convinced her to leave the Republican Party.
Many Republicans opposed to Mr Trump have stopped short of supporting Mrs Clinton, saying they would vote for a third candidate instead.
However, former Republican congressman John LeBoutillier told the BBC's Newsday programme he believed many Republican politicians would back Mrs Clinton.
"Now I think in private a lot of Republican congressmen are going to vote for Hillary, they can't stand Trump," he said.
Despite the criticism, Mr Trump tweeted on Wednesday that there was "great unity in my campaign, perhaps greater than ever before".
The property billionaire has been heavily under fire over a war of words he has been waging with the bereaved parents of a US Muslim soldier killed in Iraq.
Democratic and Republican leaders as well as veterans' groups quickly rounded on him.
Mr Trump's son Eric defended his father's comments about the Khans, telling CBS News that they had been been "blown hugely out of proportion".
The row led US President Barack Obama to make his strongest comments yet on Mr Trump, saying he was "unfit" to serve as president.
In response, Mr Trump dismissed President Obama's time in the White House as a "disaster".
"He's been weak, he's been ineffective… the worst president, maybe, in the history of our country", he said in a Fox News interview.
Mr Trump's spokeswoman Katrina Pierson later caused a social media storm by blaming both President Obama and Mrs Clinton for the death of Capt Humayun Khan, citing their policies on Iraq. Capt Khan died in 2004 when Mrs Clinton was in the US Senate and five years before Mr Obama was elected to the White House.
Mrs Clinton has been actively courting moderate Republicans. A Reuters/Ipsos opinion poll suggested she had extended her lead over Mr Trump to eight percentage points, from six points on Friday.