Mr Tillerson's spokesman said he only learned he was out of a job when he saw the president's tweet thanking him for his service as top US diplomat.
The former ExxonMobil chief had a series of public rifts with the White House since he was appointed last year.
Mr Trump also named Gina Haspel to become the CIA's first female director.
What was Trump's reason?
Speaking to reporters outside the White House on Tuesday, Mr Trump said his differences with Mr Tillerson came down to personal "chemistry".
"We got along actually quite well, but we disagreed on things," said the president.
"When you look at the Iran deal, I think it's terrible. I guess he thought it was OK.
"I wanted to either break it or do something and he felt a little bit differently, so we were not really thinking the same.
Tillerson – I will now return to private life
"With Mike, Mike Pompeo, we have a very similar thought process. I think it's going to go very well.
"Rex is a very good man, I like Rex a lot."
Mr Tillerson is the latest in a long list of senior officials who have either resigned or been fired since Mr Trump took office.
Mike Pompeo, Director of the CIA, will become our new Secretary of State. He will do a fantastic job! Thank you to Rex Tillerson for his service! Gina Haspel will become the new Director of the CIA, and the first woman so chosen. Congratulations to all!
How was Tillerson fired?
The Department of State said Mr Tillerson had not spoken to the president and was "unaware of the reason" for his firing.
Under Secretary Steve Goldstein said: "The Secretary had every intention of staying because of the critical progress made in national security."
Mr Goldstein himself was fired later on Tuesday by the White House.
Mr Tillerson said later in a news conference that he had received a call from the president on Air Force One after midday, about three hours after Mr Trump fired him via tweet.
He added he would finish his term on 31 March and would "return to private life as a private citizen".
According to the Associated Press, White House chief of staff John Kelly called Mr Tillerson on Friday and advised him to watch out for a presidential tweet about him.
Mr Kelly did not tell Mr Tillerson when the tweet would be posted or what it might say, reports the news agency.
The secretary of state was on an official tour of Africa last week when he was caught unawares by Mr Trump's agreement to meet North Korean leader Kim Jong-un.
The top US diplomat's team said he was feeling unwell on Saturday and later in the weekend the state department said he would cut short his tour by a day.
"I felt like, look, I just need to get back," Mr Tillerson told reporters on his plane home, reports the Washington Post.
The 65-year-old arrived back in Washington before dawn on Tuesday to learn he was out of a job a few hours later.
The odd couple of politics no more
Analysis by Jon Sopel, BBC North America editor
The end was a perfect metaphor for the relationship. The odd couple of politics had been yoked together for too long.
Men of different temperaments, demeanour and style had reached a parting of the ways.
The secretary of state landed back at Joint Base Andrews to have a member of staff inform him that the president had tweeted.
Because Mr Tillerson is not on Twitter, the tweet had to be printed out. Fancy being the one tasked with handing that over to the boss. Fired by a tweet.
The career of a one-time giant of corporate America had come to an ignominious end.
Mike Pompeo: Meet America's new top diplomat
Who are the nominees?
A US Senate report found that Ms Haspel ran a notorious CIA prison in Thailand, where prisoners were waterboarded in 2002.
The CIA deputy director was also accused of destroying dozens of videotapes of interrogations at the camp.
Mr Pompeo is a former hardline conservative Republican lawmaker from Kansas and a Trump loyalist.
In 2014, he defended the CIA officers who waterboarded detainees as "patriots".
Both the new secretary of state and CIA director will have to be confirmed by the Senate.
Why was Tillerson fired?
Reports had swirled since last year of a schism in the Trump administration between the commander-in-chief and his top diplomat.
Last October Mr Tillerson was forced to convene a news conference to deny claims he planned to quit, though he did not address a report that he had referred to his boss as a moron after a Pentagon meeting.
In October, Rex Tillerson responded to a report he called Mr Trump 'a moron'
Last autumn, Mr Trump publicly undercut the former Texas oilman by tweeting that he was "wasting his time" trying to negotiate with North Korea.
In December, the secretary of state departed from administration talking points when he offered to begin direct talks with Pyongyang without preconditions.
The White House – which insisted North Korea must first accept any negotiations would be about giving up its nuclear arsenal – distanced itself from his remarks.
Mr Tillerson was reported to be astonished at how little Mr Trump grasped the basics of foreign policy.
The Republican president, meanwhile, was irritated by Mr Tillerson's body language during meetings, the New York Times reported.
The secretary of state was said to roll his eyes or slouch when he disagreed with Mr Trump's decisions.