The Premier League title rivals' first meeting since City crushed United 6-1 at Old Trafford in October lived up to expectations with City fighting back after going 3-0 down, but ultimately failing to hold on to the trophy they won last May.
In contrast to United's nail-biter, Chelsea, who won the cup three times in the four seasons before City's triumph last May, beat Portsmouth 4-0 at Stamford Bridge in a replay of the 2010 final which they won 1-0.
Juan Mata put Chelsea ahead just after halftime and three of Chelsea's goals came in the dying minutes. Ramires scored in the 85th and 87th minutes before Frank Lampard added a fourth in the fourth minute of stoppage time against the Championship (second division side).
Sunderland, the third Premier League team in action, also made the last 32 with a relatively easy 2-0 win at Championship side Peterborough United with Seb Larsson opening the scoring direct from a free-kick and James McClean adding the other.
There was one shock with sleeping giant Sheffield Wednesday, three times FA Cup winners in the past but now playing in League One (third tier), beating Championship promotion hopefuls West ham United 1-0 at Hillsborough with Chris O'Grady scoring the late winner. Sam Baldock missed a penalty for West Ham early in the second half.
The third round will be completed when Arsenal face Leeds United at the Emirates Stadium on Monday.
UNITED COASTING
All eyes were focused on the Etihad Stadium where City had been unbeaten since losing to Everton in December 2010.
By halftime that record was in jeopardy with United, the 11-times record FA Cup winners, 3-0 ahead and seemingly coasting into the fourth round.
Rooney headed them in front after 10 minutes, two minutes before City captain Kompany was controversially sent off for a challenge on Nani when he appeared to make fair contact with the ball, although referee Chris Foy ruled he had slid into the tackle with his feet off the ground.
Danny Welbeck scored with an acrobatic volley on the turn after 30 minutes to put United 2-0 ahead before Rooney made it 3-0 with another header, scoring on the rebound when Costel Pantilimon initially saved his penalty.
But City, victors over United in last season's semi-final, were in no mood to surrender meekly their grip on the trophy, and stormed back after the break with Aleksandar Kolarov scoring from a freekick after 48 minutes.
Midfielder Paul Scholes, who came out of retirement on Sunday to rejoin United until the end of the season, featured as a 59th minute substitute but his loose pass enabled City to break for Sergio Aguero to make it 3-2 after 65 minutes and set up a dramatic finale.
Rooney told ITV: "We knew it was going to be a difficult game today. Obviously going 3-0 up, we maybe took our foot off the gas a little bit, City got two goals and we had to make sure we saw the game out which we managed to do so we are delighted."
Rooney dismissed any notion that he would be leaving the club following newspaper reports on Saturday morning which implied he had had a major falling out with manager Alex Ferguson.
NOTHING TO PROVE
Asked if he had a point to prove, Rooney replied: "Not at all: what's been in the press of late has been a load of nonsense, because it's me, because it's Manchester United, it's blown up out of context. There are no problems with me at this club, I want to be at this club for a long time."
Rooney said the team were surprised when they realised Scholes was in the squad.
"It was a bit of a shock to be honest, we didn't know until we were in the dressing room. It's great. It gave everyone a lift he's a fantastic player and he showed his quality when he came on."
Ferguson was not overly pleased with his team's performance.
"I think we made them look better than they were," he said, "We were very careless in the second half, but the problem was, being 3-0 up at halftime, the players thought they were through and took their foot off the pedal."
Beaten manager Roberto Mancini seemed the happier of the two saying he was delighted with the character his side showed in their second half fightback.
"I am disappointed with the result but I am very happy with my players for their performance," Mancini told reporters.
"We defended very well in the second half, they did not have a chance and we scored two goals. It is not easy to play with 10 players against United."
