The former Work and Pensions minister recited the words to Lose Yourself live on ITV's Good Morning Britain, leaving presenter Piers Morgan in stitches.
He first did his impression of the rap star, who was the best-selling artist of the 2000s in the US, during the break.
But after some gentle encouragement, the Tory politician repeated his impression on air.
Morgan jokingly described it as "one of the most iconic cultural moments I have probably ever experienced on this sofa".
He said: "Iain Duncan Smith perhaps the last person you would expect to know this, was talking about a message for Diane Abbott and began reciting Eminem lyrics"
Duncan Smith replied: "I only said that, because you were talking about Eminem earlier on, there are Lose Yourself lyrics for Diane Abbott."
He then rapped: "He opens his mouth, but the words won't come out.
"He's choking, how, everybody's joking now. The clock's run out"
It came after Diane Abbott was accused of failing to keep abreast of key figures when she underestimated Labour's losses by more than half.
The rap received a mixed response from viewers with some finding the outburst "cringe" and but others finding it amusing.