Dean Henderson was the hero for Forest after having saved two penalties, leaving them with a 4-3 win after the match had ended 1-1 in normal time.
The Reds opened the scorecards with Willy Boly scoring his first goal for Forest 19 minutes in - and it comes against his old club.
This was equalised when Raúl Jiménez scored for Wolves in the 64th minute.
It means Forest are now in the last four of a competition that has special memories, having won it four times.
A return to Wembley looks a real possibility as they will play one of Manchester United, Newcastle or Southampton in the two-legged last-four tie.