Everyone know the medrash quoted by Rashi that she was 3 years old. However, the Gemara in יבמות and the Sifri in וזאת הברכה disagree.
The Gemara in יבמות ס"א discusses who a Kohen Gadol can marry. The Torah says בתולה, the gemara says that a בתולה means a נערה, a girl who is between 12 and and 12.5. The Gemara says that we know this from the pasuk by Rivka which calls her a בתולה. We see clearly that the Gemara assumes that she was older then 3 (at least 12).
In fact, all of the Rishonim point this out (that the Gemara argues on the medrash that Rashi quotes) and say that it is a machlokes hamidrashim. The Rishonim also point out the gemara later on ס"ד. The Mishna has the din that if a couple is married for 10 years with no children they should divorce. The Gemara asks why isn't it 20 years as Yitzchak married at the age of 40 and didn't have kids until 60. According to Rashi that Rivka was 3 there is no question, for the first 10 years she couldn't have kids. You see clearly from the Gemara that the assumption is that Rivka was old enough to have children at the time of the marraige.
The Sifri in וזאת הברכה says that Rivka lived 133 years. If you work backwards from her death she got married at the age of 14 (see Tosafos in יבמות ס"א for the exact calculation).
Why did Rashi pick the medrash that she was 3 against the gemara? It would seem that Rashi thought that was the simpler pshat in the pesukim. It all seems to hinge on when the akeida was. The following facts are mentioned in the Chumash itself.
1. Yitzchak was 37 when Sara died (she was 90 when he was born and died at 127).
2. Yitzchak was 40 when he married Rivka
3. Rivka was born around the time of the Akeida (see the end of וירא).
If the Akeida was when Yitzchak was 37 then Rivka was only 3 when she married. On the other hand if the akeida was 10 years earlier then she was around 14.