At first glance the answer is of course, I damaged you and therefore I have to pay. However, R' Sternbuch and others point out that it is not so simple.
The Gemara in Bava Kama (48b) learns out from a pasuk that if my animal eats fruits that are growing in your field I don't pay for the actual fruits that were eaten, rather I pay for the devaluation of the field. Lets take a simple example. You have a field that is worth $1000 which has orange trees. My animal comes and eats 50 oranges and each orange is worth $1. The damage I caused you is $50, however, I don't pay you $50. Instead, I pay how much the field is devalued by the fact that my animal ate 50 oranges. Lets say that the field was originally worth $1000 including all of the oranges and now that my animal ate 50, the field is only worth $990. Since the value of the field has only gone down $10 all Ii have to pay is $10 (not $50). The Gemara says that the same din applies if I do the damage myself (I eat the oranges or I uproot the tree).
Based on this it is quote din the name of R' Chaim, the Chafetz Chaim and others, that the same din should apply if I break the window of your house. I don't pay for the damage I caused, rather, I need to pay the amount that your house has been devalued. The fact is that price for a house that has 1 broken window is the same as the price of the house with no broken windows. If I am asking for $300,000 for my house, the extra $50-100 for a broken window is simply not taken into account. The buyer is not going to change his offer to $299,900 because there is now a broken window. Therefore, the person who broke the window doesn't need to pay me anything because there is no damage the way the Gemara says to figure it, the value of the house is still $300,000.
The Chazon Ish (Bava Kama siman 6 sif 3) argues and says that you have to pay the replacement value. The Chazon Ish says that a house is more similar to מטלטלים. Everyone agrees that if instead of breaking the window of my house you broke my car window, you would have to pay for the damage. Damage to non-property is figured based on the actual damage and therefore if it costs $50 to replace the window that is what you owe me. The Chazon Ish thinks that the same din would apply to a house window and you would need to pay for the actual damage.
הלכה למעשה you have a machlokes haposkim. R' Sternbuch paskens like R' Chaim etc. that the damager doesn't have to pay and others pasken like the Chazon ish, it depends on the Beis Din.
I have to say this machlokes really surprised me in a number of ways. First of all, none of the Rishonim or early Acharonim discuss this question. It is only in the last 100 years that this question is being discussed. Second of all, the Gemara in Bava Kama says that we don't want to be hard or easy on the מזיק. In this situation it seems we are being way too lenient on the מזיק by allowing him to get off scot free. Intuitively, the Chazon Ish's approach seems right, why should the guy get off scot free, yet the Chazon Ish is against the simple reading of the Gemara.
