The GH in his blog has devoted a lot of time to machshava/hashkafa issues and wonders why there is no new machshava coming out of the frum world. The follwoing anecdotes may explain things.
I heard the following from R' CY Goldvicht Rosh Yeshiva of KBY. He was a talmid of both the Chazon Ish and the Brisker Rav and a big baal machshava. He once asked the Chazon Ish why he only talks in halacha to his talimdim, why doesn't he talk hashkafa? The Chazon Ish answered that in Halacha there are boundaries. We have the words of the tannaim, then amoraim, then rishonim, acharaonim and the shulchan aruch. If we go off the path we will see it right away. In machshava, when we interpret a midrash, we are trying to say pshat in the words of the tannaim with no intervening generations. Also, there is no shulchan aruch and therefore it is very easy to make a mistake.
RHS told the following story about RYBS. Ater a shiur he gave in Moriah about aggada one of his talmidim came to the Rav and asked him where does he get it from? How does he know to say such big chiddushim in aggada. The Rav answered, and why don't you ask about my gemara shiurim? The talmid answered, there you have a basis, the rishonim, the acharonim, your grandfather, here there is none of that. The Rav answered, "my father taught me to read between the lines".
We see 2 things from these stories.
1. You need to be an adam gadol to say chiddushim in machshava, otherwise you are liable to make a mistake.
2. You need a Rebbe to teach you to read between the lines.
Even more important is to be humble. I detect a great amount of hubris nowadays, people think that because we know so much science and understand how the physical world works we are greater then previous generations. This leads to a zilzul in chazal. We need to understand that they were closer to matan torah and therefore had a much better understanding of hashem and torah.
