It seems to me, based on my limited reading in such subjects, that machshavah, pnimiyus, jewish philosophy, hashkafah, kabalah etc, are all basically a similar phenomenon. Attempts by man to go beyond the mundane and create some kind of super-natural system of thought, either as a means to get closer to G-d, to understand G-d, or maybe just because its interesting.
Examples I can think of include the works of the Maharal, Rav Kook, Nefesh Hachayim, R YB Soloveitchik, AJ Heschel, all of Chasiddus etc. There doesn't seem to be much of this pre 12th? century though. Chazal did this a bit, but usually it was real short, just a pasuk and a quick 'homily'. Did no one before the Rishonim ever engage in this ?
Maybe some or all of this is somewhat divinely inspired, but its still a human creation. That doesn’t mean it doesn't have value, I just don't believe any of it came down from Sinai that’s all. When Rav Moshe Shapiro gives an awesome machshavah shiur for example, I am sure its valuable, but at the end of the day, its his invention, its not Torah Min HaShamayim. Take what appeals to you and leave the rest. (Maybe thats obvious ?)
The base side of this same tendency is all the stupid gilgulim / shaydim / kabalistic trickery / dibukim / superstition / mekubalim nonsense, which is also expressly forbidden by the Torah in my opinion, and has no value at all. Its amazing that the UO world is still so steeped in this. Are they still using pigeons in Eretz Yisrael to cure hepatitis ? Wasn't the whole point of Judaism to steer us away from all these kinds of things ?
The problem in the UO world is that they seem to be unable to distinguish between the good stuff and the bad stuff. The whole lot of it gets mushed together into one huge UO ideological cholent, and if you don't believe in any part of it, you are a kofer, or worse, a modernishe (chas vesholom).
Well, my advice is don't eat too much of that cholent, it's after effects can be quite, shall we say, unpleasant.