FWIW, it seems to me that a major benefit of the Bayesian approach is to make bad reasoning (in the form of, say, an unreasonable prior) transparent and obvious. I've never heard it claimed that the Bayesian approach was robust to sophisticated idiocy (neither on LessWrong nor mainstream writing on Bayesian methods), except in the narrow techical sense that the posterior asymptotically approximates the likelihood given infinite data (provably true under some assumptions but irrelevant to the objections in the OP).
Yup. Actually, a common theme on LessWrong was realizing that with better reasoning tools you're more able to bullshit yourself, and so you need to be extra-careful.
You can't protect yourself in 100% - it would require developing more powerful reasoning tools in an infinite regression. But what you can do is to use introspection, and triple-check your reasoning when it seems to defy common sense or leads you to weird (awful) conclusions.
That's why LW is so big on biases and heuristics by the way - you can treat them as a list of warning signs; if your reasoning seems to match some of them, it's time to take a closer look.
(A thing a lot of people missed since knowing logical fallacies started to become a mainstream thing - you should use the list to check your own reasoning, not your opponent's.)
The problem with trying to rely on heuristics to avoid biases is people often ignore the biases in the heuristics of choice. To continue the example of LW, there are many people there who seem to think highly of IQ test, and who ignore the many issues with them (the Flynn effect an the effect of incentives being a couple examples of the flaws in IQ tests).
Trying to remove biases is great. But there is a problem when someone works to remove some biases, then believes that they are inherently more rational than the public at large, and then uncritically accepts there other biases ("Someone like me who's worked hard to remove there biases must be correct when compared to the biased masses.").
Yes, there is that risk, and no doubt many fall for it. Ego / self-esteem issues may be a big part of it. But then again, every worthy goal poses risks. When you fly a plane, there's a greater risk you'll kill yourself than when you stay on the ground, and yet airplanes are being flown and we're reaping great benefits from it.
RE IQ, personally, I'm 100% confused on the topic. I used to believe that Flynn effect is basically people getting better at doing tests, but recently I heard that someone controlled for that and the effect remained. So I don't know. The topic is complicated and most of studies I heard of are the kind of psychology and social science I implicitly assume is mostly bullshit.
IQ is held in high esteem because research around g is very good and comprehensive, perhaps the crown jewel of psychology. Additionally most people's knowledge of the Flynn effect is out of date -- recent studies (here's one: http://www.sciencedirect.com/science/article/pii/S0160289615... in fact here's a boatload of references http://www.iapsych.com/iqmr/fe/MasterFlynnEffectreferencelis...) show a rise, leveling off, and then an overall fall from the beginning in performance over the last 40 years rather than the continuous rise (or at least non-decreasing) behavior most people would probably bet on from their layman understanding of the effect. (Additionally ethnic gaps have remained despite controls for everything and it is this unfortunate reality that I think is the reason for so much dismissal of IQ...)
When fairly minor monetary incentives (over $10) can lead to a 20 point increase in scores[1], I'd be wary of reading too much into the tests. As for the Flynn effect, I've read a number of different theories, but there doesn't seem to be any consensus. Given the other issues IQ tests have (like the one just mentioned), I'd be wary about assuming that it doesn't stem from underlying problems with the test itself. Some researchers seem to think it stems from familiarity with test taking in general, which seems to match the general understanding that you will do better on IQ tests if you repeatedly take them, referred to as the "practice effect" (IE, IQ tests at least in part measure familiarity with the test).
This study isn't surprising, I'm unsure what you think it implies. Indeed IQ researchers have been wary themselves and known about motivational effects for decades, along with many other objections to testing like cultural bias and the like that in good modern research are all accounted for -- obviously if I sleep during a test I'll score very low, but if I'm actually awake and care I can increase my score by a huge factor. IQ isn't a perfect correlate, but it's common to reason as if it alone has a predictive power of 0.4-0.6 for various important things, that is stronger than any other single factor we know about. When you add in Conscientiousness (the big five traits being the other contender for psychology's crown jewel, I think), which is about grit, intrinsic motivation, and the like, together with IQ (two factors now, not just one), you get predictive powers of 0.7 to educational success. I would wager that the differences seen in the paper cited by your article are almost entirely accounted for by Conscientiousness, but from the abstract (don't have the full paper) it looks like that was not controlled for at all.
The Flynn effect mystery is exactly the same as the mystery that in most countries the younger population is taller than the previous generation. The fact that the Flynn effect only occurs in the bottom half of the intelligence distrubtion is a bit of a clue as to what is the cause.
What is the cause of the population getting taller?
The brain is just another organ that is affected by nutrition in the same way height is. Improve nutrition and those individuals that are below their genetic potential due to poor nutrition will improve. Guess which half of the population has suffered from poor nutrition in the past?
You should probably know what James Flynn thinks of the Flynn effect. He doesn't try to escape the conclusions of an I.Q. test as much as extend them, despite some of the "paradoxes".
Fair enough. It's also worth pointing out that Alfred Binet, generally considered to be one of the fathers (if not THE father) of intelligence testing, felt that the idea of quantitative intelligence testing was severely flawed (and had fairly harsh words for people who believed that there was largely a single measure of intelligence).
You don't take anything hammered out by reason alone as sound information. You test reasonable-looking propositions against experience, and until they've stood up against that test by providing accurate predictions with substantial information-content, you take them as provisional.
Bayesian seems great when you first see it. It should be obvious how to apply it for something like a card game. The problem is how long would it take you to realize a deck of cards was missing the 4 of diamonds? What if the card was lost 1/2 way though the game? How about on the prior hand?
In the end it's stuck at one level of recursion and all facts are fuzzy.
It's stuck in the reality you define - you can add the last hand, all cards seen so far, and the color of people's jackets to the model if you so please.
Your still stuck with: You observed X/Y. How accurate is your count. How accurate is your estimate of accuracy. How accurate is your estimate of accuracy of your accuracy estimate. ... recursive infinity.
Did you account for the possibility you're dreaming, or in a simulation? Or in a simulation of a simulation and so on? Yes, that's the curse of real numbers. In practice you set a limit of precision required, and a threshold of evidence you need to tell you that the precision should be more precise. You can get quite far with Newtonian physics without considering the corrections of general relativity, you can get quite far in engineering by assuming linearity by only expanding the Taylor transform of a non-linear expression by a bit instead of infinitely, and you can do a lot by assuming probabilistic independence of your card counting from what's going on on Pluto right now.
It is going to take me more than a few minutes to parse this. However, I am relatively certain there is a lot of sarcasm in this snippet and I like it.
