I've been saying this for years, and this is a large reason why I find the LessWrong folks to be almost entirely full of it. Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking.
A couple of comments are saying, "no duh" to this article, but those folks likely don't realize quite how many other people are falling into this trap. "Garbage in, garbage out" is only good advice when the person you're saying it to realizes they're putting garbage in.
FWIW, it seems to me that a major benefit of the Bayesian approach is to make bad reasoning (in the form of, say, an unreasonable prior) transparent and obvious. I've never heard it claimed that the Bayesian approach was robust to sophisticated idiocy (neither on LessWrong nor mainstream writing on Bayesian methods), except in the narrow techical sense that the posterior asymptotically approximates the likelihood given infinite data (provably true under some assumptions but irrelevant to the objections in the OP).
Yup. Actually, a common theme on LessWrong was realizing that with better reasoning tools you're more able to bullshit yourself, and so you need to be extra-careful.
You can't protect yourself in 100% - it would require developing more powerful reasoning tools in an infinite regression. But what you can do is to use introspection, and triple-check your reasoning when it seems to defy common sense or leads you to weird (awful) conclusions.
That's why LW is so big on biases and heuristics by the way - you can treat them as a list of warning signs; if your reasoning seems to match some of them, it's time to take a closer look.
(A thing a lot of people missed since knowing logical fallacies started to become a mainstream thing - you should use the list to check your own reasoning, not your opponent's.)
The problem with trying to rely on heuristics to avoid biases is people often ignore the biases in the heuristics of choice. To continue the example of LW, there are many people there who seem to think highly of IQ test, and who ignore the many issues with them (the Flynn effect an the effect of incentives being a couple examples of the flaws in IQ tests).
Trying to remove biases is great. But there is a problem when someone works to remove some biases, then believes that they are inherently more rational than the public at large, and then uncritically accepts there other biases ("Someone like me who's worked hard to remove there biases must be correct when compared to the biased masses.").
Yes, there is that risk, and no doubt many fall for it. Ego / self-esteem issues may be a big part of it. But then again, every worthy goal poses risks. When you fly a plane, there's a greater risk you'll kill yourself than when you stay on the ground, and yet airplanes are being flown and we're reaping great benefits from it.
RE IQ, personally, I'm 100% confused on the topic. I used to believe that Flynn effect is basically people getting better at doing tests, but recently I heard that someone controlled for that and the effect remained. So I don't know. The topic is complicated and most of studies I heard of are the kind of psychology and social science I implicitly assume is mostly bullshit.
IQ is held in high esteem because research around g is very good and comprehensive, perhaps the crown jewel of psychology. Additionally most people's knowledge of the Flynn effect is out of date -- recent studies (here's one: http://www.sciencedirect.com/science/article/pii/S0160289615... in fact here's a boatload of references http://www.iapsych.com/iqmr/fe/MasterFlynnEffectreferencelis...) show a rise, leveling off, and then an overall fall from the beginning in performance over the last 40 years rather than the continuous rise (or at least non-decreasing) behavior most people would probably bet on from their layman understanding of the effect. (Additionally ethnic gaps have remained despite controls for everything and it is this unfortunate reality that I think is the reason for so much dismissal of IQ...)
When fairly minor monetary incentives (over $10) can lead to a 20 point increase in scores[1], I'd be wary of reading too much into the tests. As for the Flynn effect, I've read a number of different theories, but there doesn't seem to be any consensus. Given the other issues IQ tests have (like the one just mentioned), I'd be wary about assuming that it doesn't stem from underlying problems with the test itself. Some researchers seem to think it stems from familiarity with test taking in general, which seems to match the general understanding that you will do better on IQ tests if you repeatedly take them, referred to as the "practice effect" (IE, IQ tests at least in part measure familiarity with the test).
This study isn't surprising, I'm unsure what you think it implies. Indeed IQ researchers have been wary themselves and known about motivational effects for decades, along with many other objections to testing like cultural bias and the like that in good modern research are all accounted for -- obviously if I sleep during a test I'll score very low, but if I'm actually awake and care I can increase my score by a huge factor. IQ isn't a perfect correlate, but it's common to reason as if it alone has a predictive power of 0.4-0.6 for various important things, that is stronger than any other single factor we know about. When you add in Conscientiousness (the big five traits being the other contender for psychology's crown jewel, I think), which is about grit, intrinsic motivation, and the like, together with IQ (two factors now, not just one), you get predictive powers of 0.7 to educational success. I would wager that the differences seen in the paper cited by your article are almost entirely accounted for by Conscientiousness, but from the abstract (don't have the full paper) it looks like that was not controlled for at all.
The Flynn effect mystery is exactly the same as the mystery that in most countries the younger population is taller than the previous generation. The fact that the Flynn effect only occurs in the bottom half of the intelligence distrubtion is a bit of a clue as to what is the cause.
What is the cause of the population getting taller?
The brain is just another organ that is affected by nutrition in the same way height is. Improve nutrition and those individuals that are below their genetic potential due to poor nutrition will improve. Guess which half of the population has suffered from poor nutrition in the past?
You should probably know what James Flynn thinks of the Flynn effect. He doesn't try to escape the conclusions of an I.Q. test as much as extend them, despite some of the "paradoxes".
Fair enough. It's also worth pointing out that Alfred Binet, generally considered to be one of the fathers (if not THE father) of intelligence testing, felt that the idea of quantitative intelligence testing was severely flawed (and had fairly harsh words for people who believed that there was largely a single measure of intelligence).
You don't take anything hammered out by reason alone as sound information. You test reasonable-looking propositions against experience, and until they've stood up against that test by providing accurate predictions with substantial information-content, you take them as provisional.
Bayesian seems great when you first see it. It should be obvious how to apply it for something like a card game. The problem is how long would it take you to realize a deck of cards was missing the 4 of diamonds? What if the card was lost 1/2 way though the game? How about on the prior hand?
In the end it's stuck at one level of recursion and all facts are fuzzy.
It's stuck in the reality you define - you can add the last hand, all cards seen so far, and the color of people's jackets to the model if you so please.
Your still stuck with: You observed X/Y. How accurate is your count. How accurate is your estimate of accuracy. How accurate is your estimate of accuracy of your accuracy estimate. ... recursive infinity.
Did you account for the possibility you're dreaming, or in a simulation? Or in a simulation of a simulation and so on? Yes, that's the curse of real numbers. In practice you set a limit of precision required, and a threshold of evidence you need to tell you that the precision should be more precise. You can get quite far with Newtonian physics without considering the corrections of general relativity, you can get quite far in engineering by assuming linearity by only expanding the Taylor transform of a non-linear expression by a bit instead of infinitely, and you can do a lot by assuming probabilistic independence of your card counting from what's going on on Pluto right now.
It is going to take me more than a few minutes to parse this. However, I am relatively certain there is a lot of sarcasm in this snippet and I like it.
Except that at least with Bayesian methods the prior is explicitly laid out.
Frequencist methods when they are used to make predictions and get useful information out of experiments have hidden implicit priors that bias inferences in opaque ways.
The very honest frequencists will admit that their procedures are only rejecting hypotheses so small as to have no practical utility. Others will use weird and dishonest doublespeak where they call rejecting an insignificantly small hypothesis "statistical significance".
But I suppose they do redeem themselves a bit with the wording "null hypothesis" which candidly conveys the sense of having significantly rejected _nothing_.
> I've been saying this for years, and this is a large reason why I find the LessWrong folks to be almost entirely full of it. Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking.
I assume you're saying LessWrong folks are more prone to miscalculating priors than most. Could you give some examples of this?
The LessWrong folks aren’t obviously better or worse at calculating priors than anyone else. The “problem” is that their hobby is spending their free time considering outlandish scenarios, inventing arbitrary assumptions related to such scenarios, drawing questionable conclusions, and then convincing themselves that because they used logic and math, their analysis must be correct. Plenty of other folks who spend time on similar activities with a less pseudo-rigorous framing end up as conspiracy theorists or occultists; belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans is far from the kookiest thing people convince themselves about.
Oh come on. That's how you're supposed to use math. To aid your thinking. Without it, and considering "outlandish scenarios", we would not have any scientific progress.
Also, this is one strange thing - any time someone asks us (the STEM crowd), "what will I ever use math for in my life?", the default answer seems to be, "it's about having more tools for thinking, and greater clarity of thought; it'll make you smarter". But then, some of us turn around and refuse to acknowledge that people who actually learn math and try to apply it may be getting those promised results. Whether it's LW people, programmers, engineers or scientists, the moment it matters, the default conclusion is that math gives nothing.
What? Who said “math gives nothing”? I spend most of my day building things out of math. I think math and scientific inquiry are basically the most important tools invented/popularized in the past 1000 years.
The lapse here is not math, but rather spending lots of attention on abstract thought disconnected from any kind of reality check. Of course, there’s nothing inherently wrong with speculating sans evidence about the future, it’s generally a harmless hobby. In the best case it makes for fun SF novels. Convincing yourself, still without direct evidence, that your speculation reflects truth implies that something has gone off the rails in the reasoning process, however.
Using statistical analysis to understand real causal relationships in areas we have real data about is damn hard, and even plenty of people who are highly trained as statisticians screw up all the time. Academic fields like comparative politics (to take an example I spent a fair amount of time studying) are rife with poor conclusions drawn from bad analysis. The LessWrong folks are hardly unique in applying logic poorly. But they do tend to tackle more speculative questions and convince themselves more firmly of their conclusions (at least, such is my impression as an outsider).
I think this is a common problem when people working in fields that have somewhat accurate mathematical models look at fields that don't. They often don't realize how hard it is to create an accurate mathematical model for many situations, and assume that the other fields don't have them because the individuals who work in said fields aren't as good at math.
Which is why every so often you'll get things like a physicist spending a couple months studying economics in their free time and deciding that they can now unlock the secret to economics which has eluded economists.
The more you extrapolate the more frequently you will need to adjust your future predictions because of errors in your initial measurements. This goes for any kind of extrapolation (for instance: plotting a course on a map), but it goes even more for extrapolating the future from limited evidence present today. Your 'best guess' might be off by many orders of magnitude if the evidence you have today is only loosely related to the future in terms of importance and where evidence may not be nearly as independent in nature as you currently perceive.
This can lead to your best guess based on available evidence being about as good at predicting the future as randomness in spite of all the apparent effort at making the predictions mathematically sound.
> because they used logic and math, their analysis must be correct
> belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans
I don't think anyone on LW believes these are 'facts' that are 'correct'. LW commonly thinks of these ideas as risks / opportunites that might happen (except for the 1000-year human lifespans, which is a new idea to me), and that it's probably worth investing minor amounts of money in case it does. In case of cryonics, that's about $20/month for insurance that covers it; in case of AI safety, that's a couple people doing research on the problem, and some amount of money sent their way.
The way you phrase it seems like LW people are certain that cryonics will definitely let them be revived after death. That's definitely not the case - in fact, IIRC, on a survey a year or two ago LWers subscribed to cryonics assigned lower probability of it working than ones not subscribed. It's not a cargo-cult.
Substitute “plausible” for “correct” if you want to give them the benefit of the doubt. Either way it all so speculative as to be basically pure fiction. It reads very similar to me to various “scientific” defenses of particular religious traditions.
Again, as I said, I don’t think there’s anything inherently wrong with this. Little communities of people should do whatever harmless hobby is fun for them.
I just don’t find it very interesting or insightful.
If people want to spend time and attention and resources on existential risks, how about the ones which are clear and imminent, like wealth inequality, the retreat of world democracy and increasing power of entirely unaccountable and amoral multinational corporations, or global climate change, e.g. http://www.esquire.com/news-politics/a36228/ballad-of-the-sa...
But these aren't examples so much as vague caricatures. The subject matter that LessWrong considers is certainly unusual, but that alone should not be enough to call it arbitrary, questionable or outlandish.
I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes. It's not like LW gives Roko significant air time or serious treatment (EY freaking out and deleting it was partially principle of the thing, partially the fear that somebody might follow this road of thought to come up with something more terrifying and he doesn't want to take the community there, etc) somebody doing this is generally taken as a sign of serious crankery.
(FWIW, I agree with the top parent post that the hyping of Bayes Theorem is one of the LW foibles. At least the presentation of it.)
It's less about the plausibility of the thought experiment, and more about typical online drama and hysteria that ensued, which sort of belies that LW is made up of mortals like you and me. They aren't hyper-rational machines, after all.
No one claims they are. In fact, if I was to name a single overarching theme of all lesswrong discussions, it would be the fallibility of human reasoning. How is having some reddit-like drama on an open internet forum even relevant?
ps. to belie is to contradict, whereas I think you meant the drama shows that LW is made up of mortals? Just making sure I understood you correctly.
"The subject matter that LessWrong considers is certainly unusual, but that alone should not be enough to call it arbitrary, questionable or outlandish."
"But let me tell you about the time LW experienced internet drama."
>I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes.
Actually, that sounds like a fine reason to malign philosophers. In order to consider "outlandish scary scenarios", you must first be quite sure that those scenarios are realistic. If they're not, then you're wasting everyone's time.
And yes, if your expected-utility expressions fail to converge because you believe in taking every Pascal's Wager/Mugging scenario into account, or because you don't believe in time, then you've attempted to take the limit of a nonconverging sequence and no amount of philosophizing will help.
This is still working backwards: a thought experiment that increases your chances of being tortured by a future AI? Surely outlandish! But why? Which premises are truly outlandish, arbitrary, etc? What I see given the premises is no more than what Yudkowsky's already said: ".. a Friendly AI torturing people who didn't help it exist has probability ~0, nor did I ever say otherwise."
(However, I agree that to be genuinely distressed by the thought experiment possibility suggests more is going on psychologically than a rational assessment of unknowns, but this seems to be a minority of the community)
So I'm a LessWronger and know a bit about the "movement", and think you are misunderstanding what "LessWrongers think". Obviously not all LessWrongers think the same thing at all, but I'm talking about the average position of the people who believe AI safety should be worked towards.
I'd love to explain the basic position, and tell me where you disagree with it. This is the basic position:
1. Intelligence can be created, because there is nothing "special"/"magical" about humans, and our intelligence was eventually created.
2. At some point, humanity will create an "artificial general intelligence". (Since we'll just keep improving science and technology, and there's no fundamental reason why this won't eventually allow us to create an intelligence.
3. "Artificial general intelligence" basically means a machine that is capable of achieving its goals, where the goals and methods it uses to achieve them are general. I.e. not "is able to play chess really well", but rather "is able to e.g. cure cancer".
4. For various reasons, once we have an artificial intelligence, it will likely become much smarter than us. (There are many reasons and debates about this, but let's just assume that since it's a computer, we can run it much faster than a human. If you dispute this point, we can talk about it more).
5. Something being much more intelligent than us means that, in effect, it has almost absolute power over what happens in the world (like we are basically all-powerful from the vantage point of monkeys, and their fate is totally in our hands).
6. (This is, I believe, the main point): Something being "intelligent" in the sense we're talking about doesn't say anything about what its goals are, or about how its mind works. We're used to everything that's intelligent being a human being, therefore the way our mind works is basically the same across every human. With an artificial intelligence, it will work completely differently from our mind. So if we "tell it" something like "cure cancer", it won't have our intuition and background knowledge to understand that we mean "but don't turn half the world into a giant computer in order to cure it".
7. Combine the two points above, and you get the large idea - whatever the goals of the AI will be, it will achieve them. Its goals won't, by default, be ones that are good for humanity, if only because we have no idea how to program our "value system" into a computer.
8. Therefore, we need to start working on making sure that when AI does come, it's safe. Even if we create an AI, the "extra" problem of making it safe is both hard, and we have absolutely no idea how to do it right now. We have no idea how long AI will take, or how long figuring out safety will take, but since this is a humanity-threatening problem, we should devote at least some resources to working on it right now.
That's it, that's the basic idea. I'd love to hear which part you disagree with. I totally understand that not everyone will agree on some of the final details like, e.g., how many resources we should effectively devote right now (you might even claim it's 0 because anything we do now won't be useful).
But I think the overall reasoning is sound, and would love to hear an intelligent disagreement.
> 1. Intelligence can be created, because there is nothing "special"/"magical" about humans, and our intelligence was eventually created.
Human intelligence evolved through a (very long!) series of natural processes, to the best of my knowledge. To say it was "created" implies something closer to a religious or philosophical opinion, rather than something supported by science.
> 2. At some point, humanity will create an "artificial general intelligence". (Since we'll just keep improving science and technology, and there's no fundamental reason why this won't eventually allow us to create an intelligence.
This is hugely debatable. Why is AGI inevitable? Even given great amounts of computing resources, a artificial general intelligence does not just automatically appear, it must somehow be designed and programmed. Fields like computer vision have grown tremendously using techniques like deep learning, but there really isn't any evidence that I know of that a general intelligence is any closer than it was 20 years ago.
Totally agree with your first point, I just didn't want to have too many caveats and nitpicking words. If it's not clear, then of course my arugment in no way implies that human intelligence was "created" by an intelligence - it evolved. Poor wording aside, my statement remains the same.
"This is hugely debatable. Why is AGI inevitable? Even given great amounts of computing resources, a artificial general intelligence does not just automatically appear [...]"
Well no one thinks AGI will appear without anyone working on it, but lots and lots of people are working on it now. And since there are huge incentives to create one, the belief is that more people will work on it as time goes on.
"[...] there really isn't any evidence that I know of that a general intelligence is any closer than it was 20 years ago."
Well, in some sense I agree, in that we still have no idea how far off AGI is. If it's going to happen in 10 years, we should definitely prepare now. If it's 500 years away, maybe it's too early to think about it. But since neither of us knows, wouldn't you say it's worth putting some effort to working towards safety?
In another sense though, I disagree with you that we're not any closer to AGI. As you said jsut the sentence before, fields like comptuer vision have advanced tremendously. While this doesn't necessarily mean AGI is closer, it certainly seems that the fields are related, so advancement in one is a sign that advancement in the other is closer.
Yeah, you go off the rails around step 5. "Something being much more intelligent than us means that, in effect, it has almost absolute power over what happens in the world" makes no sense. Since when does intelligence get you power? Are the smartest people you know also in positions of power? Are the most powerful people all highly intelligent?
"whatever the goals of the AI will be, it will achieve them". Dude, if intelligence meant you could achieve your goals, Hacker News would be a much less whiny place.
"Since when does intelligence get you power?" You hit the nail on the head there. Its about I/O. (Just as its about I/O in the original article - garbage in, garbage out). Jaron Lanier makes this point in.
"This notion of attacking the problem on the level of some sort of autonomy algorithm, instead of on the actuator level is totally misdirected. This is where it becomes a policy issue. The sad fact is that, as a society, we have to do something to not have little killer drones proliferate. And maybe that problem will never take place anyway. What we don't have to worry about is the AI algorithm running them, because that's speculative. There isn't an AI algorithm that's good enough to do that for the time being. An equivalent problem can come about, whether or not the AI algorithm happens. In a sense, it's a massive misdirection."
As I've said before, the singularity theorists seem to be somewhere between computer scientists, who think in terms of software, and philosophers, who think in terms of mindware, and they seem to have a tendency to completely forget about hardware.
There seems to be this leap from 'superintelligent AI' to 'omnipotent omniscient deity' which is accepted as inevitable by (what for shorthand here is being called the 'lesswrong' worldview) which seems to ignore the fact that there are limited resources, limited amounts of energy, and limitations imposed by the laws of physics and information, that stand between a superintelligent AI and the ability to actuate changes in the world.
You're not engaging with the claim as it was meant. In context, no human being has ever been "much more intelligent" than me. Not in the same way that I am "much more intelligent" than the monkey von Neumann.
You might decide that this means edanm goes off the rails at step four, instead. But you should at least understand where you disagree.
I'm still not sure you could assume ultimate power and achieve everything you desired if you were the only hacker news reader on a planet of 8 billion monkeys.
> I'm still not sure you could assume ultimate power and achieve everything you desired if you were the only hacker news reader on a planet of 8 billion monkeys.
I would think it relatively easy for a human armed with today's knowledge and a reasonable yet limited resource infrastructure (for comparison to the situation of an unguarded AI) to quite easily engineer the demise of primate competitors in the neighborhood. Set some strategic fires, burn down jungles would be the first step. "Fire" might be a metaphor for some technology that an AI might master that humans don't quite have the hang of yet that can be used against them. For example, a significant portion of Americans seem way too easily manipulated by religion and fear, an AI-generated televangelist or Donald Trump figure might be a frightening thought.
Well "is able to e.g. cure cancer" is not actually very general. Which leads to the problem with 2) whats the economics behind creating a general intelligence when a specific intelligence will get you better results in a given industry. Even then specific intelligence is still going to be subject to the good-enough economic plateau that has killed so many future predictions.
Then the problems with 4 on up really concern the speed with which 4 can feasibly happen. The AI goes FOOM doomsaysers seem to think that we'll end up with an AI which is so horribly inefficient that/and it will be able to rewrite it self to be super duper intelligent without leaving its machine (and won't accidentally nerf itself in the attempt) and then that super duper intelligent computer will trick several industries into building an even more powerful body for itself etc... all of this happening before humans pull the plug. no step of which is has anything beyond speculation to support it.
In a general note the full employment theorems mean that even if general AI is economically incentivized there's still going to be dozens/hundred/thousands of different AIs carving out niches for themselves which, given that the earth/universe has limited resources, handily prevents the paper clip maximizer problem. While the future may not need humans it will still be a diverse future.
1) Define intelligence, knowledge, truth, proof (deductive and inductive)... how do concepts work?, etc. I am not being facetious here. AI is an epistemology problem not a technological one.
2) I agree but we have to solve the problem of induction first but LW/EY are certain that there is no problem of induction. How can one be certain in a Kantian/Popper framework where statements can be proved false but never true?
3a) Here is where we part ways. It is a common assumption that AI implies consciousness but I think that is an unwarranted assumption. Whatever the principles behind intelligence are, we know that consciousness minds have found a way to (implicitly) enact them. It does not follow that consciousness is necessary for intelligence (just the biological manifestation of them) and I think good arguments exist to think that they are not correlatives. If they are correlatives then it will be easier to genetically design better babies, now that evolution is in conscious control, than to start from scratch.
3b) Goals, values, aims, etc. are teleological concepts that apply to living things only because they face the alternative of life or death. Turning off your computer does not kill it in the same sense that a living thing that stops functioning dies forever. 3a) & 3b) diffuses all the scary AI scenarios about AI taking over the world. It does raise the issue of AI in the hands of bad people with evil goals and values, like the dictator of North Korea who now apparently has the H-Bomb. This is the real danger today.
4) I agree. Computer aided intelligence will allow us to accelerate the accumulation of knowledge (and its application to human life) in unimaginable ways. But it will be no more conscious than your (deductive) calculator.
5) Non Sequiturs. Possibly psychological projection of helplessness or hopelessness.
6) As the joke goes, we can always unplug it.
7) Granting your premises then the goal of LW/EY should not be AI but the scientific, rational proof and definition of ethics but their fundamental philosophic premises won't allow it.
8) For me the threat is bad, evil people in possession of powerful technologies.
>Granting your premises then the goal of LW/EY should not be AI but the scientific, rational proof and definition of ethics but their fundamental philosophic premises won't allow it.
That is the goal of MIRI, the organization that EY founded, and is a frequent topic of discussion on LW
•highly reliable agent design: how can we design AI systems that reliably pursue the goals they are given?
•value learning: how can we design learning systems to learn goals that are aligned with human values?
Not exactly what I meant. What are these human values (for humans not robots) and how do you prove they are rational and scientific? Their goal is to design AI that will accept human goals/values without defining a rational basis for those human values.
I've been saying t forever. Thanks for putting it so succinctly. LessWrong is a cult of people who want to be smart and they've essentially found a community in which certain assumptions and hypothetical scenarios combined with mathematical concepts make them think they've found the answer to everything in the Universe.
They're no better than any other cult in my book. The problem is that it's only going to get worse with the advances in AI that are going on. Yudkowski has managed to convince some wealthy people to fund his so called research and we have OpenAI operating in the same waters which somehow gives LW people more legitimacy.
No better than any other cult? How are you deciding that? The LW community hasn't killed people. Doesn't cut people off from their family. Does't emotionally/physically abuse people. Etc...
Even if they are a "cult", this puts them miles ahead of other cults, like say, Scientology which has done far, far more harm to people.
I struggle to think in what way LW has harmed anyone at all.
cult |kʌlt|
noun
1 a system of religious veneration and devotion directed towards a particular figure or object: the cult of St Olaf.
• a relatively small group of people having religious beliefs or practices regarded by others as strange or as imposing excessive control over members.
The veneration of Yudkowski and others in the LW community is more than a bit "religious". So I'd say by definition it's a cult.
LW hasn't done harm to people physically, but what it's done is spawn some very questionable ideas, perpetuate pseudo-science and pseudo-mathematics. The cult leader has no formal training, zero research in peer-reviewed journals and still calls himself a "senior research fellow" in an institute he himself started.
Hell, he even has an introductory religious text - The Sequences and the Methods of Rationality fan fiction (which by the way, he wanted to monetise before the broader fan fiction community stopped him. A clear violation of copyright law).
I'll quote the section titled "More controversial positions"
Despite being viewed as the smartest two-legged being to ever walk this planet on LessWrong, Yudkowsky (and by consequence much of the LessWrong community) endorses positions as TruthTM that are actually controversial in their respective fields. Below is a partial list:
Transhumanism is correct. Cryonics might someday work. The Singularity is near![citation NOT needed]
Bayes' theorem and the scientific method don't always lead to the same conclusions (and therefore Bayes is better than science).[21]
Bayesian probability can be applied indiscriminately.[22]
Non-computable results, such as Kolmogorov complexity, are totally a reasonable basis for the entire epistemology. Solomonoff, baby!
Many Worlds Interpretation (MWI) of quantum physics is correct (a "slam dunk"), despite the lack of consensus among quantum physicists.[23]
Evolutionary psychology is well-established science.
Utilitarianism is a correct theory of morality. In particular, he proposes a framework by which an extremely, extremely huge number of people experiencing a speck of dust in their eyes for a moment could be worse than a man being tortured for 50 years.[24]
Also, while it is not very clear what his actual position is on this, he wrote a short sci-fi story where rape was briefly mentioned as legal.
TL;DR: If it's associated with LessWrong/Yudkowsky, it's probably bullshit.
I cannot judge to which degree these theses are bullshit, but I've found LW a tremendously rich source of thinking tools and I'm convinced that reading or skimming a lot of the sequences have improved my thinking.
Regarding the rape sequence in HPMOR: It's a terribly chosen trope to convey that the fictional society has very different values from ours. Apparently it ties into various parts of the story, so that EY didn't remove it and only toned it down after it was criticized.
> I cannot judge to which degree these theses are bullshit.
Go the link, go to references, read about them. I'll outline the gist: Most of what Yudkowsky says is extremely sci-fi, no real basis in scientific fact, but stretching the current technological progress to the point where his opinions on things (stuff like transhumanism, singularity) can be justified.
What he's preaching isn't science. Certainly not rigorous experimental science. He (along with Bostrom) tends to engage in extreme hypotheticals. Which, sure if you're a philosopher, is fine. But even then, wouldn't you want your work to be judged by like-minded peers? But alas, here has a convenient excuse of being "auto didactic" to fall back on, so he can sit on his armchair and critique traditional education, and his lack of peer reviewed material.
Not to mention, and this is a bit of a pet peeve, I find that most LW people are too self-absorbed, I've literally seen a blog where the person who runs it "warns" the readers that what he writes is too complicated for people to follow. This sort of narcissistic, self congratulatory thinking is what puts me off more than anything. Writing long form posts on the Internet which use complicated words don't make you smart.
> I've found LW a tremendously rich source of thinking tools and I'm convinced that reading or skimming a lot of the sequences have improved my thinking.
There are other ways to improve your thinking. Read books. Read different kind of books, that offer counter point of views. Farnham Street Blog is a good place to start for a list of resources for thinking tools/mental models btw. :)
I don't buy into the necessity that everything has to be peer-reviewed in the old fashioned way. There is peer-review happening in the comments to some extent. I don't take a fancy to dismissing any radical ideas as pseudoscience. It's just the outer fringe of hypotheses that need to be tested against reality, and as long they are approximately humanist, enlightened and don't contradict existing physics without depending on mathematics (or disclaimers), I cannot see anything wrong with it. As a naturalist, I pretty much agree with everything I've read on LW so far, except for the parts I cannot judge (like hypotheses about physics), which I allocate a weaker priors for and a few unconvincing pieces.
> Not to mention, and this is a bit of a pet peeve, I find that most LW people are too self-absorbed, I've literally seen a blog where the person who runs it "warns" the readers that what he writes is too complicated for people to follow.
I have not yet experienced that, but there are also a lot of people on reddit and HN that I don't like, yet I differentiate within these communities between what is valuable and what is not.
> Most of what Yudkowsky says is extremely sci-fi, no real basis in scientific fact, but stretching the current technological progress to the point where his opinions on things (stuff like transhumanism, singularity) can be justified.
At the risk of seeming indoctrinated to you, this is what I believe with high certainty: If Moore's law continues another one or two decades, I think the singularity is a very real possibility. The human brain seems to be nothing more than a learning and prediction machine, nothing what transcends what we can understand in principle. Evolution did come up with complex organisms, but the complexity is limited by biochemical mechanisms and availability of energy. In addition, nature often approximates very simple things in overly complicated ways because evolution is based on incremental changes, not on an ultimate goal that prescribes a design of low complexity. I also think that AI will very likely be superintelligent and that poses a tremendous risk in the 10-40 years to come (on the order of atomic warfare and runaway climate change). By the time someone implements an approximately human-level intelligence, we better have a good idea about how to control such a machine.
> There is peer-review happening in the comments to some extent.
Lol. I guess we don't need college education as well then, there's education happening in the comments to some extent. We don't need traditional means of news, there's news happening on Twitter to some extent. I could go on with analogous line of reasoning.
Don't get me wrong, I'm not 100% in favour of the traditional education model as well, but peer reviews exist for a reason. You and I are not experts in these fields. We rely on the expertise of people who have made it their business and life to study these fields based on a rigorous method. Would you try out homeopathy had it not been rejected completely by doctors and scientists but someone on a forum told you it worked for them? What if someone wrote a very long article with fancy words (like LW tends to do) explaining how and why it works (they exist, I assure you)? Would you try it then?
> I don't take a fancy to dismissing any radical ideas as pseudoscience.
Sure, I'm not saying we should be against radical ideas. That's how scientific progress happens. I'm against LW ideas, for which there is no basis in reality as far as we know based on our current understanding of science.
> I differentiate within these communities between what is valuable and what is not.
Indeed. But I'd rather the community's entire existence not depend on bullshit.
> At the risk of seeming indoctrinated to you, ...
a) Keywords: "If", "Seems"
b) Tons of assumptions in that scenario you laid out. If you can't see it, I'm sorry but you're already too far gone.
c) Watch some MIT lectures on computer architectures about how the trend of Moore's law has already radically shifted and is flatlining.
Basically, what you've done is precisely the kind of utter crap that LW perpetuates. "If x keeps happening" without providing any reason as to why that would be true. Make some ridiculous simplifications "complexity is limited by ___", nature often does __ because ___. You basically don't provide any rational reason for why you think AI will be super intelligent and even if it were, why that would be risky. You pick numbers out of a hat (10-40 years to come).
Yes, you look pretty well indoctrinated from where I'm sitting. But I hope you see the many (so many) flaws in that last paragraph of yours (it honestly made me laugh out loud :p)
Predicting the future is hard business -- be it the stock market predicting what happens tomorrow, or weather forecast for the next month. It's presumptuous and hella stupid if you think you can predict where Science and Technology will be x years from now.
> Lol. I guess we don't need college education as well then, there's education happening in the comments to some extent. We don't need traditional means of news, there's news happening on Twitter to some extent. I could go on with analogous line of reasoning.
That's a straw man. I did say it's the fringe and it needs to be tested. I didn't say one should replace the other. Peer-review is essentially just mutual corrections, and there are mutual corrections happening in the comments, just not as thoroughly as when it's institutionalized. Most of it is not new anyway, but just summarizes research results and draws logical conclusions from it (for example this [1]). If it wasn't all brought together on LW, I possibly wouldn't have found out about the wealth of knowledge for a long time.
> a) Keywords: "If", "Seems" b) Tons of assumptions in that scenario you laid out. If you can't see it, I'm sorry but you're already too far gone. c) Basically, what you've done is precisely the kind of utter crap that LW perpetuates. "If x keeps happening" without providing any reason as to why that would be true.
It's very logical. My certainty referred to the implication, but it is hard, of course, to come up with a prior for that 'if': Exponential progress could continue in various ways, e.g. by invention of more energy efficient chips and by scaling them up, by 3D circuitry, molecular assemblers, memristors, or perhaps quantum computing. There are contradicting studies, so one should put P(Moore's law continues for another 10-20 yrs) at perhaps 50%. So, of course, this is all hedged behind this prior (which I think many people get confused by). The discussion is always concerned with implications which can be made with fairly solid reasoning, by assuming that P(..) above to be 100%.
> Make some ridiculous simplifications "complexity is limited by ___", nature often does __ because ___. You basically don't provide any rational reason for why you think AI will be super intelligent and even if it were, why that would be risky.
That's just a basic assumptions which I find plausible, and which some respectable and knowledgeable persons find plausible too (for example Stephen Wolfram and Mark Tegmark; I am aware that appeal to authority is difficult to argue from, but both have publications which I could also refer to). I agree that mentioning the complexity limitations didn't provide any information because they don't tell us whether it makes it simple enough for us to understand, it merely says that the complexity is not infinite, so I should have left it out entirely. But this is not at all representative for the best contents on LW, it was poor reasoning on my behalf. Bostrom's book Superintelligence gives a pretty good summary about why it is thought to be plausible.
> You pick numbers out of a hat (10-40 years to come).
That's based on estimates of the processing power required for brain simulations by IBM researchers and Ray Kurzweil. Simple extrapolation of Moore's law shows us that we will reach that point roughly between 2019 and 2025. 40 years is just my bet based on what I know about brain models and current obstacles in AI.
I don't understand how you can be so certain that the hypothetical scenarios they imagine can't possibly happen. Even if it really is laughable, we spend a lot of money on laughable research (homeopathy, anyone?), so why is this case so particularly bad?
Not everyone has to agree. That's why we have the scientific method and research institutes. If the current science shows that something is worth exploring in more detail, that some avenues are worth spending money on, spending money on them makes sense.
Bullshit Ideas like AI apocalypse and singularity, transhumanism, downloading brain into a computer do not fit into that criteria.
So how do ideas get to the stage where the 'research institutes' agree they're worth investigating?
It seems to me like you're saying "I don't like these ideas, no one should be working on them". That seems a worse principle then "everyone should work on ideas they find worth investigating".
I also have no idea how you distinguish 'Bullshit Ideas' from non-bullshit ideas without investigating them. Your gut is not that good at distinguishing truth from a blunder.
Do priors just start you off closer to the truth? That is to say, if you start with any prior, will enough additional pieces of evidence always let you converge on the truth?
Does anyone commonly set their priors to be a distribution? Perhaps a range or actually a normal distribution to represent a prior with uncertainty?
> That is to say, if you start with any prior, will enough additional pieces of evidence always let you converge on the truth?
Yes, with two caveats. First, you can't have assigned zero probability to the right answer. Which means that if you have a probability distribution over hypotheses, and the right answer wasn't one of the hypotheses in the distribution, you're doomed from the start. The Solomonoff prior is a mathematical construct that gets around this problem by including every hypothesis that can be expressed as a Turing machine, but it gets used more as a philosophical token than an actual computing tool because it's unfeasible to use directly. The second issue is that if you have a bad enough prior, "enough additional pieces of evidence" can be an arbitrarily large amount, and there may only be a limited amount of evidence available to collect. In particular, rerunning an experiment over and over can only provide a limited amount of evidence, because of the possibility that some systematic error affects every instance of the experiment.
Ah I dimly remember that it also has an Occams razor built-in, so the prior probability falls off with the lengths of the strings that describe the Turing machines?
In my field (Epidemiology), when doing Bayesian analysis, it is very common to set one's priors to be a distribution. Sometimes the point estimate and spread of a previously conducted study or meta-analysis, sometimes merely a uniform distribution with upper and lower bounds ("It is extremely unlikely that the relative risk of disease for this exposure is below 0.01 or above 100...")
It's been argued that frequentist analysis is essentially a Bayesian analysis with a prior distribution centered on zero with bounds from positive to negative infinity.
It has to do with calculus. If the probability of a given result approaches zero as the result itself approaches positive or negative infinity, then the area under the curve approaches 1.
Imagine 1/2 + 1/4 + 1/8 ... to infinity. The sum approaches 1 as the denominator approaches infinity. With calculus, we can determine that with mathematical methods.
It's more of a philosophical statement than an actual implementation. Mainly that frequentist analysis begins with the prior "I dunno, could be anything..."
Essentially, two genuine Bayesian rationalists (with some hand wavy preconditions) cannot agree to disagree; ie, they will eventually converge onto the same understanding of an event.
This isn't true though, the problem is there are generally more then 2 possible explanations. For instance let's imagine both of us are using an experimental telescope to observe if some event occurs. We are then looking at four possible scenarios. The telescope could work/not work correctly and the event could happen/not happen. You are confident that the telescope works correctly and also confident that the event will not occur so you give high prior probability to the first and a low to the second. I on the other hand think the telescope is rubbish and the event will almost certainly occur and do the opposite.
We sit down and wait and do not observe the event. You then come to the conclusion that the event did not occur and the telescope works correctly, while I come to the reverse conclusion.
One of the "handy wavy preconditions" is that they share priors. This nearly never happens in the real world. Almost all disagreements can be traced to differing priors.
First Qn: yes. But the rabbit is hiding in the 'enough'. Most commonly the poor uses of BT end up with a narratively argued prior + a single suite of evidence.
For a finite set of evidence (particularly chosen by someone with bias), bias + evidence can be arbitrarily far from the truth.
> Does anyone commonly set their priors to be a distribution? Perhaps a range or actually a normal distribution to represent a prior with uncertainty?
Almost everyone does this, and the solution is a posterior probability distribution. Most uses of Bayesian techniques that I'm familiar with are based on Monte Carlo simulations, where priors are drawn from the specified prior distribution, processed in some fashion, and result in samples of the posterior distribution.
This is especially helpful when you don't view the problem as having a 'true' single answer with some 'uncertainty' but instead have actual variability in the system, which is described by the posterior distribution.
Thinking about it, I'm really not sure in what instance the prior wouldn't be a distribution. It is, by definition, a probability. I suppose you could have a singular value, where the pdf is a delta function, but what's the point of doing the Bayesian inversion or estimation then? If you have a single value with prior p(x) = 0 or 1, you should end up with the posterior p(x|d)=0 or 1.
So the simplest case is where the prior is a binary: Say p(x=yes) = 0.6, p(x=no)=0.4. Or something like that. It's not a continuous distribution but it's still a distribution. The sum/integral of all cases has to be 1.
The technical term for the condition you're interested in is consistency. There is a theorem (Doob) regarding the consistency of Bayes estimates. [1]
In simple cases it means that under any non-silly prior your posterior will converge to the truth. E.g. mean of a normal distribution with a normal prior.
The complex cases include things like models that expand when given more data.
> if you start with any prior, will enough additional pieces of evidence always let you converge on the truth?
That's the general trend: pooling data tends to make priors converge. However, converging priors isn't quite the same thing as everyone converging on the truth.
Statistical expressions themselves are an incomplete explanation, we routinely use assumptions about the direction of causality that aren't captured in them. See Judea Pearl's "Why I am only half Bayesian" paper for a discussion as well as an intro to how his framework approaches such independence assumptions.
Bayesianism is a 'grand unified theory of reasoning' that all of science be should be based on assigning (and updating) probabilities for a list of possible outcomes; the probabilities are supposed to indicate your subjective degree of confidence that a given outcome will occur.
Contrast this with an alternative conception of rationality as espoused by David Deutsch.
David Deutsch in his superb books, 'The Fabric Of Theory' and 'The Beginning Of Infinity', argued for a different theory of reasoning than Bayesianism. Deutsch (correctly in my view) pointed out that real science is not based on probabilistic predictions, but on explanations. So real science is better thought of as the growth or integration of knowledge, rather than probability calculations.
So what's wrong with Bayesianism?
Probability theory was designed for reasoning about external observations - sensory data. (for example, "a coin has a 50% chance of coming up heads"). In terms of predicting things in the external world, it works very well.
Where it breaks down is when you try to apply it to reasoning about your own internal thought processes. It was never intended to do this. As statistician Andrew Gelman correctly points out, it is simply invalid to try to assign probabilities to mathematical statements or theories, for instance.
Can an alternative mathematical framework be developed, one more in keeping with the ideas of David Deutsch and the coherence theory of knowledge?
I believe the answer is yes, and I am going to sketch the basic ideas for such a framework.
The basic idea is to separate out levels of abstraction when reasoning (or equivalently, levels of recursion). In my proposed framework, there are 3 levels, and each level gets its own measure of 'truth-value'. All reasoning must terminate in a Boolean truth value (True/False) at the base level but the idea is that different forms of reasoning correspond to different levels of abstraction.
For full reflection, you need three different numbers: a Boolean value (T/F) at the base, a probability value (0-1) at the next level of abstraction, and an entirely new measure called conceptual coherence at the highest of abstraction.
As a rough working definition of conceptual coherence, I would define it thusly;
"The degree to which a concept coheres with (integrates with) the overall world-model."
It should now be clear what's wrong with Bayesianism! It only gets us to the 2nd level abstraction! There is not just uncertainty about our own knowledge of the world (probability), there is another meta-level of uncertainly; uncertainty about our own reasoning processes, or logical uncertainty. Bayesianism can't help us here. Conceptual coherence can. Lets see how:
All statements of the form:
‘outcome x has probability y’
can be converted into statements about conceptual coherence, simply by redefining ‘x’ as a concept in a world-model. Then the
correct form of logical expression is:
‘concept x has coherence value y’.
The idea is that probability values are just special cases of coherence (the notion of coherence is more general than the notion of probabilities).
To conclude, conceptual coherence is the degree with which a concept is integrated with the rest of your world-model, and I think it accurately captures in mathematical terms the ideas that Deutsch was trying express, and is a more powerful method of reasoning than Bayesianism.
A couple of comments are saying, "no duh" to this article, but those folks likely don't realize quite how many other people are falling into this trap. "Garbage in, garbage out" is only good advice when the person you're saying it to realizes they're putting garbage in.