This is the second time I've ever tried to describe at length what's so important and fascinating about Bell's Theorem, but my first time was a semi-failure (partly because I was trying to explain it to someone who rejected QM out of the gate, and didn't want to understand the maths and probabilities involved - which makes this scenario fundamentally different out of the gate, you guys seem to like QM and have some understanding of the probabilities involved), so this time I'm going to start again without using any of the material I wrote the first time. I'm going to probably be writing some shit you already know, but please bear with me. Also, sorry if this is a lot to read.
I think the best way to contextualize Bell's Theorem is to go back to the beginning, back to when QM was first being introduced to the physics community. I'm going to spin a little narrative that's perhaps not entirely true, but hopefully true enough, to set the stage for why Bell's Theorem was even thought up to begin with.
Before QM, all physical theories were what we now call "classical", including Relativity. I think of "classical" as almost being comparable to basic object-permanence. We all learn at some age that, if we put a book in our backpack and then close our backpack, we can (usually) expect to find it in our backpack later on. When we open up our backpack an hour later, we generally don't assume that the book just appeared there - we have a persistent conception of the world where, even when we weren't looking at the book, it was still in the backpack all the same. Classical Theories are object-permanence at the universal scale - every particle that exists always exists in a specific place at every moment in time, even when we're not looking at it. Our ignorance of where a particle is or how fast it's moving is just a fact about us, not a fact about the particle itself. The particle itself is always existing somewhere, and moving at some specific velocity, at every moment in time, regardless of our ignorance about it.
Then, Bohr, Planck, Heisenberg, Schrodinger and friends introduced QM to the world. They said that there are some properties of particles that, prior to measurement, we can't actually tell a Classical story about. If we shoot a particle at t=0, through a double slit for example, and measure where that particle landed at t=100, we might be tempted to ask the question "ok, so where was that particle at t=50?" If the world worked classically, then there would be an objectively true answer to that question - even if we as human beings couldn't find an answer. If we can't find an answer, that's just our own ignorance, but there still *is* an answer. QM said, actually, there *is not* an answer. Or at least, not a *singular, definite answer* -- that's the phrasing I like to use. Prior to measurement, some of these properties of things like Photons and Electrons do not in fact have singular definite answers - not even to God. If God himself were to peer into the universe and look at that particle at t=50, he wouldn't have a singular definite answer to the question "where was that particle?" (Please note that I'm using God as a narrative tool, I'm not a theist. "God" is just a stand in for the idea of some external entity who could, in principle, know the world as it really is - could answer any question about any system without disturbing that system).
Shortly after, Einstein and friends produced the EPR paper. In short, they fundamentally disagreed that the QM vision of the world was true. They said, no, if we don't know where a particle is, or its velocity, or any other measurable property of that particle, that's just a matter of personal ignorance. Quantum indeterminacy is not a real feature of reality, it's just a measure of the things we don't know. And they came up with a theoretical class of experiment to demonstrate why. They said (again, I'm using some artistic license here, this isn't exactly how it happened) -- imagine we have a pair of particles that we've arranged to have a correlated state in some measurable property. Now, we have one particle flying east and one flying west, we haven't measured this property yet so according to QM this property doesn't have a singular definite answer yet. After, say, a second we measure the particle flying west -- if their properties are correlated, as by the experimental design, then that means at the moment we measure the West particle, the East particle must also suddenly and immediately have an answer as to what it will be measured as as well, despite being 2 light-seconds away. This thought experiment, as far as I know, is the first time the idea of "entanglement", as we now call it, was discussed at length. Einstein referred to the idea that measuring one particle could affect its entangled paired particle immediately as "spooky action at a distance". The idea of instantaneous causality across space went against every intuition about physics Einstein had - it went directly against intuitive notions of Causality and his own Relativity. After all, if cause-and-effect can happen immediately across distances, that creates a real problem for the idea of Relativity of Simultaneity (please ask if you want more detail on this and why it was such a problem).
So, what we have here is 2 competing ideas: 1. the classical take of Einstein via the EPR paper, that us not knowing what a property would be measured as is just a statement about our own ignorance, not a statement about reality, and 2. the QM take, that these properties are not just something WE are ignorant about - God himself wouldn't have an answer. There objectively is not an answer to these quantum questions prior to measurement. And the tricky part here is, how could you possibly tell? How could we tell what type of universe we live in? A classical one or a QM one? What experiment could tell us the difference between "I don't know this property of this particle" and "this property of this particle genuinely does not have a singular definite value"? On the surface, those two ideas - personal ignorance vs ontoloical lack of an answer in reality - seem experimentally indistinguishable.
THIS is where Bell's Theorem steps in. Bell ingeniously figured out a way to definitely tell us if we live in a world where a lack of an answer to quantum questions was because of personal ignorance, or if in fact QM was right and ontologically the answer to this quantum questions *cannot* in fact have a singular definite answer prior to measurement. This, to me, is why Bell's Theorem is so wonderfully beautiful, and so centrally important - it took us 40-odd years to finally find a way to settle the disagreement betweein Einstein and QM. Thank you, John Bell.
When the EPR paper came out, I'm not quite sure personally how they imagined setting up an entangled pair of particles, but by the time John Bell was approaching the question, I believe physicists had a good idea about this concept of "spin" - that you could create a pair of entangled particles where, due to the perservation of the quantum equivalent of "angular momentum", you could guarantee that one particle had the opposite spin of the other particle. I actually don't believe they had experimentally confirmed this could be done at this point, I think it was still theory when Bell came up with the idea, but nonetheless this is where we can start untangling the disagreement between Einstein and QM. What Bell discovered was, Quantum Mechanics predicts certain correlations of spins at varying degrees of measurement, and the exact correlations it predicts are *incompatible* with Einstein's view of the world. If you read the wiki page on Bell's Theorem, you'll see it referred to as "local hidden variables" - local hidden variables, Bell figured out, could not (at least not without some weird loopholes) explain the statistical results that QM predicts. So let's go into why and what that means.
First, let's just talk about what you alread know: in a Bell Test, they create a pair of particles that have entangled spin. One of them goes left, one of them goes right. If you measure their spin along the same axis, if they're properly entangled then one spin will be up and one will always be down. If you measure them at a different angle, however, the statistics start to differ a little bit.
Here's where I introduce the "analogy" I've been touting prior. Local Hidden Variables and entanglement. I'm absolutely certain I'm oversimplifying this but I think the simplification I'm going to present is useful at the very least, so here it goes: Local Hidden Variables theories are basically Classical theories, which is to say they match this idea of Object Permanence I talked about - regardless of our own ignorance, every property of a particle has a singular definite value at all moments in time. Now the idea that two particles could have some correlated value isn't itself incompatible with Classical theories, so here's the first layer of the analogy: A person called FJ (me) is offering a service online. You send me 2 addresses and a dollar, and I'll send an envolope with a piece of paper in it to each address. The service is very simple: to one address I'll send an envelope with a red piece of paper inside, and to the other addrss I'll send an envelope with a green piece of paper inside. This is "classical entanglement", aka the Local Hidden Variables view of entanglement. You purchase my services and send me your address and T.Clark's address, and when you get your envelope you open it and you see a Green piece of paper. You have *immediately and instantaneously* learned information about the envelope heading to T.Clark's house. You know for a fact that the paper going to him is Red, regardless of the fact that it hasn't been measured yet. This isn't magic and it's not quantum, this is the classical view of entanglement.
So, the analogy is a "particle" is an envelope, and a "measurement" is opening the envelope here, and in a classical system, there's nothing weird or strange about the idea that the contents of one envelope could be perfectly correlated with the contents of another envelope, right? I hope that all makes sense. We consider it classical because, even if you opened the envelope, say, 24 hours after I mailed it off, you could reasonably ask the question "what color was the paper in this envlope 12 hours after it was sent?" and, common sense says, it was green when you opened it and so it was green 12 hours before you opened it as well. And 23 hours before you opened it. And 1 hour before you opened it. When you opened this envelope, you weren't generating some new fact about the color of the paper, you were discovering a fact that was true the whole time - that's what makes it classical. Complete object permanence. Objective facts regardless of your ignorance.
I send you an email a few days later letting you know I'm offering a more advanced service. I'm offering to send the 2 envelopes with a paper inside, same as before, but this time I'm going to write a number on each one. On the first paper I write a number from 0 - 359 - let's call this number X. On the second paper, I'm going to write whatever this formula outputs: (X + 180) MOD 360. So, whatever the first number is, the second number is 180 degrees rotated from the first number. If the first number is 1, the second number is 181. If the first number is 90, the second number is 270. If the first number is 320, the second number is (320 + 180) mod 360 which is 140. Now this service I'm offering is *almost* directly comparable to the experiment in a Bell test. The only thing you have left to do to make it actually comparable to a Bell test is devise a *measurement scheme* that makes it similar to a Bell test, which is actually remarkably easy.
Like before, you ask me to send one envelope to your house and one to T.Clark's house. You've agreed with T.Clark for the first round to measure every number received according to this scheme: If the number is between 0 and 180 (including 0, not including 180), you record an UP on your spreadsheet, and if it's from 180 - 360 (including 180, not including 360 aka 0) you'll record a DOWN on your spreadsheet.
So, you run the test 1000 times, say, and after you compare your spreadsheet with T.Clark's spreadsheet, you're completely unsurprised to discover that every time you've recorded UP, he's recorded DOWN, and vice versa. Again, this is just basic, classical entanglement. These envelopes are classical envelopes, filled with classical paper, written on with classical pen. You might measure a particle as UP after you receive it, but the number was already written in pen long before you received it. Nothing weird here.
But meanwhile you and T.Clark have also set up some entangled quantum photons to arrive at your houses, and you're measuring their spins, and you've noticed with the entangled particles that when you run the test above, measuring their spin at the same angle, you get the same results as the envelope experiment - every result you see as UP, T.Clark sees as DOWN, and vice versa. So in this case, our classical experiment is looking a lot like our quantum experiment.
So, you and T.Clark start changing the experiment up, and you start doing a proper Bell test. Some of the time, you measure a particle at, say, 0° and he measures it at 20°. Some of the time, you measure a particle at 20° and he measures it at 40°. And some of the time, you measure a particle at 0° and he measures it at 40°. {0° above means UP if it's [0-180) and DOWN if it's [180-360), 40° means UP if it's [40-220)}. You both record your ups and downs now, and you discover the following set of facts:
When you're measuring 0° and he's doing 20°, you BOTH record UP 5.8% of the time (as in, 5.8% of the time yours and his both register UP for the same run).
When you're measuring 20° and he's doing 40°, you BOTH record UP 5.8% of the time.
When you're measuring 0° and he's doing 40°, you BOTH record UP 20.7% of the time.
Now, you decide to run the same sort of tests using FJ's mailing service just to see what the results are, to compare your quantum measurements to a classical system. Here's what you get.
When you're measuring 0° and he's doing 20°, you BOTH record UP 5.55...% of the time.
When you're measuring 20° and he's doing 40°, you BOTH record UP 5.55...% of the time.
When you're measuring 0° and he's doing 40°, you BOTH record UP 11.111...% of the time.
You notice this, tim, and you have a little intuition: you think that in my classical mailing service, it's actually impossible for my classical mailing service to produce the 5.8, 5.8, 20.7% statistics from the quantum tests. You think that it might be the case that the 0-20 and 20-40 statistics might have to add up to the 0-40 statistics, but maybe you can't quite explain why yet. You intuitively think there's no way for me to recreate that distrubituion. So, you tell T.Clark and he disagrees (I'm sure you wouldn't actually disagree, mr Clark, it's just a story). So you make a bet - you tell T.Clark that HE can call up and tell me (FJ, the mailing man) exactly what numbers to put in the envelopes, BUT you, tim, YOU get to decide how they're going to be measured (0 and 0, 0 and 20, 20 and 40 or 0 and 40), and you decide that 12 hours after the envelope is sent off, so T.Clark and FJ have no way of knowing which angles you're going to measure them at. So the bet is this: under those conditions, can T.Clark and FJ create a scheme that will make it so that at 0 and 0, all the result are opposite, at 0 and 20 they're both up 5.8% of the time, and 20 and 40 they're both up 5.8% of the time, and at 0 and 40 they're both up 20.7% of the time?
So, at this stage, what do you think? Can FJ and T.Clark devise any system of sending these letters to you in the conditions given? Is there any strategy at all they could take to produce these distributions reliably?
Bell's Theorem says there isn't. The only out that I know of, the only loop hole, is that if T.Clark and FJ already know ahead of time how you're going to be measuring them, THEN FJ and T.Clark can conspire to rig the results (this is essentially what Superdeterminism means, it means that the mechanism creating the spin values knows ahead of time how they're going to be measured and has conspired to give the results QM predicts). But I've worded it such that that loophole is closed - "BUT you, tim, YOU get to decide how they're going to be measured, and you decide that 12 hours after the envelope is sent off" - so they can't cheat in the superdeterministic way.
If there's no way for FJ's classical mailing system to produce the statistical outputs that QM predicts - that 5.8% and 20.7% numbers - then that means we can possibly prove that we don't live in a classical world.
So what do you think? Can FJ and T.Clark devise such a system, so that the mailing system can output those numbers? Can you devise such a system?