Your example shows that simple statistical tests aren't always that simple. Especially when talking about small effects.
What you really want are confidence intervals which show what would be a significant change. You can calculate that from your A-data and from your B-data. If they overlap you probably aren't quite there yet.
Comparing A/A vs. B or A/A/...A/A vs. B/B/...B/B is a poor man's approach to visualize the distribution of the mean values.
Things get further complicated when doing a lot of tests. If you do hundreds of A/B-Tests and a handful show a weakly significant result that may actually be a statistical fluke. The likelyhood that a wrong seemingly significant result is present when doing hundreds of tests can actually be pretty high. You should rerun these tests with fresh data and check for consistency, which in itself is some kind of A/A/B/B-test.
What you really want are confidence intervals which show what would be a significant change. You can calculate that from your A-data and from your B-data. If they overlap you probably aren't quite there yet.
Comparing A/A vs. B or A/A/...A/A vs. B/B/...B/B is a poor man's approach to visualize the distribution of the mean values.
Things get further complicated when doing a lot of tests. If you do hundreds of A/B-Tests and a handful show a weakly significant result that may actually be a statistical fluke. The likelyhood that a wrong seemingly significant result is present when doing hundreds of tests can actually be pretty high. You should rerun these tests with fresh data and check for consistency, which in itself is some kind of A/A/B/B-test.