Suppose you have an experiment that consists of several tests that return either “success” or “failure”. The longer you wait the more tests you can perform, but with limited time devising a condition of when do you have enough data is important.
Another question is if its even possible to gather enough data for a stable result in a reasonable timeframe.
- We assume the tests are independent from one another.
- Success rate is quite low (~0.001 to 0.01)
- We perform two different sets of similar tests (anything said above applies to both, but the probability of success for tests from different sets can vary)
- $N_1, N_2$ stand for amount of tests in two sets, while $n_1, n_2$ — for amount of successes.
- Example — $N_1=N_2=2000, n_1=15, n_2=5$.
The main question is how likely is it that both sets of tests have same, or at least similar success rate.
My first idea was to use binomial distribution with probability of success taken from one of the sets $p=frac{n_1}{N_1}$ and arbitratily choose a cutoff measured in amount of standard deviations between norm and actual results of second set for when we can assume the second set can have the same success rate.
It works sort of alright for my needs, but obviously says nothing about data verification and I’m still in darkness on whether it is actually a correct method to tackle the problem.