Choice of parameters for an R&R study and its impact on the measurement capability

. When you perform a traditional R&R (Repeatability and Reproducibility), the first practical question that comes to mind is “How many times “n” do we repeat the measurements, and how many operators “p” should we consider?”. As the quality of the standard deviations obtained during this study depends on the parameters n and p, a large number of measurements is required. This is obviously not possible due to environment, time and cost aspects. The question is then, how to optimise the parameters “n” and “p”, for an R&R study? To answer this question, statistics are useful tools, which, combined with your practical skills, will help you to understand the outcome of the study and rationalise your choice.


What is an R&R study?
An R&R study is a simple experimental design allowing an experimental study with two sources of variations of a measurement system: the Repeatability and the Reproducibility, hence its name. The definition of Repeatability is usually well understood. It is the observed variation over several successive measurements, with fixed parameters. The VIM [1] gives the following definition: Condition of measurement, out of a set of conditions that includes the same measurement procedure, same operators, same measuring system, same operating conditions and same location, and replicate measurements on the same or similar objects over a short period of time This definition provides information on the condition of measurement, but misses its purpose. The repeatability conveys two aspects. First, it gives the minimum dispersion of the studied process. Independently of the chosen parameters (operators, etc.), the measurement process will always have at least this dispersion. It is similar to the minimum background noise of the measurement. Second, it shows the dispersion of parameters, which impacts the measurement and what we do not control.
The definition of Reproducibility is a little bit more problematic. For the VIM, all parameters must vary:

Condition of measurement, out of a set of conditions that includes different locations, operators, measuring systems, and replicate measurements on the same or similar objects
This definition is used in ISO 5725 [5] to describe the dispersion of the global result. However, it contrasts with other documents such as the MSA [2], where the reproducibility consists in varying one parameter (e.g. the operator).
In the following sections, for the sake of clarity, I only use repeatability, intermediate measurement precision, and total dispersion.

Why do we do an R&R study?
An R&R study helps to understand our measurement process. In a dispersion study, as the repeatability is our minimal dispersion, we can identify a source of variation (or noise) by comparing our results to our repeatability. Without our repeatability, we cannot conclude if this dispersion is pure noise or a proper source of variation. The intermediate measurement precision helps to characterise the impact of one specific parameter. It usually represents the inter-operator dispersion in the mechanical industry, but it can also be an interequipment, or an inter-time dispersion. Therefore, the choice of the parameter for the intermediate measurement precision depends on what we want to observe. In general, we choose the most impacting parameter on the dispersion of the measurement process.
With both repeatability and intermediate measurement precision of the most impacting parameter, we obtain an experimental estimation of the measurement uncertainty. This estimation may be coarse, but remains easy to determine; we consider the other sources of variation negligible. By comparing this "uncertainty" U to the tolerance we want to measure, we determine the process capability C. We have just described the conventional approach to estimate the capability of a measurement process: The deficiency of an R&R is that it is restraint to study one and only one parameter (addition to the repeatability). The fact that it is relatively simple to implement is, however, a real benefit of the method. If we want to study several parameters, we can use more advanced experimental designs, such as the ANOVA (ANalysis Of Variance). The last but not the least, an R&R study gives information on the dispersions only, not the trueness of the measurement.

Balanced R&R study
An R&R study is a balanced experimental design where the measurements are structured in a table (see Table 1). Note : In the following sections, « n » is the number of trials, while « p » is the number of operators.
We can ask ourselves one question. If we keep the same amount of measurements (n x p = 12), why not asking the operator A to do 9 measurements (to estimate the repeatability), and ask the three other operators to do only one measure each (to estimate the intermediate measurement precision)? The answer is simple. This approach leads to a loss of information. The obtained repeatability would be the operator A's repeatability, which cannot be compared to the others' repeatability (as we have only one measure for the operators B, C, and D). We also have little knowledge regarding the dispersion of the other operators B, C, and D (again, no repeatability to set the basis). By balancing the experimental design, we check the homogeneity of the dispersion for each operator. We also have a better knowledge of their measurement without losing too much information on the final repeatability.

Quality of the estimators
Experimental design only provides information on the repeatability and the intermediate measurement precision. The quality of these estimators depends on the large number of measurements. How many measurements are required and how to divide them up in an experimental design?

Degree of freedom
The degree of freedom is a statistical tool used to tackle this problematic. It can define the quality of an estimator based on the number of independent samples (i.e. data). The degree of freedom for a mean of n independent samples is n.
The degree of freedom for a standard deviation of n independent samples is n-1. Indeed, we look for the behaviour of the differences from the mean   i x x  : Each sample is compared to the mean, which it is part of. The n differences are thus not completely independent; if one xi changes, the mean is impacted and thus all differences as well. We say that we lose a degree of freedom.
The determination of the degree of liberty for the different parameters in an R&R study is described in the following section.

Degree of freedom for the mean of standard deviations
To determine the repeatability of an R&R study, we average the mean of the variances of each operator: with si, being the standard deviation of n trials, and p, the number of operators.
The degree of freedom is obtained by multiplying the degree of freedom of the standard deviation with that one of the mean, such as we have (n-1)*p.

Degree of freedom for the standard deviation of means
In an R&R study, we calculate the dispersion of the operators with the standard deviation of the means: with i x , the mean of the operator i, x , the mean of i x , and p, the number of operators. The degree of freedom is p-1. The information coming from the means is not taken into account. In fact, we only determine the quality of the standard deviation of 3 values. Intuitively, however, we would like to know the standard deviation for the inter-operator, which is not the same. The latter is obtained such as: with sop, the standard deviation of the inter-operator, and n, the number of trials by operator. In that case, the impact of the mean occurs with the repeatability, and the number of trials for the mean. The estimation of the degree of freedom is no longer straightforward (see following section). Nevertheless, as we have only p operators, the degree of freedom will not exceed p-1.

4.1.3
Degree of freedom for a sum of variances In an R&R study, we carry out sums of variances, such as: The estimation of the degree of freedom for sR is not simple. It is not only dependent on the degree of freedom of sOP and sr, but also on their values. For example, let us have a measurement process for which the dispersion results from the inter-operator impact. An R&R study is realized with p = 3 operators and n = 100 measurements by operator. In that case, the repeatability is well defined, but the knowledge of sR is limited by the number of operators, its degree of freedom will be close to 2 (p-1). An approximation of the degree of liberty can be obtained with Welch-Satterthwaite's equation [3], with independent standard deviations: with dli, the degree of freedom of the standard deviation.

Degree of freedom in an R&R study
In a classical R&R study, we can easily determine the degrees of freedom for the repeatability and standard deviations of means (close to the inter-operator standard deviation, but more complex to determine). For a total number of measurements, we can estimate the impact of the choice of n and p. We can see that the repeatability is far less impacted than the standard deviation of the means. Thus, in an R&R study, for a total number of measurements (often restraint for a matter of cost), it is better to increase p than n.
Of course, we need to take into account business and technical constraints. For example, it is not always convenient to monopolize 9 operators for an R&R study. It is also very important to evaluate the ratio of each standard deviations. If we know that the operator has no impact, we would rather focus on the knowledge of the repeatability and thus n.

Evaluation of the estimator
The degree of freedom provides a quality scale for the estimator, but how to obtain a proper range, which is easier to understand physically?

4.2.1
Distribution of the estimator By definition, an estimator gives a representative value of the true value we are looking for. By repeating an experience, we obtain several estimators, probably slightly different from one another. The goal is to determine the variations of the estimators around the true value. This information defines the quality of the estimator. In fact, we look for the distribution of the estimators.

4.2.2
Khi-square (χ 2 ) distribution Statistics provide interesting information. The experimental variance coming from n samples, has a distribution proportional to a χ 2 distribution with n-1 degree of freedom. The distribution depends of course on the quality of the estimator, and thus on the degree of freedom studied in the previous section.

Figure 1 : Distribution and confidence interval
As we know the properties of the χ 2 distribution, we can calculate the confidence interval of the variance.

4.2.3
Complex cases When the standard deviation comes from several variances, the study is more complex. The literature [4] gives approximate values. ISO 5725-1 [5] provides a table giving the relative error of dispersions as a function of n and p.

MSA R&R study
The MSA recommends to take for an R&R study 10 manufactured parts (q), each measured 3 times (n) by 3 different operators (p), to finally have 90 measurements. The additional factor introduces the measurand effect (shape defect for example) within the repeatability. This is why the MSA is suggesting taking 10 parts in order to have a representative sample of possible imperfection.
If we know that there is no impact of the parts on the measurement, it becomes possible to adjust the number of parts required.
Degrees of freedom of these elements: df(EV) = (n-1)*p*q = 60 (10) df( x s ) = p-1 = 2 (11) with x s : standard deviation of the operator's average, close to the value of AV The obtained EV is relatively reliable but that the quality of the AV is relatively poor. It is a pragmatic choice made for the world of automotive production measurement, mainly dimensional. The measurement uncertainty in this case is mainly due to the defect of shape of the produced parts, the repeatability. The inter-operator effect is lower, but above all, it can be difficult to monopolize several production operators in order to perform a capability measurement test. A compromise must be made in the R & R parameter choice with the following elements: -Industrial constraints (cost, time of measurements) -Main dispersion source (based on expertised) -Degree of freedom (equation (10) and (11))

Exemple 1:
There is no form defect known on the parts, so the number of part can be reduce to 5. The dispersion of the measurement is mainly due to the inter-operator effect and 5 operators can be monopolized.
The optimize experiment study : q = 5 number of parts p = 5 number of operators n= 3 number of trial Total = 75 measures df (repeatability)= 50 df( x s )=4 In this case, the repeatability remains of good quality, the inter-operator effect is better defined and the total number of measurements is reduced.

Exemple 2 :
The process is mainly impacted by repeatability, little operator effect. The parts have no form defect and it is a small series production where parts are expensive. The optimize experiment study: q = 3 number of parts p = 3 number of operators n= 10 number of trial Total = 90 measures df (repeatability)= 81 df( x s )= 2 In this case, with the same number of measures as a classical MSA, the quality of repeatability is improved.

Conclusion
To choose the parameters of an experimental study and to improve the relevance of the results, we must not only rely on the statistical properties such as the degree of freedom, but we must also understand the physical impact on the results and have expertise in the field of the expected result to guide the decision.