Power and Sample Size

How many replicates a planned experiment needs to detect the effect you care about, or what power the design you can afford actually has. Thirteen tests, each answering the same question R answers, with the number checked against R itself.

What it doesCopy Anchor Link

Every comparison you plan has four quantities tied together: how many units you measure, how large the true effect is, the false-positive rate you accept (α), and the chance of detecting the effect if it is real (power). Give any three and the fourth is fixed. The panel solves whichever one you leave open: usually the sample size, but just as often the power of a design whose size is already decided by budget or by how many animals the licence allows, or the smallest effect that design could reliably see.

The effect does not have to be typed. Point the panel at a pilot table in the project, say which column is the measurement and which rows are which group, and it measures the effect itself, with the confidence interval a pilot of that size really allows. The design can then be sized at the pilot's estimate, or at the end of its interval nearest zero: the weakest effect the pilot cannot rule out, and the size that still has its power if the pilot was lucky.

The answer comes with the curve it sits on, power against sample size with everything else held, so you can see whether one more replicate buys a lot or almost nothing, and with the R call that reproduces it, so a reviewer can check the number without trusting this tool.

When you’d use thisCopy Anchor Link

• Before an experiment, to size it. You know roughly how large an effect would matter and how variable the measurement is (from a pilot, or a previous run of the same assay). Solve for the sample size, and read the whole number of replicates you need, not the fractional one.
• When the size is fixed, to know what it can see. Four mice per group, three biological replicates per condition: solve for power and find out whether the experiment can answer its own question before spending the reagents. Solve for the effect to find the smallest difference it could detect at 80% power.
• For a grant or an ethics application, which asks for a justified sample size. The R call under the result is the justification in a form a statistician can rerun.
• For an RNA-seq study, switch What is measured to RNA-seq counts, one gene: reads are counts, not normal measurements, and the panel then sizes the replicates per group from the read depth per gene, the biological variation between samples and the fold change you want to detect.
• For a whole RNA-seq experiment, switch to RNA-seq experiment, every gene. Testing twenty thousand genes is not twenty thousand independent decisions at α = 0.05: what you control is the false discovery rate, and the per-gene level that implies is far smaller than any level you would choose by hand. This mode derives it, and uses the exact negative-binomial test rather than a normal approximation, which matters most at the two or three replicates a pilot can afford.
• Reach for something else when the question is the whole transcriptome at once, at a false discovery rate: the RNA-seq calculation here is one gene at a time, at a per-gene α.

What this computesCopy Anchor Link

The design

• required sample size per group (or in total, for the tests that count that way), as R reports it and rounded up to whole units, with the power the whole number actually gives
• statistical power of a fixed design
• minimum detectable effect at a chosen power and size, in Cohen's units (d, h, r, w, f, f²) or in the measurement's own units (a mean difference, two proportions, between- and within-group variance)
• the significance level a design would need, when that is the open quantity
• the power curve: power against sample size with every other quantity held, the target power drawn across it and the solved design marked on it

The tests

• t tests: two-sample, one-sample and paired from Cohen's d; two samples of unequal size; two-sample, one-sample and paired from a raw mean difference and standard deviation, with R's strict two-sided option
• proportions: two proportions from Cohen's h, with equal or unequal group sizes; two proportions given directly; one proportion against a null value
• a normal mean with known variance, Pearson correlation, chi-square goodness of fit and independence from Cohen's w, balanced one-way ANOVA from Cohen's f or from the between- and within-group variances, and the general linear model (multiple regression) from Cohen's f²
• Cohen's conventional effect sizes, small, medium and large, for each of those effect measures

The effect from a pilot

• Hedges' g (the default), Cohen's d or Glass's Δ of one group against another, from a table of observations, with a column optionally subtracted row by row first (a reference gene's Ct, giving delta-Ct)
• the confidence interval of that effect from the noncentral t distribution, two-sided or bounded on one side, at a level you choose
• the mean, standard deviation and count of every group, the raw difference, the standardiser (pooled SD, the average of the two variances, the reference group's SD, or the SD of paired differences), the t statistic and its degrees of freedom, and Hedges' small- sample correction J
• the sample size at the estimate and at the interval limit nearest zero, side by side, and a refusal to size at all when the interval includes zero
• paired designs, where the same subjects were measured twice

RNA-seq counts

• biological replicates per group to detect a fold change at a given read depth per gene and within-group coefficient of variation, or the power, the detectable fold change, the largest CV or the α a fixed design gives
• groups of unequal size or unequal variation, where the model takes them
• a whole grid in one run: give any quantity as a list (depths 2 to 200, two fold changes) and every combination is solved, drawn as a curve of the answer along the first list with one line per value of the second, and listed in a table
• the 5/CV² depth marked on a depth curve: past it, more reads buy little and more replicates are the lever
• designs where no CV reaches the target, because sequencing noise alone at that depth uses up the variance the fold change allows, shown as empty cells and counted in the reading

A whole RNA-seq experiment, not one gene

• biological replicates per group at a target false discovery rate across every gene tested, from the exact negative-binomial test — the conditional test on the two groups' totals that edgeR runs, with no normal approximation, so it holds at two or three replicates and at low counts where the one-gene model does not
• the per-gene significance level the false discovery rate implies, reported rather than chosen: it follows from the genes tested, the genes expected to move, and the rate you set
• the power of a design you already have, at the same false discovery rate
• average power across a pilot's own genes: give the average count and dispersion of each gene and the power is computed for every one of them and averaged, instead of for one idealised gene, with each gene's own power listed
• average power when the effects themselves differ between genes, each gene's true log-fold change drawn from a normal of a mean and a spread you give, with the proportion of genes that have no effect as the quantity the answer is most sensitive to — give several proportions and the answer is reported for each
• the search curve: power against replicates per group, every size the solver evaluated, so the point where one more replicate stops buying anything is visible rather than inferred
• the dispersion, which is the square of the biological coefficient of variation, as the knob the answer turns on most

The reading of it

• a conclusion in one sentence, generated from the run, naming the test, the solved quantity and every quantity it was given
• caveats that apply to this run only: an effect taken from Cohen's convention rather than measured, a target power below one half, a one-sided test that can never declare an effect in the other direction, and R's own warnings, word for word
• for a pilot, caveats about the estimate itself: how far apart the estimate and the cautious limit put the sample size, the upward bias of an uncorrected Cohen's d, Glass's Δ standing in for a common SD that a t test assumes, and any rows left out because they held no number
• refusals, said in the same terms R uses, when no value of the open quantity inside the search range reaches the target

Uncertainty. The calculation is exact given its inputs; the uncertainty is in the inputs. An effect size taken from a small pilot is itself an estimate, and the sample size inherits all of its error. When the effect comes from a pilot in the project, that error is on screen: the interval of the effect, and the sample size at both ends of it that matter. A typed effect carries no interval, so treat that answer as the size for the effect you typed, not for the effect you will meet.

Where it livesCopy Anchor Link

Tools → Power and Sample Size opens it in the right rail. It reads no sequence: the design is the input, and a pilot table, when you use one, is any table in the project. The panel stays open beside whatever you are working on, so the assistant can read its settings and answer about the same design.

Inputs and outputsCopy Anchor Link

Input: the test, which quantity to solve for, and every other quantity of that test. The fields shown change with the test, so only the quantities that test actually takes are on screen. With Effect from set to a pilot, the effect is measured from a table in the project instead: a header row, then one row per observation, comma- or tab-separated, with the groups named either by a column or by one label per row.

Output: the solved value, the rounded-up sample size and its power, the power curve, a table of every quantity of the test by R's own name, and the R call that reproduces the answer.

What it refuses, and says so. The check runs as you type, and the reason sits under the button before you press it.

• A quantity left out that the test needs: "give df (df): pwr.chisq.test needs every quantity but the one it solves".
• A target no value can reach: "no k within pwr.anova.test's search range gives power 0.95 with the rest as given, so R refuses too", followed by R's own message.
• A pilot whose interval includes zero, sized at its limit: the refusal quotes the interval and says that no sample size covers it, and that the estimate can still be sized. A pilot group with fewer than two usable values, or a column the table does not have, is named the same way.
• A quantity the test cannot solve, or a direction it does not take: "power.t.test takes alternative two.sided, one.sided".

DemoCopy Anchor Link

There is no recording of this panel yet. The worked example below is reproducible from the panel's defaults in a few keystrokes.

Worked exampleCopy Anchor Link

A power calculation needs no file, because it is about an experiment that has not happened yet. The numbers below are from running the panel, and each one matches R 4.3.3.

You plan to compare a treated and an untreated group on a measurement whose standard deviation, in earlier runs, was about 1.2 units, and a difference of 1 unit would matter.

1. Set Test to the two-sample t test in raw units, difference to 1 and SD to 1.2.
2. Leave Solve for on the sample size, sig level on 0.05, and set power to 0.8.
3. Press Calculate sample size.

The answer is n = 23.6047 per group, so 24 in whole units, and 24 per group gives power 0.8068. The R call under it is power.t.test(n = NULL, delta = 1, sd = 1.2, sig.level = 0.05, power = 0.8).

Now suppose only four per group are possible. Set Solve for to power and n to 4: the power is 0.1682, so the experiment would miss a real 1-unit difference five times in six. Solve for difference instead and the smallest difference four per group detects at 80% power is 2.857 units, nearly three times the effect you care about.

Switching alternative to one sided on the first design drops the size to 18.52, so 19, and the caveat says why that is not free: an effect in the other direction is then never declared, however large.

Sizing an RNA-seq studyCopy Anchor Link

The RNA-seq replicates and depth sample project opens on this. Set What is measured to RNA-seq counts, give depth 20, cv 0.4 (typical within-group variation for human samples), effect 2, alpha 0.05 and power 0.8, and press Calculate replicates: 6.86 per group, so 7. The R call is rnapower(depth = 20, cv = 0.4, effect = 2, alpha = 0.05, power = 0.8).

Inbred animals vary far less: cv 0.1 brings it to 1.96, so 2. Turned around, three human samples per group (n 3, power left empty) detect a 2-fold change with power only 0.457, and the smallest fold change they detect at 80% power is 2.85.

Give depth as 2, 4, 10, 20, 50, 100, 200 and effect as 1.5, 2 and the result is a curve: at a 1.5-fold change the replicates fall from 63.0 at depth 2 to 20.1 at depth 20, then only to 15.8 at depth 200, with the 5/CV² mark at 31.25 where the curve flattens.

Sizing from a pilotCopy Anchor Link

The Power from a pilot (qPCR) sample project opens on this. Its one file is a 24-reaction qPCR pilot from the pcr R package: 12 control and 12 treated wells, each with a target gene's Ct and a reference gene's Ct.

1. Set Effect from to Pilot limit nearest zero and pick the pilot table.
2. Set value to target, subtract to ref, and give the groups as twelve control then twelve treatment. Leave the estimator on Hedges' g.
3. In the design, set power to 0.8 and press Calculate sample size.

The pilot readout says delta-Ct averaged 3.640 (SD 0.357) in control and 2.956 (SD 0.276) in treatment, a Hedges' g of −2.072, 95% interval −3.041 to −1.074. At the estimate the design needs 4.83 per group (5 in whole units); at the limit nearest zero it needs 14.63, so 15, with power 0.810. The pilot is consistent with both, and the caveat says so: an effect this well measured still leaves a threefold range in the sample size. Switch the estimator to Glass's Δ and the limit moves to −0.774, and the cautious size to 28 per group, because the control group's SD alone is the larger of the two.

ControlsCopy Anchor Link

Algorithms you can chooseCopy Anchor Link

There is one method per test, and it is the textbook one: power from the noncentral t, F or chi-square distribution (or the normal, for the proportion and known-variance tests), and the open quantity found by root-finding to the same tolerance R uses. The choice you make is the test, and the test must be the one the experiment will be analysed with, because a sample size for a t test is not a sample size for a Mann–Whitney test or a mixed model.

To go beyond the thirteen, the R call under every result is the starting point: the same functions take further arguments in R, and the Python and R consoles in the editor can run them.

The scienceCopy Anchor Link

A test rejects the null hypothesis when its statistic passes a critical value set by α. If the effect is real, the statistic does not follow the null distribution but a noncentral one, shifted by the effect times the square root of the sample size. Power is the probability that this shifted distribution lands past the critical value. That is why sample size grows with the inverse square of the effect: halving the effect quadruples the replicates.

Cohen's effect sizes put different tests on one scale: d is a mean difference in standard deviations, h the difference of arcsine-transformed proportions, w the departure of observed from expected frequencies, f the spread of group means in error standard deviations, and f² the variance a set of predictors explains over the variance it leaves. His small, medium and large values are conventions for when nothing better is known (Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed., 1988). A measured effect from your own pilot is always better.

LimitationsCopy Anchor Link

• The answer is only as good as the effect and variance you give it. Both usually come from a small pilot or another lab's paper, and their error is not carried into the sample size.
• Balanced designs and textbook tests only. Unequal groups exist for the t test and two proportions; ANOVA is one-way and balanced. Repeated measures, mixed models, clustered designs and multiple-testing corrections across many genes are not modelled.
• A pilot gives the effect, not a guarantee. The interval assumes the pilot's observations are independent and roughly normal within each group; a pilot of technical replicates measures the assay's noise, not the biological variation the real experiment will meet.
• The one-gene RNA-seq model is an approximation. It is closed-form for a single gene at a per-gene α. When solving the replicates it assumes two groups of equal size, and when solving the CV, one CV for both groups; power counts only the tail the fold change points to, which matters only at low power. For the whole experiment at a false discovery rate, use the other RNA-seq mode, which makes no such approximation.
• The whole-experiment mode holds every prognostic gene at one count and one dispersion, unless you give a pilot's per-gene counts and dispersions. It also solves power on a whole-percent grid, so a power it reports carries two decimals and no more — that is the published method's own resolution, not a rounding here. Above an average count of 2000 it fits the rejection region as two straight lines rather than summing it exactly, which is what keeps a high-count design tractable.
• Average power across genes reaches its answer through a minimiser. The replicate count it returns is exact against the published method; the power behind it is good to about 2 × 10⁻⁷.
• Simulating a whole experiment through a voom/limma pipeline is not offered. That needs the differential-expression engine, which is not here yet.
• Search ranges are R's own. When no value inside R's range reaches the target, the panel refuses as R does rather than extending the search.

Compared with other toolsCopy Anchor Link

The numbers are the same as R's pwr package and R's power.t.test, power.prop.test and power.anova.test, by construction and by test. The panel adds the curve, the rounded-up size with its real power, a plain reading of the result and a refusal that says why. It does not cover the designs dedicated packages such as G*Power cover beyond these thirteen tests.

ExportCopy Anchor Link

The result table copies like any panel table, and the R call is the portable record of the calculation: paste it into R and the same number comes back.

APICopy Anchor Link

import { runPowerAnalysis, pwrTTest, powerTTest } from '@ports/power';

// The thirteen tests through one entry point, by test id; null marks the quantity to solve.
const a = runPowerAnalysis('t_raw', { n: null, delta: 1, sd: 1.2, power: 0.8 });
a.values;      // { n: 23.6047…, delta: 1, sd: 1.2, 'sig.level': 0.05, power: 0.8 }
a.wholeUnits;  // { quantity: 'n', value: 24, power: 0.8068… }
a.curve;       // { quantity: 'n', x: [...], power: [...] }
a.rCall;       // 'power.t.test(n = NULL, delta = 1, sd = 1.2, sig.level = 0.05, power = 0.8)'

// Or each function directly, with R's argument names in camelCase.
pwrTTest({ d: 0.5, power: 0.8, n: null }).values.n; // 63.7656…

AutomationCopy Anchor Link

• Workflow builder: the node is calculate_power, with the same parameters as the panel and typed outputs for the solved value, the power, every quantity, the whole-unit size, the power curve and the R call. estimate_effect_size is the pilot: a table in, the effect, its interval, every group, the design calculate_power takes and the sizing at both ends out. calculate_rnaseq_power is the RNA-seq calculation, with the whole grid, the curve and the R call as outputs.
• The assistant can call the same tool by the same name, and reads the panel's current settings, so "is this design enough?" is answered about the design on screen.
• HTTP API and MCP expose the same tool, and the MCP result carries the same generated conclusion and caveats the panel shows.

ProvenanceCopy Anchor Link

Every result carries the R call that reproduces it. To cite the method, cite the test's source: Champely's pwr package for the Cohen-effect tests (Champely S., pwr: Basic Functions for Power Analysis, CRAN, version 1.3-0) and R's stats package for the raw-unit tests, with Cohen (1988) for the effect-size conventions.

Every function was checked against R 4.3.3 and pwr 1.3-0 on 10,935 designs: every solved value and every refusal matches, and the refusal is R's own message. The pilot's effect sizes and intervals are the effectsize package's (Ben-Shachar, Lüdecke and Makowski, Journal of Open Source Software 5, 2815, 2020; version 1.0.3), checked against it on 705 cases: every estimate matches, and every interval limit agrees to within the tolerance of R's own search for it. Cite Hedges and Olkin (Statistical Methods for Meta-Analysis, 1985) for the small-sample correction. The RNA-seq calculation is Bioconductor's RNASeqPower (version 1.52.0), checked against it on 842 grids with every cell matching; cite Hart, Therneau, Zhang, Poland and Kocher, Calculating sample size estimates for RNA sequencing data, Journal of Computational Biology 20(12), 970–978 (2013).

The whole-experiment mode is Bioconductor's RnaSeqSampleSize (version 2.23.0), checked against it on 1,070 cases — every power, every sample size and every intermediate quantity matching value for value; cite Zhao, Li, Guo, Sheng and Shyr, RnaSeqSampleSize: real data based sample size estimation for RNA sequencing, BMC Bioinformatics 19, 191 (2018), and Robinson and Smyth, Small-sample estimation of negative binomial dispersion, with applications to SAGE data, Biostatistics 9(2), 321–332 (2008) for the exact test it powers. Average power across genes is CRAN's ssizeRNA (version 1.3.3), checked on 864 sample sizes, all matching, and 19,872 powers; cite Bi and Liu, Sample size calculation while controlling false discovery rate for differential expression analysis with RNA-sequencing experiments, BMC Bioinformatics 17, 146 (2016), and Liu and Hwang, Quick calculation for sample size while controlling false discovery rate with application to microarray analysis, Bioinformatics 23(6), 739–746 (2007) for the step that turns a false discovery rate into a per-gene level.

ScreenshotsCopy Anchor Link

No still of this panel is published yet; the worked example above reproduces the result screen in three inputs.

RelatedCopy Anchor Link

• Follows it: qPCR and every other measurement panel, once the replicates are run.
• In this group: Barcode & Index Design, for sizing a multiplexed sequencing run once the number of samples is known.