How ftdnastat works

A technical description: where the site gets its data, how it updates and checks it, and how the numbers on its pages are computed. Thresholds and rules are given as they work now.

1. Sources #

All sources are public. Names of people from the project maps never reach the site.

2. Updates and publishing #

Once a day the version of every source is checked: the tree's release date, a server response header or the root totals. Only what changed is downloaded. The site is rebuilt when the data or the code changed, and in any case once a week.

Before publishing, a build has to pass acceptance:

• every required file is in place, every page has a title, a canonical address and a twin in the other language;
• no empty or non-numeric placeholders where numbers belong, every sitemap address exists and appears once;
• pages, kits, countries and migration maps are at least 95% of the published ones, and the Y tree is not older than the published one.

A build that fails acceptance is not published. Published builds are kept in an archive.

A new FTDNA Y tree also has to pass the haplogroup predictor check before it is used: on the same data, the prediction error rate for the key branches must not grow.

3. Countries, regions and peoples #

FTDNA. A kit's country is the self-reported origin of the earliest known ancestor on the line. Regions FTDNA lists separately ("Russia (Republic of Dagestan)") are pages of their own and are not part of their parent country. Kits without a country are left out of shares.

YFull. A sample's country and region are ISO codes, its people is the native language from the customer form (ISO 639-3). The codes are matched to FTDNA countries and regions; a region that has its own FTDNA entry takes the sample away from the country, as FTDNA does. A language that matches the sample's country or region is not counted a second time.

The comparable YFull slice. Comparisons with FTDNA use YFull customers only (YF… numbers) and modern samples only: published studies and ancient DNA are sampled differently from customers.

4. Branch tree and shares #

Shares are cumulative: a branch row includes all kits of its sub-branches. So nested branches never stand side by side as neighbours, and a kit whose test did not go below a node stays on that node with an asterisk (J-M172*).

• Y, major branches. A node becomes a major branch when its subtree holds at least 0.4% of the world's kits with a country; chains where the only sub-branch holds almost everything are collapsed.
• Y, country tree. Nodes from the dictionary of labelled branches (J1 → Z1842 → CTS1460), cumulative, from 1% of the sample and at least 3 kits; a node whose only shown sub-branch holds ≥ 95% of its kits gives way to it. Up to 30 rows on the page, 14 on the card.
• mt. Letter, branch such as H1a, terminal node; the tree goes by groups of one letter.
• Small samples. A country with fewer than 50 kits is flagged and left out of findings; below n = 30 a share shows its 95% Wilson interval.

5. Branch pages #

Every branch whose subtree holds at least 50 kits with a country has a page: currently 4,297 Y and 1,182 mt branches. Chains of equivalent nodes, where the only sub-branch holds ≥ 99% of the kits (Y) or ≥ 97% (mt), become one page listing the equivalent names.

• Countries and map. The branch's share in each country and the country's share of the branch; a map of shares by country.
• Sub-branches as a tree. Sub-branches with pages of their own: every direct one, nested ones from 1% of the branch's kits (at least 3); chains collapse by the same 95% rule, up to 30 rows.
• Age. The mean TMRCA estimate of FamilyTreeDNA Discover with its range; next to it YFull (TMRCA and formed, the time the branch split from its parent, 95% intervals) and TheYtree, when their node is matched to the branch. A YFull node is matched to an FTDNA node by name or by a shared SNP.
• How the branch grew. For every moment, how many sub-branches already had their common ancestor (Discover for Y, Mitotree for mt), on a log scale. Highlighted is the window where the count multiplied most, with at least 5 sub-branches at its start; the last thousand years are left out, because there the count depends on how many relatives took a test.
• Migration path. The FamilyTreeDNA Discover model: a least-cost path between the estimated places of the ancestors.
• Ancient samples from excavations in the branch's subtree.

6. YFull: comparison, neighbours, pairs, kin across borders #

• Comparison with FTDNA. The comparable YFull slice is laid on the same FTDNA tree nodes as the kits; "agreement" is the cosine similarity of the shares, "divergence" the tree row with the largest difference.
• Similar peoples and regions. Jensen–Shannon divergence of major-branch shares; a neighbour is shown at ≥ 80% similarity and at least 150 comparable samples on both sides.
• Y × mt pairs. For people with both tests, combinations of paternal and maternal lines against what independence of the lines would give; significance from the hypergeometric tail.
• Kin across borders. Young Y branches (common ancestor no older than 500 years, at least two samples) joining people from two countries or regions. A pair's number of shared branches is compared with a random rewiring that keeps each country's number of links (1,000 rounds), so the expectation accounts for how connected a country is abroad overall. Shown are pairs with at least 5 branches, twice the expectation, p ≤ 0.01.
• New samples. YFull itself marks recently added samples and drops the mark after a few weeks; the site shows them by country, people and branch.

7. FamilyTreeDNA geographic projects #

Project members themselves turned on showing their ancestor's place on the public project map. One person may be in several projects: their records are merged into one.

• Family weight. Records with the same ancestor, branch and point count as one family and together weigh one vote.
• Cells. An ancestor's place is rounded to a 1° cell. A branch is drawn in a cell where it has at least 3 people; smaller cells count in the totals and the centre but are not drawn.
• Branch share at a place — the branch's votes against the votes of all members in the cell.
• Branch centre and 68% radius. The weighted centre of the cells in the part of the world where the branch has most votes (otherwise the diaspora in the Americas moves the centre into the ocean). A vote is weighted by the testing rate: weight = √(median member density / density in its country or region), between 1/5 and 5, so countries that test more do not pull the centre towards themselves. The radius is the distance within which 68% of the weighted votes lie.
• Branch out of place. A cell is compared with all members within 300 km around it: a binomial tail for each "cell × branch" pair, Benjamini–Hochberg correction over all pairs, q ≤ 0.05 and at least 3 people. Points shared by many people of different branches are the coordinates of a country or town, not an ancestral place, and are left out.
• Ancient against modern. For branches with ancient samples, the distance of each sample from the centre of modern carriers.

8. Ancient DNA and cultures #

Y samples come from FamilyTreeDNA Discover, mt samples from Mitotree and YFull. A sample's date is the middle of its dating range. The culture is the Discover entry; different spellings of one culture are merged.

Comparing cultures. Each sample is cut to its ancestor at the chosen tree depth, or to the ancestor whose branch already existed by the chosen year (by Discover ages). A sample whose test did not reach that depth stays on its own branch.

• "A → B" — the share of A's samples that have the same branch or a branch under it in B: a shallow sample could belong to any of its sub-branches.
• Frequency overlap — the sum of the smaller shares of shared branches; the share of a shallow branch is split among its sub-branches in proportion to their shares in both cultures together.

Culture chains are shown while neighbouring cultures stay at least 10% alike; the numbers in chain texts are checked against the data on every build.

9. Findings #

Recomputed on every build, for countries with at least 50 kits:

• signature branches — a share in the country at least three times the world's, at least 10 kits;
• similar countries — Jensen–Shannon divergence of major branches, samples of 100 or more;
• single-node clusters — ≥ 10% of the sample and ≥ 20 kits on one node: usually a surname project;
• young widespread branches (younger than 2,000 years, in 5 countries or more) and old concentrated ones (older than 4,000 years, ≥ 50% of kits in one country, branches of 200 kits or more);
• the mt-to-Y ratio, diverging paternal and maternal neighbours, disagreements between FTDNA and YFull.

10. Changes #

• FTDNA by week. Each new tree release leaves a compact slice: node, parent, kits by country. Slices are matched by the internal node number, so a rename is not a removal plus an addition.
• New this month. The difference of two slices at least 30 days apart. A node gone from the newer tree hands its kits to its nearest surviving ancestor, and sums are taken over subtrees: a kit moved into a new sub-branch is not counted as new.
• YFull by month. Samples are matched by number, branches by name; a branch that changed its name but kept ≥ 80% of its samples counts as a rename.

11. Limitations #

• These are customers of the companies, not a population: the US, Britain and Scandinavia are over-represented, and the make-up depends on who chose to test.
• Geographic project members chose whether to join and which place to give.
• Ages are model estimates; the sources are calibrated differently, so they are shown side by side rather than averaged.
• YFull and FTDNA nodes are matched by names and SNPs; some YFull nodes have no counterpart and go up to the nearest matched ancestor.