1. The data is all public, from All of Us. Impressive dataset. For a genome size of 3 billion bp, “all of Us identified more than 1 billion genetic variants, including more than 275 million previously unreported genetic variants, more than 3.9 million of which had coding consequences”. They also have longitudinal electronic health record which allowed them to evaluate 3,724 genetic variants associated with 117 diseases and found high replication rates across both participants of European ancestry and participants of African ancestry. You can start a genotyping company based on those 3724 variants if you work with people from those ancestries.
2. In simulation, the authors used a balanced design (36K * 3) that matches the size of the European cohort (109K). This is capped by the 36K Latino-ancestry dataset.
3. The concept of calibration. “To assess calibration, we compared the empirical FDR to (1 − PIP threshold), a conservative FDR upper bound (as in ref. 12), as well as (1 − mean PIP), the expected FDR (which has been reported to be slightly miscalibrated in previous fine-mapping simulations.”

Let’s break down this sentence. Calibration is to see whether the predicted FDR match the empirical FDR (FP/(FP+TP) measured in simulation where truth is known. Two ways of calibration are mentioned.

The first one is empirical FDR vs. (1-PIP threshold). Ref 12 is Weissbord 2020 Nature Genetics. In it, FDR is defined as “the proportion of false positives among SNPs with posterior causal probability (posterior inclusion probability (PIP)) above a given threshold (for example, PIP > 0.95), aggregating the results across all simulations”. This is to say, only SNPs with PIP > 0.95 are considered in the calculation of FDR, so in theory the FDR calculated this way should be much lower than the FDR calculated based on all the SNPs tested.

The second one is empirical FDR vs (1 - mean PIP). Here 1 - mean(PIP) is known as the expected FDR. To understand why, we need to firstly understand PIP, or posterior inclusion probability.

What is posterior inclusion probability?

As threshold of PIP(0.95) is almost always higher than mean(PIP), the first one is less tolerant of higher FDR (therefore a more conservative/lower upper bound).

Interestingly, Ref 12 is received on 28 October 2019 and Accepted on 02 October 2020. It cited SuSiE, which is also published in 2020, but it was submitted on 01 December 2018, and only accepted on 01 May 2020, from a different lab.

Published 16 November 2020

I don’t quite remember how I answered. In fact, to this day, I don’t know the answer.

Today let’s make some attempts at least try to understand the problem that we are facing.

Individual level vs. Population level

First of all, one need to understand that rare variants is a population-level concept, while sequencing error is at read level, and can be minimized at individual level.

Quality score

One obvious tool is quality score. Quality score of that base at fastq level, qulity score of the variant. If this variant got good coverage,

First Version

bcftools filter -g3 -G10 | bcftools view -i 'QUAL>20 & MAF>0.0000001' -m2 -M2

Firstly let’s explain each option separately.

• -g3: Filter SNPs within 3 base pairs of an indel (the default) or any combination of indel,mnp,bnd,other,overlap. This is because a SNP this close to an indel might be incorportated into this indel as another allele.
• -G10: Filter clusters of indels separated by 10 or fewer base pairs allowing only one to pass. Because one of the indels might be a result of the other. They should be considered together.
• QUAL>20: this is the QUAL column of the vcf file. It is phred-scaled, and the higher the better. QUAL = 20 means only 1% of the chance that this variant is false.
• MAF>0.00000001: Minor allele frequency for filtering rare variants. But here the number I set is so low, as long as you have 1 allele count, you will stay. So basically this is filtering for monomorphic/invariant calls.
• -m2: minimum 2 alleles
• -M2: maximum 2 alleles. So only bi-allelaic stays.

The problem with this version is that along the way, I lost a SNP because it was close to an indel that had low QUAL. Because -g3 -G10 goes first, I filtered that SNP with high QUAL for an indel with low QUAL. Therefore I reordered things in the next version.

Second Version

bcftools view -i 'QUAL>20 & MAF>0.0000001' -m2 -M2 | bcftools filter -g3 -G10

One obvious thing to improve is that MAF>0.0000001 the misleading. Might as well use 0 instead.

Now the problem becomes, should we filter the multiallellic (-m2 -M2) and monomorphic (MAF>0) variants upfront? For multiallellic indels, they might be true and with high quality scores, and they could be used in -g3 and -G10, and be discarded later. For the monomorphic indels, it could be everyone 1/1, monomorphic for the ALT — a fixed difference from the reference. This could mean that there is a mistake in the assembly and the indel is still true. I actually do not have a strong preference of whether this should go earlier or later. How do you think?

Third Version

bcftools view -i 'QUAL>20' -m2 -M2 | bcftools filter -g3 -G10 | bcftools view -i `MAF>0` -m2 -M2

What would you have done differently?

This is the full annotation of this variant through SnpEff with some editing so it is not a real variant anymore. But the problem is true.

ANN=G|missense_variant|MODERATE|START_CODON_1_1000_1002|g12345|transcript|g12345.t1|protein_coding|11/15|c.721A>G|p.His241Asp|1826/2931|721/2562|241/853||WARNING_TRANSCRIPT_MULTIPLE_STOP_CODONS,G|synonymous_variant|LOW|START_CODON_1_1000_1002|g12345|transcript|g12345.t2|protein_coding|11/14|c.1803A>G|p.Val601Val|1803/2682|1803/2682|601/893||,G|intragenic_variant|MODIFIER|GENE_1_1000_28598|GENE_1_1000_28598|gene_variant|GENE_1_1000_28598|||n.23957A>G||||||,G|non_coding_transcript_variant|MODIFIER|TRANSCRIPT_1_1000_28252|null.22587|transcript|TRANSCRIPT_1_1000_28252|pseudogene||||||||,G|non_coding_transcript_variant|MODIFIER|TRANSCRIPT_1_1000_28598|null.22586|transcript|TRANSCRIPT_1_1000_28598|pseudogene||||||||

We can see that it is very long. There are multiple transcripts that it is involved in, separated by comma (,). Let’s put them into a table:

So one variant produced five annotations. Two questions jump out:

1. why are there five entries for what is really one gene with only two transcripts
2. a missense is nice but why is g12345.t1 flagged with WARNING_TRANSCRIPT_MULTIPLE_STOP_CODONS?

Where entries 3, 4, 5 come from

The strings pseudogene, null., TRANSCRIPT_1_… and GENE_1_… do not appear anywhere in the GTF. SnpEff fabricated entries 3–5 at database-build time because of how the BRAKER GTF writes its parent lines. Compare the gene/transcript lines with their child features:

gene        … 1000 28598 … +  .  g12345                                       ← bare ID
transcript  … 1000 28598 … +  .  g12345.t1                                    ← bare ID
CDS         … 1000 1449 … +  0  transcript_id "g12345.t1"; gene_id "g12345"; ← proper attributes
exon        … 1000 1449 … +  .  transcript_id "g12345.t1"; gene_id "g12345";

SnpEff’s GTF parser expects key "value"; attribute pairs. The gene and transcript lines in the raw BRAKER gtf output instead put a bare ID in column 9, so the parser extracts no ID and falls back to synthetic, coordinate-based markers. As a result SnpEff parses the locus twice:

• the properly-attributed CDS/exon/intron lines correctly rebuild g12345 with transcripts .t1 and .t2 (entries 1 and 2) because there are proper “transcript_id” and “gene_id” as the key.
• each unparseable gene line becomes a childless phantom gene GENE_chr_start_end → intragenic_variant (entry 3), and each unparseable transcript line becomes a phantom transcript TRANSCRIPT_chr_start_end wrapped in an invented null.N gene. With no CDS attached, these default to pseudogene / non_coding_transcript_variant (entries 4 and 5).

This is genome-wide: every gene and transcript line in the file uses the bare-ID format, so the database ends up with a lot of GENE_* and null.* phantom records. The fix is to normalise the attribute column (give the parent lines real gene_id "…"; transcript_id "…"; fields) and rebuild. Entries 3–5 then disappear, leaving only the two real transcripts. Here normalise means converting records to a canonical/standard form in data/file engineering.

The real puzzle: the g12345.t1 warning

With the phantoms gone we are left with entries 1 and 2 — the same variant on two transcripts of the same gene: missense (p.His241Asp) on .t1 and synonymous (p.Val601Val) on .t2. I mentioned in the beginning that when I tried to apply this mutation to the protein sequence, the sequence did not change. Why did the missense go missing? Is it related to the WARNING_TRANSCRIPT_MULTIPLE_STOP_CODONS that .t1 carries?

Let’s take a look at the gtf of this gene. The difference between the two transcripts is that .t1 carries UTR features added by stringtie2utr and .t2 does not. Sorting .t1’s features by position shows the problem:

# transcript g12345.t1  
AUGUSTUS       start_codon     1000  1002
AUGUSTUS       CDS             1000  1449
   …
AUGUSTUS       CDS             20367  20499   (phase 2)
stringtie2utr  five_prime_UTR  20555  20577   ← a 5' UTR ~19 kb INTO the coding region
AUGUSTUS       CDS             20578  20731   (phase 1)
   …
AUGUSTUS       stop_codon      28250  28252
stringtie2utr  three_prime_UTR 28253  28598
# transcript g12345.t2 has the same CDS but no UTR lines

A 5′ UTR is by definition upstream of the start codon. This one sits in the middle of the CDS, downstream of the start codon and inside an intron. When SnpEff reads an explicit five_prime_UTR, it uses it to set the coding start — the new start codon would be right after the 5′ UTR ends. So SnpEff treats everything from the true start codon (1000) up to 20577 as untranslated and begins translation at 20578, in the wrong frame. In this way, the mutation would result in missense and multiple stop codons thus the warning.

The coordinates also confirm it. Counting the variant (genomic position 23957) from the bogus coding start at 20578:

20578–20731 (154) + 20967–21047 (81) + 21373–21495 (123)
+ 22568–22738 (171) + 22819–22953 (135) + 57 bases into 23901–23996
= 721  →  codon 241

That is precisely the c.721A>G | p.His241Asp of entry 1 — computed against the broken frame. Counting the same variant from the correct start codon (1000), the way .t2 does, gives c.1803A>G, codon 601, at the wobble (3rd) position — a synonymous A→G that leaves Val unchanged. That is entry 2.

Mystery solved.

This misplaced-UTR problem is not a one-off either; I found hundreds of them in my gtf disrupting gene models the same way.

Where the bad UTR comes from — and the fix

As suggested by the source column in the gtf, the UTRs were added by stringtie2utr.py (a BRAKER helper that decorates a gene model with UTRs inferred from a StringTie assembly). This itself is a long story. In theory we should be able to do UTR prediction in AUGUSTUS with the option --UTR=on. However it will return with error and it is a known issue in BRAKER3. Katherine the author suggested us to use stringtie2utr.py as a workaround. The flaw is in how it builds the UTRs. First, merge_features adds a StringTie exon to the gene whenever that exon overlaps a BRAKER CDS and is not shorter than it — so a StringTie exon a few bases longer than the coding exon it covers (one that pokes into the flanking intron) gets merged in. Then compute_utr_features walks each exon and, for the single CDS segment it overlaps, carves whatever sticks out into a UTR:

cds_start = int(overlapping_cds[0]…[3])   # the LOCAL overlapping CDS, not the gene's coding start
if strand == "+":
    if start < cds_start:                  # exon pokes out 5' of THAT CDS segment
        utr5 = … "five_prime_UTR" …
        utr5 = utr5.replace(str(end), str(cds_start - 1))
    if end > cds_end:                      # … or 3' of it
        utr3 = … "three_prime_UTR" …

The comparison is purely local — an exon against the one CDS it happens to overlap — with no check that the exon is the transcript’s first (or last) one, nor if the resulting UTR falls outside the gene’s overall coding span as it should.

Apparently there is now BRAKER4, but unfortunately it seems like the UTR prediction is still done through stringtie2utr.py. While improved on many fronts, merge_features still merges over-long StringTie exons, and the UTR carving still uses only local exon-to-CDS comparison — so the root cause is unchanged. This is actually intentional as the author mentioned in one of the related issues that since the gene structure is ab initio, why should we trust it over RNAseq evidence? Btw I just came across this deep learning gene prediction tool called Helixer that may be worth trying.

The fix

So now we need to fix two things:

1. UTRs that fall within coding regions.
2. Bare gene/transcript ID.

We can write a post-processing script to fix both. See postprocess_braker_gtf.py as an example. Or we can fix the UTR part in string2utr.py, see an updated version in my git repo. Along the way I found a third problem from the raw braker output. There is always an redundant mRNA line for each transcript that was generated from GeneMark, and it does not include the UTRs that were predicted. postprocess_braker_gtf.py will simply remove those lines.

Do we have to redo the whole annotation? No.

A reasonable worry at this point is that fixing the GTF means re-running everything from scratch. It doesn’t. All three cleanups touch only the gene, transcript, mRNA, and UTR lines — they never modify a CDS feature therefore the cds and proteins are unchanged, thus the annotation. However we do need to update the SnpEff database and re-run the annotation of variants.

After diagnosing with AI, it turns out that one of my screens was in the T status, or was stopped. I do remember stopping jobs within that screen, but I don’t remember and don’t know how to stop a screen. But anyways, since screen -ls was also hanging, there are two helpful cmds that you can use to see what is going on.

The first one is ls -la /run/screen/S-$USER. This one will allow you to see the full names of all the screens that you have started. This way, if the screen you want to go back to is OK (as in not in T status etc.), you will be able to see their full names and attach back by doing screen -r abc.

Of course we also want to identify the root cause. This is a snippet of code that AI asked me to run:

for s in /run/screen/S-$USER/*; do
  p=${s##*/}; p=${p%%.*}
  st=$(cut -d' ' -f3 /proc/$p/stat 2>/dev/null)
  wc=$(cat /proc/$p/wchan 2>/dev/null)
  printf '%-8s %-22s %-4s %s\n' "$p" "${s##*/}" "${st:-GONE}" "$wc"
done

This would retrieve the PID of the screens and return their status. In my case, all my screens were in the mode of S/do_select (sleeping on the socket waiting for a client) except one being on T/do_signal_stop. This means the screen daemon was hit with a job-control stop signal (SIGSTOP/SIGTSTP) and is suspended. What to do? You just need to resume this job by kill -CONT $PID. Note that the PID is the number attached to the front of the screen name when you created it.

After resuming this job, you will be able to do screen -r and screen -ls without hang.

So screen has created a lot of problem for me so far. Something it gets stuck, and you can use ctrl + A + Q to exit.

I haven’t decided whether to move to tmux.

Due to LD, a lot of SNPs in the same region will be showing the same genotype-phenotype correlation. Fine mapping is the process of narrowing down to the causual variant by distinguishing the hitchhiking ones.

Why do we need fine mapping?

I can see two senarios.

1. When we have WGS data, it is usually not necessary to test very variant (precisely due to LD). Testing SNP A will give you almost the same results from testing SNP B if they are tightly linked, or in LD. You can do some LD pruning and test the representatives. However, the representatives were chosen at random and you might have actually removed the causual variant from the testing dataset.

2. Even if we have tested every single variant, again, you will need a way to distinguish the causual ones versus the hitchikers.

So one thing worth pointing out is that the fine mapping will be carried out on the full dataset.

Tools to use

Currently I am using “Sum of Single Effects” (SuSiE). It’s R realization is called susieR. The original model is described in Wang et al. 2020. This year, a newer version called MultiSuSiE where multi-ancestry is accomodated was publised.

Then I just went ahead to do my usually data preparation using PLINK until I realized some problem, and it centers around this concept called founder.

1. What is a “founder” in PLINK?

It is basically anyone with 0 0 in the PID (column 3) and MID (columns 4) of the .fam file. Meaning, we do not have information on who their parents are. Thus they are founders themselves. Since they might not be the actual founders from their population, therefore it is Not a biological concept but purely a pedigree bookkeeping artifact. If there is no pedigree info in the whole dataset, then everyone becomes a founder. In our family data where we have grandparents, F1 and F2, only the grandparents are founders.

2. Why founder matters

By default, PLINK calculates allele frequencies based on founders only. Meaning, if we do a --maf 0.05 filtering, say for variant chr1_10000_A_T, in the founders it is all A, but maybe there are a lot of copies of T in non_founders, this variant will still be considered not meeting the maf cutoff and filtered out. This makes sense when we do have the parents or grandparents in the dataset, since mendelianly they should have all the alleles of their offsprings. But in my dataset, due to the include of hypothetical PID and MID, it would be a huge lose if only “founders” are considered. Using only founders approximates sampling independent chromosomes from the base population.

Beyond presence and absense of alleles, this is also related to how allele frequencies in this population should be calculated. Allele frequency estimation assumes you’ve drawn N independent chromosomes from the population. Since related individuals share alleles IBD, they are not independent observations. If you genotype a parent and then genotype their three children, you’re partly re-counting the parent’s alleles three more times — the children’s genotypes are predictable from the parent’s. The “effective sample size” is the count of independent draws, which is far smaller than the raw count. Using raw count makes you think your estimate is more precise than it is, and it lets a few large families dominate. This concept is realted to base population that we talked about before. So founders makes the base population.

At that point, I thought it only affects certain plink functions such as --maf or --hwe. Not until today did I realized that by default, any feature of PLINK is based on the base population or the founders. OK so the first conclusion of today is, in PLINK, founders are the base population.

3. Analyses silently affected by founder status

Basically any analysis. You need to be very careful about whether you want to just use the founders (if your pedigree in the .fam file is correct), or all the individuals (turn on --nonfounders). Sometimes you also do not want to do the latter if your dataset is heavily biased by some families like I do. Here is a limited summary table for features I usually use. But again, only founders are used for allele freq calculation by default for any featuer, any!

4. How did I discover this silent scary behavior?

1. Like I said in the beginning, after adding all those PID and MID, there are very few founders left in my dataset, and I noticed that a lot more SNPs were filtered out under the same --maf.

2. Then I realized that it also affects the LD prunning because under the same parameters (--indep-pairwise 500 50 0.8 ), higher percentage of SNPs were found in LD/heavier prunning.

3. Eventually, PCA failed:

• PLINK 1.9 --pca: silent failure with cryptic GRM error (“Failed to extract eigenvector(s) from GRM”), probably a singularity problem.
• PLINK 2 --pca approx: explicit error (“less than 50 founders available to impute allele frequencies”)

Both errors have the same root cause: the GRM and allele frequency estimation are operating on fewer than 50 individuals for a dataset with thousands of samples.

5. Solutions and tradeoffs

--nonfounders: Usually this is an easy problem to fix by turning on this option and use all individuals in the dataset. This indeed retained slightly more SNPs (less than 10%) using all thousands of individuals, however still significantly less than the previous batch with only hundreds of individuals.

• --freq + --read-freq: pre-compute frequencies from a representative subset, then feed them in — most principled for mixed datasets. Four-step workflow:
  1. Pre-filter without --maf (apply --geno and --mind only)
  2. Define a representative subset: include all true founders (PID=0, MID=0) plus one individual per unique (PID, MID) pair among non-founders. This ensures every independent lineage contributes exactly once — full siblings collapse to one representative, but half-siblings (who share only one parent and thus have different (PID, MID) combinations) each get their own representative.
  3. Compute frequencies from that subset: plink --bfile ... --keep <subset> --freq --nonfounders --out ... — --nonfounders is required here because the subset includes non-founders (e.g., the half-sib representatives); without it PLINK falls back to the 27 true founders.
  4. Apply MAF filter using the pre-computed frequencies: plink --bfile ... --read-freq <freq_file> --maf 0.005 --make-bed --out ...

• Remove relatives first for LD pruning: use --rel-cutoff (can try third degree: 0.125 or second degree: 0.25) + --make-founders (required when parents are absent from the kept subset) + --indep-pairwise; apply the resulting prune list to the full dataset. For highly structured multi-population datasets, population structure will still inflate LD — per-population pruning followed by taking the union of kept variants is the most principled approach.

  A note on consistency between MAF and LD representative selection: It is natural — and correct — to use different criteria at the two stages. For MAF estimation, the pedigree-based approach (one per unique PID/MID pair) is optimal because it uses known family structure to ensure independent lineage representation; half-siblings are included because their distinct (PID, MID) pairs represent genuinely different crosses. For LD estimation, a kinship cutoff (e.g., 0.125) uses empirical relatedness to prevent shared haplotype blocks from inflating apparent LD; half-siblings (IBD ≈ 0.25) are excluded by this threshold. The LD stage being stricter about relatedness than the MAF stage is the safe direction and is not a methodological inconsistency.

• --bad-freqs: override (not recommended — hides the problem)

Attempt 1 — default (a couple dozens of founders): retained only ~4.5% of variants vs ~12.4% for a previous version where we assigned hundreds of founders. Noisy r² from small N causes spurious high-LD calls and over-pruning.

Attempt 2 — --nonfounders (all individuals in thousands): retained even fewer variants (~4.2%). This is counterintuitive — more individuals, yet worse results. The explanation requires understanding two distinct sources of r² inflation:

• Attempt 1 suffers from small-N noise: with only ~27 individuals, r² estimates are imprecise and systematically upward-biased (r² is bounded at 0, so random errors can only push it higher, never lower). Some truly unlinked variants get flagged as in LD by chance.

• Attempt 2 suffers from kinship-induced pseudo-LD: related individuals share long IBD haplotype blocks. Two variants sitting on the same shared haplotype will co-occur systematically across all members of a family — not because of actual LD in the population, but because of shared ancestry. Within a pruning window, PLINK cannot distinguish this from real LD and prunes accordingly. This is especially bad when you have a lot of related samples in your dataset.

In my case, the kinship inflation turns out to be larger than the small-N noise inflation, so going from a couple of dozens of founders to thousands of related individuals makes things worse. Therefore we need to remove relatives first — you need a dataset where r² reflects actual population LD, not shared ancestry.

At first I tried to get a unrelated subset using --rel-cutoff 0.125 , but again only a couple of dozens of individuals are left. leaving 14 — worse than the original 27 founders. This is because 2nd-degree relatedness is pretty common in my dataset. Then I tried a lower cut off --rel-cutoff 0.25 (remove only 1st-degree + duplicates), now we have a few hundreds remaining. You then need to make all of them founders (--make-founders)

Attempt 5 — add --make-founders: promotes all individuals with absent parents to founder status. This is necessary whenever you use --keep to subset a pedigree dataset. Still retained fewer variants than expected (~3.1%), because population structure (many divergent populations) inflates within-window r² regardless of relatedness.

Validation: despite all this, PCA eigenvectors computed before and after LD pruning showed >0.99 correlation — confirming that for PCA, the exact pruning strategy matters little in practice.

5.5 A second worry, a wrong turn, and what the lever actually is

After all that founder agonizing, I hit a related worry. My dataset is dominated by two big populations (TS1 and TS3), with a bunch of smaller ones trailing behind. My fear: even with --nonfounders turned on, a global --maf 0.005 is computed by pooling everyone together. So a variant that is common inside a small population but rare across the whole pool gets dropped — exactly the variants I thought I’d most want to keep.

My first instinct was that --maf was the wrong tool and I should switch to a count threshold, --mac. The reasoning felt clean: what actually destabilizes an association test is the minor allele count — how many copies enter the regression — not the frequency, so filter on the thing you actually care about. I was fairly convinced. So I ran it.

It made almost no difference. --mac 20 --nonfounders returned essentially the same variant set as --maf 0.005 --nonfounders (a hair fewer, in fact). And once I saw that, the reason was obvious and a little embarrassing: on a single pooled sample, a frequency is a count. With ~2000 samples, --maf 0.005 means a minor allele count of about 0.005 × 2 × 2000 ≈ 20. So --maf 0.005 and --mac 20 are the same threshold written two different ways. They can only diverge at the boundary, and on how each treats missingness (--mac is slightly stricter on high-missingness sites, which is why it kept a touch fewer). Switching frequency-for-count could never have fixed population imbalance — I’d been comparing a tool to itself.

So what is the lever? It’s the denominator — who counts as the base population — not the form of the threshold. That’s the whole lesson of this post, and it’s the one knob that actually moves variants in and out:

• founders only (a couple dozen people): noisy estimate, over-removes — the broken case.
• all individuals (--nonfounders, or equivalently --mac on everyone): the pooled frequency. Repairs the over-removal.
• one representative per independent lineage (the --read-freq subset trick from section 5): weights each lineage once, so the pooled denominator no longer drowns out small populations — retains the most variants.

That last one looks like the answer to my imbalance worry, and as an estimator of allele frequency it is the principled choice. But here’s the catch I only saw after running everything: the extra variants the lineage-weighted set keeps are, by construction, the ones with very few actual copies in the full sample. They survive only because dividing by a small denominator inflates their frequency. For a pooled GWAS, those are exactly the underpowered variants — there genuinely aren’t enough copies in the data I’m analyzing to test them stably.

Which dissolves the original worry rather than solving it. In a pooled analysis, a variant that is rare in the pool is untestable in the pool — no matter how common it is inside some small population. That isn’t a filtering bug to engineer around; it’s a property of pooling. If those small-population variants are biologically interesting, the answer is a stratified or population-specific analysis (where you’d filter within that population), not a cleverer global filter.

So my actual conclusion, after the wrong turn: for everything analyzed together in GCTA and SNIPAR, use all individuals as the base population and a stringency around --maf 0.005 / --mac 20 (they’re the same thing — pick whichever you find clearer; --mac is marginally more honest about missingness). For SNIPAR’s family-based tests, where the effective number of independent units is smaller than the raw N, leaning a bit more conservative (--mac 30) is reasonable. Reserve the lineage-weighted subset for when you want an unbiased frequency estimate, not for deciding which variants enter the test.

And the meta-lesson: I almost shipped a fix to a problem the fix couldn’t touch, because the reasoning sounded right. Running it was what corrected me.

6. Key takeaway

Always check your founder count before running any frequency-dependent analysis:

grep "founders" your.log

If you have a pedigree-filled .fam file and few founders, every downstream result is quietly wrong unless you intervene. The --hwe case is worth special attention: HWE violations are expected in related samples, so filtering on HWE in a pedigree dataset silently removes valid markers.

Firstly some background on BLUP vs GBLUP. Simply put, the model is exactly the same, y = Xb + Za + e. The difference lies in Z. In BLUP, it is usually the A matrix which is based on pedigree; while in GBLUP, it is usually a genetic relationship matrix (GRM) calculated based on molecular markers such as SNPs. In my case it is a GRM calculated based on millions of WGS markers.

The tool I was using is a R package called rrblup. Let’s first define how I calculate h2 and the accuracy then why there might be overfitting. In this post we will not talk about fixed effect. Maybe in another post we will do.

For h2, it is calculated with the full data:

model <- rrblup::mixed.solve(y=y, K=GRM)
h2 <- model$Vu/(model$Vu + model$Ve) 

For accuracy: cor(y, model$u)^2. Note that in rrblup::mixed.solve(), model$u is Za, not a. The corrent form should actually be cor(Za + e, Za)^2, but since in this post there is no fixed effect, this is equivalent to cor(y, Za)^2.

Hypothesis 1: rescale of GRM. Did not help.

# Check if K is properly scaled
mean(diag(GRM))  # Should be close to 1
# If not, try:
G_scaled <- GRM / mean(diag(GRM))
model <- mixed.solve(y = y, K = G_scaled)

Hypothesis 2: population structure can create “information leakage”

From the BLUP calculation, we know that the weight is much higher from their close relatives than others in the linear combination for the prediction. Therefore when there is strong population structure (which is true in my dataset), individual i from population A is predicted mainly by its close relatives. The genetic correlation (represented in their pairwise similarity in GRM) can be confounded with population-specific effects, such as environmental factors or other cryptic relatedness. Since we only have one random effect term, everything is lumpped into this term.

Now the real problem is, why this inflates r2, but not h2?

captures both true genetic effects and This is because some of the

This tool was published along with a population genomic study on wheat (Wang 2022.

The whole program breaks into 5 major parts as suggested both in the code and the output folders.

At the heart of the problem is something called Linkage Disequilibrium (LD). This word has been the center of my universe in the recent couple of years. Everything dated back in 2010 in Okinawa; LD and coalescent is the center of every theory and every lecture, together with all these selections.

Linkage Disequilibrium and Signal Coherence

When you see a sharp peak with multiple SNPs showing strong associations, it typically reflects the underlying linkage disequilibrium (LD) structure of the genome. SNPs in close proximity tend to be inherited together, so a true causal variant should create a signal that extends across nearby correlated SNPs. An isolated significant SNP surrounded by non-significant variants suggests the signal might not be reflecting a genuine biological effect in that genomic region.

Technical Artifacts and Genotyping Errors

Isolated significant SNPs are more likely to represent technical problems like genotyping errors, batch effects, or platform-specific artifacts. These issues typically affect individual SNPs rather than entire LD blocks. Quality control procedures can miss some of these problems, especially if they’re systematic across cases and controls.

Population Stratification Issues

Inadequately corrected population structure can create spurious associations at individual SNPs, particularly those with unusual allele frequency patterns across ancestral groups. Well-designed studies use principal components or other methods to control for this, but isolated signals might indicate residual stratification.

Multiple Testing Considerations

While you mention adjusted p-values, the genomic context matters for interpretation. A single SNP reaching genome-wide significance (typically 5×10⁻⁸) in isolation is statistically significant but lacks the biological plausibility that comes with seeing the expected LD pattern around a true association.

Biological Plausibility

Clustered signals often coincide with known genes, regulatory elements, or functional annotations, providing biological context. Isolated SNPs in gene deserts or without obvious functional relevance require more scrutiny.

However, isolated SNPs aren’t automatically false positives - they could represent rare variants with large effects, structural variants not well-captured by standard arrays, or associations in regions of low LD. The key is to evaluate them with additional evidence like replication studies, functional annotation, and deeper sequencing.