To perform heritability analysis (e.g., estimate SNP heritability or partition heritability), requires that the samples are "unrelated" (in practice, this means at most distantly related, with no pair closer than, say, second cousins). This ensures that the heritability estimates reflect only the heritability contributed by predictors in the dataset (or predictors in local linkage disequilibrium with these). By contrast, if your dataset includes (substantial) relatedness, there will likely be long-range linkage disequilibrium (e.g., between predictors on different chromosomes), and you will end up with inflated estimates.
Furthermore, heritability analyses generally require a large sample size. For example, to reliably estimate SNP heritability (standard deviation less than 5%) using a single kinship matrix typically needs at least 7,000 unrelated samples; if you wish to use multiple kinship matrices (i.e., perform Genomic Partitioning), the required number of samples is even higher.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
We recommend that you do not perform heritability analysis, if after performing careful Quality Control, you have fewer than 5000 samples (see the exception below). If your sample size just exceeds this limit (say, you have 5000-7000 samples), you should consider increasing the minor allele frequency threshold (e.g., restrict to SNPs with MAF>0.05, instead of those with MAF>0.01), as this will slightly increase the precision of estimates.
Note that you can consider relaxing the above sample size requirement if you have multiple phenotypes, and you are interested in overall patterns, rather than results for any single phenotype. This is because even though individual heritability estimates will be imprecise, the average across many phenotypes can still be informative. An example is if you are analysing transcriptome data, where each sample is recorded for thousands of gene expressions.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
If your dataset is too small to perform heritability analysis, then we suggest you focus on association testing. LDAK is able to perform both Single-Predictor Analysis and Gene-based Analysis, both classically and within a mixed-model framework (the latter is useful when your dataset contains related samples and/or population structure).
