The multi-scale structure and thermal properties of eight widely different types of high-amylose starches (HASs) having amylose contents (AC) in the range of 34.4%–97.3% originating from maize, wheat, barley, and potato were analyzed to unveil possible relationships among different levels of structures and thermal properties. The starches were found to cluster in four groups: (I) two HASs from maize, Gelose50 and Gelose80, with high gelatinization enthalpy (△H) and low onset (T₀) and peak (T_p) gelatinization temperatures, (II) two HASs from potato and wheat, with medium and high △H and extremely low T₀ and T_p, (III) two HASs from maize, NAFU50 and NAFU60, with medium △H and medium T₀ and T_p, (IV) two HASs from maize and barley, Hylon VII and AOBS, with low △H but high T₀ and T_p. The degree of molecular branching and the extent of the granule V-type crystalline polymorph were the critical factors determining their thermal properties, while botanical source and AC were not found important. HASs from wheat and barley showed relatively low lamellar and crystalline order, which was related to high content of amylopectin or AM-like chains with degree of polymerization (DP) 6–12 and long amylose chains, both of which can contribute to prevent the formation of double helices. Our data pinpoint the importance of amylopectin short chains, amylose long chains, and degree of branching on HAS starch granule structural order and thermal stability, which are potentially useful in boosting the development of HAS-based products and be beneficial for developing new HAS crops.