Engineering quantum anomalous Hall phases with orbital and spin degrees of freedom Hongbin Zhang,∗ Frank Freimuth, Gustav Bihlmayer, Marjana Leˇzai´c, Stefan Blu¨gel, and Yuriy Mokrousov† Peter Gru¨nberg Institut and Institute for Advanced Simulation, Forschungszentrum Ju¨lich and JARA, D-52425 Ju¨lich, Germany (Dated: today) Combining tight-binding models and first principles calculations, we investigate the quantum anomalousHall(QAH)effectinducedbyintrinsicspin-orbitcoupling(SOC)inbuckledhoneycomb lattice with sp orbitals in an external exchange field. Detailed analysis reveals that nontrivial 3 topological properties can arise utilizing not only spin but also orbital degrees of freedom in the 1 strong SOC limit, when the bands acquire non-zero Chern numbers upon undergoing the so-called 0 orbital purification. Asaprototypeof abuckledhoneycomblattice with strongSOCwechoose the 2 Bi(111) bilayer,analyzingitstopological propertiesindetail. Inparticular, weshowtheemergence n ofseveralQAHphasesuponspinexchangeoftheChernnumbersasafunctionofSOCstrengthand a magnitude of the exchange field. Interestingly, we observe that in one of such phases, namely, in J thequantum spinCherninsulatorphase,thequantizedchargeandspinHallconductivitiesco-exist. 8 Weconsider the possibility of tuningthe SOCstrength in Bi bilayer via alloying with isoelectronic 1 Sb, and speculate that exotic properties could be expected in such an alloyed system owing to the competition of the topological properties of its constituents. Finally, we demonstrate that 3d ] dopants can be used to induce a sizeable exchange field in Bi(111) bilayer, resulting in non-trivial l l Chern insulator properties. a h - s I. INTRODUCTION In the quantized case of the AHE in two dimensions, e the value of the anomalous Hall conductivity (AHC) is m proportional to the so-called (first) Chern number, ob- t. In recent years, dissipationless charge and spin trans- tained as an integral of the Berry curvature of the oc- a portphenomenahavedrawnveryintensiveattention. Af- cupied states in the Brillouin zone. This integer Chern m ter the topological nature of the anomalous Hall effect number, which characterizes the “phase” complexity of (AHE) in ferromagnetswasdiscoveredandunderstood,1 - the manifold of Bloch electrons as a whole, has its roots d the existence of the spin Hall2,3 and quantum spin Hall in solid state physics since the discovery of the quantum n (QSH) effects4,5 was theoretically proposed and exper- o Hall effect and its theoretical understanding by Thou- imentally demonstrated. Nevertheless, the experimen- c less et al. in 1982.27 Shortly after that, Haldane sug- tal observationof the so-calledquantum anomalousHall [ gested a spinless model to realize the QAHE without effect (QAHE),6 characterized by a quantization of the an external magnetic field,6 an effort, which conceptu- 1 anomalousHallconductivity,isstillmissing,despitevari- v oustheoreticalproposals7–11andrecentexperimentalad- ally led to the theoretical suggestion of graphene as a 6 first topological insulator.4 In the latter case, the Chern vances in this direction.12 2 numbersofspin-upandspin-downelectronsarenon-zero 4 Numerous studies on the QSH effect have spawned a due to a microscopic realization of the Haldane’s model, 4 fascinating field focusing on the topologically nontriv- but they are opposite to each other. The generality of . 1 ial states of matter, in particular, topological insulators this counteracting nature of the Chern numbers in TIs 0 (TIs).13,14 Remarkable physical properties of TIs orig- lead eventually to the concept of the spin Chern num- 3 inate from the gapless surface states, guaranteed by a bers, which can be used to characterize the TI phases 1 finite bulk gap and time-reversal symmetry. It is plau- in time-reversal broken situations.28 Ultimately, the re- : v sible to seek QAH effects by introducing time-reversal alizationoftheQAHEishingedonthepresenceofpoints i broken perturbations to TIs while keeping the topolog- in the electronic structure, at which otherwise separate X ical nontriviality, e.g. via a proximity effect.14 Among spin bands interact with each other, and exchange cor- ar all known TIs,15 two-dimensional (2D) systems are of responding spin Chern numbers, to give a global non- particular interest, such as graphene,4 HgTe quantum vanishing Chern number in the system. The mechanism wells,5AlSb/InAs/GaSbquantumwells,16silicene,17and behind such a Chern number exchange is strongly de- Bi(111) bilayers (BLs).18,19 In the past, a lot of work pendent on the details of symmetry, hybridization and has been done basedon the extended Kane-Melemodel4 microscopic structure of the spin-orbit interactionin the in the context of graphene.9,20–24 In these studies, only system, making search for systems which could exhibit the σ band of the pz orbitals are considered, with key non-zero QAHE difficult, yet definitely rewarding since terms to induce topological nontriviality being effective the physics involved is possibly very rich. projected spin-orbit and Rashba spin-orbit interactions, arising from second-order perturbative processes.25 The Among other materials, Bismuth (111) bilayer (BL) strength of intrinsic atomic spin-orbit coupling of car- has been shown to host nontrivial topological properties bon is limited, though it can be enhanced in fine-tuned duetothestrongSOCofBi.18,19,29Inourpreviouswork, systems.26 weusedtheconceptofthespinChernnumbertocharac- 2 terize the topological phase transitions in Bi(111) bilay- ξ(l+s− +l−s+)/2, couples p and {p ,p } orbitals via z x y ers with respect to the strength of an external exchange a flip of spin and a ±1 change in the orbital quantum field.29 Here, we extend this work further, and demon- number. Further,wetaketheSlater-Kostertight-binding stratethemicroscopicoriginoftheQAHphasesfoundin parameters of Bi, provided in Ref. [30] for the values of Bi(111)bilayers when utilizing both orbitaland spin de- hopping integrals of the model, t . i,j greesoffreedom. Basedonageneraltight-bindingmodel Beforeproceeding further with the analysis ofEq.(1), forbuckledhoneycomblatticeswithsp-orbitals,weillus- we would like to remind the reader of the Haldane trate different Haldane-inspired6 mechanisms which can model,6 which is a generic model exhibiting an emer- leadto a nonzeroChern number in eachindividualband gence of the QAH effect on a honeycomblattice without upon employing the spin and orbital complexity (sec- an external magnetic field. The key mechanism in this tion II). We find that in addition to pz orbital based model is the complex hopping between the next near- physics,inthestrongSOClimit,orbitalangularmomen- estneighbor(NNN) orbitals,whichleadstoasublattice- tum purification can also lead to nontrivial topological dependent “magnetic field” with zero flux through the properties. We further clarify the ways behind achiev- overall hexagonal plaquette. In our work, it is the on- ing the QAHE at the so-calledspin-mixing points in the site atomic SOC that induces such complex hopping via electronic structure. Finally, using first-principles tech- different processes as discussed below. niques, we show that various topologicalphases occur in a realization of a buckled sp honeycomb model − the Bi(111) bilayer − with respect to the SOC strength and A. pz orbitals magnitude of an external exchange field (section III). In particular we suggest the existence of a so-called quan- First, we consider the case of the p bands, partic- tum spin Chern insulator phase, which is characterized z ularly relevant in graphene physics, well-separated from by the quantization of both the transverse charge and {p ,p }andsstates. Generally,therearetwowaystoin- x y spin conductivities. We conclude by discussing how to ducenontrivialChernnumbersinthep bandsviagener- realize the tuning of SOC strength by alloying Bi with z ating an effective next nearest neighbor (NNN) complex Sb, and the effect of an exchange field by introducing hopping due to intrinsic atomic SOC. First, on a buck- magnetic substrates and/or dopants. led honeycomb lattice, p orbitals can hybridize directly z with the {p ,p } orbitals, and complex hoppings can be x y induced via the spin-conserving part of SOC which acts II. MODEL ANALYSIS between p and p states. As illustrated in Fig. 1(a), x y in this mechanism the corresponding virtual transitions To illustrate how nonzero Chern numbers and non- read:31 trivial QAH phases can arise, we first consider a generic tight-binding Hamiltonian of sp-orbitals on a buckled |pA ↑it→NN|pB ↑iξl→zsz|pB ↑it→NN|pA ↑i (3) honeycomb lattice in x-y plane: z x,y x,y z H = t c†c + c†(ǫ I+Bs )c +H (1) where V indicates the direct hopping between pz and X i,j i j X i i z i SOC px,y orbitals on the neighboring sites, while superscripts i,j i AandBdenotethenearestneighboratomicsitesinsub- where the first term presents the kinetic hopping with lattice A and B. ti,j (i,j = s,px,py,pz) as the hopping parameters. The For the resulting NNN hopping we obtain tNNN ∼ ξ. second term reflects an orbital-dependent on-site energy Importantly, the effective hoppings are opposite in sign ǫi and the interaction with the Zeeman exchange field for pz orbitals of a given spin on A and B sublattice, B directed along the z-axis, with I (sz) as the identity hence giving rise to a finite gap ∆1 ∼ 2ξ at the K (K′) (Pauli) matrix. The third term in Hamiltonian (1) is high-symmetry points in the Brillouin zone, as shown in the on-site SOC Hamiltonian. In order to facilitate the Fig. 1(a), in which a small magnetic field was applied in analysis, s, pz, and {px,py} bands are artificially sepa- ordertoseparatethespin-upandspin-downbandsinen- rated from each other by imposing a rigid shift of ǫi. To ergy. The gapis topologicallynontrivial,in directaccor- identify different origins of the QAHE, the spin-orbitin- dance to Haldane’s arguments.6 The two spin-channels teractionisfurtherdecomposedintospin-conservingand hereareindependent,sinceonlythespin-conservingpart spin-flip parts: ofSOCisinvolved,althoughthecomplexNNNhoppings H = ξl·s=ξl s +ξ(l+s−+l−s+)/2 (2) forthespin-downandspin-upelectronsarecomplexcon- SOC z z jugate of each other.31 This means that the Chern num- where l (s) is the orbital (spin) angular momentum op- bersofthespin-upbandsareoppositetotheChernnum- erator, and ξ is the electron-shell averaged atomic SOC bersofthespin-downbands,asshowninFig.1(a),which strength. Since in this work we choose the direction of suppresses the QAHE of occupied electrons. the spin-polarizationto be aligned along z direction, the Secondly,on-sitespin-flipSOCcangiverisetocomplex spinconservingpartoftheSOCHamiltonian,ξl s ,cou- next neighbor hopping too, even if there is no direct hy- z z ples {p ,p } orbitals, while the spin flip part of Eq. 2, bridizationbetweenp and{p ,p }orbitals,Fig.1(b). In x y z x y 3 FIG. 1: Topological analysis of pz ((a)-(c)), s ((d)-(f)), and {px,py} ((g)-(i)) bands in a buckled honeycomb bilayer. The electronicstructureisobtainedusingthetight-bindingparametersforbulkBiaslistedinRef.[30],withanadditionalartificial rigid shift of on-site energies yielding separation of s, pz, {px,py} states in energy. Only nearest neighbor (NN) hoppings are considered, with Vppπ = 0 (this does not affect our conclusions, see the main text). Without specification, the spin-orbit couplingparameterξ=4.5eVandexchangefieldB=0.1eVareused. Leftcolumn(a,d,g)(middlecolumn(b,e,h))corresponds tothecasewithonlyspin-conservingSOC(spin-flipSOC)included,whilethefullSOCisconsideredintherightcolumn(c,f,i). Red (blue) in (a-f) and upper inset of (i) stand for the spin-down (spin-up) states, while in (g-i) and lower inset of (i) for the states with orbital magnetic number m = +1 (m = −1). Dashed horizontal lines indicate the Fermi energy level adjusted such thatrelevant bands(forinstance, pz in (a-c))are half-filled. NumbersdenotetheChern numberofeach individualband. Insets in (a-g) ((i)) display the electronic structure at the Dirac critical point at ±0.01 eV (±0.06 eV) with respect to the Fermi energy, and sketches illustrate different channels for complex nearest neighbor hopping (red (yellow) circles denote pz (s) orbitals, while px,y orbitals are indicated by blue ellipsoids; black (red) arrows depicts tNN hoppings (SOC hybridization), respectively.). Text below each panel denotes the hybridization of {px,py} orbitals to s and pz orbitals by kinetic hoppings, where + (−) indicates (no) hybridization. For instance, “s−{px,py}+pz” in (a),(i) means that there is no hybridization between s and {px,py}, while {px,py} and pz are coupled. 4 thiscase,thecorrespondingvirtualtransitionsare:25,31,32 theChernnumberisone(two). Thissavesusfromevalu- atingtheChernnumberexchangeexplicitly,althoughthe |pA ↑iξ→flip|pA ↓it→NN|pB ↓it→NN|pA ↓iξ→flip|pA ↑i sign of IndxBerry cannot be determined from this simple z x,y x,y x,y z (4) argument. In the case of Fig. 1(c), the Chern number of where ξ = ξ(l+s− + l−s+)/2, and V stands for the the upper valence band is reducedby 2, while the Chern flip direct hybridization between p orbitals on neighbor- number of the lower conduction band is increased by 2, x,y ing A and B sites. In analogy to the case with spin- as comparedto the situation in Figs.1(a) and (b), when conservingSOC consideredpreviously,the effective hop- the hybridizationbetweenthe pz bandsofoppositespins pings within A and B sublattices are of opposite sign is introduced. The reason for this is that, although the and the resulting gap is also topologically nontrivial. In dispersion of the bands at the point of degeneracy is ob- this case t ∼ξ2, and the corresponding gap ∆ at K viously linear, Fig. 1(c), there are two such points in the NNN 2 (K′) (Fig. 1(b)) scales quadratically with respect to the Brillouin zone (K and K’). SOCstrength. Ifthereisnohybridizationbetweenthep z In previous studies of the QAHE in the context of states of the opposite spin as what occurs in the 2D pla- graphene,20–24 an extended Kane-Mele model4 was em- narhoneycomblattice,thissecondsituationwithoutcon- ployed with an additional Rashba term. The effective sidering an external exchange field actually presents the Rashba spin-orbitinteractionoriginates fromthe combi- topologicalinsulatorstateasfoundingraphene,4consist- nation of intrinsic SOC and potential gradient perpen- ingoftwoindependentspincopiesoftheHaldanemodel. dicular to the honeycomb lattice plane which breaks the Thus, the resulted QAH conductivity for the half-filled inversion symmetry, and can be created by e.g. apply- situationisagainzero,althoughthep bandsofopposite z ing afinite electric fieldperpendicular to the honeycomb spin are meeting at the Fermi energy. plane25 or imposing a finite curvature of the honeycomb Note that, while the arising non-zero Chern numbers sheet.35 Inthiscase,whenthetime-reversalsymmetryis of the bands in all situations discussed in Fig. 1 could brokenvia,e.g.,non-zeromagnetizationduetoadatoms, lead to a QAH effect upon inducing a very large (larger theQAHEcanbeinduced.9,10 Nevertheless,theeffective than the bandwidth) exchange splitting in the system, complex hoppings due to the Rashba spin-orbitarisebe- we do not discuss such “trivial” possibility here. Thus, tweenp orbitalsofoppositespinonthenearestneighbor z to induce nontrivial QAHE in the p bands, the spin-up z sites − a situation, topologically distinct from the com- andspin-downp bandshavetobecoupled,inducingex- z plex NNN hoppings considered in this work. change of the Chern numbers. This can be achieved by allowingforthehybridizationbetweenp and{p ,p }or- z x y bitals and upon taking the complete SOC into consider- ation. When two entangled bands of different spin char- acter from conduction and valence bands overlap with B. s orbitals each other, spin-mixing occurs and the Chern number exchange takes place when the gap is opened, Fig. 1(c). Forexample,forpz orbitals,apossibletransitionprocess The second-order perturbation processes discussed is: above can also take place if the s bands are taken in- steadofp states. Itisachievedbyhybridizationbetween |pAz ↑iξ→flip|pAx,y ↓iξl→zsz|pAz ↓i. (5) s and p ozrbitals, Fig. 1(d)-(f). For instance, the effec- tivecomplexhoppingsbetweensorbitalscanbe induced Themagnitudeofthe gap,openedbetweenupanddown byspin-conservingSOCviahybridizingwith{p ,p }or- p -bands, as shown in Fig. 1(c), is proportional to ξ2 x y z bitals(Fig.1(d))orspin-flipSOC(Fig.1(e)). Comparing according to our calculations. to the p orbitals,the only difference is that for the case z The variation of Chern numbers upon band touching with spin-flip SOC (Fig. 1(e)), the correspondingvirtual is determined by the Berry indices at degenerate points transitions are in a generalized parameter space (k,η), which is defined as:33,34 IndxBerry = 21π ZS2dsΩ·n, (6) Not|esAth↑aitt→tNhNe|prBxo,lye↑oifξ→|flipp|BpBzi↓ainξd→flip||ppBBxi,yin↑itt→hNiNs|csaAse↑ic.(a7n) x,y z withS2asatwodimensionalsurfaceenclosingthedegen- be exchanged, i.e., electrons can hop from | sAi to | pBi z eracy point, n is the surface normalvector pointing out- and then couple with | pB i. Furthermore, similarly to x,y wards, Ω is the Berry curvature.34 In our case, η can be thecasesforp bands(Fig.1(a-c)),thespin-upandspin- z the strength of the exchange fields or SOC.Since S2 can down s bands are decoupled with Chern numbers of op- be chosen arbitrarily small, the magnitude of Indx posite sign (Fig. 1(d) and (e)), and a nontrivial QAH Berry canbeobtainedbyexaminingthebanddispersionatthe effect for the half-filled case takes place only when the degeneracy point.33,34 For instance, if the dispersion of two spin-channels are entangled, i.e., due to hybridiza- the crossing bands is linear (quadratic), the variation of tion with p orbitals (Fig. 1(f)). z 5 C. {px,py} orbitals: orbital purification nificantly reduced if Vppπ hoppings are taken into ac- count, which reduces the band width of the p bands. In the latter case, for Bi, ξ becomes 2.4 eV if all relevant More interestingly, nontrivial topological properties op hopping integrals for the bilayers, as listed in Ref. [30], ariseinthe{p ,p }bandsaswell,Fig.1(g-i). Ifonlythe x y are included in our tight-binding calculations. Our first spin-conserving SOC is considered, every {p ,p } band x y principlescalculationsoftheBi(Sb)bilayer,ontheother acquires a nonzero Chern number when the magnitude hand,showthatthemagnitudeoftheatomicSOCforBi, of SOC is larger than a critical value ξ (Fig. 1(g)). op ξ (ξ forSb),isabout2.8eV(0.9eV).Thus,Bibilayer This is different from the cases considered for p and s Bi Sb z resides in the strong SOC limit, where the orbitalpurifi- bands, where the strength of SOC determines only the cationtakes place,orbitally polarizedp ±ip bands are size of the gap, while it is irrelevant for the topological x y formed and exhibit non-zero Chern numbers. To verify properties ofindividual bands. Ouranalysisrevealsthat this explicitly, we evaluate the expectation value of the the origin of the nontrivial Chern numbers can be at- tributed to the so-called orbital purification, caused by orbital momentum operator Lz in Bi bands. As shown inFig.2,thefirstandthe thirdvalencebandsofBi(111) strongSOC(Fig.1(g)). Forinstance,forthemiddlefour bilayerareindeedorbitallypurifiedintheentireBrillouin bands in Fig. 1(g), the Chern numbers are nonzero only zone. ThecorrespondingChernnumbersare±2and±1, whenSOCis largerthanξ =5.4eV,whenthe valueof op respectively. The orbital purification in these bands oc- the orbital angular momentum L in each of the bands z curs until the SOC strength of Bi atoms is scaled down becomes consistently either strictly positive or negative to 70% of its atomic value, that is, approximately 2 eV, ateachkpoint(seethecoloringofthebandsinFig.1(g)). when the Bi bilayerlooses its topologicalinsulator prop- Thisleadstothepredominantlym=±1(p ±ip )char- x y erties, see Fig. 3(a). In this sense, we can call the Bi acter of each band, which constitutes the essence of or- bilayeranorbital topological insulator, asopposedto the bital purification. The complex NNN hoppings in this case of graphene, in which all the topological properties case are induced via the following mechanism: are due to p states. z |pA±ipAi→V |pB∓ipBi→V |pA±ipAi (8) x y x y x y due to virtual transitions between the bands of differ- ent m on different sublattices, mediated by kinetic hop- ping. Sinceonlyspin-conservingSOCisconsideredhere, the spin-upandspin-downchannelsarenotentangled,if no direct hybridization of {p ,p } and other orbitals is x y present, which corresponds to the situation discussed by Wu36,37 for spinless honeycomb optical lattices. On the other hand, for the lowest (top most) two bands,theorbitalangularmomentumpurificationoccurs for an even smaller SOC strength of about 0.3 eV, when onlyspin-conservingSOCisconsidered. Thereexistsalso another possibility for these bands to acquire a complex NNNhoppingsbyspin-flipSOCasshowninFig.1(h). In this case, the Chern numbers for the middle four bands are always zero even if the spin-flip SOC strength is in- FIG.2: OrbitalangularmomentumLz purificationinBi(111) creased to very large values, while for the lowest (top bilayerfromfirstprinciplescalculations. Red(blue)standsfor most)two bands,the Chernnumbers arequantizedwith the expectation value of the Lz operator. A small artificial nonzero value. To obtain a nontrivial QAHE, entangle- exchange field of 10 meV was applied perturbatively to split mentbetweentwospinchannelsisagainessential,where originally degenerate bands. In each pair of resulting nearly Chern number exchange occurs via spin-mixing as ob- degenerate bands the expectation value of Lz, sz and the servedforthesandp casesabove. For{p ,p }orbitals, Chern numbers are opposite to each other. Integers next to z x y suchentanglementcanbeachievedbyhybridizationwith thebandsstandfortheabsolutevalueoftheChernnumbers. p -orbitals, with corresponding Chern number exchange z of 1 due to a linear band dispersion at the points of de- generacy. Following previous arguments, this leads to a A comment on achieving nontrivial topologicalphases non-zero QAH conductivity with C = −1 at half-filling, in optical lattices is in order, given that the idea of or- Fig. 1(i). bital angular moment purification was first proposed in The orbital purification is determined by the compe- the context of ultracold atoms.36 Quite recently, both tition between the strength of the on-site SOC and the Abelian38andnon-Abelian39–41gaugefields(seeRef.[42] band width due to kinetic hoppings. In the model anal- for a recent review) have been experimentally realized ysis above, we neglected the V hoppings in Bi, with for trapped neutral atoms. Since ultracold atoms are ppπ resulting ξ of about 5.4 eV. This critical value is sig- spinless particles, in this context the relevance of the op 6 models discussed here is limited to the cases shown in generalizedto the caseswith spin-flipSOCby Prodan.28 Fig. 1(a),(d),(g) when neglecting the spin degree of free- Compared to the more universal approach of Ref. [47], dom. Moreover, the hybridization between different or- the spin Chern number can be easily constructed. It is bitals (as in Fig. 1(a) and (d)) is not necessary, since in applicabletothe situationwithouttime-reversalsymme- optical lattices complex hoppings are carried by dressed try, and robust when the spectra of Ps P are gapped in z states. Experimentally,ultracoldatomsinhigherorbitals the BZ, where P is the projection operator onto the oc- such as p bands have been recently achieved (see e.g. cupiedstates. Inthefollowing,weuseC (C )todenote + − Ref. [43]), which makes them a promising candidate for the“Chern”numberofthespin“up”(“down”)projected realizing various topological properties. occupiedbands,andthespinChernnumberisdefinedas Tosummarize,wedemonstratedthatbothorbitaland Cs = 21(C+−C−). For more details see Ref. [29]. spindegreesoffreedomcanbeutilizedtoinducenonzero In this section we tackle the question of finding and Chern numbers in an individual band, while nontrivial characterizingpossibletopologicalphasesofBiandSbbi- QAH effect is due to the entanglementof two spin chan- layersfromfirstprinciples,asthecorrespondingHamilto- nels accompanied by spin exchange of the Chern num- nianofthesesystemsisalteredbyscalingtheSOCmatrix ber. In real materials, all bands are in general coupled elementsandapplyinganexternalexchangefield. Ourab by hybridization and, hence, Chern number exchange is initiocalculationswereperformedusingthefull-potential expected at every non-accidental band crossing. In this linearizedaugmentedplanewavemethodasimplemented sense, graphene is a peculiar material,44 since the well- intheJu¨lichdensityfunctionaltheorycodeFLEUR.48The separated in energy σ and π bands are not coupled by Wannier functions technique was used on top of self- kinetichoppingduetographene’splanarstructure. Sub- consistent first principles calculations to derive an accu- lattice staggering as found in silicene17 is too small to rate tight-binding Hamiltonian of the system.49–51 The induce significant variation of the electronic structure, relaxed bulk in-plane lattice constant and the distance thus the simplified 4-band model considering only the betweenthe twolayersforSb(111)constitute4.30˚Aand π bands is good enough to account for the topological 1.55 ˚A, respectively, while for Bi(111) bilayer the corre- phasetransitionsingraphene-relatedsystems.20–23 How- sponding values are 4.52 ˚A and 1.67 ˚A. The details of ever, for Bi(111) bilayer and its derivatives, all three p- calculatingthe anomalous(spin) Hallconductivities, are orbitals reside in the same energy scale, the hybridiza- described in Ref. [29]. Moreover, we find that the spec- tion betweenthem is stronglyenhanced due to buckling, tra of s projected onto the occupied states are always z andthe strengthofSOC isordersofmagnitude largeras globallygappedforthecasesconsideredinthefollowing, compared to that of carbon or silicon atoms. Therefore, despite the fact that the strength of SOC in Bi, ξ , is Bi rich physics with competing orbital and spin degrees of verystrong(≈2.8eV),leadingtowell-definedspinChern freedom is expected in the latter case. In the next sec- numbers. A uniform exchange field was applied on top tion, we will demonstrate that non-trivial QAH phases ofthefirstprincipleselectronicstructure,asdescribedin can be achieved in Bi(111) BL by applying an exchange Ref. [29]. The spin-orbit strength ξ in our calculations field, where the variation of Chern numbers can be ex- was scaled by hand via multiplying all atomic SOC ma- plained using the Chern number exchange scheme illus- trix elements with a uniform scaling parameter during trated above. the self-consistencycycle. Inorderto clarifythe effectof a more realistic exchange field, we have performed addi- tional calculations of a Bi(111) bilayer on ferromagnetic substrates (europium chalcogenides)and in the presence III. TOPOLOGICAL STATES OF Bi AND Sb BILAYERS of magnetic dopants in the system, as described later. In the previous section we illustrated within a tight- binding model how non-zero Chern numbers can arise A. Phase diagram of Bi(111) bilayer due to intrinsic SOC and how to induce the QAH effect via applying an exchange field. One problem remains The calculated phase diagram of the Bi(111) BL with unsolved, however, that is the problem of characterizing respecttothestrengthofatomicSOC,ξ,andthemagni- varioustopologicalphasesconsistently. Forinstance,TIs tude of the exchange field, B, is shownin Fig. 3(a). The are characterizedby the Z index45 and QAH insulators emergingdistincttopologicalphasescanbecharacterized 2 bythe(first)Chernnumber,anditisstillnotcompletely bythevalueoftheAHC(whichequalstheChernnumber clearhowtoproperlycharacterizethe2Dinsulatingtopo- times e2/h in the insulating regime) and the spin Chern logicalphaseswhichdonotfit intothe these twoclasses, number. The topologicalinsulatingphasesareseparated e.g.insulatorswithbrokentime-reversalsymmetrywhich by a metallic phase, with topological phase transitions originate from TIs and have a zero Chern number. An- occurring during closing the bulk band gap and reopen- other issue is how such “intermediate” phases can be ing it again as B and the SOC strength are varied. As probed and distinguished experimentally from topolog- confirmed by the calculation of the spin Chern number, ical and Chern insulators. To this end, we use the spin when the time-reversal symmetry is not broken (B = 0 ChernnumberfirstintroducedbyShengetal.46 andlater eV), the Bi(111) BL is a trivial insulator (C =0, C =0) s 7 FIG. 3: Quantum anomalous Hall phases in Bi/Sb(111) bilayers, and electronic structure of the Bi(111) bilayer on magnetic substratesandwithmagneticdopants. (a)((c))displaysthephasediagramoftheBi(111)(Sb(111))bilayerwithrespecttothe strengthofatomicSOCandmagnitudeofexchangefieldB. NumbersdenotetheChernnumberintheQAHphase,“TI”stands fortheTI phase,while“TRBTI” standsfor thetime-reversalbrokenTIphase. Thehorizontaldashed linein (a)indicatesthe case with B =0.5 eV, for which the AHC and spin Hall conductivity (SHC) are shown in (b) with respect to the strength of SOC. Inset in (a) displays the dispersion of the edge states (red lines) and the projected bulk states (gray shaded region) in Bi(111) BL ribbon with B =1eV and 70% of theBi atomic SOCstrength in theC =+1phase,while that in (c) displaysthe dispersion of the edge states in Sb(111) BL ribbon with B = 1 eV and 100% of the atomic SOC of Sb in C = −1 phase. (d) displays the band structure of the Bi(111) BL on top of EuSe(111) terminated with Se atoms. Black (red) lines denotes the majority(minority)bands. TheexchangesplittingatKpointforthefirsttwopairsofthevalencebandsisabout60meV.The dashed horizontal line indicates EF, and the inset displays the positions of the atoms. Left panel of (e) shows the electronic structures of Bi(111) BL in 2×2 superstructure with one Fe atom located at the hollow site in between the two Bi layers, whereas AHC conductivity is shown in the right panel. The horizontal dashed line in (d) indicates the position of the Fermi level which is not in the gap due to the fact that Se assimilate electrons. Shaded region in (e) marks the gap with the Chern numberC =+2. (f) is analogous to (e),but with theSOC strength of Bi scaled down to 20% of its atomic value. fortheSOCstrength≤56%ofitsatomicBivalue,while in which the originally separated bands directly overlap. it turns into a TI (C =−1, C =0) when ξ is largerthan On the other hand, for Bi BL with ξ > 0.67ξ , the s Bi 67%ofξ .19 IntheTIphase,C =−C =−1,resulting originaltopologicallynontrivialinsulatinggapisfinitefor Bi + − in a spin Chern number of −1. finitevaluesofB,withC =−C =−1untilenteringthe + − Breaking the time-reversal symmetry by applying an metallic region. The resulting spin Chern number is in exchange field induces topological phase transitions; If this case Cs =−1, which manifests the occurrence of the the magnitude of the induced exchange splitting is large time-reversal broken TI (TRBTI) phase, observed also enough to make the two bands, which were originally for graphene in [52], and discussed at length in e.g. [29]. below and above the gap, to overlap, according to the ThetransitionintotheQAHphaseuponincreasingthe mechanism depicted in Fig. 1(c),(f),(i). Before that, the B alwaystakes place via anintermediate metallic phase, originalinsulatingphaseisnotdestroyed,andtoacertain Fig. 3(a). In our previous work we have shown the ap- extentpreservesitstopologicalpropertiesinthepresence pearance of a QAH phase with C = −2 for Bi BL with of a small exchange field. For example, for 0.2ξ <ξ < full atomic SOC and B ≥0.42 eV.29 As we can see from Bi 0.56ξ thetrivialinsulatorphaseatB =0retainsasthe Fig.3(a),inaBiBLwithintermediateSOCstrength,ap- Bi B is increaseduntil the system enters the metallic phase plyingstrongexchangefieldcanalsoinduceQAHphases 8 which are derived from either a metallic, trivial insu- We call such an emergent topological phase a quantum lator, or topological insulating phases at B = 0. For spin Chern insulator phase. instance, in a Bi BL with SOC strength scaled to one The variation of the Chern numbers in the phase dia- half of the atomic value, a QAH phase with C = +1 gramoftheBiBLcanbeexplainedbytheChernnumber emerges for B ≥0.25 eV. It is an exotic phase, in which exchange mechanism at the critical points, as discussed C− = +1 while C+ = 0, as compared to the QAH phase in the previous section.33 For phase transitions at con- with C = −2, in which C− = C+ = −1. In the former stant ξ and varying B the Chern numbers are changed case, the spin Chern number Cs = 12(C+−C−) is not an by+1(forintermediateξ)and−2(forlargeξ),Fig.3(a), integer, and there exist only one chiral edge state local- corresponding to linear and quadratic dispersions at the ized at each edge of a one-dimensional Bi(111) ribbon, critical points, respectively, as confirmed by our calcula- asshowninthe insetofFig.3(a). Ourcalculationsshow tions(notshown). Moreinterestingly,theChernnumber that these edge states are spin-polarized in the same di- is changedby −3as we go fromthe C =+1 phaseto the rection, leading to the quantized value of the AHC (in C = −2 phase at a constant large B when varying the units of +e2/h) and also finite and large values of the strengthofSOC.Ouranalysisrevealsthatthistransition SHC, as demonstrated explicitly by calculations for the canbe decomposedintothreesteps,ateachofwhichthe case of B =0.5 eV in Fig. 3(b). topology of the Berry curvature distribution in the Bril- The peculiarity of the topologically nontrivial phases louin zone changes, as analyzed in the following, Fig. 4 with nonzero C lies in their finite SHC. It can be under- (B =1 eV). s stood in an intuitive way following two equations below: We focus on the neighborhood of the Γ-point, since most of the contribution to the variation of the Berry AHC=σ↑+σ↓ curvature and the Chern number exchange comes from (9) SHC=σ↑−σ↓ the coupling of the highest occupied bands to the low- est unoccupied bands in this region. For ξ ≤ 0.79ξ Bi in the C = +1 phase the distribution of Berry curva- where σ↑ (σ↓) denotes the conductivity of the majority ture exhibits a hot-loop structure with two pronounced (minority) electronsin units of e2/h (e/4π) for the AHC singularity-like points at the k = 0 axis. Enhancing (SHC), respectively. These relations can be defined and x the SOC strength brings down the conduction band and hold true only when the spin-flip band transitions due results in a singularity directly at Γ-point as the bands to spin-non-conserving part of SOC are absent. In the touch each other. Upon reopening the gap at the touch- trivial insulator phase, both the AHC and SHC are zero ing Γ-point the Chern number of the valence band is since C =C =0. When time-reversalsymmetry is not + − changed from +1 to +2 (ξ = 0.80ξ in Fig. 4). If the broken,i.e.forthe BiBLinthe TI phase,itimplies that Bi valencebandatthisξ wasseparatedbyaglobalgapfrom σ↑ =−σ↓ =−1,leadingtozeroAHCandtoaquantized the conduction band, the Chern number of the system SHC of−2e/4π.18 Nevertheless,due to the spin-flippart would be +2. As the ξ is increased further, the conduc- ofSOC,theSHCoftheBiBLinthe TIphaseisreduced tion band goes further down in energy, while the point to about 0.7 e/4π, as we found in our previous work of band crossing with the valence band and correspond- (Fig. 3 in Ref. [29]). In the QAH phase with C = −2, ing singularities in the Berry curvature split and move C = C = −1, and, correspondingly, σ↑ = σ↓, leading + − away from the Γ-point, resulting in a hot loop and four to zero SHC without the spin-flip transitions. However, “monopoles” at ξ = 0.82ξ . The Chern number of the theexchangesplittinginthesystemleadstothefactthat Bi valence band at each of such singular points is changed the spin-flip scattering between majority and minority by −1, therefore, the resulting QAH phase for ξ ≥ 82% states is notbalanced,leading to a smallbut finite SHC, of ξ has the Chern number C = −2. At 82% of the as shown in Fig. 3(b) for Bi BL at B =0.5 eV. Bi Bi SOC strength the band structure at the Γ-point is On the contrary, for the QAH phase with the Chern almostindistinguishablefromthatat79%,seeFig.4,al- numberC =+1,themajorityspinchannelisswitchedoff though the Chern number, sign of the Berry curvature, sinceC =0,andthe resultingSHC isenhancedinmag- + and orbital character of valence and conduction bands nitude as compared to the QAH phase with C =−2, see have changed completely. Fig. 3(b). As we artificially suppress the spin-flip band transitions by switching offthe spin-non-conservingpart of SOC in our calculations, we observe that the value of the SHC acquires a quantized value of the magnitude of B. Tuning ξ in Bi bilayers: alloying with Sb − e (not shown), despite the fact that the area of the 4π C = +1 phase shrinks as the electronic structure of the We turn now to the question of how to achieve tuning bilayerismodifiedwhenthespin-flipSOCisswitchedoff. of SOC strength and inducing a finite exchange field in That is, we demonstrated that it is possible to achieve a Bi(111) bilayer. Tuning the strength of SOC can be a coexistence of the QAHE and quantized (in the spin- achieved by alloying Bi with its isoelectronic but lighter conserving sense) spin Hall effect by applying a strong element Sb, as demonstrated in e.g. BiTl(S Se ) 1−δ δ 2 exchange field (at constantSOC strength) to a topologi- where Se was substituted with the isoelectronic S.53 cal insulator, trivial insulator, or even a metal at B =0. In Fig. 3(c) we show the calculated phase diagram of 9 FIG. 4: Variation of the distribution of the Berry curvature (first and third row) and the band structure (second and fourth row) at the Γ-point during a phase transition from C = +1 to C = −2 in Bi(111) BL at B = 1 eV as a function of the SOC strength. Numbersbelow each panel indicate the percentage of thestrength of SOC in unit of ξBi. Black (red) lines stand for thebands with ky =0 (kx =0). The numbersstand for theChern numbersof the topmost valencebands. Sb(111) BL with respect to the magnitude of the ex- ξ ≈2.7 eV, which is very close to ξ . Bi change field B and the strength of SOC ξ. The general Clearly,there existalso significantdifferences between features of the phase diagram of Sb BL are similar to the phase diagrams in Fig. 3(a) and (c). First, the TI those of the Bi BL. For instance, at weak SOC strength phasewithtime-reversalsymmetryatB =0isreachedin and small magnetic field, both bilayers are trivial insu- SbBLonlywhentheSOCstrengthisscaledabove305% lators, while QAH phases emerge with increasing ξ and of its atomic value, ξ ≥ 2.75 eV, which is larger than B. TheQAHphasewithChernnumberC =−2emerges the critical value in Bi BLs ξ ≈ 1.9 eV. Second, the Bi in Sb BL for B ≥ 0.4 eV if the SOC strength of Sb is boundary between the trivial insulator and the metallic scaled by about three times of its atomic value, to reach phases is moved towards larger ξ and B regime in Sb 10 BL, as compared to Bi BL. Third, and most important, magnetic semiconductors with diverse magnetic order- in the Sb BL in the intermediate ξ regime (0.8ξ ≤ξ ≤ ing and a wide range of lattice constants.54 We report Sb 1.8ξ ) and in a larger exchange field, the QAH phase here on first principles calculations of Bi(111) bilayer Sb bares a Chern number with the sign opposite to that on top of EuSe slabs along (111) direction terminated in the Bi BL: C = −1 for Sb as compared to C = +1 with Se atoms, Fig. 3(d), assuming ferromagnetic order for Bi. We observe that the occurrence of the C = −1 in EuSe with magnetization direction perpendicular to QAHphaseisextremelysensitivetothefinedetailsofthe the slab, but the conclusions below also hold for other electronicstructureofthebilayer. Forinstance,atB =1 europium chalcogenides. EuSe has cubic structure with eV,a“−1”QAHEphaseisalsopresentinasmallregion the lattice constant of 6.19 ˚A, which provides the small- of the SOC strength 2.6ξ ≤ ξ ≤ 2.65ξ . Moreover, estlatticemismatchofaround3%toBi(111)BLascom- Sb Sb for 1.80ξ ≤ξ ≤2.60ξ and B ≈1 eV, the Sb bilayer pared to other europium chalcogenides. To describe Eu Sb Sb is gapped with a tiny gap of a few meV, which can not 4f-electrons properly, we employed the LDA+U scheme be measured in a realistic experiment in which disorder with U =7.0 eV and J =1.2 eV. As shownin Fig. 3(d), and temperature effects are inevitable. the states of the system around E are originated from F The reason for the differences in the phase diagrams Bi bands, with the largestexchange splitting induced by ofSb(111)and Bi(111)BLs canbe attributed to the dif- hybridization with the magnetic substrate accounting to ference in the fine details of their electronic structure. about 60 meV. Such magnitude of the exchange split- Forinstance,themagnitudeoftheV nearestneighbor ting is definitely not enough to reach the QAH phase in ppσ hopping parameter in Bi accounts to 80% of that in Sb, Fig. 3(a), as verified by explicit AHC calculations (not while the values of the V next nearest neighbor hop- shown), although it might be used to probe the time- ppσ ping parameter in Bi is only half of that in Sb, as listed reversal broken TI phase in the system, if the Fermi en- in Ref. [30]. This underlines that, to a certain extent, ergy of the hybrid systems can be tuned to locate in the Bi Sb alloys cannot be treated within a simple SOC gap. ThesmallexchangesplittingintheBLisduetothe 1−x x strengthscalingpicture, andwesuspectthatmoreinter- hybridizationwiththehalf-filledshellof4f Euelectrons, esting and non-trivial topological phases might occur in which is quite localized leading to insignificant exchange these alloysin a strong exchangefield due to the compe- interactions in the 6p orbitals of Biatoms. In this sense, titionbetweenC =±1phases,whichhavetobecaptured usingasubstratewithpartiallyfilledd-shell,e.g.MnSe,55 from more accurate first principles calculations. might be favorable. TheQAHphasewithC=−1foundinSbBL(Fig.3(c)) Ontheotherhand,dopingwithoradsorptionof3d,4d istopologicallydifferentfromtheQAHphasewithC=+1 or5dtransition-metalatomscaninduce strongexchange in Bi BL (Fig. 3(a-b)). According to the bulk-edge cor- splitting and hence a non-trivial QAH effect in TIs, as respondence, the bulk properties (Chern numbers) are shown in Refs. [7–10]. To demonstrate that this is in- reflectedinthe chiralityofthe edgestatesinafinite sys- deed the case for Bi bilayer, here we consider the case tem. Obviously, the chiralities of the edge states for the of Fe adatoms. Compared to graphene, Bi(111) bilayer C =±1QAHphasesareoppositetoeachother,asshown has a larger in-plane lattice constant (4.52 ˚A, as com- in the insets ofFig. 3(a) andFig. 3(c). That is, the edge paredto2.46˚Aingraphene). Ourstructuralrelaxations statelocatedontheupperedgeofaSb(111)ribbon(inset show that the Fe atoms can be stabilized at the hollow ofFig.3(c)) isright-propagating,while thatinaBi(111) site of the hexagonal plaquette in the middle of two Bi ribbon (inset of Fig. 3(a)) is left-propagating. The SHC atomic layers at the spatial inversion point of the stag- is of finite magnitude for both C =±1 phases (Fig. 3(b) gered honeycomb lattice. Fig. 3(e) shows the electronic for C =+1), but of opposite sign due to different chiral- structure of Fe adatoms in p(2 × 2) superstructure on ities of the edge states. Overall, the ±1 QAHE phases Bi(111)BL,togetherwiththecalculatedanomalousHall in Bi and Sb BLs are the quantum spin Chern insulator conductivity. Among two global gaps which are formed phases with opposite values of the anomalous and spin upon Fe deposition, the band gap at the Fermi energy Hall conductivities. is trivial, while the band gap at about 0.6 eV below E F exhibits a non-zero Chern number C = +2 (Fig. 3(e)). This is a new QAH phase, as compared to the phase di- C. Tuning B in Bi bilayers: magnetic substrates agram of Bi BL in Fig. 3(a), whose emergence is due to and doping a strong hybridizationbetween the Fe d- and Bi p-states inthevicinityoftheFermienergy,inanalogytothecase InordertoinduceafiniteexchangefieldinBi/Sb(111) of graphene.10,56 A similar mechanism also plays out for bilayers, two realistic ways can be suggested: (i) either the p(2×2) Co doped Bi(111) BLs, in which case the by depositing the bilayers on top of a suitable magnetic QAH gap is opened at about 0.3 eV below E and the F insulating substrate whose surface layer exhibits an in- Chern number is +2. Furthermore, varying the strength plane ferromagnetic spin structure, or (ii) by doping the of SOC on Bi atomic sites can drive the system into yet systems with magnetic atoms. We first briefly consider different topological phases. For instance, as exempli- (i). fied in Fig. 3(f), if we scale the strength of SOC of Bi Europium chalcogenides (EuX, X=O, S, Se, Te) are atoms to 20% of their atomic value, the gap at E of Fe F