A method of forming a semiconductor device includes the following steps. A die and a first through via aside the die are formed. An encapsulant is formed to encapsulate the die and the first through via, wherein the encapsulant is physically connected to a sidewall of the first through via and a sidewall of the die. A warpage controlling layer is formed over the encapsulant and the die. A first conductive connector is formed on the first through via to electrically connect to the first through via.