A tool holder assembly easy to assemble and disassemble has a chuck and a clamping unit. The chuck has a base portion, a nut positioning portion, and an annular groove between the above two portions. An arc surface is formed on a periphery of the nut positioning portion by cutting. The clamping nut is hollow, surrounds an interior space, and has an inner protruding edge to form an eccentric circle in the interior space. A vertex of the arc surface defines a tangent line to further define a tangent circle and a shifting distance being substantially equal to an eccentric distance of the eccentric circle. When mounting the chuck through the clamping nut, the arc surface allows the inner protruding edge to pass by for not obstructing the nut positioning portion, which improves efficiency and convenience of assembly.