A cutter holder assembly has a cutter holder, a cutting unit, a joint connector, and a nut. The cutter holder has a receiving hole, a conical hole, and a connection hole disposed therein and communicating with one another. The cutting unit has an assembling portion inserted in the conical hole, an engaging hole being polygonal, an extension pole extending from the assembling portion, and at least one machining tool assembled to the extension pole. The joint connector is disposed in the receiving hole and has a connection section threaded within the connection hole and an engaging section engaging with the engaging hole. The nut is connected to the cutter holder. Wherein, the joint connector is fixed within the cutter holder and engages with the cutting unit to firmly connect the cutter holder and the cutting unit and to promote machining efficiency and quality of the cutter holder assembly.