A bullet-carried rust removal mechanism and a rust removal device are provided. The bullet-carried rust removal mechanism includes bullets, the bullets can reciprocate linearly under the driving of an external driving mechanism to thereby repeatedly knock a surface to be derusted, and an end of each of the bullets away from the driving mechanism is a knocking portion for derusting conveniently. The bullet-carried rust removal mechanism has a relatively small volume and a position to be derusted can be accurately positioned during derusting.