We propose a mechanism that can convert a sizable fraction of neutron stars into black holes with mass ∼1 M, too light to be produced via stellar evolution. We show that asymmetric fermionic dark matter of mass on the teraelectron volt scale, with attractive self-interaction within the range that alleviates the problems of collisionless cold dark matter, can accumulate in a neutron star and collapse, forming a black hole that converts the rest of the star to a solar mass black hole. We estimate the fraction of neutron stars that can become black holes without contradicting neutron star observations. Such solar mass black holes could be in binary systems, which may be searched for by existing and forthcoming gravitational wave detectors. The (non-)observation of binary mergers of solar mass black holes may thus test the nature of dark matter.