The strong direct detection limits could be pointing to dark matter – nucleus scattering at loop level. We study in detail the prototype example of an electroweak singlet (Dirac or Majorana) dark matter fermion coupled to an extended dark sector, which is composed of a new fermion and a new scalar. Given the strong limits on colored particles from direct and indirect searches we assume that the fields of the new dark sector are color singlets. We outline the possible simplified models, including the well-motivated cases in which the extra scalar or fermion is a Standard Model particle, as well as the possible connection to neutrino masses. We compute the contributions to direct detection from the photon, the Z and the Higgs penguins for arbitrary quantum numbers of the dark sector. Furthermore, we derive compact expressions in certain limits, i.e., when all new particles are heavier than the dark matter mass and when the fermion running in the loop is light, like a Standard Model lepton. We study in detail the predicted direct detection rate and how current and future direct detection limits constrain the model parameters. In case dark matter couples directly to Standard Model leptons we find an interesting interplay between lepton flavor violation, direct detection and the observed relic abundance.