We explain all features of lepton and quark mixing in a scenario with the flavor symmetry Δ(384) and a charge conjugation-parity (CP) symmetry, where these are broken in several steps. The residual symmetry in the neutrino and up quark sector is a Klein group and CP, while a Z3 and a Z16 symmetry are preserved among charged leptons and down quarks, respectively. If the Klein group in the neutrino sector is further broken down to a single Z2 symmetry, we obtain predictions for all lepton mixing parameters in terms of one real quantity, whose size is determined by the value of the reactor mixing angle. The Dirac and Majorana phases are fixed, in particular sinδ≈-0.936. A sum rule, relating these CP phases and the reactor and atmospheric mixing angles, θ13 and θ23, is given. In the quark sector, we have for the Cabibbo angle θC=sinπ/16≈0.195 after the first step of symmetry breaking. This is brought into full accordance with experimental data with the second step of symmetry breaking, where either the Z16 group is broken to a Z8 symmetry in the down quark sector or the Klein group to one Z2 symmetry only among up quarks. The other two quark mixing angles are generated in the third and last symmetry breaking step, when the residual symmetries in the up and/or down quark sector are further broken. If this step occurs among both up and down quarks, the amount of CP violation in the quark sector is determined by the lepton sector and explaining the current neutrino oscillation data entails that the Jarlskog invariant JCPq is in very good agreement with experimental findings. Last, a sum rule is derived that contains the CP phase δq and θC of the quark sector and the lepton mixing parameters θ13, θ23, and δ.