Business ecosystem is popularly used to investigate a complex social system with the business perspective, and particularly contributes to the understanding of actors and their relations in the innovation research. However, the aspect of business ecosystem modeling is limited discussed in the literature, although the importance has emerged significantly in recent years due to the emphasis on cross-disciplinary research and digitalization with artificial intelligence. Therefore, this paper proposes a framework for business ecosystem modeling with the discussion of system engineering and ecological modeling. The domain of smart grid is selected to demonstrate how system engineering, especially standards and ontologies contribute to the business ecosystem modeling. The proposed framework of the business ecosystem modeling includes three parts and nine stages that combines theories from system engineering, ecology, and business ecosystem. Part I-Business ecosystem architecture development includes four stages which aims to identify a target business ecosystem and its elements (actors, roles, and interactions). Part II-Factor analysis includes two stages to identify potential changes (and the dimensions of the changes) in the ecosystem. Part III- Ecosystem simulation and reconfiguration aims to use simulations to investigate the transition of an ecosystem and the re-configurated ecosystem. The framework not only provides a systematic approach for modeling a business ecosystem but also provides a methodological foundation for research on the aspect of complex systems in the business ecosystem field.