Building Generalized Foundation Model for Embodied AI (EAI) is a significant challenge. It necessitates not only comprehending the world through multi-modal sensors but also the capacity to adapt to novel tasks, environments, and hardware configurations through continual interaction with the world. Large-scale generative models exhibit the potential for learning-to-learn, enabling them to generalize to new tasks and acquire new skills without parameter tuning. Based on it, General-Purpose In-Context Learning (GPICL) offers a promising approach to addressing diverse tasks and hardware configurations using a single foundational model. Furthermore, it has been observed that the extent of generalization is not solely reliant on parameter scale but is more significantly influenced by the diversity of data sets, the length of context, and the size of memory states. Those insights suggests potential new pathways for developing a generalized foundation model for EAI.