Emotion awareness research in SE context has been growing in recent years. Currently, researchers often rely on textual communication records to extract emotion states using natural language processing techniques. However how well these extracted emotion states reflect people’s real emotions has not been thoroughly investigated. In this paper, we report a multi-level, longitudinal empirical study with 82 individual members in 27 project teams. We collected their self-reported retrospective emotion states on a weekly basis during their year-long projects and also extract corresponding emotions from the textual communication records. We then model and compare the dynamics of these two types of emotions using multiple statistical and time series analysis methods. Our analyses yield a rich set of findings. The most important one is that the dynamics of emotions extracted using text-based algorithms often do not well reflect the dynamics of self-reported retrospective emotions. Besides, the extracted emotions match self-reported retrospective emotions better at the team level. Our results also suggest that individual personalities and team’s emotion display norms significantly impact the match/mismatch. Our results should warn the research community about the limitations and challenges of applying text-based emotion recognition tools in SE research.