Cyber threat reports are a key resource for understanding adversary behavior, often interpreted through frameworks such as MITRE ATT&CK. However, deriving tactics, techniques, and procedures (TTPs) from these reports is a manual and time-consuming process, which has motivated many automated extraction approaches. In practice, it remains difficult to compare these methods, as many open-source implementations are difficult to run, lack usable artifacts, or are evaluated under varying conditions.
In this work, we examine automated TTP extraction with a focus on evaluating these systems in a consistent and repeatable way. We first study the existing literature and open source implementations to identify practical barriers to reuse and comparison. We then develop an evaluation system that standardizes how tools are executed and assessed, along with a dataset that combines prior sources with additional annotated reports. Using this setup, we adapt several previously proposed systems and evaluate them under a unified experimental setting, including tests on data not used in their original studies. We find that only a limited portion of prior results can be reproduced, and that performance drops substantially when evaluated on previously unseen data.
Our results highlight gaps between reported and observed performance and suggest that these systems require further maturation before they can be reliably operationalized. By providing a more consistent evaluation process and dataset, this work aims to make it easier to reproduce prior results and to support more meaningful comparisons of future TTP extraction systems.