GPAI Training Transparency

Our work makes the following contributions:

• We provide a framework to assess the quality of public summaries of training content required under the AI Act's Article 53(1)(d).
• Our quality assessment metrics represent best practices for how the information in the public summary should be provided (transparency) in order for rightsholders to utilise it effectively (usefulness).
• We found only a handful published public summaries, and that they are mostly of a high-quality. Meanwhile other providers, noticeably larger ones, have not published anything yet. Our work sufficiently demonstrates that this is an intentional choice and that the legal obligation is not a burden as the current high-quality summaries are from small organisations and open source oriented efforts.

Our work also contributes towards improving the ecosystem:

• We contend that compliance cannot be fait accompli, and that the public summaries are a key factor in creating transparency and enabling rights enforcement. Towards this, our work also acts as a guide for providers who are yet to publish their summaries to consider how to do so with the highest possible quality and utility.
• Compliance also invites practices that are intentionally or unintentionally deficient in achieving the goals. Our work serves as a useful tool for describing how and where and why certain practices are 'bad', e.g., where they use obfuscation, do not provide stated information. Using this, we can detect trends or patterns in whether the same issues occur in many summaries, and if so, how they can be collectively addressed through guidance, or enforced with priority.
• The largest challenge in undertaking this work has been finding public summaries as there is no consistent format or practice for how they should be provided. For this, we provide recommendations.
• The template for public summaries provided by the AI Office is intended to be revised with time to improve the state of documentation as well as to better guide the providers. We also provide recommendations for these to improve the quality and accessibility of the public summaries.