What we test
Test catalog
The 51 tests we run against your agents, across the six categories we assess: hallucinations, token efficiency, data privacy, security, robustness at scale, and permissions compliance. Expand any test for its detail.
Hallucinations
10 testsCatch fabricated, harmful, or out-of-scope outputs before they reach a business record, with recurring third-party evaluation
D001Prevent hallucinated outputs
Implement safeguards or technical controls to prevent hallucinated outputs
Implementing factual accuracy controls. For example, deploying available fact-checking mechanisms, flagging uncertain or low-confidence responses.
Code or configuration showing groundedness validation - may include filters checking responses against source documents, fact-checking API integration, or logic comparing generated content to retrieved context for factual accuracy.
Establishing information source validation. For example, requiring citations for factual claims, implementing source reliability checks.
UI or output format showing citations and source attributions provided to users - may include inline citations, source links, reference lists, or attribution labels identifying where information originated.
Maintaining uncertainty communication. For example, displaying confidence levels, providing appropriate disclaimers for generated information.
UI or output format showing confidence levels, uncertainty disclaimers, or warnings for generated information - may include confidence score displays, low-certainty warnings, or standard disclaimers about potential inaccuracies.
- MEASURE 2.5: Validity and reliability
- LLM05:25 - Improper Output Handling
- LLM09:25 - Misinformation
- AIS-09: Output Validation
- MDS-10: Model Continuous Monitoring
- MDS-11: Model Failure
D002Hallucinations
Appoint expert third-parties to evaluate hallucinated outputs at least every 3 months
Appointing qualified third-party assessors. Including selecting assessors with relevant technical capabilities for identified risk areas, maintaining records of assessor qualifications and independence.
Conducting regular testing. Including defining testing scope and methodologies based on risk taxonomy and performing assessments at least every quarter.
Maintaining documentation. Including testing scope, results, and remediation actions taken, tracking follow-up activities and resolution timelines.
Third-party evaluation report showing hallucination testing - must include risk taxonomy tested, testing methodology and findings, and improvement tracking with remediation timelines and documentation.
- GOVERN 4.3: Testing and incident sharing
- MANAGE 2.2: Deployed system value
- MEASURE 1.3: Independent assessment
- MEASURE 2.1: TEVV documentation
- MEASURE 2.5: Validity and reliability
- MEASURE 4.1: Context-specific measurement
- MEASURE 4.2: Trustworthiness validation
- LLM09:25 - Misinformation
- AIS-09: Output Validation
- AIS-05: Application Security Testing
- MDS-06: Adversarial Attack Analysis
- MDS-07: Robustness against Adversarial Attack / Model Hardening
C003Prevent harmful outputs
Implement safeguards or technical controls to prevent harmful outputs including distressed outputs, angry responses, high-risk advice, offensive content, bias, and deception
Implementing content filtering for harmful output types. For example, detecting and blocking distressed responses, angry language, offensive content, biased statements, and deceptive information.
Content filtering rules, moderation API configuration, or classifier settings showing detection and blocking logic for harmful output types - may include filtering rules in code, third-party moderation tool configuration (e.g., OpenAI Moderation API, Perspective API), or custom classifier model settings with harm category definitions.
Implementing guardrails for advice generation. For example, restricting high-risk recommendations in sensitive domains, requiring disclaimers for guidance.
System prompts, guardrail rules, or domain restrictions showing safety controls on advice generation - may include defensive prompting, domain-specific output restrictions (e.g., medical/legal/financial advice blocklists), or conditional response templates that add warnings for sensitive topics.
Implementing bias detection and mitigation controls. For example, monitoring for discriminatory patterns, implementing fairness checks in outputs.
Documentation of bias eval results testing for stereotypical responses across demographic attributes, manual review logs documenting bias assessments, or output filtering rules blocking discriminatory patterns - may include automated fairness evaluation tools or bias monitoring dashboards if implemented.
Evaluating harm mitigation controls using performance metrics.
Test results, metrics dashboard, or evaluation report showing performance of harm controls - may include false positive/negative rates, coverage analysis of test scenarios, benchmark results against harm datasets (e.g., ToxiGen, RealToxicityPrompts), or confusion matrices showing filtering accuracy across harm categories.
- Article 9: Risk Management System
- MEASURE 2.11: Fairness and bias
- LLM05:25 - Improper Output Handling
- LLM09:25 - Misinformation
C004Prevent out-of-scope outputs
Implement safeguards or technical controls to prevent out-of-scope outputs (e.g. political discussion, healthcare advice)
Detecting and blocking out-of-scope requests. For example, detecting conversations outside intended use cases, blocking prohibited topics, providing redirection messages when users hit boundaries, and escalating or restricting access for repeated violations.
Blocking rules, defensive prompting, or filtering configuration showing how out-of-scope requests are detected and handled - may include topic blocklists, redirection message templates, escalation rules for repeated attempts, or system prompts defining allowed topics.
Tracking out-of-scope violations and updating boundaries. For example, logging boundary violations, adjusting restrictions based on misuse patterns.
Logs showing out-of-scope attempts with frequency data. May include documentation of boundary updates made in response to violations, monitoring dashboard of flagged requests, change log showing restriction updates with rationale, or incident reports triggering scope adjustments.
Providing user guidance on system capabilities and limitations. For example, communicating what the AI system can and cannot do, intended use cases, and topics or requests outside the system's scope.
User-facing guidance explaining system capabilities and limitations - may include onboarding tooltips or welcome screens, help documentation or FAQs describing intended use, UI warnings when approaching scope boundaries, or published usage guidelines.
- Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems
- MAP 2.2: Knowledge limits
- MAP 3.4: Operator proficiency
- LLM05:25 - Improper Output Handling
C005Prevent agent-specific high risk outputs
Implement safeguards or technical controls to prevent agent-specific high-risk outputs as defined in risk taxonomy
Implementing detection and blocking mechanisms aligned with organizational risk taxonomy. For example, deploying filtering based on defined risk categories and severity thresholds.
Implementing response actions for detected risks. For example, blocking high-severity outputs, flagging medium-risk content for review, logging violations for monitoring and analysis.
Filtering rules, system configuration, or code showing detection logic mapped to AI risk taxonomy categories and corresponding response actions per severity level - may include risk classifiers with block/flag/log rules, content moderation API configuration defining actions by risk type, or defensive prompting.
Establishing escalation procedures for flagged high-risk content. For example, defining when human review is required and establishing approval workflows for edge cases.
Documentation or workflow configuration showing human review and escalation procedures for flagged content - may include runbook defining escalation criteria and review SLAs, workflow diagram showing approval process, or ticketing system configuration (Jira, Linear) with content review queues and assignment rules.
Implementing automated real-time interventions. For example, blocking or modifying outputs based on severity.
Code or system configuration showing automated response mechanisms - may include logic blocking or modifying outputs based on risk scores, or dynamic warning messages triggered by content flags.
- Article 9: Risk Management System
- MANAGE 1.4: Residual risk documentation
- LLM05:25 - Improper Output Handling
C010Harmful outputs
Appoint expert third parties to evaluate system robustness to harmful outputs including distressed outputs, angry responses, high-risk advice, offensive content, bias, and deception at least every 3 months
Appointing qualified third-party assessors. Including selecting assessors with relevant technical capabilities for identified risk areas, maintaining records of assessor qualifications and independence.
Conducting regular testing. Including performing assessments of harmful outputs at least every quarter, defining testing scope and methodologies based on risk classifications and industry benchmarks like ToxiGen, coordinating with internal security and testing teams.
Maintaining documentation. Including testing scope, results, and remediation actions taken, tracking follow-up activities and resolution timelines.
Third-party evaluation report showing harmful output testing - must include documentation of assessor qualifications, testing methodology and findings, and improvement tracking with remediation timelines and documentation.
- GOVERN 4.3: Testing and incident sharing
- MANAGE 2.2: Deployed system value
- MEASURE 1.3: Independent assessment
- MEASURE 2.1: TEVV documentation
- MEASURE 2.6: Safety evaluation
- MEASURE 2.11: Fairness and bias
- MEASURE 4.1: Context-specific measurement
- MEASURE 4.2: Trustworthiness validation
- Article 9: Risk Management System
- GRC-11: Bias and Fairness Assessment
- A&A-02: Independent Assessments
- TVM-06: Penetration Testing
C011Out-of-scope outputs
Appoint expert third parties to evaluate system robustness to out-of-scope outputs at least every 3 months (e.g. political discussion, healthcare advice)
Appointing qualified third-party assessors. Including selecting assessors with relevant technical capabilities for identified risk areas, maintaining records of assessor qualifications and independence.
Conducting regular testing. Including defining testing scope and methodologies based on risk taxonomy and performing assessments of out-of-scope outputs at least every quarter.
Maintaining documentation. Including testing scope, results, and remediation actions taken, tracking follow-up activities and resolution timelines.
Third-party evaluation report showing out-of-scope output testing - must include documentation of assessor qualifications, testing methodology and findings, and improvement tracking with remediation timelines and documentation.
- GOVERN 4.3: Testing and incident sharing
- MANAGE 2.2: Deployed system value
- MAP 2.2: Knowledge limits
- MEASURE 1.3: Independent assessment
- MEASURE 2.1: TEVV documentation
- MEASURE 2.6: Safety evaluation
- MEASURE 4.1: Context-specific measurement
- MEASURE 4.2: Trustworthiness validation
- A.6.2.4: AI system verification and validation
- IBM 69: Output - Improper usage
- IBM 83: Non-Technical - Incomplete usage definition
C012Customer-defined risk
Appoint expert third-parties to evaluate system robustness to additional high-risk outputs as defined in risk taxonomy at least every 3 months
Appointing qualified third-party assessors. Including selecting assessors with relevant technical capabilities for identified risk areas, maintaining records of assessor qualifications and independence.
Conducting regular testing. Including defining testing scope and methodologies based on risk taxonomy and performing assessments of high-risk areas at least every quarter.
Maintaining documentation. Including testing scope, results, and remediation actions taken, tracking follow-up activities and resolution timelines.
Third-party evaluation report showing testing of customer-defined risk - must include documentation of assessor qualifications, testing methodology and findings, and improvement tracking with remediation timelines and documentation.
- GOVERN 4.3: Testing and incident sharing
- MANAGE 2.2: Deployed system value
- MEASURE 1.3: Independent assessment
- MEASURE 2.1: TEVV documentation
- MEASURE 2.6: Safety evaluation
- MEASURE 4.1: Context-specific measurement
- MEASURE 4.2: Trustworthiness validation
- A.6.2.4: AI system verification and validation
- A&A-02: Independent Assessments
- TVM-06: Penetration Testing
E002AI failure plan for harmful outputs
Document AI failure plan for harmful AI outputs that cause significant customer harm assigning accountable owners and establishing remediation with third-party support as needed (e.g. legal, PR, insurers)
Implementing customer communication protocols. For example, disclosure procedures, explanation of corrective actions, and follow-up commitments with executive approval for significant incidents.
Establishing immediate mitigation steps with designated staff responsibilities. For example, system freeze capabilities, output suppression, customer notification, and system adjustments.
Can be standalone document or integrated in existing incident response procedures/policies
Defining harmful output categories with reference to risk taxonomy. For example, discriminatory content, offensive material, inappropriate recommendations, ideally with concrete examples.
Coordinating external support engagement. For example, legal counsel consultation, PR support, and insurance claim procedures.
May include harmful output category definitions referenced to risk taxonomy, external support contact list (legal counsel, PR firms, insurance providers), support engagement procedures or runbooks, or escalation criteria for involving external parties.
- Article 20: Corrective Actions and Duty of Information
- Article 73: Reporting of Serious Incidents
- A.8.4: Communication of incidents
- GOVERN 4.3: Testing and incident sharing
- MANAGE 1.3: Risk response planning
- MANAGE 4.3: Incident communication
E003AI failure plan for hallucinations
Document AI failure plan for hallucinated AI outputs that cause substantial customer financial loss assigning accountable owners and establishing remediation with third-party support as needed (e.g. legal, PR, insurers)
Implementing customer communication protocols. For example, disclosure procedures, explanation of corrective actions, and follow-up commitments with executive approval for significant incidents.
Establishing immediate mitigation steps with designated staff responsibilities. For example, system freeze capabilities, model adjustments, output validation improvements, customer notification, and enhanced monitoring.
Can be standalone document or integrated in existing incident response procedures/policies
Defining hallucination incident types.
Coordinating potential external support. For example, legal consultation for significant claims, financial review when needed, and insurance coverage activation.
May include hallucination incident categories (e.g. factual errors, incorrect recommendations), external support contact list (legal counsel, financial reviewers, insurance providers), support engagement procedures, or escalation criteria for involving external parties.
- Article 20: Corrective Actions and Duty of Information
- Article 73: Reporting of Serious Incidents
- A.8.4: Communication of incidents
- GOVERN 4.3: Testing and incident sharing
- MANAGE 1.3: Risk response planning
- MANAGE 4.3: Incident communication
Token efficiency
2 testsKeep outputs scoped and cost-disciplined, backed by a quality management system proportionate to the deployment
B009Limit output over-exposure
Implement output limitations and obfuscation techniques to safeguard against information leakage
Reducing or limiting the number of results shown in outputs to relevant only to balance security and utility. For example, character limits, limits on inference time.
Code or configuration implementing output restrictions - may include character or token limits, inference time limits, result count restrictions, or timeout configurations preventing excessive output. Can be demonstrated by product demo showing system timeout when requesting output exceeding limits.
Providing user-facing notices or documentation about output limitations.
Product interface showing user notices about output limitations - may include messages indicating truncated or suppressed outputs for security or privacy reasons, user documentation explaining limitation policies, or help articles describing output restrictions.
Limiting the fidelity of model outputs in certain use cases. For example, applying output rounding, threshold bands, or obfuscation techniques to reduce the risk of model inversion.
Code implementing output fidelity limitations - may include rounding logic for numerical outputs, threshold bands reducing precision, or obfuscation techniques preventing model inversion, precision-sensitive data disclosure, or adversarial model extraction attacks.
- AML-M0002: Passive AI Output Obfuscation
- MEASURE 2.10: Privacy risk assessment
- LLM02:25 - Sensitive Information Disclosure
- LLM05:25 - Improper Output Handling
- LLM08:25 - Vector and Embedding Weaknesses
- LLM09:25 - Misinformation
E013Implement quality management system
Establish a quality management system for AI systems proportionate to the size of the organization
Defining quality objectives, metrics, and risk management approach for AI systems. For example, establishing performance targets, safety thresholds, risk assessment methodologies, and measurement processes appropriate to system risk level.
Documentation showing quality objectives, metrics, and risk management approach - may include quality metrics dashboard or reports, risk assessment documentation for AI systems, performance targets and safety thresholds, or measurement methodologies defining how quality is evaluated.
Establishing change management, approval processes, and documentation standards. For example, defining review and approval requirements for AI system changes, assigning accountability for quality decisions, documenting design and development procedures.
Documentation showing change management and approval processes - may include change approval workflows or procedures, RACI matrix assigning accountability for quality decisions, design and development procedure documents, or documentation standards and templates for AI systems. May be fulfilled by evidence submitted to E004: Assign accountability.
Implementing defect tracking, continuous improvement, and post-market monitoring. For example, maintaining issue tracking systems, conducting root cause analysis, documenting corrective actions, establishing post-market monitoring processes.
Issue tracking system or monitoring records - may include issue tracker (Jira, Linear, GitHub) with defects and corrective actions, root cause analysis reports, post-market monitoring logs or dashboards, or continuous improvement documentation showing lessons learned.
Establishing data management and record-keeping systems. For example, documenting data governance procedures, maintaining technical documentation, implementing record retention policies for model training data and system outputs.
Documentation showing data management and record-keeping practices - may include data governance policies, technical documentation standards, record retention procedures, or data lineage tracking systems for training data and system outputs.
Documenting communication procedures with regulatory authorities and stakeholders. For example, establishing protocols for regulatory reporting, stakeholder notifications for incidents, and procedures for authority interactions.
Procedures document or communication protocols - may include incident reporting templates or protocols to regulatory authorities, stakeholder notification procedures for serious incidents, guidelines for interacting with competent authorities or notified bodies, or escalation procedures for regulatory communications.
- Article 9: Risk Management System
- Article 10: Data and Data Governance
- Article 11: Technical Documentation
- Article 12: Record-Keeping
- Article 16: Obligations of Providers of High-Risk AI Systems
- Article 17: Quality Management System
- Article 18: Documentation Keeping
- Article 19: Automatically Generated Logs
- Article 26: Obligations of Deployers of High-Risk AI Systems
- Article 43: Conformity Assessment
- Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems
- Article 73: Reporting of Serious Incidents
- GOVERN 1.4: Risk management governance
- GOVERN 1.3: Risk management processes
- A.5.2: AI system impact assessment process
- A.6.2.7: AI system technical documentation
- A.5.3: Documentation of AI system impact assessments
- A.5.4: Assessing AI system impact on individuals or groups of individuals
- A.4.2: Resource documentation
- 4.4: AI management system
- 6.1.4: AI system impact assessment
- 7.1: Resources
- 7.5.1: Documented information — General
- 8.1: Operational planning and control
- 8.4: AI system impact assessment
- A.6.2.3: Documentation of AI system design and development
- 9.1: Monitoring, measurement, analysis and evaluation
- 10.1: Continual improvement
- 10.2: Nonconformity and corrective action
Data privacy
10 testsProtect against data leakage, IP leakage, PII exposure, and training on user data without consent
A001Establish input data policy
Establish and communicate AI input data policies covering how customer data is used for model training, inference processing, data retention periods, and customer data rights
Defining and communicating input data usage policies. Including specifying how customer data is used for inference and model training, establishing data retention periods, and documenting customer data rights.
Typically demonstrated by Terms of Service, Privacy Policy or Data Processing Agreement
Implementing technical controls to enforce data retention and deletion policies. For example, automating data deletion based on retention schedules, using secure removal mechanisms, and managing data lifecycles.
Implementation of automated deletion schedule or data lifecycle system - may include cron job or scheduled task deleting expired data, deletion script in Python/Bash with retention period logic, data lifecycle management tool configuration (e.g., AWS S3 lifecycle rules, database TTL settings), or deletion audit logs from database or storage system.
Documenting processes for handling end-user data subject rights. For example, handling requests for opt-in/opt-out rights, access, portability, or deletion of input data.
May be included in DPA, GDPR appendix, External Privacy Policy or similar internal or external policies documenting processes for data handling
- Article 11: Technical Documentation
- A.7.2: Data for development and enhancement of AI system
- A.7.3: Acquisition of data
- MEASURE 2.10: Privacy risk assessment
A002Establish output data policy
Establish AI output ownership, usage, opt-in/out and deletion policies to customers and communicate these policies
Establishing output ownership and usage rights policies. For example, specifying customer ownership of AI-generated outputs versus AI inputs, defining permitted uses of outputs (commercial use, redistribution, modification), documenting usage restrictions or limitations, and clarifying how ownership applies to different output types or use cases.
Disclosing opt-in/opt-out and deletion policies for AI outputs. For example, documenting how customers can opt out of output storage or reuse, explaining deletion request processes, specifying retention periods and data handling practices, and clarifying how customers can control or revoke permissions for their outputs.
Typically demonstrated by Terms of Service, Data Processing Agreement, Master Service Agreement, Privacy Policy, or AI Addendum. May be a combination of these policies.
Implementing technical controls to enforce AI output opt-in/opt-out and deletion policies. For example, automating customer preference enforcement through consent management configuration, processing opt-out and deletion requests within defined workflows, and validating that opted-out outputs are excluded from storage and downstream reuse.
Implementation of opt-in/opt-out enforcement mechanism or output deletion workflow - may include consent management system enforcing customer preferences (e.g., feature flag gating output storage, consent database linked to processing pipeline), opt-out or deletion request processing script or automated task, or audit logs from consent and deletion systems with timestamps and execution records.
- DSP-16: Data Retention and Deletion
- DSP-08: Data Privacy by Design and Default
- STA-10: Primary Service and Contractual Agreement
- IBM 32: Training Data - Data privacy rights alignment
- IBM 90: Non-Technical - Generated content ownership and IP
A003Limit AI agent data access
Implement safeguards to limit AI agent data access based on task, user role, agent role and context
Enabling agent identity management. For example, assigning each agent a unique, cryptographically verifiable identity; supporting standard identity federation protocols (e.g., OAuth 2.0, OIDC) for enterprise IAM integration; publishing agent cards declaring each agent's capabilities, tools, and permission scopes.
Documentation showing how identity governance is enabled - may include platform configuration showing unique, cryptographically verifiable identity assignment per agent instance, API or SDK documentation for identity federation endpoints (e.g., OIDC token exchange), or a sample agent card declaring agent capabilities and scopes.
Enabling agent access and governance through permission-ready architecture. For example, exposing per-agent permission scopes mappable to enterprise roles; supporting just-in-time permissions scoped to specific subtasks; preventing silent inheritance of elevated permissions from orchestrators or parent agents; enforcing segregation of duties across systems; and integrating data loss prevention controls on agent actions and tool calls.
Documentation showing how permission governance is enabled - may include API or SDK documentation for permission scope configuration mappable to RBAC or ABAC policies, platform settings or code demonstrating just-in-time credential issuance scoped to individual subtasks or tool calls, architecture documentation showing how inherited permission chains from orchestrators or parent agents are bounded or surfaced for review, a segregation of duties matrix identifying and preventing conflicting agent permission combinations across integrated systems, or DLP policy configuration showing inspection and blocking rules applied to outbound data transmitted through agent actions or tool calls.
- MAP 2.1: Task definition
- LLM06:25 - Excessive Agency
- LLM08:25 - Vector and Embedding Weaknesses
- LLM10:25 - Unbounded Consumption
- DSP-07: Data Protection by Design and Default
- AIS-11: Agents Security Boundaries
- DSP-08: Data Privacy by Design and Default
- DSP-22: Privacy Enhancing Technologies
- IAM-17: Knowledge Access Control - Need to Know
- IAM-19: Agent Access Restriction
- MDS-04: Model Documentation Requirements
A004Protect IP & trade secrets
Implement safeguards or technical controls to prevent AI systems from leaking company intellectual property or confidential information
Providing user guidance on protecting confidential information. For example, instructing employees not to input trade secrets, proprietary code, or confidential business information into AI systems, communicating data handling policies for AI tool usage, or establishing clear guidelines on what information can and cannot be shared with AI agents.
Policy document, training materials, or user guidelines instructing users on protecting confidential information when using AI systems.
Leveraging foundation model provider protections. For example, using providers with zero data retention policies, requiring contractual commitments that inputs are not used for training, selecting models with enhanced privacy guarantees for sensitive use cases.
Provider contracts, terms of service, or documentation showing IP protection commitments. Often found in third party's terms of use/service, DPA or AI Addendum/Schedule.
Implementing technical controls to detect proprietary information in outputs.
Code or configuration detecting proprietary information patterns in AI outputs - may include labelling proprietary files, filtering rules for internal identifiers/data labels/API keys, scanning logic for trade secret terminology, or rejection demonstrations showing appropriate responses to proprietary requests.
Establishing output monitoring for high-risk IP scenarios. For example, logging AI responses that accessed confidential data sources, implementing human review workflows for outputs flagged as potentially containing sensitive information.
Logs, audit trails, or review workflow documentation for AI outputs potentially containing sensitive information - may include logs of responses accessing confidential sources, flagged output review queues, or human approval workflows for high-risk disclosures.
- AML-M0020: Generative AI Guardrails
- Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems
- LLM03:25 - Supply Chain
- LLM05:25 - Improper Output Handling
- LLM08:25 - Vector and Embedding Weaknesses
A005Prevent cross-customer data exposure
Implement safeguards to prevent cross-customer data exposure
Establishing explicit consent and disclosure for combined data usage. For example, informing customers when their data will be combined with competitor data, disclosing data anonymization and abstraction policies, providing opt-out mechanisms.
Typically demonstrated by Data Processing Agreement or Terms of Service
Implementing customer data isolation controls. For example, enforcing strict logical and physical separation of customer data, applying tenant-specific encryption, validating data flow boundaries in shared infrastructure, establishing technical barriers between customer datasets during training.
Configuration or code demonstrating enforced isolation of each customer's data. For example, tenant/app-ID enforcement in the database schema, namespace or per-tenant isolation in the retrieval/vector store for RAG, logical isolation appropriate to the architecture.
Implementing specific privacy-enhancing technologies (PETs) to reduce competitive exposure.
May include tokenization, hashing, or anonymization techniques (robust to prevent re-identification or reversal) making data algorithmic-usable but not human-readable, differential privacy implementation obfuscating individual contributions, federated learning configuration avoiding centralized raw data, or data masking/pseudonymization protecting customer identities.
- MEASURE 2.10: Privacy risk assessment
- LLM02:25 - Sensitive Information Disclosure
- LLM05:25 - Improper Output Handling
- LLM08:25 - Vector and Embedding Weaknesses
- I&S-06: Segmentation and Segregation
- DSP-22: Privacy Enhancing Technologies
- UEM-08: Storage Encryption
A006Prevent PII leakage
Establish safeguards to prevent personal data leakage through AI outputs and logs
Implementing safeguards to prevent personal data leakage through AI system outputs and logs. For example, filtering prompts and outputs for personal identifiers before storage or display, implementing automated PII detection and redaction in system logs, preventing retention of outputs containing sensitive personal information, or blocking responses that would expose personal identifiers.
Configuration or code output demonstrating filtering of LLM inputs and/or outputs for personal identifiers - may include keyword checks or regex patterns detecting PII (e.g. names, emails, SSNs, phone numbers), scrubbing functions removing personal data before storage or logging, output filtering blocking responses containing personal identifiers, log redaction configuration removing PII from application or system logs, or structured logging with PII isolation controls.
Integrating with existing data loss prevention (DLP) systems to monitor and block outputs containing personal data in violation of policy.
Output pipeline integration with a DLP system to scan and block PII policy violations - may include DLP integration code scanning AI outputs before delivery to users, DLP configuration rules for PII detection, or logs showing blocked outputs containing personal data.
- AML-M0020: Generative AI Guardrails
- Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems
- MEASURE 2.10: Privacy risk assessment
A007Prevent IP violations
Implement safeguards and technical controls to prevent AI outputs from violating copyrights, trademarks, or other third-party intellectual property rights
Documenting foundation model provider IP protections which may serve as primary infringement safeguards. For example, indemnification clauses or copyright/trademark guardrails.
Foundation model provider contract, terms of service, or data processing agreement showing IP protection commitments including copyright/trademark handling policies, indemnification clauses, liability coverage, and any documented limitations or exclusions. May include vendor questionnaire responses or certification documents addressing IP protections.
Establishing supplementary content filtering mechanisms where provider protections have gaps or limitations. For example, detecting copyrighted material in outputs, implementing trademark screening.
Code, API configuration, or filtering system showing detection of copyrighted material, trademark screening, or content validation checks applied to AI outputs - this could be pattern matching logic, third-party API integration (e.g. copyright detection services), or custom filtering rules.
Implementing user guidance and guardrails to reduce IP risk. For example, usage policies that explain prohibited content types, user warnings in product, restricting output generation in known infringement domains.
Implementing restrictions in AI acceptable use policy.
User-facing IP risk guidance - may include warning messages when attempting high-risk operations, help center articles about IP infringement guidance, or UI elements explaining prohibited use cases.
- AML-M0020: Generative AI Guardrails
- GOVERN 6.1: Third-party risk policies
- MAP 4.1: Legal risk mapping
- LLM03:25 - Supply Chain
- LLM05:25 - Improper Output Handling
A008Prevent leakage of credentials and secrets
Implement safeguards to detect and prevent leakage of secrets in AI system inputs, outputs, logs, and credential storage
Implementing safeguards to detect credentials in user inputs. For example, scanning user prompts and pasted content for API keys, access tokens, private keys, and connection strings using pattern-matching or entropy-based detection; defining handling procedures when detected such as warning the user, refusing to persist, or flagging to the deployer.
Code or configuration demonstrating input-side secrets detection. For example, regex patterns or detection libraries (e.g., detect-secrets, TruffleHog rulesets) scanning user prompts and pasted context, entropy-based detection for high-randomness strings, or integration with third-party secret detection APIs. Evidence should also show the defined handling response (e.g., user warning UI, rejection logic, or alert configuration).
Implementing safeguards to keep secrets out of generated code artifacts. For example, guiding the model via system prompts to reference environment variables or secret-management tools rather than hardcoding credentials, post-generation scanning of output files for common credential patterns, or blocking/flagging generations that contain detected secrets before they are written to disk.
Code, configuration, or system prompt excerpt demonstrating secrets hygiene - may include system prompt guidance directing the model toward envvar or secret-store patterns, scanning logic applied to generated files, sample generations on credential-requiring prompts showing envvar references rather than literals, or block/flag logic preventing persistence of outputs containing detected secrets.
Implementing safeguards to securely store user-provided credentials to enable agent-connected services. For example, storing OAuth tokens, API keys, and connection strings in a dedicated secret manager, encrypting credentials at rest, scoping credential access per tool and per session, or fetching credentials only at the point of tool invocation rather than persisting them in the agent's context window.
Configuration showing secure handling of user-provided credentials — may include secret manager integration (e.g., AWS Secrets Manager, HashiCorp Vault, cloud-native KMS), encryption-at-rest configuration for credential storage, access scoping configuration limiting which tools or sessions can retrieve specific credentials, or code showing just-in-time credential fetching at tool-call time.
Implementing user-facing warnings when potential secrets are detected in user inputs. For example, alerting users when credentials are detected in their prompts.
Documentation of user-facing warnings when secrets are detected in inputs — may include user-facing documentation showing warning messages displayed when credentials are detected in user input before model inference.
Implementing safeguards to prevent secrets from being retained in platform logs, conversation history, and stored artifacts. For example, redacting detected credential patterns before log storage, masking secrets in conversation history displayed to users or support staff, or applying scrubbing functions to prompts and outputs before persistence.
Code or configuration showing secrets redaction in stored data - may include log redaction rules alongside PII patterns (extending A006.1-style logic to credentials), masking logic applied to conversation storage, scrubbing functions in output persistence pipelines, or storage configuration showing sanitization before writes. Can be demonstrated alongside or as part of existing PII redaction controls.
E005Document data storage security
Document data storage security practices considering data sensitivity, regulatory requirements, security controls, and operational needs
Documenting data storage security. For example, assessments around cloud vs. on-premises processing.
Documenting data storage security practices against data sensitivity, regulatory requirements, and operational needs - may include security trust center documentation, deployment decision memos such as cloud vs. on-prem evaluations, risk assessment reports, and records of periodic reviews when requirements changed.
- AML-M0017: AI Model Distribution Methods
- MAP 4.2: Internal risk controls
- LLM03:25 - Supply Chain
E011Record processing locations
Document AI data processing locations
Maintaining AI infrastructure location documentation. For example, geographic locations of foundation model processing locations and inference endpoint regions, documenting third-party AI service provider data handling locations.
Reviewing and updating documentation regularly.
Subprocessor list showing third-party AI provider locations, infrastructure documentation listing cloud regions and inference endpoints, or data flow diagram with geographic processing locations and version history or review dates.
Implementing transfer compliance procedures. For example, assessing data transfer requirements for AI training data and inference processing, maintaining approved transfer mechanisms for foundation model providers and AI infrastructure, mitigating transfer risk for cross-border AI model training.
Demonstrated by DPA, data transfer impact assessments, approved transfer mechanism documentation (Standard Contractual Clauses, adequacy decisions), cross-border data flow approvals for AI training/inference, or risk assessments for international AI processing.
- Article 11: Technical Documentation
- A.7.5: Data provenance
- GOVERN 1.6: AI system inventory
Security
11 testsProtect against adversarial attacks like jailbreaks and prompt injections, and harden the deployment surface
B001Adversarial robustness
Implement adversarial testing program to validate system resilience against adversarial inputs and prompt injection attempts in line with adversarial threat taxonomy
Establishing a taxonomy for adversarial risks. For example, drawing on NIST's AI 100-2e2023 attack classifications and aligning these to system architecture and use cases.
Conducting comprehensive adversarial testing at least quarterly. For example, performing structured red-teaming, prompt injection assessments, jailbreaking attempts, adversarial perturbation testing, semantic manipulation, and simulated malicious tool invocations.
Maintaining secure testing documentation. For example, recording test cases, methods, outcomes, and system behaviors with restricted access controls, implementing secure storage for sensitive testing materials.
Establishing improvement processes based on findings. For example, assigning owners and remediation timelines based on test severity, tracking fixes through risk registers or issue management systems, documenting updates to safeguards and procedures.
Third-party evaluation report showing adversarial robustness testing - must include risk taxonomy tested, testing methodology and findings, secure documentation practices, and improvement tracking with remediation timelines and documentation.
Aligning adversarial testing with broader security testing programs. For example, integrating AI-specific test cases into broader penetration testing, sharing threat models across red/blue teams, aligning test cycles with security audit and compliance calendars.
Penetration test reports with AI-specific test cases, shared threat models, and testing calendars, or documentation of broader security program incorporating AI adversarial testing requirements.
- AML-M0003: Model Hardening
- AML-M0004: Restrict Number of AI Model Queries
- GOVERN 4.3: Testing and incident sharing
- MEASURE 2.1: TEVV documentation
- MEASURE 2.6: Safety evaluation
- MEASURE 2.7: Security and resilience
- LLM01:25 - Prompt Injection
- LLM04:25 - Data and Model Poisoning
- LLM05:25 - Improper Output Handling
- LLM08:25 - Vector and Embedding Weaknesses
B002Detect adversarial input
Implement monitoring capabilities to detect and enable responding to adversarial inputs and prompt injection attempts
Establishing detection and alerting. For example, implementing monitoring for prompt injection patterns, jailbreak techniques, adversarial input attempts, and exceeding rate limits, configuring alerts and threat notifications for suspicious activities.
Monitoring system, SIEM, or detection code showing rules and alerts for adversarial inputs - may include prompt injection detection patterns, jailbreak technique signatures, rate limit monitoring with threshold alerts, or notification configurations (Slack, PagerDuty, email)
Implementing incident logging and response procedures. For example, logging suspected adversarial attacks with relevant context, escalating to designated personnel based on severity, and documenting response actions in a centralized system.
Incident management system or logs showing adversarial attack handling - may include log entries with timestamps and user/session context, escalation runbooks defining severity thresholds, or incident tickets in Jira/PagerDuty/ServiceNow documenting response actions and workflows.
Maintaining detection effectiveness through quarterly reviews. For example, updating detection rules based on emerging adversarial techniques, analyzing incident patterns and documenting system improvements.
Quarterly review documentation showing detection updates - for example, review meeting notes with incident pattern analysis, updated detection rules with version history, or tracking records showing rule improvements (e.g. GitHub/Jira tickets).
Implementing adversarial input detection prior to AI model processing where feasible. For example, using pre-processing filters to flag likely threats before model processing.
Pre-processing filtering logic or gateway - may include pattern-matching or heuristic code checking inputs before model processing, WAF or API gateway rules blocking adversarial patterns, or IP-based filtering.
Integrating adversarial input detection into existing security operations tooling. For example, forwarding flagged inputs to SIEM platforms, correlating detection with authentication and network logs, enabling SOC teams to triage AI-related security events.
SIEM platform, SOC tooling, or log forwarding configuration showing adversarial detection integration - may include Splunk/Datadog/Elastic SIEM ingesting AI adversarial alerts, correlation rules linking AI events with authentication or network logs, SOC dashboard displaying AI security event triage, or code forwarding flagged inputs to security platforms.
- AML-M0003: Model Hardening
- AML-M0015: Adversarial Input Detection
- AML-M0024: AI Telemetry Logging
- AML-M0021: Generative AI Guidelines
- Article 15: Accuracy Robustness and Cybersecurity
- Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems
- GOVERN 1.5: Risk monitoring and review
- MEASURE 2.4: Production monitoring
- MEASURE 2.7: Security and resilience
- MEASURE 3.1: Emergent risk tracking
B003Manage public release of technical details
Implement controls to prevent over-disclosure of technical information about AI systems and organizational details that could enable adversarial targeting
Documenting limitations on technical information release. For example, limiting public disclosure of model architectures, algorithms, training data details, system configurations, and performance metrics, requiring approval before sharing technical specifications or implementation details.
Controlling organizational information to balance transparency with security. For example, limiting disclosure of AI team details, development timelines, and other information that could reveal technical capabilities, reviewing public communications for sensitive information.
Policy document, SOP, or handbook section defining limitations and approval requirements for publicly sharing AI system technical details - may include communication policy limiting disclosure of model architectures or configurations, engineering handbook with approval workflows for technical specifications, or internal procedures controlling release of organizational AI information.
Establishing approval processes. For example, requiring designated review for public content referencing AI capabilities in e.g. publications, presentations, and marketing materials, and documenting approved disclosures with business justification.
Approval email, ticket, or review documentation for public AI communications - may include approval requests in email or Jira/Slack for blog posts or press releases, marketing review records for AI capability disclosures, or periodic security review logs for public-facing AI content.
- AML-M0000: Limit Public Release of Information
- AML-M0001: Limit Model Artifact Release
- LLM02:25 - Sensitive Information Disclosure
- LLM07:25 - System Prompt Leakage
- AIS-09: Output Validation
- AIS-15: Prompt Differentiation
B004Prevent AI endpoint scraping
Implement safeguards to prevent probing or scraping of external AI endpoints
Implementing systems distinguishing between high-volume legitimate usage and adversarial behavior. For example, using behavioral analytics and user profiling to calibrate detection thresholds and prevent false positives against trusted users.
Anomaly detection system or configuration file - may include behavioral analytics dashboard (Datadog, Elastic, Splunk) with user scoring rules, rate limiting configuration with tier-based thresholds (config.yaml, API gateway settings), user allowlists or reputation tables, or code implementing session-based threshold logic.
Implementing rate limiting and query restrictions. For example, establishing per-user quotas to prevent model extraction, blocking excessive query patterns, implementing progressive restrictions for suspicious behavior, or using economic disincentives for high-volume usage.
Rate limiting configuration for API endpoints - may include per-user quota settings, query throttling rules, progressive restriction policies, WAF configuration (Cloudflare, AWS WAF, Azure Application Gateway) with blocking rules for excessive patterns, or pricing tier settings implementing usage-based cost increases.
Conducting simulated external attack testing of AI endpoints. For example, performing automated attack simulations, testing endpoint protection effectiveness against high-volume and distributed attacks, and documenting methodologies appropriate to organizational threat profile.
Third-party penetration test report for AI endpoints including attack simulations tested (e.g. scraping attempts, brute force, reconnaissance), rate limiting and endpoint protection validation, distributed attack testing, test methodology, and findings on protection effectiveness.
Maintaining endpoint security through remediation. For example, tracking identified vulnerabilities, implementing protective measures based on testing outcomes, and regularly updating endpoint defenses and detection thresholds.
Issue tracking system (GitHub, Jira, Linear) showing endpoint vulnerability lifecycle - must include vulnerability identification, remediation proposal, implementation, and production deployment with timestamps and approval records.
- AML-M0003: Model Hardening
- AML-M0004: Restrict Number of AI Model Queries
- Article 15: Accuracy Robustness and Cybersecurity
- MEASURE 2.7: Security and resilience
B005Implement real-time input filtering
Implement real-time input filtering using automated moderation tools
Integrating automated moderation tools to filter inputs before they reach the foundation model. For example, integrating third-party moderation APIs, implementing custom filtering rules, configuring blocking or warning actions for flagged content, and establishing confidence thresholds based on risk category and severity
Moderation tool integration showing API configuration, filtering rules, action settings (block/warn/modify), and confidence thresholds for different violation categories - this could be screenshots of configuration files, admin dashboard settings, or API integration code. Example moderation tools: OpenAI Moderation API, Claude content filtering, VirtueAI/Hive/Spectrum Labs
Documenting the moderation logic and rationale. For example, explaining chosen moderation tools, threshold justifications, and decision criteria for different risk categories.
Document explaining moderation approach including tool selection rationale, threshold settings with justifications, action logic for different violation types, and examples of how different input categories are handled.
User-facing messages or UI flows showing how blocked inputs are communicated to users - this could be error messages, warning dialogs, or alternative suggestions provided when content is filtered.
Logging flagged prompts for analysis and refinement of filters, while ensuring compliance with privacy obligations.
Logging system showing how flagged inputs are captured, what metadata is included/excluded for privacy, retention policies, and audit trail - may include privacy documentation explaining logging disclosures to users.
Periodically evaluating filter performance and adjusting thresholds accordingly. For example, accuracy, latency, false positives/negatives.
Report or dashboard showing analysis of filter performance metrics (false positives, false negatives, accuracy, latency) and documented threshold adjustments made based on performance data - should include timestamps and rationale for changes.
- LLM01:25 - Prompt Injection
- LLM04:25 - Data and Model Poisoning
- LLM10:25 - Unbounded Consumption
- AML-M0015: Adversarial Input Detection
- AML-M0021: Generative AI Guidelines
- MEASURE 2.7: Security and resilience
B008Protect AI system deployment environment
Implement security measures for AI system deployment environments including encryption, access controls and authorization
Enforcing caller authentication across API endpoints and agentic interfaces. For example, applying scoped API tokens or signed requests for model API access; enforcing OAuth 2.0 or OIDC token validation with appropriate scoping for MCP server connections; implementing mutual authentication for agent-to-agent interfaces.
Configuration or code showing caller authentication controls - may include scoped API token or signed request configuration for model API endpoints, OAuth token scoping or OIDC validation middleware for MCP server connections, or mutual authentication configuration for agent-to-agent interfaces (e.g. A2A protocol authentication config).
Securing data in transit across model API endpoints and agentic interfaces. For example, enforcing TLS for all model API endpoint traffic, MCP server connections, and agent-to-agent communication channels; implementing credential rotation policies for long-lived service connections.
Configuration or code showing transport security controls - may include TLS/HTTPS certificate configuration for model API endpoints, MCP server traffic, or agent-to-agent connections, or credential rotation policy documentation for service-level MCP or A2A connections.
Verifying model integrity before and during deployment. For example, using cryptographic checksums or signed artifacts to detect tampering, scanning model files for malicious payloads.
Deployment pipeline or code implementing model integrity checks - may include cryptographic checksum verification, model artifact signature validation, hash comparison before deployment, model scanning configuration detecting malicious payloads (e.g. Pickle, ONNX) using tools like Cisco's pickle-fuzzer, Trail of Bit's Fickling, or deployment logs recording model version hashes.
Enforcing data integrity across agentic interfaces. For example, implementing cryptographic message signing for agent-to-agent communication; applying schema validation and input sanitization to MCP tool call inputs and outputs.
Configuration or code showing data integrity controls for agentic interfaces - may include cryptographic message signing configuration for agent-to-agent interfaces (e.g. signed agent cards), or schema validation configuration applied to MCP tool call inputs and outputs.
Securing model hosting environments. For example, using up-to-date and minimal container images, scanning for known vulnerabilities in dependencies and base images, and applying infrastructure-level isolation techniques based on risk level (e.g. container namespaces, VM separation, or dedicated GPU access).
Container configuration or infrastructure setup for model hosting - may include Dockerfile with minimal base images and up-to-date dependencies, vulnerability scanning results from Trivy or Snyk for container images, or infrastructure configuration showing isolation techniques (container namespaces, VM separation, network policies, dedicated GPU allocation).
- AML-M0005: Control Access to AI Models and Data at Rest
- AML-M0012: Encrypt Sensitive Information
- AML-M0019: Control Access to AI Models and Data in Production
- Article 15: Accuracy Robustness and Cybersecurity
- LLM07:25 - System Prompt Leakage
B010Promote secure patterns in generated code
Implement safeguards to promote secure patterns and prevent known vulnerabilities in generated code
Implementing safeguards to promote secure patterns for common web vulnerability classes in generated code. For example, defaulting to parameterized queries or safe ORM patterns for database access, applying output encoding and contextual escaping to prevent cross-site scripting, or enforcing HTTPS/TLS in generated network and API code.
Code, configuration, or system prompt excerpt demonstrating secure defaults for common vulnerability classes, may include system prompt guidance directing the model toward parameterized queries and safe ORM patterns, framework scaffolding that applies output encoding by default, or sample generations on database or UI prompts showing safe patterns used without explicit user request.
Implementing safeguards to promote secure patterns for authentication and authorization in generated code. For example, defaulting to vetted authentication libraries or patterns on auth-related prompts, applying access control checks and least-privilege patterns by default, or avoiding generation of custom cryptographic or authentication primitives.
Code, configuration, or system prompt excerpt demonstrating secure defaults for auth, may include system prompt guidance directing the model toward established auth libraries (e.g., Auth0, Clerk, framework-native auth), or sample generations on auth-related prompts showing vetted patterns rather than custom implementations.
Implementing safeguards to prevent dependency risks in generated code. For example, avoiding wildcard version specifications (e.g., \* or latest) in generated manifests, verifying that suggested packages exist in their respective registries before including them in generated code, or applying guidance that avoids known-abandoned or typosquatted packages.
Code, configuration, or system prompt excerpt demonstrating safe dependency specification, may include system prompt guidance directing the model toward pinned versions and verified packages, registry validation logic checking generated package names against known registries, or sample generations showing appropriate version specifications across ecosystems.
Implementing safeguards to promote secure defaults for session management in generated code. For example, applying HttpOnly, Secure, and SameSite flags to session cookies by default, or defaulting to CSRF protections on state-changing endpoints.
Code, configuration, or sample generations showing secure session and cookie defaults in generated code, may include system prompt guidance, framework scaffolding configuration, or sample generations on session-management-related prompts.
Implementing safeguards to promote defensive programming patterns in generated code. For example, defaulting to input validation on user-supplied data, applying safe error handling that avoids exposing stack traces or internal details, or defaulting to rate limiting on generated public-facing endpoints.
Code, configuration, or sample generations showing input validation and error handling defaults, may include system prompt guidance, framework scaffolding patterns, or sample generations on prompts generating user-facing endpoints.
Implementing safeguards to prevent generated logging code from capturing secrets or personal information. For example, defaulting logging patterns to exclude sensitive fields, or applying guidance that avoids logging full request bodies or auth headers.
Code, configuration, or sample generations showing safe logging defaults in generated code, may include system prompt guidance, logging templates used in scaffolding, or sample generations on logging-related prompts.
C006Prevent output vulnerabilities
Implement safeguards to prevent security vulnerabilities in outputs from impacting users
Establishing output sanitization and validation procedures before presenting content to users. For example, encoding or stripping potentially malicious content, validating structured outputs against safe schemas, blocking unsafe URLs, and enforcing secure rendering modes.
Code or configuration implementing output sanitization - may include HTML/JavaScript/shell syntax encoding functions, URL validation or rewriting rules blocking unsafe links, schema validation checking structured outputs (JSON/YAML/XML) against whitelists, CSP header configuration, or template rendering with auto-escaping enabled.
Implementing content handling and security labelling based on trust level. For example, marking untrusted or third-party content, distinguishing external data from system-generated content, and applying differentiated security controls based on content source.
UI or code showing trust-based content handling - may include visual indicators marking third-party content (badges, styling, warning icons), metadata tags tracking content source and trust level, or code applying conditional security controls based on content origin (e.g., stricter sanitization for external sources).
Detecting advanced output-based attack patterns. For example, identifying prompt injection attempts, model subversion techniques, payloads targeting downstream systems, or obfuscated exploits designed to bypass filters.
Detection rules or monitoring system identifying advanced attack patterns in outputs - may include pattern matching for prompt injection chains or jailbreak tokens, payload signature scanning detecting command injection or SQL queries, or anomaly detection flagging obfuscated exploits bypassing basic filters.
- AML-M0020: Generative AI Guardrails
- Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems
- LLM05:25 - Improper Output Handling
E001AI failure plan for security breaches
Document AI failure plan for AI privacy and security breaches assigning accountable owners and establishing notification and remediation with third-party support as needed (e.g. legal, PR, insurers)
Assigning a breach response lead from existing staff. For example, IT manager, security officer, or designated executive with authority to engage external counsel and specialists as needed.
Defining breach notification procedures. For example, customer communications, regulatory reporting requirements, and vendor notifications based on applicable privacy laws.
Implementing security remediation measures. For example, system freeze capabilities, vulnerability fixes, access control updates, and coordination with external security consultants when internal expertise is insufficient.
Establishing evidence collection requirements with guidance on preserving evidence for potential legal review. For example, system logs, user activity records, and basic documentation.
Can be standalone document or integrated in existing incident response procedures/policies
- Article 20: Corrective Actions and Duty of Information
- Article 73: Reporting of Serious Incidents
- A.8.4: Communication of incidents
- A.8.5: Information for interested parties
- GOVERN 4.3: Testing and incident sharing
- MANAGE 1.3: Risk response planning
- MANAGE 4.3: Incident communication
F001Prevent AI cyber misuse
Implement or document guardrails to prevent AI-enabled misuse for cyber attacks and exploitation
Results of testing from foundation model developer on offensive cyber capabilities and mitigations.
Provider model cards, cybersecurity assessment reports from model developers, or foundation model documentation describing offensive cyber capabilities and mitigations
Implementing malicious use detection and blocking. For example, deploying available content filtering to detect requests for malicious code generation, attack planning, and vulnerability exploitation guidance, configuring automated blocking of cyber attack assistance requests, maintaining databases of prohibited use patterns.
Content filtering rules blocking cyber attack requests, keyword or pattern matching detecting malicious code generation attempts, automated blocking configuration for exploit development queries, or prohibited use pattern database.
- MEASURE 2.7: Security and resilience
- A.5.5: Assessing societal impacts of AI systems
- GRC-02: Risk Management Program
- GRC-09: Acceptable Use of the AI Service
- GRC-10: AI Impact Assessment
- GRC-12: Ethics Committee
- TVM-11: Guardrails
F002Prevent catastrophic misuse
Implement or document guardrails to prevent AI-enabled catastrophic system misuse (chemical / bio / radio / nuclear)
Results of testing from foundation model developer on CBRN capabilities and mitigations.
List of foundation models used with CBRN capability information - may include provider model cards with CBRN assessments, weapons of mass destruction risk evaluations from model developers, or other documentation describing CBRN-related capabilities and mitigations.
Establishing catastrophic misuse monitoring. For example, monitoring AI system interactions for patterns indicating weapons development or mass harm intent, implementing real-time alerting for detected catastrophic misuse attempts, documenting suspicious queries and system responses.
Monitoring dashboard or alert configuration for catastrophic misuse patterns - may include usage monitoring flagging CBRN-related queries, alert rules for weapons development patterns, logs of detected and blocked catastrophic misuse attempts, or incident records documenting suspicious CBRN-related interactions.
- A.5.5: Assessing societal impacts of AI systems
- GRC-02: Risk Management Program
- GRC-09: Acceptable Use of the AI Service
- GRC-10: AI Impact Assessment
- GRC-12: Ethics Committee
- TVM-11: Guardrails
- IBM 60: Output - Harmful output
- IBM 65: Output - Dangerous use
Robustness at scale
7 testsProve the agent stays reliable as usage grows: pre-deployment testing, continuous monitoring, logging, and review
C001Define AI risk taxonomy
Establish a risk taxonomy based on system capabilities and deployment context
Defining risk categories with severity levels and examples based on industry and deployment context. For example, classifying harmful outputs such as distressed outputs, angry responses, high-risk advice, offensive content, bias, and deception, identifying other high-risk use cases such as safety-critical instructions, legal recommendations, financial advice.
Aligning risk taxonomy with external frameworks and standards.
Establishing severity grading appropriate to organizational context and risk tolerance. For example, implementing consistent scoring methodology across risk categories, defining thresholds for flagging and human review.
Internal policy document, risk framework, or taxonomy defining AI risk categories with severity levels and examples specific to deployment context. Example taxonomies to draw upon include NIST AI RMF functions, EU AI Act article 9, ISO42001 controls.
Maintaining taxonomy currency with documented change management. For example, updating based on emerging threats or incidents.
Meeting notes, change log, or review documentation showing annual reviews of the risk taxonomy. Could include review dates, participants, decisions made (categories added/removed/modified, threshold adjustments), rationale for changes, approvals records, and version history showing taxonomy updates over time with timestamps. Can be standalone or part of broader internal audit/review or change management procedures.
- Article 9: Risk Management System
- A.5.2: AI system impact assessment process
- A.5.3: Documentation of AI system impact assessments
- A.5.4: Assessing AI system impact on individuals or groups of individuals
- A.5.5: Assessing societal impacts of AI systems
- 4.1: Understanding the organization and its context
- 6.1.1: Actions to address risks and opportunities — General
- 6.1.2: AI risk assessment
- 6.1.3: AI risk treatment
- 6.1.4: AI system impact assessment
- 8.2: AI risk assessment
- 8.3: AI risk treatment
- 8.4: AI system impact assessment
- GOVERN 1.3: Risk management processes
- GOVERN 1.4: Risk management governance
- GOVERN 4.2: Risk documentation
- GOVERN 6.1: Third-party risk policies
- MANAGE 1.2: Risk prioritization
- MANAGE 1.3: Risk response planning
- MANAGE 1.4: Residual risk documentation
- MAP 1.5: Risk tolerance
- MAP 5.1: Impact assessment
- MEASURE 1.1: Risk metrics selection
- MEASURE 2.10: Privacy risk assessment
- MEASURE 2.11: Fairness and bias
- MEASURE 3.1: Emergent risk tracking
C002Conduct pre-deployment testing
Conduct internal testing of AI systems prior to deployment across risk categories for system changes requiring formal review or approval
Conducting pre-deployment testing with documented results and identified issues. For example, structured hallucination testing, adversarial prompting, safety unit tests, and scenario-based walkthroughs.
Completing risk assessments of identified issues before system deployment. For example, potential impact analysis, mitigation strategies, and residual risk evaluation.
Obtaining approval sign-offs from designated accountable. For example, documented rationale for approval decisions and maintained records for review purposes.
Test results with identified issues and severity ratings, risk assessment with mitigation decisions, and approval sign-offs with rationale - may be combined in deployment gate documentation or provided as separate documents (e.g., test suite outputs from GitHub Actions/pytest, Jira/Linear tickets with risk assessment and approval, staging environment test reports, deployment checklist with sign-offs).
Integrating AI system testing into established software development lifecycle (SDLC) gates. For example, including threat modelling and risk evaluation during design phases, requiring risk evaluation and sign-off at staging or pre-production milestones, aligning with CI/CD or MLOps pipelines, and documenting test artefacts in shared repositories."
CI/CD pipeline configuration or workflow showing AI testing integrated as deployment gate - may include GitHub Actions/Jenkins/GitLab CI config files requiring test passage, pull request templates with testing checklists, or branch protection rules enforcing pre-deployment validation.
Implementing pre-deployment vulnerability scanning of AI artifacts and dependencies. For example, scanning AI models and ML libraries for security vulnerabilities, validating runtime behavior for unsafe operations, and analyzing outputs for harmful content before deployment.
Security scanning tools or CI/CD pipeline showing vulnerability analysis of AI artifacts and dependencies - may include GitHub/GitLab security tab with dependency alerts, Snyk or Dependabot vulnerability findings, pip-audit or safety check terminal output showing CVE scans, model file scanning results, or CI/CD logs showing security scan execution.
- AML-M0016: Vulnerability Scanning
- Article 9: Risk Management System
- Article 27: Fundamental Rights Impact Assessment for High-Risk AI Systems
- A.6.2.5: AI system deployment
- A.6.2.4: AI system verification and validation
C007Flag high risk outputs for human review
Implement an alerting system that flags high-risk outputs for human review
Defining high-risk output criteria drawing on risk taxonomy.
Document or policy defining high-risk outputs requiring human review - should specify criteria for flagging (e.g. financial advice thresholds, medical/legal/safety domains, reputational harm triggers). Can be standalone or included in existing AI risk taxonomy/AI risk policy.
Implementing automated detection mechanisms for high-risk outputs. For example, using content filtering, risk scoring, or classification models to identify outputs requiring review or flagging.
Detection code, configuration file, or rules engine showing high-risk output filtering - may include keyword lists or regex patterns flagging sensitive topics, scoring logic assigning risk values to recommendations, if/then rules defining high-risk conditions, ML model configuration (e.g., classification thresholds in config.yaml), or API response showing confidence scores with risk thresholds.
Establishing human review workflows for flagged high-risk outputs. For example, assigning reviewers, defining escalation procedures for complex cases, managing review queues with response time tracking, documenting review decisions, and reviewing workflow effectiveness regularly.
Workflow documentation or ticketing system configuration showing human review process for flagged outputs - may include runbook with reviewer assignments and escalation paths, queue management in Jira/Linear/support ticketing with pending review tracking, SLA targets for review response times, or procedure document defining review decision documentation requirements.
- AML-M0020: Generative AI Guardrails
- GOVERN 3.2: Human-AI oversight
- MAP 3.5: Human oversight
- GRC-15: Human supervision
C008Monitor AI risk categories
Implement monitoring of AI systems across risk categories
Establishing ongoing monitoring of AI outputs across risk categories. For example, conducting regular evaluations prioritized by risk severity, sampling outputs for review, and tracking system behavior patterns.
Monitoring dashboard, logging system, or evaluation reports showing ongoing AI output tracking - may include output sampling logs with review results, behavior trace logs showing system patterns, prompt-response logging configuration, evaluation schedules prioritized by risk severity, or monitoring metrics dashboard tracking trends over time.
Maintaining documentation. For example, recording identified scenarios with clear examples, updating risk taxonomy based on monitoring findings and incidents.
Document or change log showing identified risk scenarios with examples - may include incident reports triggering taxonomy changes, risk scenario database with concrete examples, or version history of risk taxonomy showing updates with rationale linked to monitoring findings.
Integrating AI output monitoring with existing security tools. For example, forwarding alerts and flagged outputs to SIEM platforms, applying standard logging formats (e.g. JSON, syslog) to support automated threat detection workflows.
Monitoring dashboard, logging system, or evaluation reports showing ongoing AI output tracking - may include output sampling logs with review results, behavior trace logs showing system patterns, prompt-response logging configuration, evaluation schedules prioritized by risk severity, or monitoring metrics dashboard tracking trends over time.
- Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems
- A.5.4: Assessing AI system impact on individuals or groups of individuals
- A.6.2.6: AI system operation and monitoring
- A.9.4: Intended use of the AI system
- 6.1.1: Actions to address risks and opportunities — General
- 6.1.2: AI risk assessment
- 6.1.3: AI risk treatment
- 8.2: AI risk assessment
- 8.3: AI risk treatment
- 9.1: Monitoring, measurement, analysis and evaluation
- A.9.2: Processes for responsible use of AI systems
- GOVERN 1.5: Risk monitoring and review
- MANAGE 3.1: Third-party monitoring
- MANAGE 4.1: Post-deployment monitoring
- MEASURE 2.4: Production monitoring
- MEASURE 4.3: Performance tracking
C009Enable real-time feedback and intervention
Implement mechanisms to enable real-time user feedback collection, intervention and actioning mechanisms
Enabling user intervention capabilities. For example, providing mechanisms for users to pause, stop, or redirect system behavior, implementing feedback collection tools for users to report issues or concerns, ensuring technical controls persist across devices and interaction contexts.
Ensuring accessibility of feedback and intervention mechanisms. For example, adhering to WCAG 2.1 standards for color contrast, screen reader compatibility, keyboard navigation, and clear messaging for users with disabilities.
Intervention controls (stop/pause/redirect buttons, feedback forms, issue reporting mechanisms) with accessibility features integrated (e.g. keyboard navigation, high contrast modes, screen reader labels)
Reviewing user feedback and intervention logs at regular intervals, analyzing findings using structured methodologies (e. g., categorizing by risk domain, frequency, and severity), and integrating corrective actions into product backlogs or compliance workflows, with records maintained for traceability.
Logs, reports, or dashboard showing review, analysis and actioning of user feedback and intervention patterns - may include feedback summary reports, intervention frequency analysis, categorization by risk domain, documentation of system changes made in response to patterns, or integration with product backlog/compliance workflows.
- Article 14: Human Oversight
- A.8.3: External reporting
- GOVERN 3.2: Human-AI oversight
- MAP 3.5: Human oversight
- MEASURE 3.3: User feedback systems
E008Review internal processes
Establish regular internal reviews of key processes and document review records and approvals
Reviewing decision processes every quarter including AI system changes, foundational model selection, security assessment.
Maintaining a centralized repository of decision records and internal review of these record. For example, supporting evidence reviewed, remediation plans.
Documenting and tracking remediation of any risks identified.
Centralized repository, policy, or tickets showing quarterly internal reviews - e.g. review meeting notes or calendars, decision logs in Jira/Notion/Confluence, risk registers with remediation status, threat modelling outcomes, or audit trails of review activities.
Collecting and implementing external feedback on AI systems. For example, system risks, new threat patterns, new mitigation strategies.
Documentation showing external feedback collected and implemented - may include external security advisories reviewed, threat intelligence integrated, third-party recommendations adopted, or records of external input incorporated into system improvements.
- Article 43: Conformity Assessment
- A.3.3: Reporting of concerns
- A.2.3: Alignment with other organizational policies
- A.2.4: Review of the AI policy
- 6.3: Planning of changes
- 7.5.2: Creating and updating documented information
- 9.2.1: Internal audit - General
- 9.2.2: Internal audit programme
- 9.3.1: Management review - General
- 9.3.2: Management review inputs
- 9.3.3: Management review results
- GOVERN 1.7: AI system decommissioning
- GOVERN 5.2: Feedback integration
- GOVERN 5.1: External feedback
- MANAGE 4.2: Continual improvement
- MEASURE 1.2: Metric appropriateness
- MEASURE 2.13: TEVV effectiveness
E015Log AI system activity
Maintain logs of AI system processes, actions, and agent outputs where permitted to support incident investigation, auditing, and explanation of AI system behavior
Capturing system activity details to support incident investigation and behavior explanation. For example, logging inputs, processing steps, outputs, and metadata for AI systems.
Logging code or configuration showing what system activity is captured - may include code logging inputs and outputs, logging configuration file specifying what to log, or example log entries showing captured information (timestamps, inputs, outputs, user actions).
Implementing log storage with appropriate retention periods, access controls, and data sanitation to support auditing and incident response.
Log storage system showing retention policies, access controls and sanitation practices - may include log management platform (Datadog, Splunk, CloudWatch) with retention period settings and PII-masking, access control configuration showing who can view logs, or storage settings with automatic deletion rules.
Implementing technical controls to ensure logs are tamper-evident and independently verifiable. For example, ensuring that captured records cannot be modified or deleted after creation, ensuring sequence integrity so that gaps, omissions, and reordering are detectable during incident investigation or audit.
Log immutability controls - for example, write-once-read-many (WORM) storage configuration, cryptographic hashing of log entries, append-only database settings, or third-party log management platform features.
Capturing full execution chains of agentic workflows to support investigation of agent-specific incidents. For example, logging agent provenance metadata, tool call parameters and results, sub-agent delegations and their outcomes, approval/authorization events (e.g., human-in-the-loop approvals), and reasoning traces where available.
Logging code or configuration demonstrating agent execution logging - may include log fields capturing agent provenance metadata per execution (e.g. agent type identifier, creator or deployment origin); structured log entries capturing tool call parameters and their results; delegation chain records showing sub-agent handoffs with identity, task context, and outcome at each step; approval/authorization records linked to execution (e.g., approver identity, timestamp, decision outcome); or reasoning trace output from the agent framework.
- AML-M0024: AI Telemetry Logging
- Article 12: Record-Keeping
- Article 19: Automatically Generated Logs
- A.6.2.8: AI system recording of event logs
Permissions compliance
11 testsVerify agents act only within authorized scope, with accountability, oversight, and vendor and regulatory controls
B006Prevent unauthorized AI agent actions
Implement safeguards to prevent AI agents from performing actions beyond intended scope and authorized privileges
Implementing technical restrictions that limit agent capabilities to authorized scope. For example, restricting agent access to approved backend services, APIs and MCP servers, enforcing network segmentation or API gateway rules, or implementing service-level authorization preventing access to sensitive systems.
Configuration showing technical limitations on agent backend access - may include API gateway rules restricting accessible services, network policies defining allowed endpoints, MCP server allowlist or registration configuration restricting which MCP servers and tools the agent may connect to, service-level authorization configuration, or architecture diagram showing agent isolation boundaries including MCP server placement and network segmentation.
Deploying monitoring and alerting for agent actions that exceed security boundaries. For example, logging all agent service interactions, alerting on access attempts to unauthorized systems or APIs, or anomaly detection flagging unusual connection patterns.
Implementation of monitoring configuration tracking agent security-relevant actions - may include logging setup capturing agent service calls and authentication attempts, alert rules for unauthorized system access, security monitoring dashboard showing agent infrastructure interactions, or example logs demonstrating boundary violations are detected.
Implementing additional safeguards to contain runtime risk. For example, enabling sandboxed execution environments with configurable filesystem, network, and credential restrictions for agent-executed code and first-party MCP servers, monitoring MCP tool definitions for unauthorized changes after initial approval, providing pre-execution authorization hooks that verify runtime tool calls against defined policy before execution proceeds, or scanning agent configuration artifacts such as hooks, skills and rules for prompt injection or unauthorized behavior.
Configuration or code demonstrating runtime containment controls — may include sandbox or container configuration showing configurable filesystem, network, and credential restrictions for agent code execution environments and first-party MCP servers, tool definition integrity controls showing how unauthorized post-approval changes are detected, pre-execution hook or policy engine configuration showing tool calls are verified at runtime, or scanning of agent configuration artifacts for prompt injection or malicious patterns.
- MAP 2.1: Task definition
- LLM08:25 - Vector and Embedding Weaknesses
- LLM10:25 - Unbounded Consumption
- AIS-11: Agents Security Boundaries
- IAM-19: Agent Access Restriction
- DSP-07: Data Protection by Design and Default
B007Enforce user access privileges to AI systems
Establish and maintain user access controls and admin privileges for AI systems in line with policy
Implementing system-level access controls tailored to AI systems. For example, using role-based or attribute-based access to restrict access to model configuration, training datasets, tool-calling capabilities, or prompt logs, based on job function and system sensitivity.
Restricting administrative and configuration privileges to authorized personnel. For example, limiting ability to alter system behavior, tools, or models.
IAM platform, permission files, or admin panel showing role-based or attribute-based access restrictions for AI system resources (model configurations, training datasets, tool-calling capabilities, prompt logs) - may include IAM role assignments, permission policies, or authorization code validating user permissions before accessing sensitive AI components.
Conducting access reviews and updates at least quarterly. For example, validating access assignments, updating based on policy or role changes, documenting access changes with AI-specific context (e.g. model access justification, changes to agent capability boundaries, or access to sensitive prompt/response history).
Quarterly access review documentation - may include access review meeting notes, tracking records of access changes with justifications, or reports documenting role changes and access modifications based on policy updates.
- AML-M0005: Control Access to AI Models and Data at Rest
- AML-M0019: Control Access to AI Models and Data in Production
- LLM02:25 - Sensitive Information Disclosure
- LLM06:25 - Excessive Agency
- LLM10:25 - Unbounded Consumption
- DSP-07: Data Protection by Design and Default
- DSP-08: Data Privacy by Design and Default
- IAM-01: Identity and Access Management Policy and Procedures
- IAM-02: Strong Password Policy and Procedures
- IAM-03: Identity Inventory
- IAM-04: Separation of Duties
- IAM-05: Least Privilege
- IAM-06: User Access Provisioning
- IAM-07: User Access Changes and Revocation
- IAM-08: User Access Review
- IAM-09: Segregation of Privileged Access Roles
- IAM-10: Management of Privileged Access Roles
- IAM-13: Uniquely Identifiable Users
- IAM-14: Strong Authentication
- IAM-15: Passwords and Secrets Management
- IAM-16: Authorization Mechanisms
- IAM-17: Knowledge Access Control - Need to Know
- IAM-18: Output Modification and Special Authorization
- LOG-04: Audit Logs Access and Accountability
- LOG-09: Log Protection
- LOG-12: Access Control Logs
D003Restrict unsafe tool calls
Implement safeguards or technical controls to prevent tool calls in AI systems from executing unauthorized actions, accessing restricted information, or making decisions beyond their intended scope
Implementing tool call validation and authorization. For example, restricting tool calls to approved functions and MCP servers, validating parameters before execution.
Code or configuration showing function and tool allowlists, parameter validation logic, or authz checks before tool execution - may include tool permission schemas, input validation functions, or access control lists restricting available tools per agent/user.
Enforcing rate limits and transaction caps for autonomous tool use.
Code or configuration showing rate limits and transaction caps on tool usage - may include per-tool usage quotas, time-windowed limits, or circuit breakers preventing excessive autonomous tool calls.
Establishing execution monitoring and logging. For example, tracking all tool calls, monitoring for unauthorized access attempts or scope violations.
Logging configuration, monitoring dashboard, or audit logs showing tracked tool calls - may include log entries capturing the originating MCP server, tool name, tool version, input parameters, and timestamps per invocation, alerts for unauthorized tool access attempts, or monitoring system flagging scope violations.
Requiring human approval for sensitive tool operations. For example, requiring human confirmation before executing high-risk actions, multi-step tool calls, implementing approval workflows for operations beyond autonomous boundaries.
Approval workflow, code requiring human confirmation, or ticketing system for sensitive, high-risk, or multi-step tool operations
Reviewing patterns of AI tool usage. For example, identifying anomalies, updating tool permissions, and retiring unused or high-risk functions during scheduled evaluations.
Reports or documentation showing periodic review of tool usage patterns, permission updates, and function retirement decisions - may include usage analytics identifying anomalies, change logs showing permission adjustments, or records of deprecated/retired tools with rationale.
- AML-M0004: Restrict Number of AI Model Queries
- AML-M0024: AI Telemetry Logging
- Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems
- GOVERN 6.1: Third-party risk policies
D004Tool calls
Appoint expert third-parties to evaluate tool calls in AI systems, including executing unauthorized actions, accessing restricted information, or making decisions beyond their intended scope at least every 3 months
Appointing qualified third-party assessors. Including selecting assessors with relevant technical capabilities for identified risk areas, maintaining records of assessor qualifications and independence.
Conducting regular testing. Including defining testing scope and methodologies based on risk taxonomy and performing assessments of tool calls at least every quarter.
Maintaining documentation. Including testing scope, results, and remediation actions taken, tracking follow-up activities and resolution timelines.
Third-party evaluation report showing tool call testing - must include risk taxonomy tested, testing methodology and findings, and improvement tracking with remediation timelines and documentation.
- GOVERN 6.1: Third-party risk policies
- GOVERN 4.3: Testing and incident sharing
- MANAGE 2.2: Deployed system value
- MEASURE 1.3: Independent assessment
- MEASURE 2.1: TEVV documentation
- MEASURE 2.6: Safety evaluation
- MEASURE 4.1: Context-specific measurement
- MEASURE 4.2: Trustworthiness validation
- LLM06:25 - Excessive Agency
- A.6.2.4: AI system verification and validation
E004Assign accountability
Document which AI system changes across the development & deployment lifecycle require formal review or approval, assign a lead accountable for each, and document their approval with supporting evidence
Defining AI system changes requiring approval including model selection, material changes to the meta prompt, adding / removing guardrails, changes to end-user workflow, other changes that drive material. For example, +/-10% performance on evals.
Assigning an accountable lead as approver for each of these changes. Can follow a RACI structure to formalize roles of those consulted and informed.
Documentation or policy defining which AI system changes require approval with assigned accountable leads, and approval records showing sign-offs with supporting evidence. Can be a change management policy, overview table in e.g. Notion, approval logs from Jira/Linear/GitHub, or deployment gate documentation.
Implementing code signing and verification processes for AI models, libraries, and deployment artefacts to ensure only digitally signed components are approved for production use.
Code signing configuration, CI/CD pipeline requiring signed artifacts, or verification process for AI components - may include model signing process, signature verification in deployment pipeline, artifact registry showing signed models/libraries, or policy enforcement blocking unsigned components from production.
- AML-M0013: Code Signing
- Article 17: Quality Management System
- Article 18: Documentation Keeping
- A.3.2: AI roles and responsibilities
- A.4.6: Human resources
- A.6.2.2: AI system requirements and specification
- A.10.2: Allocating responsibilities
- 5.1: Leadership and commitment
- 5.3: Roles, responsibilities and authorities
- 7.2: Competence
E006Conduct vendor due diligence
Establish AI vendor due diligence processes for foundation and upstream model providers covering data handling, PII controls, security and compliance
Defining assessment criteria for foundational or upstream AI models. For example, data handling and ownership practices, PII controls, security measures, compliance status, open-source.
Conducting documented assessments. For example, scoring results, verification activities such as certifications reviewed and references contacted, and approval decisions.
Maintaining assessment records with sufficient detail for audit purposes and retaining due diligence evidence before vendor approval.
Vendor assessment records showing evaluation criteria, scoring results, verification activities, approval decisions with accountable leads, and retained evidence supporting the assessment. May include vendor questionnaires, security reviews, compliance documentation, or due diligence reports.
- Article 23: Obligations of Importers
- Article 24: Obligations of Distributors
- A.10.3: Suppliers
- MAP 4.2: Internal risk controls
E009Monitor third-party access
Implement systems to monitor and log third-party API connections, sessions, and data access
Configuring logging for third-party interactions. For example, capturing API connections, user access sessions, data exchanges, and service integrations.
Capturing access metadata. For example, user identification, authentication timestamps, accessed resources, session duration, origin IP addresses, and resource usage patterns.
Logging system or SIEM configuration showing third-party interactions being monitored with captured metadata - may include cloud logging interface (Google Cloud Logging, AWS CloudWatch, Azure Monitor) showing logged API requests with timestamps/IPs/user agents, access logs capturing authentication events and resource access, or SIEM dashboard displaying third-party connection monitoring with relevant metadata fields.
Generating alerts on anomalous third-party access patterns against defined detection rules. For example, alerting on unexpected call volume or out-of-scope endpoints for service connections, repeated failed logins for human access, or credential use outside approved scope - with each alert routed to a responsible owner for disposition.
Detection-rule or SIEM configuration for anomalous third-party access - may include alerting rules or correlation logic defining anomalous-access conditions, plus sample fired alerts showing disposition (case assignment, triage notes, or resolution records).
- AML-M0024: AI Telemetry Logging
- Article 72: Post-Market Monitoring by Providers and Post-Market Monitoring Plan for High-Risk AI Systems
- GOVERN 1.5: Risk monitoring and review
- MANAGE 4.1: Post-deployment monitoring
E010Establish AI acceptable use policy
Establish and implement an AI acceptable use policy
Defining prohibited AI usage for end-users. For example, jailbreak attempts, malicious prompt injection, unauthorized data extraction, generation of harmful content, and misuse of customer data.
Policy document defining acceptable and/or prohibited AI usage - can be standalone document or parts of, e.g., terms of service
Implementing detection and monitoring tools. For example, prompt analysis, output filtering, usage pattern anomalies, and suspicious access attempts.
Code, configuration, or monitoring system detecting acceptable use policy violations - may include prompt analysis logic, output filtering rules, anomaly detection for usage patterns, or alerting on suspicious access attempts.
Implementing user feedback when policy is breached. For example, showing alerts or error messages when inputs violate acceptable use.
User-facing alerts or error messages displayed when acceptable use policy is violated - may include in-product warning messages, blocked request notifications, or error screens explaining policy violations.
Real-time monitoring, blocking, or alerting capabilities.
Maintaining logging and tracking systems. For example, incident creation, violation tracking with case assignment and resolution documentation.
Conducting regular effectiveness reviews. For example, quarterly analysis of violation trends, tool performance assessment, policy updates based on emerging threats, and user training adjustments.
Documentation or screenshots showing additional AUP enforcement mechanisms - may include real-time blocking/alerting systems, violation tracking logs with incident management, effectiveness review reports analyzing violation trends and policy updates, or training materials addressing emerging misuse patterns.
- A.2.2: AI policy
- A.9.2: Processes for responsible use of AI systems
- A.9.4: Intended use of the AI system
- A.2.4: Review of the AI policy
- A.9.3: Objectives for responsible use of AI system
- 4.1: Understanding the organization and its context
- 4.3: Determining the scope of the AI management system
- 5.2: AI policy
- GOVERN 1.2: Trustworthy AI policies
- MAP 1.6: System requirements
- MAP 3.3: Application scope
- MAP 3.4: Operator proficiency
- MEASURE 2.4: Production monitoring
- LLM10:25 - Unbounded Consumption
E012Document regulatory compliance
Document applicable AI laws and standards, required data protections, and strategies for compliance
Identifying relevant regulations. For example, data protection laws. For example, GDPR, CCPA, sector-specific requirements, emerging AI standards. For example, EU AI Act.
Documenting compliance procedures and strategies appropriate for company size and operations.
Reviewing the repository every 6 months and when additional requirements may be triggered. For example, regulations change or business operations expand into new jurisdictions.
Compliance register, assessment memo or review tickets (e.g. in Notion), or policy listing applicable regulations with compliance strategies - should include review dates or version history showing periodic updates.
- Article 16: Obligations of Providers of High-Risk AI Systems
- Article 18: Documentation Keeping
- Article 21: Cooperation with Competent Authorities
- Article 22: Authorised Representatives of Providers of High-Risk AI Systems
- Article 25: Responsibilities Along the AI Value Chain
- Article 26: Obligations of Deployers of High-Risk AI Systems
- Article 43: Conformity Assessment
- Article 44: Certificates
- Article 47: EU Declaration of Conformity
- Article 48: CE Marking
- Article 49: Registration
- A.2.3: Alignment with other organizational policies
- A.8.5: Information for interested parties
- 10.2: Nonconformity and corrective action
- GOVERN 1.1: Legal and regulatory compliance
- GOVERN 1.7: AI system decommissioning
- MAP 1.1: Context understanding
- MAP 4.1: Legal risk mapping
E016Implement AI disclosure mechanisms
Implement clear disclosure mechanisms to inform users when they are interacting with AI systems rather than humans
Implementing AI disclosure for text-based interactions. For example, displaying clear notices when users interact with AI chatbots, virtual assistants, or automated messaging systems.
Text-based AI disclosure - may include chatbot interface with "You're chatting with AI" notice, messaging system showing AI agent identifier, website chat widget with AI disclosure banner, or automated email/SMS with AI generation notice.
Implementing AI disclosure for voice-based interactions. For example, providing audio notifications at the beginning of voice calls or interactions.
Transcript or audio recording of voice AI disclosure.
Labelling AI-generated media and documents in a machine-readable and detectable format. For example, marking AI-generated images, videos, audio, or documents with metadata, watermarks, or labels indicating artificial generation.
AI generation labeling implementation - may include Content Credentials or C2PA metadata embedded in files, visible watermarking system with AI generation marks, classifier output detecting and flagging AI-generated content, or metadata tagging system marking files as artificially generated.
Disclosing when autonomous AI agents or systems are performing actions. For example, notifying users when AI systems are making decisions, processing requests, or executing tasks without human oversight.
AI automation disclosure in product - may include "Powered by AI" or "AI Agent" labels in interface, workflow dashboard displaying AI-automated tasks, status indicators showing "AI is handling this" or "Automated by AI," or notification messages stating "AI agent completed your request."
Establishing reactive disclosure capabilities when users ask if they are interacting with AI.
Chatbot or voice agent transcript responding to "Are you AI?"
- Article 13: Transparency and Provision of Information to Deployers
- Article 50: Transparency Obligations for Providers and Deployers of Certain AI Systems
- A.8.2: System documentation and information for users
- MAP 2.2: Knowledge limits
- MAP 3.4: Operator proficiency
- MEASURE 2.8: Transparency and accountability
E017Document system transparency policy
Establish a system transparency policy and maintain a repository of model cards, datasheets, and interpretability reports for major systems
Creating transparency documentation for major AI systems. For example, documenting system characteristics, data provenance, and model behavior for systems meeting documentation criteria.
Transparency documentation artifacts such as a model card, datasheet, or AI bill of materials - for example, records of the system name and version, training or fine-tuning data sources and characteristics, dated records of fine-tuning or adaptation changes and records of AI-specific components beyond the model including frameworks, orchestration layers, tool connectors.
Defining policies for sharing transparency documentation with external stakeholders. For example, establishing when reports are shared, specifying recipient categories, determining what information is disclosed to each stakeholder type.
Documenting sharing procedures including approval workflows, version control, and distribution tracking. For example, establishing approval requirements before external sharing, maintaining version control of shared documents, tracking which stakeholders received which versions.
Policy document defining transparency sharing practices - may include sharing triggers, recipient categories with disclosure levels (regulators, customers, affected parties, public), or matrix mapping stakeholder types to shared documentation (model cards, datasheets, performance reports, incident summaries).
Documenting platform-level and deployer-level security responsibilities for AI systems. For example, delineating which security obligations are managed by the platform versus the deploying organization.
Documentation delineating platform and deployer security responsibilities - may include shared responsibility documentation defining platform-level and deployer-level security obligations.
- AML-M0023: AI Bill of Materials
- AML-M0025: Maintain AI Dataset Provenance
- Article 11: Technical Documentation
- A.4.2: Resource documentation
- A.4.3: Data resources
- A.4.4: Tooling resources
- A.4.5: System and computing resources
- A.6.2.3: Documentation of AI system design and development
- A.2.2: AI policy
- A.2.4: Review of the AI policy
- 4.3: Determining the scope of the AI management system
- 5.2: AI policy