Module 5 (v 2.0): Key Considerations in planning AI Deployment

Question 1

What are the main areas of consideration to evaluate when procuring AI from a third party?

Answer 1

• Data considerations
• Security and safety
• Potential for bias
• Product type
• Technical specs
• Model performance results
• Monitoring and maintenance
• Terms of use

Question 2

What are some data considerations to evaluate when procuring AI from a third party?

Answer 2

Data considerations:
- Do they have legal rights to the data used?
- Was personal data minimized during collection and de-identified before being used for training or testing?
- If they will collect data your organization uses with the model, how will they use it?

Question 3

What are some security and safety considerations to evaluate when procuring AI from a third party?

Answer 3

Security/safety:
- What are the model’s identified risks?
- What is the potential for the model to fail, be misused, be attacked or be used for a high-risk activity?
- Are incident response plans in place?

Question 4

What are some bias related considerations to evaluate when procuring AI from a third party?

Answer 4

Potential for bias:
- What steps have been taken to minimize bias?
- Have they established that the AI function does not lead to statistical inaccuracies, bias or discrimination in results that apply to people?

Question 5

What are some product type considerations to evaluate when procuring AI from a third party?

Answer 5

Product type:
- Is it meant for the organization’s internal use, or is it meant to be external-facing?
- Does it generate content?

Question 6

What are some technical specification consideration to evaluate when procuring AI from a third party?

Answer 6

Technical specs:
- What are the model types the AI function provides? - - What types of datasets are used to train the AI?

Question 7

What are some model performance results aspects to evaluate when procuring AI from a third party?

Answer 7

Model performance results:
- What has been done to ensure model stability?
- What has been done to prevent inaccurate outputs?

Question 8

What are some monitoring and maintenance related questions to ask when procuring AI from a third party?

Answer 8

Monitoring and maintenance:
- How will the developer maintain the model?
- How will the developer monitor the model?

Question 9

What are some questions regarding terms of use to evaluate when procuring AI from a third party?

Answer 9

Terms of use:
- Is the organization allowed to fine-tune the model?
- If the model generates content, what is specified regarding intellectual property and model outputs?