Welcome to the forefront of conversational AI as we explore the fascinating world of AI chatbots in our dedicated blog series. Discover the latest advancements, applications, and strategies that propel the evolution of chatbot technology. From enhancing customer interactions to streamlining business processes, these articles delve into the innovative ways artificial intelligence is shaping the landscape of automated conversational agents. Whether you’re a business owner, developer, or simply intrigued by the future of interactive technology, join us on this journey to unravel the transformative power and endless possibilities of AI chatbots.
We use cookies to analyse traffic and to improve your browsing experience on our website. To find out more, read our privacy policy.
Selecting an AI product
Privacy by design
Transparency
Privacy risks when using AI
Accuracy
This guidance is targeted at organisations that are deploying AI systems that were built with, collect, store, use or disclose personal information. A ‘deployer’ is any individual or organisation that supplies or uses an AI system to provide a product or service.[1] Deployment can be used for internal purposes or used externally impacting others, such as customers or individuals, who are not deployers of the system. If your organisation is using AI to provide a product or service, including internally within your organisation, then you will be a deployer.
This guidance is intended to assist organisations to comply with their privacy obligations when using commercially available AI products. Common types of AI tools and products currently being deployed by Australian entities include chatbots, content-generation tools (including text-to-image generators), and productivity assistants that augment writing, coding, note-taking, and transcription. While most of the content in the guidance is specific to situations in which an organisation considers purchasing an AI product, it also addresses the use of AI products which are freely available, such as publicly accessible AI chatbots.
The OAIC has separate guidance on privacy and developing and training generative AI models.
This guidance is not intended to be a comprehensive overview of all relevant privacy risks and obligations that apply to the use of AI. It aims to highlight the key privacy considerations and APP requirements that your business should have in mind when selecting and using an AI product. It does not address considerations from other regulatory regimes that may apply to the use of AI systems.[2]
While the Privacy Act applies to all uses of AI which involve the handling of personal information, this guidance will be particularly useful in relation to the use of generative AI tools and general-purpose AI tools involving personal information, as well as other uses of AI with a high risk of adverse impacts.
It is important to recognise that generative AI systems may carry particular privacy risks due to their probabilistic nature, reliance on large amounts of training data and vulnerability to malicious uses.[3] However, a number of significant privacy risks can arise in relation to both traditional and generative AI systems, depending on the design and use. It can also be difficult for users to clearly distinguish between generative and traditional AI systems, which can sometimes contain similar features. Further, in some use cases, generative and traditional AI are being combined – for example, in the marketing context traditional AI can be used to identify customer segments for personalised campaigns, with generative AI then used to create the personalised marketing content.
Organisations should therefore take a proportionate and risk-based approach to the selection and use of any AI products. This guidance sets out questions aimed at assisting your organisation with this process.
This guidance does not have to be read from start to finish – you can use the topic headings to navigate to the sections of interest to you. Most sections conclude with a list of practical tips, which draw together the key learnings from each section. We have also included case studies and examples in each section to illustrate the way that the APPs may apply.
Finally, there is a Quick Reference Guide and two Checklists to help guide your business in the selection and use of AI products and summarise the obligations discussed.
While there is no single agreed definition of artificial intelligence (AI), as a general term it refers to the ability of machines to perform tasks which normally require human intelligence.[4] Although AI has existed in different forms for many decades, there have been significant technological advances made in recent years which have led to the emergence of AI models which apply advanced machine learning to increasingly sophisticated uses. [5]
More technically, AI refers to ‘a machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments’. Different AI systems vary in their levels of autonomy and adaptiveness after deployment’.[6] An AI model is the ‘raw, mathematical essence that is often the ‘engine’ of AI applications’ such as GPT-4, while an AI system is ‘the ensemble of several components, including one or more AI models, that is designed to be particularly useful to humans in some way’ such as the ChatGPT app.[7]
There are many different kinds of AI.[8] General-purpose AI is a type of AI model that is capable of being used, or capable of being adapted for use, for a variety of purposes, both for direct use as well as for integration in other systems.[9] ChatGPT and DALL-E are examples of general-purpose AI systems (as well as being generative AI, see further below). There are also narrow AI models, which are focused on defined tasks and uses to address a specific problem.[10] Examples of narrow AI systems include AI assistants such as Apple’s Siri or Google Assistant, and many facial recognition tools.
Generative AI refers to ‘an AI model with the capability of learning to generate content such as images, text, and other media with similar properties to its training data’ and systems built on such models.[11] Large language models (LLMs) and multimodal foundation models (MFMs) are both examples of generative AI. An LLM is ‘a type of generative AI that specialises in the generation of human-like text’.[12] Some examples of products or services incorporating LLMs are Meta AI Assistant, ChatGPT, Microsoft Copilot and HuggingChat.[13]
An MFM is ‘a type of generative AI that can process and output multiple data types (e.g. text, images, audio)’.[14] Some examples of products or services incorporating MFMs that are image or video generators include DALL-E 3, Firefly, Jasper Art, Synthesia, Midjourney and Stable Diffusion. Some examples of products or services incorporating MFMs that are audio generators include Riffusion, Suno, Lyria and AudioCraft.
Artificial intelligence (AI) has the potential to benefit the Australian economy and society, by improving efficiency and productivity across a wide range of sectors and enhancing the quality of goods and services for consumers.[15] However, the data-driven nature of AI technologies, which rely on large data sets that often include personal information, can also create new specific privacy risks, amplify existing risks and lead to serious harms.[16]
It is important that organisations understand these potential risks when considering the use of commercially available AI products. Entities that choose to use AI need to:
Existing AI systems can perform a range of tasks such as:
These applications are continuing to change and expand as the technologies develop.
The risks of AI will differ depending on the particular use case and the types of information involved. For example, the use of AI in the healthcare sector may pose particular privacy risks and challenges due to the sensitivity of health information.
The Privacy Act 1988 and the Australian Privacy Principles (APPs) apply to all uses of AI involving personal information, including where information is used to train, test or use an AI system. If your organisation is covered by the Privacy Act, you will need to understand your obligations under the APPs when using AI. This includes being aware of the different ways that your organisation may be collecting, using and disclosing personal information when interacting with an AI product.
What is personal information?
Personal information includes a broad range of information, or an opinion, that could identify an individual. This may include information such as a person’s name, contact details and images or videos where a person is identifiable. What is personal information will vary, depending on whether a person can be identified or is reasonably identifiable in the circumstances. Personal information is a broad concept and includes information which can reasonably be linked with other information to identify an individual.
Sensitive information is a subset of personal information that is generally afforded a higher level of privacy protection. Examples of sensitive information include photographs or videos where sensitive information such as race or health information can be inferred, as well as information about an individual’s political opinions or religious or philosophical beliefs.
Importantly, information can be personal information whether or not it is true. This may include false information generated by an AI system, such as hallucinations or deepfakes.[18]
The use of both traditional and generative AI technologies carries a number of significant privacy risks. Specific risks are discussed throughout this guidance, but can include:
Generative AI can carry particular privacy risks, such as the following:
Example – AI chatbot regurgitating personal information
A work health and safety training company used an AI chatbot to generate fictional scenarios of psychosocial hazards to be used in a course delivered at an Australian prison. One of the ‘fictional’ case studies generated by the chatbot was in fact a real scenario involving a former employee of the prison, and included the full names of persons involved as well as details from an ongoing court case.[29]
This example highlights the risks of AI systems regurgitating personal information from their training data even when prompted for fictional examples, creating a range of potential privacy compliance and ethical risks. This risk can be exacerbated when the system has been trained on limited data relevant to the prompt.
The National AI Centre has developed a Voluntary AI Safety Standard to help organisations develop and deploy AI systems in Australia safely and reliably. The standard consists of 10 voluntary guardrails that apply to all organisations across the AI supply chain. It does not seek to create new legal obligations, but rather helps organisations deploy and use AI systems in accordance with existing Australian laws. The information in this guidance is focussed on compliance with the Privacy Act, but will also assist organisations in addressing the guardrails in the Standard. For more information, see: www.industry.gov.au/publications/voluntary-ai-safety-standard.
When looking to adopt a commercially available AI product, your organisation must ensure that it has enough information to understand how the product works and the potential risks involved, including in relation to privacy. Failing to conduct appropriate due diligence may create a range of risks, including that your organisation will deploy a product which is unsuited to its intended uses and does not produce accurate responses.
Practising ‘privacy by design’ is the best way to ensure your organisation engages with AI products in a responsible way that protects personal information and maintains trust in your products or services. This means building the management of privacy risks into your systems and processes from the beginning, rather than at the end.
A Privacy Impact Assessment will assist your organisation to understand the impact that your use of a particular AI product may have on the privacy of individuals and identify ways to manage, minimise or eliminate those impacts. For more information, see our Guide to undertaking privacy impact assessments and our Undertaking a privacy impact assessment e‑learning course.
The following are matters which should be considered when selecting an AI product. This is focused on privacy-related issues and is not a comprehensive list – organisations should consider whether there are obligations or due diligence requirements arising from other frameworks which may also be relevant.
When selecting an AI product, your organisation should clearly identify the ways it intends to use AI and evaluate whether the particular product is appropriate. This includes considering whether the product has been tested and proven for such uses.
You should consider whether the intended uses are likely to constitute high privacy risk activities. This may be the case, for example, if the AI system will be used in relation to making decisions that may have a legal or similarly significant effect on an individual’s rights. In circumstances of high privacy risk, it is particularly important to ensure the appropriateness of the product for the intended purpose. This will include considering whether the behaviour of the AI system can be understood or explained clearly by your organisation.
To meet your organisation’s privacy obligations, particularly with respect to accuracy under APP 10, you should ensure you understand what data sources the AI system has been trained on, to assess whether the training data will be sufficiently diverse, relevant and reliable for your organisation’s intended uses. This will also help you to understand the potential risks of bias and discrimination. For example, if the AI product has not been trained on Australian data sets you should consider whether your organisation’s use of the product is likely to comply with your accuracy obligations under APP 10. Before deploying AI products, particularly customer-facing products such as chatbots, you should carefully test them to understand the risks of inaccurate or biased answers.
You should take steps to understand the limitations of the system as well as the protections that the developer has put in place. However, while developers may implement mitigations or safeguards to address these limitations – such as to prevent an AI system from producing or ‘surfacing’ personal information in its output – these will not be foolproof. Your business will need to implement its own processes to ensure the use of AI complies with your privacy obligations – these are discussed further below.
If your organisation intends to fine-tune an existing AI system for your specific purposes, the OAIC’s guidance on developing and training generative AI models sets out further privacy considerations and requirements.
For more information about your accuracy obligations under APP 10, see ‘What obligations do we have to ensure the accuracy of AI systems?’ below.
Example – Accuracy considerations when selecting an AI product
AI technology is increasingly being applied to healthcare, from AI-generated diagnostics to algorithms for health record analysis or disease prediction.[32] While AI may have potential benefits for healthcare, users should be aware of the potential for bias and inaccuracy. For example, an AI algorithm that has been trained with data from Swiss patients may not perform well in Australia, as patient population and treatment solutions may differ.
The impact of an algorithm applying biased information may lead to worsening health inequity for an already vulnerable section of the population or supply incorrect diagnoses.[33] It is important that your organisation is aware of how the AI software was developed, including the source data used to train and fine-tune the model, to ensure it is appropriate for your intended use.
When assessing a new product, you should consider your organisation’s obligations under APP 11 to take reasonable steps to protect the personal information you hold from misuse, interference and loss, as well as unauthorised access, modification or disclosure. You should take active steps to ensure that the use of an AI system will not expose your organisation’s personal information holdings to additional security risks.
Consider the general safety of the product, including whether there are any known or likely security risks. It may be helpful to run a search for any past security incidents. You should also assess what security measures have been put in place by the product owner to detect and protect against threats and attacks. For example, you may consider whether a generative AI product fine-tuned with your organisation’s data has been developed to block malicious or unlawful queries intended to make the product regurgitate its training data.
Organisations should also think about the intended operating environment for the AI product. For example, if a generative AI product is to be integrated into the organisation’s systems and have access to its documents then you will need to consider whether this will occur within a secure environment on the entity’s premises or hosted through the cloud. If the generative AI system will be deployed through the cloud, it will be necessary to consider the security and privacy risks of this, including where the servers are located and whether personal information could be disclosed outside of Australia. Deploying a system locally or on premises is likely to be more privacy-preserving as it limits the risks of third party access to the data.
For example, an AI coding assistant may be used to assist with software development such as by taking over recurring tasks in the development process, formatting and documenting code, and supporting debugging. Despite these benefits, studies have shown that programs generated by coding assistants often carry security vulnerabilities and the coding assistants themselves are vulnerable to malicious attacks.[34] When considering deployment of an AI coding assistant tool that may handle (or otherwise have access to) personal information, organisations should undertake a risk analysis and put in place appropriate mitigation measures where necessary.
Carefully review the terms and settings which will apply to your organisation’s use of the product. In particular, you should understand whether the service terms provide the developer with access to data which your organisation inputs or generates when using the AI. If it does, you will need to consider whether the use of the product will be compliant with your privacy obligations, particularly APP 6 which restricts the disclosure of personal information for secondary purposes.
Some commercial AI products will include terms or settings that allow the product owner to collect the data input by customers for further training and development of AI technologies. This can create privacy risks, with the potential for personal information input into an AI product then surfacing in response to a prompt from another user. You should also consider whether a third party receives personal information through the operation of the commercial AI product. For example, some commercial generative AI products have interfaces with search engines or other features that would result in disclosure of personal information entered in prompts to a third party.
If the operation or features of the AI system require disclosure of personal information to the developer or third parties, you will need to consider whether it is appropriate to proceed with using the product and if so, what measures should be placed around your use of the system to ensure you comply with APP 6. Relevant measures could include turning off features that would disclose personal information, providing any required notice of disclosure to the individual, controls to prevent personal information from being entered or prohibiting your staff from entering personal information in the AI inputs. Any controls or prohibitions on including personal information in AI inputs will need to be accompanied by robust training and auditing measures.
For more information on your APP 6 obligations when using AI systems, see ‘Can we input personal information into an AI system, such as through a prompt in an AI chatbot?’ below.
Practical tips – privacy considerations when selecting an AI product
When choosing an AI product, your organisation must conduct due diligence and ensure you identify potential privacy risks. You should be able to answer the following questions:
Conducting a Privacy Impact Assessment may assist you with considering these questions.
When using AI products, organisations should be aware of the different ways that they may be handling personal information and of their privacy obligations in relation to this information. It is important to understand the specific risks associated with your organisation’s intended uses of AI, and whether or how they can be managed and mitigated.
Organisations should always consider whether the use of personal information in relation to an AI system is necessary and the best solution in the circumstances. AI products should not be used simply because they are available.
Key privacy considerations are set out below in relation to both the data input into an AI system, and the output data generated by AI.
The sharing of personal information with AI systems may raise a number of privacy risks. Once personal information has been input into AI systems, particularly generative AI products, it will be very difficult to track or control how it is used, and potentially impossible to remove the information from the system.[35] The information may also be susceptible to various security threats which aim to bypass restrictions in the AI system, or may be at risk of re-identification even when de-identified or anonymised.
It is therefore important that your business proceeds cautiously before using personal information in an AI prompt or query, and only does so if it can comply with APP 6.
When personal information is input into an AI system, it may be considered either a use of personal information (if the data stays within your organisation’s control) or a disclosure of personal information (if the data is made accessible to others outside the organisation and has been released from your organisation’s effective control) under the Privacy Act. The use and disclosure of personal information must comply with APP 6.
APP 6 provides that an individual’s personal information can only be used or disclosed for the purpose or purposes for which it was collected (known as the ‘primary purpose’) or for a secondary purpose if an exception applies. Common exceptions include where:
Your organisation should identify the anticipated purposes for which you will use personal information in connection with an AI system, and whether these are the same as the purposes for which you collected the information. If you intend to use personal information in AI systems for other, secondary purposes, you should consider whether these will be authorised by one of the exceptions under APP 6.
As outlined in the example below, organisations should frame purposes for collection, use and disclosure narrowly rather than expansively. However, it can be helpful to distinguish between the use of AI which is facilitative of, or incidental to a primary purpose (such as the use of personal information as part of customer service, where AI is the tool used), from purposes which are directly AI-related (such as the use of personal information to train an AI model).
Example – Can an organisation enter personal information into a publicly available generative AI chatbot?
Employees at an insurance company experiment with using a publicly available AI chatbot to assist with their work. They enter the details of a customer’s claim – including the customer’s personal and sensitive health information – into the chatbot and ask it to prepare a report assessing whether to accept the claim.
By entering the personal information into the AI chatbot, the insurance company is disclosing the information to the owners of the chatbot. For this to comply with APP 6, the insurance company will have to consider whether the purpose for which the personal information was disclosed is the same as the primary purpose for which the information was collected. This should have been specified in a notice provided to the customer at the time of collection in accordance with APP 5.
How broadly a purpose can be described will depend on the circumstances, however in cases of ambiguity, the OAIC considers that the primary purpose for collection, use or disclosure should be construed narrowly rather than expansively.[36]
If the purpose of disclosure is not consistent with the primary purpose of collection, the company will need to assess whether an exception under APP 6 applies. If the company wants to rely on the ‘reasonable expectations’ exception under APP 6.2(a), they must establish that the customer would reasonably have expected their personal information to be disclosed for the relevant purpose. The company should consider whether their collection notice and privacy policy specify that personal information may be disclosed to AI system developers or owners, in accordance with APP 5. It may be difficult to establish reasonable expectations if customers were not specifically notified of these disclosures, given the significant public concern about the privacy risks of chatbots.
It is important to be aware that an organisation seeking to input personal information into AI chatbots will also need to consider a range of other privacy obligations, including in relation to the generation or collection of personal information (APP 3), accuracy of personal information (APP 10) and cross-border disclosures (APP 8).
Given the significant and complex privacy risks involved, as a matter of best practice it is recommended that organisations do not enter personal information, and particularly sensitive information, into AI chatbots.
If your organisation is seeking to rely on the ‘reasonable expectations’ exception to APP 6, you should be able to show that a reasonable person who is properly informed would expect their personal information to be used or disclosed for the proposed secondary purpose.
You should consider whether the proposed use would have been within the reasonable expectations of the individual at the time the information was collected. This may be the case if the secondary use was expressly outlined in a notice at the time of collection and in your organisation’s privacy policy.
Whether APP 5 notices or privacy policies were updated, or other information was given at a point in time after the collection, may also be relevant to this assessment. It is possible for an individual’s reasonable expectations in relation to secondary uses to change over time. However, particular consideration should be given to the reasonable expectations at the time of collection, given this is when the primary purpose is determined.
Given the significant privacy risks that may be posed by AI systems, and strong levels of community concern around the use of AI, in many cases it will be difficult to establish that a secondary use for AI-related purposes (such as training an AI system) was within reasonable expectations.
If your organisation cannot clearly establish that a secondary use for an AI-related purpose was within reasonable expectations and related to the primary purpose, to avoid regulatory risk you should seek consent for that use and/or offer individuals a meaningful and informed ability to opt-out.
Importantly, you should only use or disclose the minimum amount of personal information sufficient for the secondary purpose. The OAIC expects organisations to consider what information is necessary and assess whether there are ways to minimise the amount of personal information that is input into the AI product. It may be helpful to consider whether there are privacy-preserving techniques that you could use to minimise the personal information included in the relevant prompt or query without compromising the accuracy of the output.
Further information about APP 6 requirements and exceptions is in the APP Guidelines at Chapter 6: APP 6 Use or disclosure of personal information.
Example – Can our organisation use existing customer data with an AI system?
Your organisation may have existing information that you wish to input into a new AI product. For example, you may plan to use AI to assist with your customer service functions, such as providing tailored marketing recommendations. To do so, the AI system needs to run individual analysis on your customer data.
If your organisation is using a proprietary AI system rather than a publicly available chatbot, for example, and has protections in place to ensure that information entered into the system will not be disclosed outside the organisation (such as to the system developer), this will constitute a use rather than a disclosure of personal information.
To assess whether your organisation’s proposed use of customer data will be compliant with your APP 6 obligations, you should consider the purposes for which the information was collected. This should have been specified in a notice provided to the customer at the time of collection in accordance with APP 5.
If one of the primary purposes of collection was providing tailored marketing recommendations, the proposed use of AI to assist with this analysis is likely consistent with the primary purpose. As discussed above, how broadly a purpose can be described will depend on the circumstances, however in cases of ambiguity, the OAIC considers that the primary purposes of collection, use or disclosure should be construed narrowly rather than expansively.[38]
Your organisation then wishes to use the customer data you hold to further fine-tune the AI system you are using. If the customer’s personal information was not collected for this purpose, you will need to either obtain the customer’s consent to use the information in this way or establish that the reasonable expectations exception under APP 6.2(a) applies. To establish reasonable expectations, you should consider whether you have notified the individual that their personal information would be used in this way, such as via your organisation’s APP 5 notices and privacy policy.
Further OAIC guidance on the development and training of generative AI models.
Practical tips – using or disclosing personal information as an AI input
If your organisation wants to use personal information as an input into an AI system, you should consider your APP 6 obligations:
Given the significant and complex privacy risks involved, as a matter of best practice it is recommended that organisations do not enter personal information, and particularly sensitive information, into publicly available AI chatbots.
If your organisation collects personal information through public facing AI systems, such as through the use of AI chatbots, this collection must comply with APP 3 and APP 5 requirements. For example, if you have a customer facing AI chatbot on your website and you collect the information your customers send to the chatbot, you will need to ensure this collection complies with APP 3. In particular, APP 3 requires the collection of personal information to be reasonably necessary for your entity’s functions or activities and to be carried out by lawful and fair means.
The requirements of APP 3 are discussed under ‘Can we use an AI product to generate or infer personal information?’, directly below.
Where you use AI to infer or generate personal information, the Privacy Act will treat this as a ‘collection’ of personal information and APP 3 obligations will apply. Under the Privacy Act, the concept of collection applies broadly, and includes gathering, acquiring or obtaining personal information from any source and by any means, including where information is created with reference to, or generated from, other information an entity holds.[38] To meet your APP 3 obligations when using AI, you must ensure that:
If you are using AI to generate or infer sensitive information about a person, you will also need to obtain that person’s consent to do so (unless an exception applies).
If personal information is created through the use of AI which your organisation is not permitted to collect under APP 3, it will need to be destroyed or de-identified.
The use of AI systems to infer personal information, such as making inferences about a person’s demographic information or political views based on data from their social media, creates risks by enabling organisations to generate and use personal information that has not been knowingly disclosed by the individual, including information which may not be accurate.
Generating personal information through AI will also attract other APP obligations, including in relation to providing notice to individuals (APP 5) and ensuring the accuracy of personal information (APP 10). These obligations are discussed under the ‘Transparency’ and ‘Accuracy’ sections of this guidance.
Using sensitive information in relation to AI products
Sensitive information is a subset of personal information that is generally afforded a higher level of privacy protection under the APPs. Examples of sensitive information include photographs or videos where sensitive information such as race or health information can be inferred, as well as information about an individual’s political opinions or religious or philosophical beliefs.
It is important that organisations using AI products identify whether they may be collecting sensitive information in connection with these systems. For example, AI image and text manipulators can generate sensitive biometric information, including information that may be false or misleading.
If you use AI to generate or collect sensitive information about a person, to comply with APP 3 you will usually need to obtain consent.[39] Consent goes beyond just a line in your Privacy Policy – you must make sure that the individual is fully informed of the risks of using or generating sensitive information, and this consent must be current and specific to the circumstances that you are using it in.[40]
Organisations cannot infer consent simply because you have provided individuals with notice of a proposed collection of personal information.
What is ‘reasonably necessary’ for the purposes of APP 3 is an objective test based on whether a reasonable person who is properly informed would agree that the collection is necessary. You should consider whether the type and amount of personal information you are seeking to obtain from the AI system is necessary, and whether your organisation could pursue the relevant function or activity without generating the information (or by generating less information).
If there are reasonable alternatives available, the use of AI in this way may not be considered reasonably necessary.
Using AI to generate personal information may not be lawful for the purposes of APP 3 if it is done in breach of legislation – this would include, for example, generating or inferring information in connection with, or for the purpose of, an act of discrimination.[41]
For the purposes of APP 3, the collection or generation of personal information using AI may be unfair if it is unreasonably intrusive, or involves intimidation or deception.[42] This will depend on the circumstances. However, it would usually be unfair to collect personal information covertly without the knowledge of the individual.[43]
For example, organisations seeking to use AI to generate or infer personal information based on existing information holdings for a customer, in circumstances where the customer is not aware of their personal information being collected in this way, should give careful consideration as to whether this is likely to be lawful and fair.
In considering whether the generation of personal information has been fair, it may also be relevant to consider whether individuals have been given a choice. The OAIC encourages entities to provide individuals with a meaningful opportunity to opt-out of having their personal information processed through AI syste
Example – can our business use AI to take meeting minutes?
Commonly available virtual meeting platforms can record meetings and use AI to generate transcripts or written minutes. To comply with APP 3, it is important that you carefully consider the content of the meeting discussion, and ensure that any personal information collected is reasonably necessary to your organisation’s functions or activities. If you find that your system recorded any personal or sensitive information that is not reasonably necessary, such as a personal conversation or confidential client information, you will need to ensure that the information is destroyed or de-identified.
You will also need to obtain consent from participants if the AI system collects sensitive information. Given the uncertainty of what may be discussed in a meeting, seeking consent should be pursued as a matter of best practice. Depending on the circumstances, this could be done by advising meeting attendees of the AI system being in use and providing them with the option to object at the outset. However, this approach will only be appropriate if the information about the system and opt-out option is clear and prominent to all attendees so that it is likely they will have seen and read it, and where the consequences for failing to opt-out are not serious.[44]
Whether the participants in the meeting are made aware of the meeting being recorded and a transcript being produced, and whether consent is given to this occurring, will also be relevant factors in considering whether the AI product’s collection of personal information is by lawful and fair means.[45]
If you are generating personal information through an AI product, to ensure you comply with APP 3 your organisation must be able to show that it would be unreasonable or impracticable to collect this information directly from the individual.[46]
What is ‘unreasonable or impracticable’ will depend on circumstances including whether the person would reasonably expect personal information about them to be collected directly from them, the sensitivity of the personal information being generated by the AI system, and the privacy risks of collecting the information through the use of the particular AI product. While the time and cost involved of collecting directly from the individual may also be a factor, organisations will need to show the burden is excessive in all the circumstances.[47]
Practical tips – using AI to collect or generate personal information
If your organisation is collecting personal information through an AI system such as a chatbot, or using an AI product to generate or infer personal information, you should consider your APP 3 obligations:
Organisations subject to the Privacy Act must comply with their accuracy obligations under APP 10 when using AI systems. APP 10 requires entities to take reasonable steps to ensure that:
As discussed elsewhere in this guidance, AI technologies carry inherent accuracy risks. Both traditional and generative AI models built on biased or incomplete information can then perpetuate and amplify those biases in their outputs, with discriminatory effects on individuals. Developers of AI systems may also unknowingly design systems with features that operate in biased ways.
Accuracy risks are particularly relevant in the context of generative AI. There are a number of factors which contribute to this:
These risks can be compounded by the tendency of generative AI tools to confidently produce outputs which appear credible, regardless of their accuracy. Generative AI also has the potential to emulate human-like behaviours and generate realistic outputs, which may cause users to overestimate its accuracy and reliability.
Organisations looking to use AI must be aware of these accuracy risks and how they may impact on the organisation’s ability to comply with their privacy obligations. As discussed above, APP 10 requires an entity to take reasonable steps to ensure that:
The reasonable steps that an organisation should take will depend on circumstances that include the sensitivity of the personal information, the nature of the organisation holding the personal information, and the possible adverse consequences for an individual if the quality of personal information is not ensured.[51]
There are a number of measures which organisations should consider when seeking to ensure the accuracy of AI outputs. Some examples of these are set out below. However, given the significant accuracy risks associated with AI, and its potential to be used to make decisions that may have a legal or similarly significant effect on an individual’s rights, it is possible that these measures may not always be sufficient to constitute ‘reasonable steps’ for the purposes of APP 10.
How an organisation intends to use outputs from an AI system will also be relevant to the accuracy required. When using a generative AI system for example, if it intends to use the outputs as probabilistic guesses about something that may or may not be true rather than factually accurate information about an individual and this is in a low-risk context, 100% statistical accuracy may not be required. To comply with your APP 10 obligations, you should design your processes to factor in the possibility that the output is not correct. You should also not record or treat these outputs as facts but ensure that your records clearly indicate that they are probabilistic assessments and identify the data and AI system used to generate the output.
When considering potential uses of AI products, organisations should carefully consider whether it will be possible to do so in a way that complies with their privacy obligations in respect of accuracy.
Example – AI in recruiting
Your organisation may wish to use AI for recruiting purposes, such as to source and screen candidates, analyse resumes and job applications and conduct pre-employment assessments. Given the known risk of AI systems producing biased or inaccurate results, including in relation to recruitment, [52] you should consider whether the use of AI is necessary and the best option in the circumstances. If you do use AI to assist with recruitment, you must implement controls to ensure that outputs are accurate and not biased and that the risk of any inaccuracy is considered.
Controls should include assessing the particular AI product and how it has been trained, to ensure that it is appropriate for the intended uses (see ‘Selecting an AI product’ above). You should also ensure that there is human oversight of the AI system’s processes, and that staff are appropriately trained and able to monitor for any inaccurate outputs. This should include periodic assessments of the AI output to identify potential biases or other inaccuracies.
Practical tips – accuracy measures
To comply with your APP 10 obligations, your organisation must take reasonable steps to ensure the accuracy of the personal information it collects, generates, uses and discloses when using an AI product. You should think about:
Organisations subject to the Privacy Act have a number of transparency obligations:
The complexity of AI systems, particularly large AI models such as LLMs, creates challenges for businesses in ensuring that the use of AI is open and transparent. The ‘black box’ problem posed by some AI technologies means that even AI developers may be unable to fully explain how a system came to generate an output.[55] This can make it difficult for businesses to communicate clearly with individuals about the way that their personal information is being processed by an AI system.
Despite these challenges, it is important that organisations take steps to ensure they are transparent about their handling of personal information in relation to AI systems. Ensuring that you manage your organisation’s use of AI systems in an open and transparent way will ensure you meet your APP 1 obligations, as well as increase your accountability to your customers, clients and members of the public, and help to build community trust and confidence.
What constitutes reasonable steps under APP 1 will depend on the circumstances, including the nature of the personal information your organisation holds and the possible adverse consequences for an individual if their personal information is mishandled – more rigorous steps may be required as the risk of adversity increases.[55]
Transparency is critical to enabling individuals to understand the way that AI systems are used to produce outputs or make decisions which affect them. Without a clear understanding of the way an AI product works, it is difficult for individuals to provide meaningful consent to the handling of their personal and sensitive information, understand or challenge decisions, or request corrections to the personal information processed or generated by an AI system. Entities must ensure that the output of AI systems, including any decisions made using AI, can be explained to individuals affected.
Organisations should establish policies and procedures to facilitate transparency, enhance accountability, and ensure good privacy governance. These may include:
Example – Decision-making in AI
Your organisation may be using AI to assist in decision-making processes. For example, you may use AI software that makes predictive inferences about a person to come to a decision on a client’s insurance claim, home loan application or to provide financial advice. The use of AI in relation to decisions that may have a legal or similarly significant effect on an individual’s rights is likely a high privacy risk activity, and particular care should be taken in these circumstances, including consideration of the accuracy and appropriateness of the tool for the intended purpose. In particular, if the behaviour of the AI system cannot be understood or explained clearly by your entity, it may not be appropriate to use for these purposes.
To ensure your use of AI is transparent to your clients, you should ensure that the use of personal information for these purposes is clearly outlined in your Privacy Policy.
If your organisation is using an AI product to assist in the decision-making process, you must understand how the product is producing its outputs so that you can ensure the accuracy of the decision. As a matter of best practice, you should be able to provide a meaningful explanation to your client. As discussed under ‘Accuracy’ below, you should ensure that a human user within your entity is responsible for verifying the accuracy of these outputs and can overturn any decisions made.
It is critical to ensure that you can provide the client with a sufficient explanation about how the decision was reached and the role that the AI product played in this process. This will enable the client to be comfortable that the decision was made appropriately and on the basis of accurate information.
Practical tips – transparency measures
Your organisation must be transparent about its handling of personal information in relation to AI systems. Key steps include:
Taking these steps will help to meet your transparency obligations under APP 1 and APP 5 as well as enhance the accountability of your organisation and build trust in your products or services.
Throughout the lifecycle of the AI product, your organisation should have in place processes for ensuring that the product continues to be reliable and appropriate for its intended uses. This will assist your organisation to comply with the requirement under APP 1 to take reasonable steps to implement practices, procedures and systems that will ensure it complies with the APPs and is able to deal with related inquiries and complaints. This should include:
Example – Importance of ongoing monitoring of embedded AI tools
Your organisation implements the use of an AI assistant called ‘Zelda’ which performs a range of tasks, including generating summaries of customer interactions, providing information on request, and editing written work. Zelda takes on the tone of a friendly colleague and staff find the tool very useful for their work.
Over time, as the model’s underlying processes and training data become outdated, Zelda increasingly produces errors in its output. Unless your organisation has robust processes in place to provide human oversight and review of these outputs, ensure the tool remains fit for purpose, and train staff to understand its limitations, there is a risk that staff will come to over-rely on the assistant and overestimate its accuracy, leading to potential compliance risks.
Question
Considerations
Is the AI system appropriate and reliable for your entity’s intended uses?
To assist with meeting your organisation’s privacy obligations, particularly regarding accuracy under APP 10, consider:
Can you clearly identify the data that the system has been trained on, to ensure that its output will be accurate?
To help assess whether your use of the AI system will be compliant with your APP 10 obligations:
What are the potential security risks associated with the system?
It is important to consider your entity’s security obligations under APP 11 when selecting an AI product. Consider:
What is the intended operating environment for the AI system?
As part of assessing your APP 11 obligations, consider:
Will your inputs be accessible by the system developer?
Your use of the AI system must be compliant with your APP 6 obligations regarding the disclosure of personal information. Ensure you understand:
Question
Considerations
Are you using or disclosing personal information in the context of an AI system?
To assess whether you will be compliant with your entity’s obligations under APP 6, consider:
Are you collecting, generating or inferring personal information using an AI system?
To assess whether you will be compliant with your entity’s obligations under APP 3, consider:
Have you taken reasonable steps to ensure the accuracy of personal information in relation to the AI system?
To comply with your APP 10 obligations, your business should consider:
Has your organisation put in place appropriate transparency and accountability measures?
To comply with your business’s obligations under APP 1 and APP 5, you should consider:
What ongoing assurance processes need to be put in place?
Your organisation will need to ensure it has established appropriate processes and systems to ensure it complies with its APP obligations throughout the lifecycle of the AI system. Consider:
[1] Department of Industry, Science and Resources (DISR), Proposals paper for introducing mandatory guardrails for AI in high-risk settings, DISR, September 2024, p. 32.
[2] Generative AI in particular raises a number of other issues which are in the remit of other Australian regulators, such as online safety and consumer protection. For further information see: Digital Platform Regulators Forum (DP-REG), Working Paper 2: Examination of technology – Large Language Models, DP-REG, 25 October 2023 and DP-REG, Working Paper 3: Examination of technology –Multimodal Foundation Models, DP-REG, 19 September 2024.
[3] JM Paterson ‘Regulating Generative AI in Australia: Challenges of regulatory design and regulator capacity’ in P Hacker, Handbook on Generative AI, Oxford University Press, 2024.
[4] Australian Signal Directorate’s Australian Cyber Security Centre (ACSC), ‘An introduction to Artificial Intelligence’, ACSC, November 2023, accessed 2 September 2024; Business.gov.au, ‘Artificial intelligence (AI)’, July 2024, accessed 2 September 2024.
[5] Productivity Commission, Making the most of the AI opportunity: AI update, productivity and the role of government, Research paper 1, Productivity Commission, 2024, accessed 2 September 2024, p. 2.
[6] Organisation for Economic Co-operation and Development (OECD), What is AI? Can you make a clear distinction between AI and non-AI systems?, OECD AI Policy Observatory website, 2024.
[7] DISR, Proposals paper for introducing mandatory guardrails for AI in high-risk settings, DISR, September 2024, p. 53 (glossary).
[8] For further information on how some artificial intelligence terms interrelate see the technical diagram in F Yang et al, GenAI concepts, ADM+S and OVIC, 2024.
[9] DISR, Proposals paper for introducing mandatory guardrails for AI in high-risk settings, DISR, September 2024, p. 54.
[10] DISR, Proposals paper for introducing mandatory guardrails for AI in high-risk settings, DISR, September 2024, p. 55. Unlike general-purpose AI models, narrow AI systems cannot be used for a broader range of purposes without being re-designed.
[11] DISR proposals paper – p 26.
[12] DISR, Safe and responsible AI in Australia – Discussion paper, DISR, June 2023, p 5.
[13] DISR, Safe and responsible AI in Australia – Discussion paper, DISR, June 2023, p 5.
[14] DISR, Safe and responsible AI in Australia – Discussion paper, DISR, June 2023, p 5.
[15] Digital Platform Regulators Forum (DP-REG), Working Paper 2: Examination of technology – Large Language Models, DP-REG, 25 October 2023,; OAIC, Submission to the Department of Industry, Science and Resources – Safe and responsible AI in Australia discussion paper, OAIC, 18 August 2023, para [3].
[16] DP-REG, Working Paper 3: Examination of technology – Multimodal Foundation Models, DP-REG, 19 September 2024, p. 3.
[17] DP-REG, Working Paper 3: Examination of technology – Multimodal Foundation Models, DP-REG, 19 September 2024, pp. 7-9.
[18] An hallucination occurs where an AI model makes up facts to fit a prompt’s intent – this occurs when an AI system searches for statistically appropriate words in processing a prompt, rather than searching for the most accurate answer: F Yang et al, GenAI concepts, ADM+S and OVIC, 2024. A deepfake is a digital photo, video or sound file of a real person that has been edited to create a realistic or false depiction of them saying or doing something they did not actually do or say: eSafety Commissioner (eSafety), Deepfake trends and challenges — position statement, eSafety, 19 August 2024.
[19] UK ICO, ‘What about fairness, bias and discrimination?’, Guidance on AI and data protection, UK ICO, accessed 7 August 2024; OECD, ‘AI, data governance, and privacy: Synergies and areas of international co-operation’, pp 21-22.
[20] Note that although AI poses significant issues in relation to explainability, these are not covered more broadly in this guidance which focuses on the privacy risks of AI.
[21] OECD, ‘AI, Data Governance and Privacy: Synergies and areas of international co-operation’, OECD Artificial Intelligence Papers, No. 22, OECD, June 2024, accessed 2 September 2024, p. 21. For more on de-identification see OAIC, De-identification and the Privacy Act, OAIC, 21 March 2018.
[22] Bell, G., Burgess, J., Thomas, J., and Sadiq, S, Rapid Response Information Report: Generative AI – language models (LLMs) and multimodal foundation models (MFMs), 24 March 2023; National Cyber Security Centre (UK), ChatGPT and large language models: what’s the risk?, 14 March 2023.
[23] G Bell et al, Rapid Response Information Report: Generative AI – language models (LLMs) and multimodal foundation models (MFMs), 24 March 2023; Confederation of European Data Protection Organizations, Generative AI: The Data Protection Implications, 16 October 2023, p 12.
[24] Organisation for Economic Development (OECD), ‘AI, Data Governance and Privacy: Synergies and areas of international co-operation’, OECD Artificial Intelligence Papers, No. 22, OECD, June 2024, accessed 2 September 2024; DP-REG, Working Paper 3: Examination of technology – Multimodal Foundation Models, DP-REG, 19 September 2024, p. 24; UK Information Commissioner’s Office (UK ICO), Generative AI first call for evidence: The lawful basis for web scraping to train generative AI models | ICO, UK ICO, 2024, accessed 2 September 2024.
[25] A number of these issues are directly relevant to other areas of law. For considerations from some other regimes, see National Artificial Intelligence Centre, Voluntary AI Safety Standard, DISR, August 2024, pp 11-12; DP-REG, Working Paper 3: Examination of technology – Multimodal Foundation Models, DP-REG, August 2024; DP-REG, Working Paper 2: Examination of technology – Large Language Models, DP-REG, 25 October 2023.
[26] The eSafety Commissioner (eSafety) is Australia’s independent regulator and educator for online safety. eSafety provides more information on the online safety risks and harms of generative AI. eSafety Commissioner (eSafety), Deepfake trends and challenges — position statement, eSafety, 19 August 2024.
[27] Data poisoning involves manipulating an AI model’s training data so that the model learns incorrect patterns and may misclassify data or produce inaccurate, biased or malicious outputs: Australian Signals Directorate’s Australian Cyber Security Centre (ACSC), Engaging with Artificial Intelligence (AI), ACSC, accessed 2 September 2024, p. 6.
[28] E Bender et al, ‘On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?’ [conference presentation], ACM Conference on Fairness, Accountability, and Transparency, 3-10 March 2021, accessed 29 August 2024, pp 616-617.
[29] I Shumailov et al, ‘AI models collapse when trained on recursively generated data’, Nature.
[30] Bridget McArthur, ‘AI chatbot blamed for psychosocial workplace training gaffe at Bunbury prison’, ABC News, updated 21 August 2024, accessed 2 September 2024.
[31] See discussion in DP-REG, Working Paper 1: Literature summary – Harms and risks of algorithms, DP-REG, 24 November 2023.
[32] DISR, Safe and responsible AI in Australia – Discussion paper, DISR, June 2023, p. 8.
[33] Thomas Davenport and Ravi Kalakota, ‘The potential for artificial intelligence in healthcare’, Future Healthcare Journal, 2019, 6(2), accessed September 2024.
[34] Mirja Mittermaier et al, ‘Bias in AI-based models for medical applications: challenges and mitigation strategies’, npj digital medicine, 2023, 113, accessed 2 September 2024.
[35] Federal Office for Information Security (France), AI Coding Assistants, September 2024, pp. 9-11.
[36] Confederation of European Data Protection Organisations (CEDPO), Generative AI: The data protection implications, CEDPO October 2023, accessed 2 September 2024, p. 6.
[37] Australian Privacy Principles guidelines (oaic.gov.au) p. 22.
[38] Australian Privacy Principles guidelines (oaic.gov.au) p. 22.
[39] Chapter B: Key concepts | OAIC see B.26-B.31.
[40] APP 3.3 requires entities to obtain consent before collecting sensitive information unless an exception under APP 3.4 applies. For more information, see ‘Chapter 3: APP 3 Collection of solicited personal information’, APP Guidelines, OAIC, 22 July 2019.
[41] Australian Privacy Principles guidelines (oaic.gov.au) – the meaning of consent is discussed at pp. 9-13.
[42] Australian Privacy Principles guidelines (oaic.gov.au) 3.61.
[43] Australian Privacy Principles guidelines (oaic.gov.au), 3.62-3.63.
[44] Noted in Guide to data analytics and the Australian Privacy Principles | OAIC.
[45] Australian Privacy Principles guidelines (oaic.gov.au), B.43.
[46] Entities should also consider and comply with other obligations and laws that apply to the use of recording devices such as State and Territory surveillance device laws: Existing surveillance device laws | ALRC.
[47] See APP 3.6.
[48] Australian Privacy Principles guidelines (oaic.gov.au), 3.64 – 3.66.
[49] Andrew Ng In Conversation with Toby Walsh (Recording), UNSW Sydney, 21 February 2024 (accessed 11 October 2024).
[50] I Shumailov et al, ‘AI models collapse when trained on recursively generated data’, Nature, 2024, 631:755-759; OECD, ‘AI, data governance, and privacy: Synergies and areas of international co-operation’ p. 21.
[51] See discussion of capability in Y Benigo et al, ‘International Scientific Report on the Safety of Advanced AI: Interim Report’, DIST research paper series, number 2023/009, 2024 pp21-22, 26-27.
[52] Chapter 10: APP 10 Quality of personal information | OAIC
[53] For example, see: Sarah Ayoub, ‘Recruitment by robot: how AI is changing the way Australians get jobs’, The Guardian (Australia), 23 October 2023, accessed 2 September 2024.
[54] For example, ‘human in the loop’ processes enable humans to make decisions, provide approvals, or correct and prevent potential errors, while ‘human in command’ ensures that humans have ultimate control over the AI system and make the important decision: F Yang et al, GenAI concepts, ADM+S and OVIC, 2024.
[55] OECD, ‘AI, Data Governance and Privacy: Synergies and areas of international co-operation’, OECD Artificial Intelligence Papers, No. 22, OECD, June 2024, accessed 2 September 2024, p. 36; Office of the Victorian Information Commissioner (OVIC), Artificial Intelligence and Privacy – Issues and Challenges, OVIC, accessed 2 September 2024.
[56] Australian Privacy Principles guidelines (oaic.gov.au), [1.6].
We’d love to hear more about your rating
Monday to Thursday 10 am to 4 pm (AEST/AEDT)
Follow us
Acknowledgement of Country
The OAIC acknowledges Traditional Custodians of Country across Australia and their continuing connection to land, waters and communities. We pay our respect to First Nations people, cultures and Elders past and present.
© Commonwealth of Australia
