What does model memorisation imply for privacy in AI-supported marketing?

Training-data extraction, differential privacy and the boundaries of application-level protection

This retrospective review considers source versions available by 25 February 2026. The cutoff defines the evidence included; preparation and source checking took place on 21 October 2026.

Keywords: memorisation; privacy; training-data extraction; differential privacy; data minimisation

1. Privacy as a property of a complete processing system

A marketing model can process personal information without being trained on it. A prompt may contain an interview transcript, a retrieval service may supply a customer record, and a logging system may retain the resulting conversation. Training is only one point at which information can enter or remain in the system. This distinction is central to interpreting assurances that customer inputs are excluded from model training. Such an assurance addresses an important use of data, but leaves questions about access, storage, transmission and authorised purpose unresolved.

The research question is how evidence on memorisation and extraction should inform the analysis of AI-supported marketing systems. The relevant outcomes include verbatim recovery of training text, inference about membership in a dataset and disclosure through ordinary application access. Each can harm privacy, but each requires different evidence. A model reproducing an address does not automatically establish that the address came from a private training dataset. A model failing to reproduce it does not establish that every other route of disclosure is closed.

2. Method and scope

This narrative review was prepared on 21 October 2026 from primary-source retrieval concerning training-data extraction, memorisation, membership inference and differential privacy. Historical governance concepts draw on the original 2016 GDPR text and NIST Privacy Framework version 1.0. Thirteen references are retained. Author abstracts and publication records support the research descriptions. The differential-privacy monograph was identified bibliographically; its definition was cross-checked against official indexed NIST excerpts [13]. The legal text distinguishes concepts without evaluating a particular deployment or claiming a current jurisdiction-specific compliance assessment.

The selection is not a systematic review of privacy-preserving machine learning. No model was queried, no canary inserted, and no personal dataset examined. The review does not estimate leakage from a current supplier's proprietary model. It also does not decide whether a particular campaign has a lawful basis. The legal material is confined to the regulator's published explanation of the concepts needed to distinguish technical confidentiality from permission to process data.

3. Distinguishing the privacy outcomes

3.1 Memorisation and extractability

Memorisation concerns information learned about particular training examples. Extractability asks whether an adversary can recover that information through an available interface. The distinction affects what an experiment can establish. A researcher who already knows a sensitive sequence can test whether a model completes it; an attacker without that prior knowledge faces a different search problem. Nasr and colleagues explicitly study extractable memorisation, reporting methods that recover training data without prior knowledge of the underlying dataset [2].

Carlini and colleagues' GPT-2 extraction study recovered hundreds of verbatim sequences, including public personal information, conversations and code [1]. This demonstrates that training on publicly collected text can leave recoverable details in a model. Public availability of the original text does not make the technical finding irrelevant to privacy, although it changes the circumstances of collection and disclosure. The experiment is evidence of a leakage mechanism in the studied system, not proof that any arbitrary prompt submitted to a commercial tool becomes available to other users.

3.2 Membership is a different question

Shokri and colleagues examine whether an attacker can determine if a record was included in a model's training set through black-box access to the model. Their experiments concern classification systems, including machine-learning services [5]. Knowing that a person appears in a dataset may itself be sensitive even if the attacker cannot reconstruct the entire record. Conversely, a generated detail can be factually correct without proving dataset membership. The two attacks therefore need separate success criteria and separate statements about attacker knowledge.

In a marketing setting, membership could concern inclusion in a research panel or customer segment, while extraction could concern the words of an interview. These are illustrative applications of the distinction, not reported findings from the cited classification experiment. Their purpose is to clarify why a single privacy score is inadequate. A system could resist one attack and remain vulnerable to another because the model, interface and adversary expose different information.

4. What extraction research establishes

4.1 Scale, repetition and context

Quantifying Memorization Across Neural Language Models reports that memorised output increased with model capacity, duplication of examples and the amount of context supplied to the model. The authors also report that generalising these relationships across model families is more complicated [3]. The qualifications matter: size is one variable in a training and evaluation configuration. It is not a sufficient basis for assigning a universal privacy ranking to models with different data mixtures, deduplication, post-training procedures and interfaces.

Repeated material and longer known prefixes affect the opportunities available to an extraction procedure. This suggests different measurement questions for a public-language model and a model adapted on a small set of repeated company documents. The cited research does not supply a leakage percentage for the latter situation. Estimating it would require a defined corpus, an attack model and a comparison of sequences exposed under the actual adaptation procedure. General relationships help identify what to test; they cannot substitute for that test.

4.2 Production interfaces and alignment

Nasr and colleagues report extraction from several kinds of language model, including a production chatbot. They developed a divergence attack for the aligned chatbot and concluded that alignment did not eliminate memorisation in the systems tested [2]. The historical result separates a model's usual conversational behaviour from information that can be elicited under a specialised attack. It does not establish whether the same attack works against a revised service, and it does not identify the handling of a particular organisation's current prompts.

A defence can affect the probability that a query produces memorised material without deleting every learned trace of that material. This is why output filtering, training-data selection and privacy-preserving learning address different stages. A successful filter is evidence about observable output under its test conditions. A training guarantee is a statement about the learning mechanism under defined assumptions. Treating the former as the latter would make the security claim stronger than the evidence supports.

4.3 Fine-tuning as an additional interface

The Janus study reports an attack that uses a fine-tuning interface to recover personal information from pretraining data, with amplification relative to the baselines studied [8]. The abstract also discusses the difficulty of distinguishing true disclosure from hallucinated personal details. The relevant finding is that adaptation privileges can change the attack surface. Its exact magnitude is specific to the models and evaluation described; the present review does not generalise it to every fine-tuning provider or dataset.

Fine-tuning on marketing data creates a separate assessment question from using a general model with a short prompt. Both may expose information, but they place it in different parts of the system. A document supplied as context need not change model weights. A document used in adaptation can affect later behaviour beyond that individual request. An adequate account therefore identifies which transformation occurs, who can invoke it and which later interfaces can access the resulting artifact.

5. Testing memorisation without overinterpreting it

The Secret Sharer proposes a testing methodology for unintended memorisation of rare or unique sequences. The authors show that inserted secrets can support quantitative assessment and describe an application to a commercial text-completion system [4]. Such canaries provide controlled ground truth: the researcher knows the sequence and its placement. They are useful because ordinary output inspection may miss a low-frequency leakage pathway. The result depends on the design of the test and the sequence distribution.

A canary result answers a controlled question about a particular training and querying process. Real customer records have different lengths, frequencies and contextual cues. A model that rarely emits a random identifier may still reproduce a familiar address embedded in repeated text. The reverse can also occur. A robust report states how secrets were generated, how often they appeared, what the attacker knew and how candidate outputs were scored. Without those details, a result called no leakage is difficult to compare with another experiment.

Testing also requires separating correctness from memorisation. A generated telephone number can be fabricated. A correct name may be a common fact encountered through several sources. A meaningful extraction claim needs evidence connecting the output to training material under the study's definition. Membership tests likewise require suitable member and non-member comparisons. The privacy literature supplies methods for such distinctions, while an informal conversation with a model generally cannot identify the complete origin of its answer [1,4,5].

6. Differential privacy and the meaning of a guarantee

Dwork and Roth formalise differential privacy as a limit on how much the output distribution of a randomised mechanism changes when neighbouring datasets differ in the contribution being protected [7,13]. The definition concerns the mechanism and its parameters. It allows reasoning about privacy without assuming that an adversary lacks all background knowledge. It also requires specifying the protected unit: a record-level claim and a person-level claim can cover different amounts of information when one person contributes many records.

Abadi and colleagues develop algorithms for training neural networks with differential privacy and analyse privacy costs [6]. Their experiments show that useful deep learning can be performed under a quantified privacy budget. The existence of the method does not imply that an ordinary model was trained with it. A deployment claiming this protection must identify the mechanism, parameters, accounting and data contribution limits. Those details determine the guarantee; the phrase privacy-preserving is not a substitute.

Privacy and utility remain separate quantities. Introducing privacy protection can change model quality or the resources needed to achieve a given quality target. The relevant comparison keeps the task and evaluation data fixed while reporting both privacy parameters and predictive performance. It should also distinguish the protection of training contributions from the confidentiality of inference-time prompts. A differentially private training algorithm does not, by itself, prevent a logging system from retaining an unredacted customer transcript.

7. Minimisation across the AI data lifecycle

Article 5 of the original GDPR text distinguishes purpose limitation from data minimisation [11]. Minimisation concerns whether information is adequate, relevant and limited in relation to a specified purpose. The smallest possible input is not necessarily an adequate one. A technical review can distinguish what processing is for from what information that purpose requires. Omitting essential context can impair an analysis, while adding unrelated records expands processing without establishing their contribution to the task.

AI systems can copy information into training sets, test sets, prompts, retrieval indexes and output records. Article 32 of the original GDPR text relates security measures to processing risk [9]. This review distinguishes a model endpoint from information handling across its surrounding workflow. An inventory can separate original records, transformed or duplicated versions and generated outputs containing personal details. Extraction research does not establish that any particular inventory or security control is complete.

Removing obvious identifiers can reduce exposure, but its effect depends on what remains and who has auxiliary information. A qualitative interview can contain distinctive biographical details even after a name is removed. This review does not quantify re-identification risk for such text or treat pseudonymisation as demonstrated anonymity. The defensible statement is narrower: a transformation changes the data available to a system, and its adequacy must be assessed against the data and purpose rather than inferred from the transformation's label.

8. Training exclusion, hosting and authorised processing

A service can promise not to train on a customer's inputs while still processing those inputs to provide an answer. It may also retain operational information under its applicable terms. The current paper makes no claim about any particular supplier's present contract. The analytical distinction is between training use, inference processing and retention. Each is a separate data flow that needs an accurate description. A marketing team cannot infer all three from a statement about only one.

Local deployment changes the destination of model computation. It may avoid sending prompts to an external inference provider, but a local application can still use cloud logging, remote retrieval or shared storage. The privacy boundary follows the complete data path. Conversely, a hosted service may provide access controls or contractual restrictions that differ from an unmanaged local installation. The literature reviewed here establishes no universal ranking of those complete configurations.

Article 6 of the original GDPR text specifies conditions for lawful processing [10]. This differs from whether information is technically protected against disclosure. Confidentiality controls do not by themselves establish the purpose and authority for processing. Training a general model and producing a customer response are distinct workflow operations. This review neither infers a lawful basis from a security mechanism nor determines authorisation for a particular operation or jurisdiction.

9. Privacy risk and the measurement boundary

The NIST Privacy Framework situates privacy within organisational risk management [12]. For the present research question, that broader setting prevents a model test from becoming a claim about the entire organisation. A low extraction rate is valuable evidence about one mechanism. It leaves unresolved whether a person expected the processing, whether the data were accurate, whether staff access was appropriate and whether an objection or deletion request can be handled across copies.

Evaluation boundaries also determine which costs are visible. A system may restrict access to raw interviews but generate detailed summaries retained in an unrestricted folder. The original access control has not necessarily protected the derived information. That observation follows from tracking the information flow and does not depend on a model memorising the interviews. It illustrates why model privacy and ordinary application confidentiality should be analysed together while retaining separate outcome measures.

9.1 Distinguishing a privacy test from a release decision

An extraction test supplies evidence about what an adversary can recover under stated conditions. Releasing a model or enabling a data flow requires a broader decision about the information at stake and the consequences of disclosure. A negative test result is informative but conditional on the attacks attempted. A positive result establishes an exposure pathway without automatically measuring its prevalence in ordinary use. Preserving that distinction allows technical findings to inform governance without overstating either reassurance or alarm. It also makes later retesting meaningful when the model or interface changes.

10. Limitations and conclusion

The reviewed attacks concern particular historical model families and interfaces. They establish technical possibilities and conditional relationships, not a current leakage rate for every AI tool. Most empirical sources were consulted at abstract level, so detailed statistical appraisal and reproduction remain outside scope. The official guidance may change as law and practice develop; the cited pages were consulted on 21 October 2026. No claim is made that a single technical intervention settles all privacy obligations.

The literature demonstrates recoverable training information, membership leakage and changes in privacy risk through adaptation. It also provides formal and empirical methods for limiting or measuring specific forms of exposure. Applying those findings to marketing requires a complete account of processing: where personal data enter, whether weights change, which interfaces can reveal information and where copies remain. Training exclusions, local hosting and privacy-preserving learning each address a defined part of that account. Their scope should remain explicit.

Source access and appraisal register

The research studies [1], [2], [3], [4], [5], [6] and [8] were consulted through author abstracts and publication records. The monograph [7] was identified from its author-hosted front matter; definition pages were not fully retrieved. Terminology was cross-checked against official indexed NIST excerpts [13]. References 9–11 identify three articles of the same 2016 instrument, not independent empirical studies. The fixed Privacy Framework [12] supplies organisational context. No independent model audit, complete legal assessment or full privacy-accounting calculation is reported.

References