OpenAI is reported to be running an internal project in which people read and evaluate real user conversations to improve the quality of ChatGPT’s answers. An automatic filter that masks personal information is applied, but concerns have been raised that sensitive details may not be fully removed.

On Sept. 15 local time, 404 Media reported that OpenAI is conducting an internal evaluation effort called Project Lily. Internal documents obtained by the outlet say hundreds of contract workers read real ChatGPT conversations, identify users’ intent and then assess the quality of the model’s responses and any problems.

Reviewers are not shown users’ account names, but they may be provided with entire conversations. Some tasks also include a “user memory summary” that compiles past context ChatGPT has retained. The outlet said this can contain information that could be used to infer an individual, such as usage purpose, interests and an approximate region.

OpenAI uses a Privacy Filter before review to detect and mask personally identifiable information such as names and addresses, email addresses, phone numbers, bank account numbers and other identifiers. But even in model documentation the company has published, it acknowledges it can miss rarely used names, region-specific terms, or personal information that varies by context.

Personal ChatGPT users can prevent new conversations from being used for model training by turning off “Improve the model for everyone” under Settings and Data Controls. ChatGPT Business, Enterprise and Edu, by default, do not use inputs and outputs for model training.

“Temporary Chat” is also not used for model improvement. It may be stored for up to 30 days for safety and abuse prevention, and can be reviewed on a limited basis if necessary. OpenAI advises users not to enter sensitive information they do not want reviewed by people.

As human evaluation continues to be used to improve the quality of AI services, the issue of transparency over how far real users’ private conversations can enter training and evaluation processes appears to be resurfacing.