A recent discussion paper from the Bank of Japan's Institute for Monetary and Economic Studies, co-authored with the Bank of Korea, investigates the feasibility of using generative artificial intelligence (GenAI), specifically ChatGPT, to simulate survey responses regarding user perceptions of payment applications. The study benchmarks its findings against an existing human-based survey conducted by Brits and Jonker (2023) in the Netherlands, which focused on the privacy paradox and privacy calculus associated with financial apps.

The researchers designed "generative agents" with demographic profiles—such as age, gender, and user status—mirroring the distribution observed in the actual survey. Using a structured prompt methodology (Role, Task, Format), they simulated responses to questions about the benefits and risks of using mobile payment apps.

The findings reveal that GenAI can replicate several key patterns from human surveys. Notably, agents categorized as more privacy-concerned consistently viewed payment apps less favorably and perceived higher risks, aligning with the concept of privacy calculus. Furthermore, ChatGPT successfully captured the significant difference in perceptions between users and non-users without explicit instruction, with users reporting higher benefits and lower risks.

However, the study highlights critical limitations. Most significantly, the variation in AI-generated responses, measured by standard deviation, was substantially lower than in the human survey. This suggests GenAI struggles to replicate the full spectrum of human opinions, potentially overlooking minority viewpoints. Additionally, a strong bias was observed: a disproportionately high number of generative agents were classified as "privacy fundamentalists," indicating a systemic skew towards privacy concern that does not match the distribution in the actual population. Attempts to increase variation by adjusting the AI's "temperature" parameter or adding more detailed persona attributes (like income and education) yielded only minimal improvements.

The research concludes that while GenAI shows promise as a complementary tool for preliminary survey design and brainstorming—offering cost and time efficiencies—its inherent limitations in response variance and potential for embedded biases make it unsuitable as a complete substitute for human respondents. Policymakers, in particular, are cautioned that relying solely on AI-generated data could lead to misleading conclusions by failing to adequately represent diverse, especially minority, perspectives.