Neural Transparency: Mechanistic Interpretability Interfaces for Anticipating Model Behaviors for Personalized AI

Overview

This research addresses the opacity problem in personalized large language model interactions by proposing a mechanistic interpretability interface that exposes internal neural activations during chatbot personality design. Users commonly employ system prompts to customize LLM-based chatbots without visibility into how design choices manifest as model behaviors, creating potential for harmful outcomes including sycophancy, toxicity, and misaligned objectives. The study introduces a method for extracting behavioral trait vectors through contrastive system prompt analysis and projecting these vectors to generate quantified persona scores visualized through interactive sunburst diagrams, enabling non-technical users to anticipate model behaviors before deployment.

Methods and approach

The approach operates by calculating differential neural activations between contrasting system prompts designed to elicit opposing behavioral traits (empathy, toxicity, sycophancy, and others). Behavioral trait vectors are derived from these activation differences. System prompt final token activations are then projected onto these trait vectors to generate persona scores, which are normalized across traits for comparability and rendered via interactive visualization. The interface was evaluated through an online user study with 80 participants comparing the neural transparency interface against a baseline without transparency mechanisms. Quantitative metrics assessed user calibration accuracy for trait predictions, while qualitative analysis examined user experiences with the visualization and interaction patterns during chatbot design iterations.

Key Findings

Quantitative findings demonstrated systematic user miscalibration: participants misjudged trait activations in 11 of 15 analyzable traits when comparing predicted to actual model behaviors. While the neural transparency interface did not significantly alter design iteration patterns or the number of refinements users made, it produced measurable increases in user trust and received positive qualitative reception. Qualitative analysis revealed nuanced user experiences with the visualization interface, identifying both effective aspects of the approach and specific areas where interface design and interaction mechanics require refinement to optimize user comprehension and utility.

Implications

The results indicate that users lack reliable intuitions about how system prompts translate to model behaviors, establishing a substantive need for transparency tools in everyday human-AI interaction contexts. The finding that the interface increased trust despite not altering iteration patterns suggests that transparency mechanisms satisfy user preferences for explainability and control independent of their influence on design outcomes. The work demonstrates feasibility of operationalizing mechanistic interpretability techniques for non-technical users, creating a methodological pathway for integrating interpretability research into practical human-AI interaction design. Future iterations should focus on refining visualization design and interaction patterns to enhance user calibration accuracy and decision-making quality during personality configuration.