Anyone else paranoid using AI for analysis?

I'm a data scientist by training with my own process for AI-assisted analysis, SOPs, asserts, sanity checks. Just want to see if others feel what I feel.

Claude Code for products: incredible, tight feedback loop, works or it doesn't.

Claude Code for analysis: paranoid every time. Wrong analysis looks identical to right analysis, silently dropped rows, miscoded variables, a slightly wrong groupby, the code runs, the number has decimals, and you have no idea if it's real unless you read every line.

And I feel one step removed from the data now. I used to write every line myself and notice the weird distribution, the unexpected category, the row that didn't belong. That peripheral awareness is where real insight comes from. With the LLM in the loop, I touch the data less, and I catch less.

1. Do you also feel one step removed from the data compared to before these tools existed?

2. What are you doing to safeguard and double-check AI-assisted analysis?

3. Has AI-assisted analysis ever caused you to ship a wrong number to a stakeholder? What happened?