Transforming Transformations with AI

The stakes were high. Tasked with bringing the first AI solution into a complex ETL platform and coming off a just-canceled project. My team and me knew this was our chance to shine.

Using the score, we ended with a shortlist of opportunities for the data science team to explore through experiments.
Following Adverity’s Discoverer process: a blend of Shape Up and Design Sprint, we focused on learning fast and so many experiments quickly reached dead ends due to insufficient data or high technical costs.

Our plan seemed clear and feasible. We took an LLM (via OpenAI’s API), fine-tuned it using data from documentation and support tickets, and developed a conversational assistant.

But reality hit fast.
During early testing with customer-facing teams, data transformation experts reported frequent hallucinations. They also pointed us to the root cause: outdated documentation.

Like other internal tech debts, it had never made it to the top of the roadmap. Now it was front and center, undermining our AI and inflating support volume.

We hit pause and got back to the drawing board. If documentation couldn’t be trusted, we needed a new foundation. This time, we partnered directly with internal data transformation experts. Using a simple feedback UI, they reviewed assistant responses, flagged inaccuracies, and provided corrections. Every manual label made the model smarter.
We also rebalanced the training data, giving more weight to real, resolved support tickets: ground truth over idealized help articles.

Yes, it raised the project’s complexity and cost. But investors were watching closely, and AI was still the centerpiece of our product vision. We made our case and secured extra funding.
Better yet, our revised strategy created a second benefit: expert-reviewed responses could now be fed back into our documentation, improving it in parallel.

One system. Two wins.

We had traction. While the assistant wasn’t flawless, the number of cases requiring expert intervention dropped noticeably. It handled more of the repetitive, time-consuming queries freeing up our support team to focus on complex edge cases and faster response times.

Each resolved interaction became part of a continuous improvement loop. Expert-reviewed answers were fed back into the model, sharpening its accuracy. Fewer follow-up questions meant fewer API calls, which translated into lower costs and better performance.

— Efficiency improved. Precision grew. And trust followed.—

Designing a conversational assistant was about crafting a personality, building trust, and setting expectations in a space full of unknowns.

I faced new challenges:

• How do we reflect the brand’s tone of voice in the assistant’s responses?
• How do we gracefully handle irrelevant questions or edge cases?
• How do we signal limitations without eroding trust?

To address this, we teamed up with data scientists again this time to define a set of known data transformation intents. These acted as anchors for response quality and let us craft replies that were not only context-aware but also human in tone.

We designed fallback interactions that balanced clarity with charm. Even in error states, the assistant stayed helpful and a bit playful ;)

It was here that conversation design became a part of our design system. And it was the UX community through shared resources like shapeof.ai that guided me through unfamiliar territory. Thanks.

Once the assistant launched, feedback came fast and with it, a clear direction for what came next. Users wanted more than explanations. They wanted execution.

We envisioned a future where the assistant could offer personalized suggestions and apply data transformations directly, moving from advisor to action-taker.
This step came with new challenges:

• Trust: users needed full visibility before allowing any automated action. The assistant would have to explain its intended steps clearly, enabling users to confirm or reject them.
• Security was non-negotiable. Because the assistant handled sensitive data, we had to guarantee strict isolation between customer environments: no overlap, no leakage.

The project moved again into the pitching phase with the vision of an assistant that not only understands your data problems, but solves them: securely, transparently, and in real time.