The Wave of Agentic AI: Lessons Learned from Nate’s $50 Million ‘Experiment’

Agentic AI promises to transform our world with automation and technological advancements. However, in reality, most use cases still fall short in real-world environments.

A strong example of this is Nate, a company that claimed AI agents powered its services. In reality, a team of people were actually doing the work manually. Yet, they managed to raise $50,000 from false promises of agentic AI doing real work.

Andrzej Kinastowski, Head of Delivery and Managing Partner at Office Samurai, shared this example during a talk on agentic AI at the recent KYP.ai Forward event.

Nate’s story underscores the importance of distinguishing between what technology can theoretically do and what it can practically accomplish today. For humankind and our economy, it’s essential to distinguish the capabilities and limitations of these technologies.

“I’m not saying AI agents or assistants don’t work—they do. But you need to be careful about what’s hype versus what delivers value today and what’s still years away from being useful to your business,” he emphasizes.

Illusions of Agentic AI

Artificial Intelligence (AI) has great potential. However, in today’s reality, the intelligence of AI models is often more of an illusion than a fully fleshed way of working. Nate’s Experiment revealed that AI-generated recommendations can sometimes be inaccurate. Another example Andrzej shared was a time he asked AI to recommend the best lawn mower, and it suggested models that don’t exist! This illustrates the gap between what AI can do and how it currently operates.

• AI can produce outputs that sound plausible but are incorrect. (e.g., the claims that the RFK Jr health report, from the United States Secretary of Health and Human Services, cited false studies with real authors and titles).
• The confidence of AI’s responses can mislead us. This is especially true for common answers, which are accurate for many of us. (e.g. It gives a general answer to business questions like cost of acquisition (CAC) or average conversion rate. However, it does not take into account your revenue or total lifetime value, giving you an irrelevant average related to your context).

This serves as a reminder that while AI technology advancements are impressive, they often lack the depth of understanding or the critical checks and balances against misinformation.

Reality Check: What AI Can Actually Do Today

AI has come a long way, and it’s already driving real results. But it’s not magic—and it’s not fully autonomous. The best outcomes occur when companies understand what AI can do now, what still requires human input, and how to effectively balance both.

That’s where solutions like KYP.ai and Cyber Ola come in.

You Need Process Intelligence Before Agentic AI Can Deliver

Tools like KYP.ai automatically capture and analyze real work across your teams and systems, without adding extra steps. The platform then reveals where your teams can save time and effort, helping you speed up processes, free up resources for higher-value work, and deliver better outcomes—faster and more consistently.

Agentic AI can’t improve what it doesn’t understand. First, you need clear, detailed data on where finance or operations teams spend their time and which tasks are repeated most often. That insight enables AI agents to perform the right work at scale and eliminate wasted effort without risk.

Lessons from Cyber Ola’s Success

Cyber Ola provides a successful example of AI applications in business operations. By integrating AI with existing systems, Cyber Ola efficiently handles repetitive tasks, showcasing a practical implementation of AI technology. And, she’s the first one to admit when a task is too complicated or when she lacks the data to answer accurately. In these scenarios, she responds with ‘Ask the real Ola,’ referencing Office Samurai’s Human Resources department.

Many of us are eager to use AI to improve our daily work. However, we are not yet at the point where reality matches the hype.

AI Agents like Cyber Ola are an innovative, ever-evolving way to:

• Provides a seamless interface for employees to access information.
• Reduces workload and frustration for human staff by automating routine tasks and providing quick answers.

Future Insights and Practical Applications

Looking forward, the potential for AI to transform business operations is exponential. As AI continues to evolve, it will enable companies to automate more complex tasks, reduce errors, and unlock new levels of efficiency. By combining AI with real-time process intelligence, companies can see, understand, and change inefficiencies. They can implement automation and agentic AI with data to back each decision as a wise investment. This approach minimizes risks and accelerates ROI by focusing efforts where they matter most. Ultimately, it empowers teams to deliver faster results and higher-quality outcomes, enabling them to connect more deeply with their work and with each other.

Building Effective AI Agents

Building effective AI agents requires a strategic approach. Once you identify the processes that fit well with AI agents, use AI agent builders. Some options include SAP Joule, Google Vertex AI Agent Builder, and Microsoft Power Automate with AI Builder.

Only when you have the right inputs can you have the desired outputs.

Here’s how businesses can develop AI agents that deliver value:

1. Identify key processes that are compatible for AI Agents (use KYP.ai to fast-track your process discovery and automation candidates).

2. Select the most suitable technology based on specific business requirements. Only automate what’s worth it to you: time savings, cost reduction, higher yields, etc. There are many ways to automate, but you need end-to-end details and the most compatible types of automation (IDP, RPA, Agentic AI, Gen AI, etc.).

3. Measure before and after, then improve again. Find ways to measure top-and bottom-line savings. Share lessons learned, results achieved, and key milestones.

Successful implementation involves ongoing testing and refinement to ensure AI agents meet organizational goals.

AI isn’t perfect and still requires human oversight. To maximize the benefits of AI, businesses should continually monitor its impact, upskill staff on how to leverage AI to add value, and offer prescriptive, personalized use cases to encourage greater adoption and value. This balanced approach ensures reliable results, prevents mistakes, and maximizes the value of automation efforts.

To learn more about how KYP.ai or Office Samurai can help identify your Agentic AI potential, please reach out.