Picture an employee typing a question into the company’s internal AI assistant. “Who’s on a PIP right now? What’s their salary? Who’s getting paid the most?” If that assistant was connected to the company wiki through RAG and nobody thought about permissions, it might answer every one of those questions.
That scenario came up in our recent webinar, Cybersecurity in the Era of AI, where I sat down with Ishmael Samuel, Co-founder and CTO of Paragon, and Tony Camp, Head of Product at Manifest Cyber. Paragon builds integration infrastructure that connects software products and AI agents to tools like Slack, Salesforce and Jira. Manifest Cyber helps companies find the vulnerabilities hiding in the software they build and buy. Between them, they see AI agent security from both ends of the pipeline.
I jokingly said at the start of the session that we could have this conversation once a week and it would still be relevant, but I stand by that comment. Here are some of the main ideas from the webinar that stuck with me, with a few quotes that are too good to paraphrase.
Key insights
- AI is accelerating existing security risks, making vulnerabilities easier to discover and exploit at scale.
- AI agents need clear access boundaries based on acceptable risks, with human oversight at critical moments.
- Security fundamentals matter more than ever, especially as AI increases code production and the pressure on existing controls.
- Engineering partners are part of your security posture, from the AI tools they use to how they access and protect your systems.
Less has changed than you think, and that’s the problem
I opened by asking what has fundamentally changed since AI became part of everyday engineering workflows. Tony’s answer was well-put. “Sometimes I feel like everything has changed. But at the same time, I think less has changed than what a lot of people think. The risks have been there forever.”
What changed is how fast those existing risks get found and exploited. Finding a vulnerability used to require deep technical skill. Today, attackers can point a model at a codebase and get results cheaply. In Tony’s words, “these models are finding old exploits, and they’re taking advantage of them at a pace we’ve never seen.”
Ishmael framed the biggest AI security risks in terms of what AI exposes inside a company. Teams now produce code faster than humans can review it and faster than systems can validate it. “It doesn’t change the underlying security posture. It just reveals the underlying weaknesses.” His answer was to double down on the fundamentals, meaning identity, least privilege, isolation, logging and observability, and to design systems that hold up under worst-case scenarios.
I agree with both of them. The risks that have existed for decades are mutating, and speed is what makes them dangerous now.
Everyone ships code now
The second big change is who is writing software. “You now have non-engineers, non-technical people shipping code into production,” Ishmael said. Internal tools, and sometimes customer-facing applications, are being built by people who have never sat through a security review, and the attack surface has grown with them.
I see the operations side of this all the time. Employees connecting AI tools to each other often have no idea what information they are moving between systems, or what they might be exposing along the way. These tools are so user friendly that anyone can adopt them, which is exactly why companies need guidelines that keep people from running wild with them.
Tony described the developer version of the same problem with uncomfortable accuracy. “You’re letting an agent decide what’s getting pulled in. If it works, most of the time you’re like, cool. If it passes your CI checks and your tests, you’re like okay, awesome... and then you deploy it. And you might not understand everything that got pulled in.”
He admitted engineers rarely understood every dependency back in the early npm days either. The difference is that the decision now gets made by an agent, at volume, sometimes by people who don’t know what a dependency is.
AI agent security starts with the worst-case scenario
When I asked about agents that interact with other systems, Ishmael broke the risk into two questions. What can the agent say, meaning what information can it expose? And what can the agent do, meaning what can it create, update, modify or delete?
His working assumption is blunt. “You have to assume that the AI is going to be wrong at least once, and it’s going to be very, very wrong at least once.”
So I asked them both where they draw the line on access. Tony described himself as firmly in the pro-access camp, within limits. “I wouldn’t be comfortable with maybe giving it my credit card number and letting it go ham or placing orders on my behalf.”
Ishmael’s rule was perhaps the most practical takeaway of the session. “I work my way backwards from the worst-case scenario, and if this is an acceptable risk, then I’ll give agents as much access. With those limits in place, Ishmael runs multiple AI agents on the same project, setting clear boundaries for their autonomy and keeping human oversight at critical stages.
Protecting sensitive data in RAG systems
This is where the PIP example came from. Ishmael described a Notion workspace where leadership keeps documents with salaries and performance plans. If that content gets pulled in via RAG and an agent is exposed to it, any employee might be able to ask who’s on a PIP, who’s getting paid the most or who’s up for promotion.
The lesson is that RAG security depends on the permissions agents inherit and on the people behind the agents, in addition to the model vendor.
This word of warning was particularly relevant to what we’ve worked on at Devlane. When we built Lainey, the internal AI assistant our organization uses every day. Access runs only through each employee’s Google profile, and the assistant respects each person’s individual permissions inside every connected tool. Two people can use the same assistant connected to the same tools and still only see what they were already authorized to see. We’ve built similar controls for a number of clients who asked for the same thing.
AI model governance starts with lineage
Tony highlighted an important challenge in AI model governance, particularly when it comes to understanding how models behave. With a traditional software library, developers can inspect the code, but much of a model’s behavior is encoded in weights that can’t be interpreted in the same way.
That makes lineage the starting point for AI model governance. Tony boils it down to who trained the model, what it was trained on and what it was built on top of. Research has shown that a model’s biases persist even when it runs on your own infrastructure and can bleed into the code it generates. Fine-tuning starts from a base model, and nobody tells you which behaviors survived.
His conclusion mirrors a rule every engineer already follows. You shouldn’t pull in a library without knowing its dependencies and who maintains it, and the same discipline belongs on every model you use to write code.
Guardrails haven’t changed, they’ve gotten more expensive
I asked Ishmael about a challenge that can come up with engineers who are just starting out, especially when working with unfamiliar codebases. He pointed out that AI can be particularly useful here, helping developers understand how a system fits together without having to rely on a senior engineer for every question.
The rest of his answer read like a (useful) list of security best practices that predate AI. Linting, security scanning, static and dynamic analysis, unit, end-to-end, smoke and integration tests, dependency validation and build checks. “They’ve only gotten more important.”
Tony also highlighted the infrastructure costs associated with AI-assisted development. As agents generate more pull requests, CI pipelines run more frequently, increasing compute costs and making them an important consideration when scaling AI-driven workflows.
Preparing for security incidents in AI-driven development
One idea ran through every answer Tony gave. “You’re going to get hit. It’s going to happen, especially with everyone building, it’s just inevitable.”
It matches what I see with our clients, especially those with a CISO. Their involvement has become much more day to day, because speed opens channels for vulnerabilities that used to be locked down.
That inevitability should change where teams invest. Do you know everything an incident touched? Do you have a list of the products pulling in a given dependency? Can you remediate quickly? Tony’s advice for product leaders was to be ambitious and move fast, and before shipping a feature, to write down how you would find and fix it on the day something goes wrong. “Sharpen your recovery playbook. If you can’t do those things, you’re not ready to ship yet.”
Ishmael’s closing advice for CTOs came from the governance side. Inventory every system, whether it’s a model, a SaaS provider or an MCP server, give each one central access and revocable permissions through SSO or MFA, include all of it in regular audits, and confirm that the worst case you’ve credentialed is one you can live with.
A vendor security assessment for engineering partners
This is perhaps the topic I care about most, so I asked what companies must prioritize when bringing in an external technology partner. For Tony, much of it comes down to controlling access and being deliberate about how much of your environment you share with a partner. He also raised some important questions about AI tooling. “Which AI tools does your partner and your engineering partners use? Where does all your code go, and what do they have access to?”
I’d add one more. Ask whether the partner runs a formal information security management system, and how it shows up in their day-to-day work. For us, security is part of how we work, and that’s why we hold an ISO 27001 certification, have an in-house CISO and follow security processes that cover every engineer on every account. We also adapt to the requirements of each industry we work in. For our health tech clients, where HIPAA requirements apply, a Business Associate Agreement is included in our master service agreement. We also align our practices with the security and compliance requirements of our clients in fintech, retail and other industries. If you’re evaluating vendors, don’t be shy about asking these questions. They can tell you a lot about how seriously a partner takes security.
Ishmael shared how Paragon approaches security when working with us. “Just to promote Devlane a bit, this is something that we love about working with Devlane. We’ve been with them for several years now... And we’ve been able to have all of our engineers and engineering managers through Devlane go through our security training... install MDMs on the laptops. They get MFA, SSO, all the things... and it’s been phenomenal.”
When it comes to security, Paragon applies the same standards to Devlane engineers as its internal team, including security training, device management and access controls. Tony added that processes and audit logs only go so far. “It all starts with a level of trust.”
Putting it all together, these are the questions I’d ask when evaluating an engineering partner’s security practices.
AI code review and the pull request problem
AI makes it easy to produce more pull requests, so I asked how they measure whether AI is paying off. Tony admitted nobody has a great answer yet. Lines of code and PR counts were already imperfect measures of engineering impact, and AI makes those numbers look spectacular, but “those PRs don’t mean much if it’s not solving a customer problem.” The better signals come later, in customer impact and new revenue.
All that volume also has to be reviewed. Tony described teams overwhelmed by PRs who “gloss over it and smash that merge button a little bit quicker.”
Ishmael went further. “I’m gonna take a provocative stance on this, in that I don’t think engineers are gonna be reviewing PRs one or two years from now.” His team still reviews every PR today, but he runs five tiers of AI code review on each one, from security assessments to QA agents that build a Docker image, hit the API endpoints and read the server logs. “The fidelity of the reports is better than anyone in my company, including myself, can do. So I’m very confident that within the next few years, code reviewing is going to be akin to churning your own butter.”
All I could say was, “That is provocative.” I’ll keep an impartial stance on that one.
Conclusion
After an hour with Ishmael and Tony, what stayed with me is how much of AI security comes back to the basics. AI is accelerating risks that have existed for decades, and the teams handling it well are strengthening their fundamentals, setting clear boundaries for AI agents and holding external partners to the same security standards as their own teams.
These are conversations we’re having with our clients at Devlane, too, especially as AI becomes part of more products and engineering workflows. Security is an important part of how we approach every engagement, backed by our ISO 27001 certification, internal security processes and the experience we’ve built working with companies across different industries.
Our senior engineers integrate into existing teams and work within each client’s security policies, tools and access controls. We also design and build custom software, including AI agents and RAG systems, with permissions, data protection and security considerations built into the development process from the start.
If you’re planning a new software product, exploring how to integrate AI into your business or looking to expand your engineering team with experienced developers, we’d love to hear what you’re working on. Let’s talk about how we can support your next project.
.webp)
Frequently Asked Questions
What are the biggest AI security risks for engineering teams?
According to Ishmael Samuel and Tony Camp, AI mostly accelerates existing risks. Attackers can find and exploit old vulnerabilities faster, teams ship more code and dependencies than they can review, and non-engineers now push code to production. The fundamentals of identity, least privilege, isolation and observability matter more than before.
How much access should AI agents have?
Ishmael Samuel recommends working backwards from the worst-case scenario. If the worst outcome of a given level of access is acceptable, give agents that access and no more, and keep a human in the loop at critical moments.
What should a vendor security assessment for an engineering partner include?
Start with how much access the partner needs and how much of your environment you’ll share with them. Ask which AI tools they use and where your code goes, confirm whether they hold certifications like ISO 27001, and check that they can meet your industry’s data protection requirements, such as HIPAA in health tech.
Is self-hosting an LLM a good way to protect data privacy?
In most cases, no. Self-hosting large models is expensive and struggles with concurrent workloads, while major cloud providers offer tenant isolation and sign BAAs at a fraction of the cost.
Other Blog Posts
Forward Deployed Engineer: What the Role Is and Why AI Is Driving Demand
.webp)
Cybersecurity in the era of AI: what engineering leaders told us about AI agent security
.webp)
What agentic AI means, and how to introduce it into your software development projects
