The pace of software development has never been faster, driven by the emergence of Generative AI tools that promise to accelerate coding and automation. Yet, with speed comes risk, and the fundamental challenge of application security remains: how do we ensure the code we ship is both fast and secure?
We recently had the privilege of hosting Jim Manico on the Scale to Zero podcast. As the founder and lead instructor at Manicode Security, Jim is an application security architect and secure coding educator who trains hundreds of developers on these exact practices. His deep expertise in secure coding, coupled with his advisory roles for multiple security startups, provided us with a comprehensive masterclass on AppSec—especially how it applies to the bleeding edge of AI development.
You can read the complete transcript of the epiosde here >
How Confident Should Developers Be in AI-Generated Code?
Generative AI tools like GitHub Copilot and AWS Code Whisperer are now staples in the developer workflow, offering code snippets and security recommendations trained on vast repositories of open-source code. Jim’s immediate answer to whether this code can be trusted is a firm: don’t trust any code from AI.
The quality of AI-generated code, including its security, hinges entirely on the prompt.
The Rigor of the Prompt
Jim advises developers to be extremely specific about their requirements. Instead of asking for a general script, ask for a script “with rigorous security baked in”. The more detail you provide about security, complexity, and even the desired architecture, the better the initial output will be.
The Necessity of Trust but Verify
Regardless of how well-crafted the prompt is, AI-generated code must not bypass standard quality gates and security checks.
- Integrate into DevSecOps: Any AI-generated code should be treated like code written by a junior developer—it must go through the entire DevSecOps lifecycle.
- Mandatory Scanning: At a minimum, run a static analysis engine (like Semgrep or Bear) and fix any security bugs detected.
- Deep Review for Critical Code: For code that is security-critical, deeper reviews are mandatory.
- Licensing Concerns: Developers must also remember that the licensing issues surrounding AI-generated code are still not settled in society, adding another layer of risk.
Monitoring Code Complexity (Cyclomatic Complexity)
A key metric Jim monitors is Cyclomatic Complexity.
- What it Is: High cyclomatic complexity indicates a codebase that is hard to maintain, test, and understand.
- The Security Correlation: High complexity is a red flag that almost always correlates with a higher chance of security issues and functional bugs.
- The Goal: Jim prefers simpler, “more plain” development with lower cyclomatic complexity because it is easier to maintain and secure. When prompting AI, you can even request the code to have low cyclomatic complexity.
How Can Developers Use AI Tools Without Compromising Business-Critical Information?
The risk of accidentally exposing sensitive or business-critical data by feeding it into a Generative AI tool (a key concern that was recently in the news ) is real.
Jim’s recommendation is straightforward: do not put any kind of sensitive data or business-critical information in the request used to generate code.
- Generic Utility vs. Custom Logic: Use AI for general utility stuff. If you are working on custom business logic that is highly unique or sensitive, use AI far less.
- High-Level Requirements Only: When using AI for sensitive development, keep the requirements generic. Even if you feed in requirements to ChatGPT, you must carefully review the output, security test it, and perform automated analysis depending on its criticality.
The speed and convenience of AI are compelling—it is an amazing tool to speed up development —but privacy and confidentiality must be maintained through strict information control.
What are the Next-Level Security Defenses for AI Data Stores?
AI development introduces new challenges around protecting the vast, often sensitive, datasets used for training and operation. Traditional security controls are necessary, but advanced cryptographic and privacy techniques are essential for next-generation AI systems.
Advanced Data Privacy Techniques
When building AI systems from scratch, Jim points to high-end solutions for protecting data confidentiality.
- Differential Privacy: This is a mathematical framework that quantifies privacy risk involved in data release. The core idea is to add noise to data or query results so that the presence or absence of a single record does not significantly affect the outcome. This approach maintains data utility while preventing individual privacy compromise, helping with compliance standards like GDPR.
- Encrypted Computation: Using advanced cryptography (out of IBM and other labs), developers can perform computation on data even while it remains encrypted, thereby preserving data confidentiality within the AI data store.
Regardless of the advanced methods, strong access control remains a fundamental necessity in AI systems.
How Do We Secure the AI Software Supply Chain?
The AI ecosystem is heavily reliant on bleeding-edge, third-party, and open-source libraries (e.g., LangChain, vector databases). This reliance immediately resurrects the painful problems of software supply chain security.
The Painful Choice: Build vs. Buy
Jim describes the dependency on third-party components in a bleeding-edge industry as an unwinnable situation:
- The Update Burden: If you use too many libraries, you speed up initial development but pay for it with an update burden. A security bug in a big data library can force an update that breaks the entire system, leading to painful unwinding and debugging.
- The Dev Time Burden: If you write too much code yourself, you get longer development times.
Risk Mitigation Strategies
Jim advocates for a risk-averse approach to third-party libraries:
- Judicious Selection: Be really judicious about which libraries you select and vet them carefully before betting your company on them.
- Err on Building: Jim prefers to err on the side of writing his own code and only use a library when it is absolutely necessary.
- Automation is Non-Negotiable: A robust DevOps pipeline must include a Software Composition Analysis (SCA) tool (e.g., Dependabot, Snyk, JFrog) to prevent code merges if any third-party library has a known security problem.
- Stay Behind the Bleeding Edge: Maintain and update components for security and functionality bugs over time, but Jim’s personal strategy is to stay about a month behind the bleeding edge to let the worst initial problems shake out.
The Wrapper Class Engineering Practice
A powerful software engineering practice that also serves as a security control is writing wrapper classes around the use of utility libraries.
- Decoupling: By wrapping a library, if its functionality or security needs to change, you only have to modify the innards of your wrapper, rather than changing your whole code base.
- Swapping out Components: This technique allows you to easily swap out an entire component with a new one (e.g., swapping a complicated UI widget) without breaking dependent code.
What is the Definitive Security Guide for LLM Development?
While the general OWASP Top 10 applies broadly to web applications, Jim highlighted a specific, essential resource for securing AI systems.
- OWASP Top 10 for LLM Applications (v1.0.1): This document, authored by Steve Wilson (“Virtual Steve”), is outstanding and a great starting place for developing secure AI systems.
The top risks covered include::
- Prompt Injection
- Insecure Output Handling
- Training Data Poisoning
- Model Denial of Service
- Supply Chain Vulnerabilities
- Sensitive Information Disclosure
- Insecure Plugin Design
- Excessive Agency
- Over-reliance
- Model Theft
Can Generative AI Build Secure Software Architecture?
Beyond generating code snippets, can AI be used to design secure architecture? Jim tested this live during the podcast by asking ChatGPT to: “Build me a very super secure architecture for an AWS app that uses an S3 bucket to save uploaded files”.
The resulting output provided an excellent starting point, including recommendations for:
- Technology Stack: Front-end (React/Angular/Vue), Backend (Java/Python/Node REST API), Database (AWS RDS), and File Storage (S3).
- Network Security: VPC Isolation, Security Groups, NACLs, and VPN.
- S3 Specifics: Enable server-side encryption in the S3 bucket using AWS KMS and implement strict bucket policies and IAM roles to grant least privilege access.
- Operational Notes: CI/CD and backups.
AI as a Starting Point, Not a Replacement
Jim concluded that AI provides a good starting point to map out and research detailed architectural choices. It does not, however, replace individual experts with experience. The process remains: start with a detailed prompt, get the initial build-out, then bring in specialized architects (e.g., AWS architects) to vet the design and identify missing elements.
What Core Secure Coding Mistakes Do Developers Still Make?
Even with AI and secure architecture, the code itself must be written securely. Jim, whose company focuses on teaching developers to write secure code, highlighted fundamental errors that persist.
Input Validation is Not Security
A common misconception is that input validation (e.g., ensuring a user enters a valid email) secures an application.
- Validation is Hygiene: Input validation is a hygiene layer that restricts data to the least amount of characters and patterns possible.
- Security is Safe Usage: Security happens when you use the data safely. A valid email address can still contain a SQL injection payload.
- Contextual Defense: Defense must be based on how the data is used:
- Databases: Use parameterized queries to protect against SQL injection.
- LDAP: Strictly validate data to exclude dangerous LDAP characters.
- Webpages: Validate, sanitize, or escape the data when using it in a webpage.
Data Storage and Misconfiguration
The most critical mistakes often revolve around data handling and system configuration:
- Not Parameterizing Queries: This remains a primary source of security issues.
- Storing Unnecessary Data: Storing sensitive data, such as bank accounts or PII, that is not strictly needed.
- Not Encrypting Sensitive Data: Failing to encrypt ultra-sensitive data in the database.
- Weak Basics: Misconfigured XML parsers, using very old libraries with serialization problems (e.g., parsing JSON), and failing to classify data.
- Referencing the ASVS: Jim directs developers to the Application Security Verification Standard (ASVS), which contains around 300 requirements detailing what can go wrong.
How Do Security Headers and Frameworks Provide Defense-in-Depth?
To defend against attacks like Cross-Site Scripting (XSS) and replay attacks, application developers must implement defense-in-depth measures, including security headers and using frameworks securely.
Content Security Policy (CSP)
Jim is a strong advocate for implementing a Content Security Policy.
- XSS Defense: CSP is a powerful way to stop Cross-Site Scripting in most modern browsers.
- Nonce-Based Policy: Jim prefers a nonce-based CSP, where each allowed script is explicitly “notched” with a unique, single-use token. Everything else is blocked from running.
Securely Using Modern Frameworks
Frameworks like React and Angular provide security features, but developers often disable them unintentionally or incorrectly.
- Don’t Disable Escaping: Developers must be careful not to disable the framework’s security features, such as using React’s
dangerouslySetInnerHTMLor Angular’sbypassSecurityTrustHtml. - Sanitize When Necessary: If using these functions is unavoidable, you must use an HTML sanitizer (e.g., DOM Purify with React).
- URL Protocol Restriction: If allowing an untrusted URL in an image tag, block dangerous protocols like JavaScript or data and only allow HTTPS.
- Framework Security Knowledge: It is critical to use React securely (which requires specialized knowledge) and Angular securely, integrating CSP into the framework’s structure.
What is the Best Practice for Multi-Factor Authentication (MFA)?
When building authentication capabilities, organizations must consider the risk profile of the data.
- Discourage SMS: The NIST standard (Special Publication 800-63) discourages SMS as the primary factor because it is the least secure form of multi-factor authentication.
- Layered MFA: For sensitive data, SMS can be implemented as a secondary choice, but a more secure option (like an app-based mobile authenticator) should be the first choice.
- Hardware Tokens for Critical Infrastructure: For infrastructure-level security (Level 3 risk), NIST recommends a hardware token like a YubiKey as the main authentication factor.
The key is to use the right kind of multifactor for the level of risk the application can tolerate.
How Can DevSecOps Practices Enable Speed and Security Simultaneously?
Finally, the philosophy underpinning modern development: how do we go fast without compromising security?
- Speed Reduces Exposure Windows: Jim rates the philosophy of prioritizing speed highly, arguing that the ability to push live fast allows teams to reduce the exposure window of bugs. If a post-deployment scanner finds a bug, it can be fixed just as fast.
- The Price of CI/CD is Discipline: Fast deployment is only safe with a disciplined and mature DevOps lifecycle.
- CI is the Medicine for CD: Continuous Integration (CI) is the necessary precursor—the “medicine”—that allows for Continuous Deployment (CD).
- Robust Automated Testing: CI must include a robust automated security and functionality testing framework, including unit tests for sensitive features, code scanners, infrastructure scanners, and Docker security scanning.
- Gating Merges: The pipeline should only stop a developer from merging if they introduce a new bug.
- Varying Review Depth: While fast deployment is suitable for a lot of standard enterprise code, sensitive code (like authentication or credit card repositories) should still require manual review.
For teams looking to establish or mature their pipelines, Jim recommends checking out Defect Dojo, an open-source DevSecOps pipeline tool that integrates well with platforms like GitHub.
Conclusion: The Future is Knowledge-Driven
Our conversation with Jim Manico confirmed that while AI is radically reshaping the speed of development, the principles of application security remain constant: disciplined practices, contextual awareness, and continuous education. For our audience, Jim recommended setting up Google Alerts for security topics and checking out recent secure coding books, specifically mentioning Secure Coding for Software Engineers by James Ma and Secure Python Coding Fundamentals by Michael Murray.
Ultimately, in this era of rapid change, secure development is not about stopping progress; it’s about having the knowledge and discipline to build resilient and trusted software systems.
