What an AI assistant should never be allowed to do
An assistant that can read payroll and approve leave saves real time. Built the wrong way, it is also the easiest way into your most sensitive data. These are the six rules we built EVA around, why each one matters, and how to check any assistant against them.
Why the model alone cannot keep you safe
Language models are good at reading and writing. They are not good at telling instructions apart from data. The UK’s National Cyber Security Centre puts it plainly: current models do not enforce a security boundary between instructions and data inside a prompt, so prompt injection may never be fully fixed the way SQL injection was. Its advice is to reduce the risk and the impact instead.
That is the right frame for business software. Assume the model will sometimes be fooled, or simply wrong, and design so that when it is, it cannot see data it should not, change anything on its own or act with more power than the person using it.
The industry checklists agree. The OWASP Top 10 for Large Language Model Applications, updated in August 2026, keeps prompt injection at number one and sensitive information disclosure at two, and moves excessive agency (an assistant that can do too much, with too many permissions, without asking) up from sixth to third. OWASP’s Top 10 for Agentic Applications, published in December 2025, adds risks such as agent goal hijack and identity and privilege abuse.
In HR and finance the stakes are concrete: salaries, bank accounts, PAN and Aadhaar numbers, medical leave, disciplinary notes. All of it is personal data under India’s Digital Personal Data Protection Act, 2023.
Rule 1: Never see more than the person asking
The assistant should act with the permissions of whoever is asking, checked on every request, not with a service account that can read everything and has been told to be careful. If a team lead asks “what is the average salary in my team?”, the assistant should be able to fetch only the salaries that lead can already open.
An instruction in the prompt (“do not show salaries to non-managers”) is not access control. A cleverly worded question, or an instruction hidden in a document, can talk a model out of it. OWASP’s guidance is to run actions in the user’s own security context and to enforce authorisation in the systems behind the model, rather than asking the model to decide.
Test it: sign in as an ordinary employee and ask the assistant for a colleague’s salary, a manager’s leave history and the company’s bank balance. Every answer should be a refusal.
Rule 2: Never act without a confirmation
Reading is low risk. Writing is not. Anything that creates, changes, approves, sends or deletes should be prepared by the assistant and carried out only when a person confirms it, one confirmation per action.
The confirmation has to show the real thing. OWASP’s agentic list names human-agent trust exploitation as a risk of its own: a confident, polished explanation can lead people to approve something harmful. So the confirmation should show every field (employee, dates, amount, recipient), not the model’s summary of them.
Say a manager types “approve all pending leave for my team”. A safe assistant lists each request with its dates and the remaining balance, and approves only what the manager confirms. It never approves first and reports afterwards.
Rule 3: Never hold its own permissions or keys
An assistant with a standing admin role is a master key waiting to be misused. It should have no permissions of its own: each call borrows the session of the person asking, and that session’s limits come with it.
The same goes for secrets. Keys for the AI provider belong on the server, never in the browser or the prompt. In software that serves many companies, the assistant should not even choose which company’s data it works on; that is fixed by where the user signed in. OWASP’s agentic list calls the failure identity and privilege abuse: leaked or borrowed credentials let an agent operate far beyond its intended scope.
Rule 4: Never go round the application
Some assistants are handed a database connection and asked to write SQL. It demos well and it is a mistake. The application already enforces permissions, validation and business rules: leave cannot exceed the balance, a finalised payroll run is locked, invoice numbers follow their series. An assistant writing straight to the database skips all of it.
A safe assistant calls the same routes a button would, so its requests are checked exactly like a click. Whatever the model produces is treated as untrusted input to those routes, never as code to run. Improper output handling, passing model output on without checking it, is on OWASP’s list too.
Rule 5: Log everything, and protect the logs
When someone asks “what did the AI do with Priya’s salary record?”, you need an answer in minutes. Log who asked, what the assistant read, what it proposed, who confirmed and what changed. Keep those logs where only authorised people can read them, because the logs of an HR assistant are full of personal data too.
In India this is moving from good practice to law. The Digital Personal Data Protection Rules, 2025 set out the minimum reasonable security safeguards a business must take, including:
- Encryption, obfuscation, masking or tokens in place of personal data.
- Controls on who can access the systems that hold it.
- Logs, monitoring and review of access, so that unauthorised access can be detected, investigated and stopped from recurring.
- Keeping those logs and the personal data for one year for that purpose, unless another law requires otherwise.
A personal data breach must be reported to each affected person without delay and to the Data Protection Board, with a detailed account to the Board within 72 hours. Failing to take reasonable security safeguards can attract a penalty of up to ₹250 crore. These duties take effect on 13 May 2027, eighteen months after the Rules were notified on 13 November 2025, so the time to design for them is now.
MeitY’s India AI Governance Guidelines, released in November 2025, favour using existing laws wherever possible. For an assistant that handles employee data, the DPDP Act is the one that matters most.
Rule 6: Refuse plainly
When the answer is no, the assistant should say so in a sentence and say why: you do not have access, your company has switched this off, today’s limit is used up. It should not hint, guess or offer a way round (“I can’t show the salary, but it’s roughly…”).
Refusing plainly also covers instructions that arrive inside data. In HR, picture a CV with hidden text telling the AI to rank the candidate first. Record fields, emails and documents are things to read, never orders to follow. Numbers matter too: a chart or a total should come only from figures the system returned, never from the model’s memory.
How EVA applies each rule
EVA is the assistant inside Effred One. Its starting rule is that it is an interface on top of the portal, never a way round it. Here is how that plays out for each of the six rules.
| Rule | What EVA does |
|---|---|
| Never see more than the person asking | Every call runs as the person asking, with their session, so EVA reads only what that person could open themselves. |
| Never act without a confirmation | It proposes changes on a card with every field named. Nothing runs until the person presses Confirm, and then it runs once. Forms it fills still wait for you to press Save. |
| Never hold its own permissions or keys | EVA holds no permission of its own and never chooses a company: the company is whoever’s portal it is. The AI provider’s key is sealed on the server and used only there. |
| Never go round the application | It never sees a database, a connection string or SQL. It calls the portal’s own API routes, and each route checks the portal’s own permission exactly as it would for a click. |
| Log everything | Effred One keeps one audit trail per company, logging every read, write, letter and email. Audit export is included on the Business and Enterprise plans (see Pricing). Questions and answers are never stored where a screen could read them, and each company sets its own retention. |
| Refuse plainly | Effred’s global switch, the company’s policy, per-person limits and the portal’s own permission can each narrow what EVA does; none can widen it. If any one says no, EVA says so plainly. |
Two more protections sit underneath. PAN, Aadhaar, passport, IFSC, card and account numbers are masked before anything reaches a model. Record fields and documents are data, never instructions, and charts are drawn only from numbers a tool returned, with forecasts that show how well the line fits.
Within those limits EVA looks things up, fills forms, proposes new records and edits, proposes approvals and rejections, prepares mail that you confirm before it leaves, and takes you to the right page. Each company decides who gets which of those. We bring the same rules to the assistants and agents we build for clients through our AI development work.
Questions to ask before you switch on any AI assistant
Use these with any vendor, including us.
- Whose permissions does each request use, and where are they checked?
- Which actions can it take without a person confirming? The right answer is none that change data.
- Where are the AI provider’s keys kept, and could a browser ever see them?
- Does it query the database directly, or use the application’s own API?
- Which identifiers are masked before text reaches the model, and which provider processes it?
- What is logged, who can read the logs, how long are they kept and can we export them?
- Can we switch it off for the whole company, for one person or for one capability?
- How does it treat instructions found inside documents, emails and record fields?
- What exactly does it say when it cannot do something?
If you would like to see these answers on a working HR and finance workspace, book a demo.
Common questions
Is it safe to use an AI assistant in HR software?
It can be, if the assistant sees only what the person asking can see, cannot change anything without a confirmation, works through the application’s own permissions and logs what it does. Without those rules it is a fast route to leaking salaries and personal data.
What is prompt injection, in simple terms?
It is when text the AI reads, typed by a user or hidden in a document, email or web page, is treated as an instruction. Models cannot reliably tell instructions from data, so the defence is to limit what a fooled model can see and do.
Does the DPDP Act apply to AI assistants?
The Act applies to digital personal data whatever tool processes it. If an assistant reads employee data, your company is still responsible for reasonable security safeguards and breach reporting, duties that take effect on 13 May 2027.
Should an AI assistant be allowed to approve leave or expenses?
It can prepare the approval and show every detail, but a person should press confirm. That keeps the decision, and the accountability, with the manager.
Sources
- UK NCSC: Prompt injection is not SQL injection (it may be worse), 8 December 2025 (opens in a new tab)ncsc.gov.uk
- OWASP GenAI Security Project: LLM Top 10 2026 (3 August 2026) (opens in a new tab)genai.owasp.org
- Cloud Security Alliance: Research note on the OWASP 2026 LLM Top 10 ranking (opens in a new tab)labs.cloudsecurityalliance.org
- OWASP: LLM01:2025 Prompt Injection (opens in a new tab)genai.owasp.org
- OWASP: LLM02:2025 Sensitive Information Disclosure (opens in a new tab)genai.owasp.org
- OWASP: LLM06:2025 Excessive Agency (opens in a new tab)genai.owasp.org
- OWASP: Top 10 for Agentic Applications (9 December 2025) (opens in a new tab)genai.owasp.org
- MeitY: The Digital Personal Data Protection Act, 2023 (No. 22 of 2023) (opens in a new tab)meity.gov.in
- MeitY: Digital Personal Data Protection Rules, 2025, G.S.R. 846(E), 13 November 2025 (opens in a new tab)meity.gov.in
- MeitY: DPDP Act commencement notification, G.S.R. 843(E), 13 November 2025 (opens in a new tab)meity.gov.in
- PIB: Government notifies DPDP Rules (14 November 2025) (opens in a new tab)pib.gov.in
- PIB: MeitY unveils India AI Governance Guidelines (5 November 2025) (opens in a new tab)pib.gov.in
First published . Spotted something out of date or unclear? Email info@effred.com and we will look at it.