I let an AI interview me about how I use AI. It found the one thing I do badly.
Why I bothered
I spend most of my working day with AI. I build software around it, run content and lead pipelines on it, and use it as an operator in my terminal. So when someone asks me "how good are you with AI?", my honest answer until recently was: no idea. I had opinions, not a measurement.
That's the gap AISA tries to close. Before writing about it, I looked at how assessment tools are reviewed properly: the psychometric standards (SIOP's Principles and its 2023 guidance on AI-based assessments, the EFPA test review model), buyer's guides for pre-employment testing, and how G2 and TrustRadius structure software reviews. This review borrows from all three: what it's like to use, whether it measures something real, and whether it's worth your (or your company's) time.
1. What AISA measures
Compared with web-based multiple-choice surveys or simple quizzes, AISA is in a league of its own in the depth of information it collects and the thoroughness of the assessment.
AISA measures AI fluency: not whether you know what a transformer is, but how you actually work with AI. Scores come from 11 criteria in 5 weighted dimensions:
- Workflow & Application (25%): Integration into real work, task decomposition, domain use
- Prompting & Communication (23%): Prompt design, iterative dialogue, context management
- Critical Thinking (22%): Evaluating output, knowing the limits
- Technical Understanding (20%): How models work, the tool landscape
- Safety & Responsibility (10%): Oversight, risk, accountability
The format is the point. There's no quiz. You talk to an AI interviewer called Aisa for 20 to 40 minutes, and a separate AI system scores the transcript against a published rubric. AISA's own line sums it up: "A quiz tests recall; a conversation tests what you do."
That puts it in a different category from the two things most people use today:
- Self-assessment surveys ("rate your AI skills 1-5"). AISA's own data says people overestimate their score by about 19 points on average.
- Short knowledge quizzes (8 to 20 multiple-choice questions). Fast, but they test vocabulary, not practice.
It's built for individuals who want an honest baseline, and for companies that want to know where their teams actually stand.
2. The experience
Getting started is quick. Two onboarding questions (your work context, the skill you most want to develop), then a nice touch: you place a bet on your own score on a slider over the real population distribution. It sets up the reveal at the end and quietly makes the point that self-perception is unreliable. Sign-up is Google or an email magic link, no password, and it keeps your progress.
The conversation is the product, and it's good. This is what stayed with me most: it felt like a real interview. Not a chatbot running through a list. Aisa picks up a specific detail from your answer and pulls on it. I mentioned a weighting parameter in one of my systems and got asked how I landed on those exact numbers. I described a fallback chain and got asked what happens when the primary model disappears overnight. Vague answers get follow-ups; interesting ones get "walk me through a specific time that happened."
A few design details worth noting:
- A "glass" meter fills as the report gets enough evidence. You can stop at the minimum or keep going for a deeper report. There's no clock pressure.
- It throws in challenges. In one, Aisa showed me a deliberately bad AI-generated weekly business summary, tailored to a business I'd mentioned, and asked what was wrong with it and what the real risk was. That's a better test of critical evaluation than any multiple-choice question.
- You can take it in practically any language, and 14 languages (Turkish included) support voice dictation.
- Replies take a while, often 15 to 30 seconds. For a reflective interview that's fine; a casual user might find it slow.
My session ran about 45 minutes because I kept going. You don't need that long; 20 minutes is enough for a report.
The reveal is staged like a small ceremony: your predicted score, then the real one, your persona, the certificate, the leaderboard. It's fun. It's also where the upsells and "challenge a friend" prompts show up, and they show up often. More on that below.
3. How AISA measures it
Does it measure something real? This is the question that matters, and the one most reviews skip. Using the SIOP lens (does it measure a job-relevant construct, is it consistent, is it fair, is it documented):
What AISA does well
- Separation of roles. The AI that interviews you is not the AI that scores you, and a stronger model runs a final calibration pass over the full transcript. Telling Aisa how you'd like to be scored doesn't move the score.
- An evidence hierarchy. Demonstrated practice can score up to 10 per criterion; describing a practice caps at 5; knowing the vocabulary caps at 4. That's exactly the right instinct for a conversational format, where it's easy to talk a good game.
- Traceability. This is the standout. Every criterion in the report opens into the scorer's reasoning, which rubric anchor it matched, how the score moved during the session, and verbatim quotes from your own answers with turn numbers. You can disagree with a score, but you can always see why you got it. Most commercial assessments give you a number and a paragraph.
- Honesty about limits. The FAQ has a section called "What does AISA not measure?" It says the assessment doesn't measure general intelligence, domain expertise or leadership, and samples one day of your behaviour rather than months. It also says plainly that it is not proctored and that it can't claim to be ungameable. I wish more vendors wrote like this.
Where the evidence base is heading
AISA publishes its own quality framework, scored against the AERA/APA/NCME testing standards, and it's open about what's next: its methodology page lists test-retest reliability and longitudinal outcome data as ongoing work. If you're evaluating it for your organisation, ask for the most recent numbers; the fact that the framework is public makes that conversation easy.
4. The report: it told me something I didn't know
My result: 94/100, "The Architect" ("builds highly complex integrated systems using AI"), in the 99th percentile of 1,890 people assessed. The population average was 46.
Flattering, sure. But the headline number isn't why the report was useful. The dimension breakdown was:
- Workflow & Application 94, Technical Understanding 93, Safety 90
- Critical Thinking 85, Prompting & Communication 83
And inside Prompting, one criterion stood out: Iterative Dialogue at 57%. The recommendation was blunt and specific:
"Give specific targeted corrections rather than repeating the whole prompt: pinpoint what was wrong and why, then ask for only that part to change."
That's exactly right, and I'd never have said it about myself. I build elaborate systems around the model, and then, when a single output is off, I re-prompt from scratch instead of telling it precisely what was wrong. The report found the one habit my architecture was compensating for. I went in with no idea of my level; I came out with a number, a map of where it comes from, and one concrete thing to change. That's the whole value proposition, and it delivered.
The free package includes the full report (with the complete transcript and PDF download), a verifiable certificate with a public URL valid for 12 months, one-click LinkedIn add, and an email-signature badge.
5. For companies
The team offering rolls individual results up to team, function and organisation level, adds industry and role benchmarks, and supports pre/post measurement around a training programme, which I think is the strongest use case. Most L&D teams roll out AI training with no baseline and no way to show it worked; this gives you both, with evidence you can inspect.
Pricing is simple: $10 per assessment credit, same rate at any team size, no subscription. The site lists EU data residency, GDPR compliance and no training on your data.
Questions I'd ask as a buyer:
- How will we use it: development, training impact, or hiring? For each use, ask AISA for the validation evidence behind it and, as with any assessment, combine it with your other selection and development methods.
- What are the data processing terms for our jurisdiction, where is data stored, and which sub-processors are involved?
- What integrations do we need (SSO, HRIS, LMS)? These aren't described on the site yet.
6. Pros, cons, and my scores
Pros
- A genuinely good interviewer: adaptive, specific, hard to bluff with vague answers
- Evidence-backed report with verbatim quotes and visible scoring reasoning
- The free tier is the real product, not a teaser: report, transcript, certificate
- Unusually candid about its own limitations
- Clear, flat pricing for teams
Cons
- Small rough edges: the coaching upsell showed a beginner-level sample lesson ("prompt chaining") to someone rated Expert; buttons lack accessibility labels; the public leaderboard is opt-out rather than opt-in
- Replies can be slow
My scores
- Ease of use: 4.5 / 5
- Assessment experience: 5 / 5
- Report depth & usefulness: 5 / 5
- Measurement transparency: 4.5 / 5
- Value for money (individuals): 5 / 5 (it's free)
- Readiness for enterprise use: 3.5 / 5
- Overall: 4.5 / 5
7. Who should (and shouldn't) use it
Take it if you:
- use AI daily and want an honest, evidence-based picture of your strengths and blind spots
- lead a team and want a baseline before investing in AI training, then a re-test after
- want a credential that shows how you work with AI, not just that you finished a course
Think twice if you:
- are not comfortable talking about your work in some depth; the report can only score what you show it
- expect a quick quiz; this takes 20 minutes minimum and rewards going deeper
Verdict
AISA does the hard thing well: it turns a conversation into a score you can audit, and it's honest about what that score can and can't tell you. I'll be following the reliability and outcome data it's building. As a mirror for your own practice, it's the best I've tried. It took me about 45 minutes to learn something about my own AI habits that years of daily use hadn't shown me.
It's free: try it at aisa.to, and make your prediction before you start. The gap between your prediction and your score is part of the lesson.