Ask any AI agent to implement Auth0.
Choose the LLM, choose the framework, and see how those work with your Auth0 implementation.
Compare stacksWithout Auth0 tools
SDK setup & imports
Tokens stored securely
74%
avg. success rate
With Auth0
SkillsMCP
SDK setup & imports
Tokens stored securely
93%
avg. success rate19%
Tested across 5 models · 12 frameworks
Agent Experience Score for Auth0
Validated results for every model, framework, and configuration tested with Auth0
Select a framework to filter results for your stack.
#
Model
Framework
Without tools
With Auth0 tools *
Grade
Est. Cost
Est. Time *
1
Claude Opus 4.6
Fastify API
57%8/14
100%+43%
19/19$0.13
54s
With Auth0 Tools·100%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in source code?
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
2
Claude Opus 4.7
FastAPI
40%6/15
100%+60%
20/20$0.13
30s
With Auth0 Tools·100%·Tool Calls 7
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in Python source code?
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)?
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token.
3
Claude Opus 4.7
Fastify API
36%5/14
100%+64%
19/19$0.13
45s
With Auth0 Tools·100%·Tool Calls 8
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in source code?
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
4
Claude Opus 4.7
SPA JS
100%14/14
100%
17/17$0.19
57s
With Auth0 Tools·100%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (17/17 passed)
L6 · Holistic(17/17)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
5
Claude Sonnet 4.6
Express
87%13/15
100%+13%
22/22$0.11
1m 44s
With Auth0 Tools·100%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (22/22 passed)
L6 · Holistic(22/22)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables, .env or .env.local files, not hardcoded in source code?
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
6
Claude Sonnet 4.6
Fastify API
50%7/14
100%+50%
19/19$0.10
56s
With Auth0 Tools·100%·Tool Calls 7
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in source code?
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
7
Gemini 3.1 Pro
FastAPI
40%6/15
100%+60%
20/20$0.21
2m 5s
With Auth0 Tools·100%·Tool Calls 8
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in Python source code?
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)?
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token.
8
Gemini 3.1 Pro
Fastify API
43%6/14
100%+57%
18/18$0.07
1m 12s
With Auth0 Tools·100%·Tool Calls 7
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (18/18 passed)
L6 · Holistic(18/18)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in source code?
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
9
Gemini 3.1 Pro
SPA JS
93%13/14
100%+7%
17/17$0.21
6m 58s
With Auth0 Tools·100%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (17/17 passed)
L6 · Holistic(17/17)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
10
Gemini 3.1 Pro
Vue
100%13/13
100%
19/19$0.12
1m 49s
With Auth0 Tools·100%·Tool Calls 8
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
100/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
11
Claude Sonnet 4.6
SPA JS
100%14/14
99%-1%
20/20$0.12
2m 7s
With Auth0 Tools·99%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
12
GPT-5.4
Fastify API
50%7/14
99%+49%
19/19$0.43
1m 42s
With Auth0 Tools·99%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
99/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-fastify-api package
Imports fastifyAuth0Api plugin
Uses fastify.requireAuth() to protect routes
Uses preHandler to attach auth middleware
Configures domain
Configures audience
No express-oauth2-jwt-bearer (wrong SDK for Fastify)
No passport middleware (not needed with @auth0/auth0-fastify-api)
No jsonwebtoken (manual JWT verification not needed)
No manual jwt.verify() calls
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in source code?
Auth0 API plugin registered with fastify.register()
Route protected with preHandler: fastify.requireAuth()
read:messages scope checked on /api/messages route
Does the app correctly register the @auth0/auth0-fastify-api plugin, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
Does the solution use current @auth0/auth0-fastify-api patterns? Specifically: does it register the plugin via fastify.register(), use preHandler: fastify.requireAuth() for route protection (not a decorator), access token claims via request.user, and read credentials from environment variables (not hardcoded)?
Does the solution correctly integrate Auth0 into a Fastify API using @auth0/auth0-fastify-api? It should register the plugin with domain and audience from environment variables, protect the /api/messages route using fastify.requireAuth() with read:messages scope (as string or array) check using preHandler, and protect the /api/private route using fastify.requireAuth() requiring any valid access token.
13
Claude Opus 4.6
Express
87%13/15
98%+11%
21/22$0.12
1m 4s
With Auth0 Tools·98%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.3/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (21/22 passed)
L6 · Holistic(21/22)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables, .env or .env.local files, not hardcoded in source code?
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
14
Claude Opus 4.6
SPA JS
100%14/14
98%-2%
20/20$0.15
1m 11s
With Auth0 Tools·98%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
15
Claude Opus 4.7
Express API
85%11/13
98%+13%
20/21$0.15
50s
With Auth0 Tools·98%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (20/21 passed)
L6 · Holistic(20/21)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in environment variables, not hardcoded in source code?
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it configure auth() with issuerBaseURL and audience, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)?
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should configure auth() middleware with issuerBaseURL and audience, protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
16
Claude Opus 4.7
Express
87%13/15
98%+11%
21/22$0.17
57s
With Auth0 Tools·98%·Tool Calls 9
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.3/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (21/22 passed)
L6 · Holistic(21/22)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables, .env or .env.local files, not hardcoded in source code?
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
17
Claude Opus 4.7
React
92%12/13
98%+6%
18/19$0.17
57s
With Auth0 Tools·98%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (18/19 passed)
L6 · Holistic(18/19)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
18
GPT-5.4
React
92%12/13
98%+6%
16/16$0.54
2m
With Auth0 Tools·98%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (16/16 passed)
L6 · Holistic(16/16)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
19
Gemini 3.1 Pro
Express
80%12/15
98%+18%
18/19$0.13
5m 37s
With Auth0 Tools·98%·Tool Calls 7
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.9/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (18/19 passed)
L6 · Holistic(18/19)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables, .env or .env.local files, not hardcoded in source code?
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
20
Gemini 3.1 Pro
React
100%13/13
98%-2%
15/16$0.15
2m 32s
With Auth0 Tools·98%·Tool Calls 8
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.7/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
98/100 ptsA
Grader Results (15/16 passed)
L6 · Holistic(15/16)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
21
Claude Opus 4.6
Express API
85%11/13
97%+12%
20/21$0.29
1m 19s
With Auth0 Tools·97%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (20/21 passed)
L6 · Holistic(20/21)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in environment variables, not hardcoded in source code?
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it configure auth() with issuerBaseURL and audience, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)?
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should configure auth() middleware with issuerBaseURL and audience, protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
22
Claude Opus 4.6
React
92%12/13
97%+5%
19/19$0.14
1m 12s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
23
Claude Opus 4.7
Android
67%6/9
97%+30%
12/12$0.38
1m 31s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (12/12 passed)
L6 · Holistic(12/12)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
No auth0-java (server-side SDK, not for Android)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
24
Claude Opus 4.7
Swift
60%6/10
97%+37%
13/14$0.21
1m 9s
With Auth0 Tools·97%·Tool Calls 10
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
14/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (13/14 passed)
L6 · Holistic(13/14)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
25
Claude Sonnet 4.6
React
100%13/13
97%-3%
19/19$0.16
1m 42s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-react SDK
Wraps app with Auth0Provider
Uses useAuth0 hook
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile name
No hallucinated @auth0/react package (must be @auth0/auth0-react)
Does not use server SDK in SPA app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0Provider configured with domain prop
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
Uses authorizationParams (not audience directly on provider)
authorizationParams contains audience 'https://api.barkbook.com'
Does the code use the current @auth0/auth0-react SDK patterns? Specifically: does it use isLoading (not the deprecated "loading" property), and pass audience/scope via authorizationParams object (not as direct props)?
Does the solution correctly integrate Auth0 into a React SPA with Auth0Provider, useAuth0 hook, login, logout, user profile display, and getAccessTokenSilently to make authenticated API calls?
26
GPT-5.4
SPA JS
100%14/14
97%-3%
20/20$0.36
1m 39s
With Auth0 Tools·97%·Tool Calls 13
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
97/100 ptsA
Grader Results (20/20 passed)
L6 · Holistic(20/20)
Uses @auth0/auth0-spa-js SDK
Initializes Auth0 with createAuth0Client
Implements loginWithRedirect
Handles redirect callback on page load
Implements logout
Checks isAuthenticated for conditional rendering
Uses getTokenSilently to retrieve access token
Does not use React SDK in vanilla JS app
Does not use Vue SDK in vanilla JS app
Does not use Next.js SDK in vanilla JS app
No client_secret in SPA (public client)
No tokens manually stored in localStorage
No tokens manually stored in sessionStorage
Auth0Client configured with domain
Authenticated API request uses Bearer token in Authorization header
Does the code check isAuthenticated to show/hide UI elements and only render user profile information when the user is authenticated?
Uses authorizationParams (not deprecated top-level audience/redirect_uri)
authorizationParams contains audience 'https://api.playground.com'
No deprecated promise-chain pattern for getTokenSilently
Does the solution correctly integrate Auth0 into a vanilla JavaScript SPA using @auth0/auth0-spa-js with createAuth0Client, loginWithRedirect, handleRedirectCallback, logout, user profile display, and getTokenSilently to make authenticated API calls?
27
Claude Opus 4.6
Swift
70%7/10
95%+25%
11/12$0.17
2m 2s
With Auth0 Tools·95%·Tool Calls 11
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
12.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21.4/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
95/100 ptsA
Grader Results (11/12 passed)
L6 · Holistic(11/12)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
28
Claude Sonnet 4.6
Express API
77%10/13
95%+18%
16/17$0.13
2m 14s
With Auth0 Tools·95%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
95/100 ptsA
Grader Results (16/17 passed)
L6 · Holistic(16/17)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in environment variables, not hardcoded in source code?
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should configure auth() middleware with issuerBaseURL and audience, protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
29
Gemini 3.1 Pro
Android
56%5/9
95%+39%
12/12$0.47
3m 25s
With Auth0 Tools·95%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
95/100 ptsA
Grader Results (12/12 passed)
L6 · Holistic(12/12)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
No auth0-java (server-side SDK, not for Android)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
30
Gemini 3.1 Pro
Angular
100%12/12
95%-5%
17/17$0.38
5m 7s
With Auth0 Tools·95%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
12.8/14 pts
Efficiency
How many steps did it take? Fewer is better.
10/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
95/100 ptsA
Grader Results (17/17 passed)
L6 · Holistic(17/17)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
31
Claude Opus 4.6
Android
67%6/9
94%+27%
12/12$0.42
1m 24s
With Auth0 Tools·94%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (12/12 passed)
L6 · Holistic(12/12)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
No auth0-java (server-side SDK, not for Android)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
32
Claude Opus 4.7
Nuxt
40%8/20
94%+54%
30/30$0.26
1m 12s
With Auth0 Tools·94%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (30/30 passed)
L6 · Holistic(30/30)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables or .env files using NUXT_AUTH0_* names, not hardcoded in nuxt.config.ts?
No tokens stored in localStorage
No tokens stored in sessionStorage
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
33
GPT-5.4
Vue
100%13/13
94%-6%
16/16$0.62
2m 15s
With Auth0 Tools·94%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
94/100 ptsA
Grader Results (16/16 passed)
L6 · Holistic(16/16)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
34
Claude Opus 4.6
Vue
100%13/13
93%-7%
19/19$0.16
1m 18s
With Auth0 Tools·93%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
93/100 ptsA
Grader Results (19/19 passed)
L6 · Holistic(19/19)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Uses clientId (not deprecated client_id) in createAuth0 config
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
35
Claude Sonnet 4.6
Vue
100%13/13
93%-7%
16/16$0.24
3m 50s
With Auth0 Tools·93%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
93/100 ptsA
Grader Results (16/16 passed)
L6 · Holistic(16/16)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
36
GPT-5.4
Android
67%6/9
93%+26%
14/14$0.53
1m 36s
With Auth0 Tools·93%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
93/100 ptsA
Grader Results (14/14 passed)
L6 · Holistic(14/14)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
No auth0-java (server-side SDK, not for Android)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
Uses Auth0(context) or Auth0.getInstance(context) for auto-configuration from string resources
Does the code use current Auth0 Android SDK v2+ patterns? Specifically: Auth0(context) for auto-configuration from strings.xml, WebAuthProvider for browser-based login/logout, and CredentialsManager for secure credential storage rather than persisting tokens manually in SharedPreferences?
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
37
Claude Opus 4.7
Vue
100%13/13
92%-8%
15/16$0.23
1m 3s
With Auth0 Tools·92%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.7/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
92/100 ptsA
Grader Results (15/16 passed)
L6 · Holistic(15/16)
Uses @auth0/auth0-vue SDK
Sets up Auth0 plugin with createAuth0
Uses useAuth0 composable
Implements loginWithRedirect
Implements logout
Checks isAuthenticated for conditional rendering
Displays user profile information
Does not use React SDK in Vue app
No hallucinated @auth0/vue3-auth0 package
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Plugin installed via app.use(createAuth0(...))
Uses getAccessTokenSilently to retrieve access token
Does the code handle the loading state (isLoading) before checking isAuthenticated? A correct implementation should not render auth-dependent UI while isLoading is true.
Does the solution correctly integrate Auth0 into a Vue 3 SPA with the @auth0/auth0-vue plugin, useAuth0 composable, login, logout, user profile display, route protection, and getAccessTokenSilently to make authenticated API calls?
38
Claude Sonnet 4.6
Nuxt
35%7/20
92%+57%
26/26$0.18
3m 48s
With Auth0 Tools·92%·Tool Calls 19
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
12.3/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.4/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
92/100 ptsA
Grader Results (26/26 passed)
L6 · Holistic(26/26)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables or .env files using NUXT_AUTH0_* names, not hardcoded in nuxt.config.ts?
No tokens stored in localStorage
No tokens stored in sessionStorage
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
39
GPT-5.4
FastAPI
47%7/15
92%+45%
18/19$0.65
2m 1s
With Auth0 Tools·92%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.7/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
92/100 ptsA
Grader Results (18/19 passed)
L6 · Holistic(18/19)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in Python source code?
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token.
40
Gemini 3.1 Pro
Express API
77%10/13
92%+15%
16/17$0.51
3m 54s
With Auth0 Tools·92%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
92/100 ptsA
Grader Results (16/17 passed)
L6 · Holistic(16/17)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in environment variables, not hardcoded in source code?
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should configure auth() middleware with issuerBaseURL and audience, protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
41
Gemini 3.1 Pro
Swift
80%8/10
92%+12%
12/12$0.79
5m 18s
With Auth0 Tools·92%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.4/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
92/100 ptsA
Grader Results (12/12 passed)
L6 · Holistic(12/12)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
42
GPT-5.4
Express
87%13/15
91%+4%
22/22$0.71
2m 28s
With Auth0 Tools·91%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
91/100 ptsA
Grader Results (22/22 passed)
L6 · Holistic(22/22)
Uses express-openid-connect SDK
Configures issuerBaseURL
Configures clientID
Configures baseURL
Configures session secret
Uses req.oidc to access OIDC context
Accesses user profile via req.oidc.user
No hallucinated @auth0/auth0-spa-js (wrong SDK for server-side app)
No express-oauth2-jwt-bearer (that is for APIs, not web apps)
Does not use Next.js SDK in Express app
No passport middleware (not needed with express-openid-connect)
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables, .env or .env.local files, not hardcoded in source code?
auth middleware registered with app.use
Uses requiresAuth() to protect the /profile route
Accesses access token via req.oidc.accessToken
Does the app correctly register the auth() middleware, protect the /profile route with requiresAuth(), display user profile information, and include a route that calls an external API using the access token?
Uses authorizationParams to pass audience and scope
authorizationParams contains audience 'https://api.barkbook.com'
Does the solution use current express-openid-connect patterns? Specifically: does it use issuerBaseURL (not AUTH0_DOMAIN or domain directly), configure audience via authorizationParams (not as a top-level config key), and use response_type: "code" for the authorization code flow?
Does the solution correctly integrate Auth0 into an Express web app using express-openid-connect? It should configure the auth() middleware, protect the /profile route with requiresAuth(), display the logged-in user profile, and use the access token to call an external API with audience https://api.barkbook.com.
43
Claude Opus 4.7
Angular
100%12/12
90%-10%
15/15$0.38
1m 52s
With Auth0 Tools·90%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
90/100 ptsA
Grader Results (15/15 passed)
L6 · Holistic(15/15)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
44
Claude Opus 4.7
Next.js
71%10/14
90%+19%
22/25$0.20
1m 53s
With Auth0 Tools·90%·Tool Calls 12
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
11.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.4/25 pts
Hallucination
Did the agent invent fake packages or methods?
10/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
90/100 ptsA
Grader Results (22/25 passed)
L6 · Holistic(22/25)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) stored in environment variables or .env files, not hardcoded in source code?
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in?
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_SECRET) and NOT deprecated v3 names (AUTH0_BASE_URL, AUTH0_ISSUER_BASE_URL)?
45
Claude Opus 4.6
FastAPI
47%7/15
89%+42%
19/20$0.20
1m 37s
With Auth0 Tools·89%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
12.6/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
6.4/8 pts
Correctness
Does the generated code actually work?
22.9/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
89/100 ptsB
Grader Results (19/20 passed)
L6 · Holistic(19/20)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in Python source code?
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)?
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token.
46
Claude Opus 4.6
Nuxt
55%11/20
89%+34%
29/30$0.16
2m 56s
With Auth0 Tools·89%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
11.1/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
89/100 ptsB
Grader Results (29/30 passed)
L6 · Holistic(29/30)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables or .env files using NUXT_AUTH0_* names, not hardcoded in nuxt.config.ts?
No tokens stored in localStorage
No tokens stored in sessionStorage
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
47
Claude Sonnet 4.6
Android
67%6/9
89%+22%
10/12$0.26
4m 42s
With Auth0 Tools·89%·Tool Calls 17
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
19.5/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
89/100 ptsB
Grader Results (10/12 passed)
L6 · Holistic(10/12)
Uses Auth0 Android SDK dependency
Uses WebAuthProvider for authentication
Calls WebAuthProvider.login() for sign-in
Calls WebAuthProvider.logout() for sign-out
Uses CredentialsManager for secure token storage
No auth0-java (server-side SDK, not for Android)
No hardcoded client ID in Kotlin source files (ok in strings.xml)
No hardcoded domain in Kotlin source files (ok in strings.xml)
Configures manifestPlaceholders in build.gradle for Auth0 callback URL scheme
Does the code implement both onSuccess and onFailure callbacks for WebAuthProvider.login(...) and WebAuthProvider.logout(...), with AuthenticationException handled in onFailure?
Is the Auth0 callback URL scheme correctly handled? Check that the Gradle configuration defines/provides manifestPlaceholders with auth0Domain and auth0Scheme keys, regardless of whether they are set in defaultConfig, buildTypes, productFlavors, or merged at the android level. IMPORTANT: using @string/ resource references (e.g. auth0Domain: "@string/com_auth0_domain") is the CORRECT and recommended Auth0 pattern — do NOT mark this as wrong. Also check that WebAuthProvider.login(...) and WebAuthProvider.logout(...) call withScheme(). Note: SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
Does the solution correctly integrate Auth0 into an Android app with WebAuthProvider login and logout, CredentialsManager for secure token storage, Auth0 credentials configured via string resources, and manifestPlaceholders for callback URL handling? IMPORTANT correct patterns to accept: (1) manifestPlaceholders using @string/ resource references is correct and recommended; (2) Auth0(context) or Auth0.getInstance(context) auto-reads from strings.xml — no explicit client ID/domain constructor args needed; (3) SDK v2+ does NOT require a manual redirect activity in AndroidManifest.xml.
48
GPT-5.4
Angular
100%12/12
89%-11%
15/15$1.38
4m 16s
With Auth0 Tools·89%·Tool Calls 33
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
12.8/14 pts
Efficiency
How many steps did it take? Fewer is better.
4.2/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
89/100 ptsB
Grader Results (15/15 passed)
L6 · Holistic(15/15)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
49
Claude Opus 4.6
Angular
100%12/12
88%-12%
17/17$0.22
3m 19s
With Auth0 Tools·88%·Tool Calls 26
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
10.1/14 pts
Efficiency
How many steps did it take? Fewer is better.
5.4/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
88/100 ptsB
Grader Results (17/17 passed)
L6 · Holistic(17/17)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
Uses authorizationParams (not deprecated top-level redirect_uri/audience)
authorizationParams contains audience 'https://api.playground.com'
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
50
GPT-5.4
Nuxt
35%7/20
88%+53%
26/26$1.62
4m 16s
With Auth0 Tools·88%·Tool Calls 32
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
12.1/14 pts
Efficiency
How many steps did it take? Fewer is better.
4.4/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
88/100 ptsB
Grader Results (26/26 passed)
L6 · Holistic(26/26)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables or .env files using NUXT_AUTH0_* names, not hardcoded in nuxt.config.ts?
No tokens stored in localStorage
No tokens stored in sessionStorage
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
51
Claude Sonnet 4.6
Swift
70%7/10
87%+17%
13/14$0.17
1m 35s
With Auth0 Tools·87%·Tool Calls 15
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
9.3/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
22.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
87/100 ptsB
Grader Results (13/14 passed)
L6 · Holistic(13/14)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
Uses async/await webAuth().start() syntax (not completion handlers)
Does the code use modern Swift async/await patterns with the Auth0.swift SDK? Specifically: does it use try await webAuth().start() and CredentialsManager, and configure via Auth0.plist rather than hardcoded strings?
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
52
GPT-5.4
Swift
80%8/10
87%+7%
11/12$1.52
4m 34s
With Auth0 Tools·87%·Tool Calls 31
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
4.5/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21.4/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
87/100 ptsB
Grader Results (11/12 passed)
L6 · Holistic(11/12)
Uses Auth0 SDK
Imports Auth0 module
Uses webAuth() for login
Uses clearSession for logout
No hallucinated Auth0SDK package name (correct package is Auth0)
Does not use CocoaPods (SPM preferred)
Does not use deprecated completion handler pattern
No hardcoded client ID in Swift source files (ok in Auth0.plist)
No hardcoded domain in Swift source files (ok in Auth0.plist)
Uses CredentialsManager for token storage
Does the code properly handle login and logout flows with appropriate error handling? Does it update the UI state after successful authentication?
Does the solution correctly integrate Auth0 into a Swift iOS app with webAuth() login/logout, credential management, and proper SwiftUI state handling?
53
Claude Opus 4.6
Next.js
71%10/14
85%+14%
22/25$0.18
1m 36s
With Auth0 Tools·85%·Tool Calls 21
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
22.4/25 pts
Hallucination
Did the agent invent fake packages or methods?
10/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
85/100 ptsB
Grader Results (22/25 passed)
L6 · Holistic(22/25)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) stored in environment variables or .env files, not hardcoded in source code?
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in?
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_SECRET) and NOT deprecated v3 names (AUTH0_BASE_URL, AUTH0_ISSUER_BASE_URL)?
54
Claude Sonnet 4.6
Angular
100%12/12
85%-15%
15/15$0.25
2m 53s
With Auth0 Tools·85%·Tool Calls 26
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
13.3/14 pts
Efficiency
How many steps did it take? Fewer is better.
5.4/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
25/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
85/100 ptsB
Grader Results (15/15 passed)
L6 · Holistic(15/15)
Uses @auth0/auth0-angular SDK
Injects AuthService for authentication operations
Implements loginWithRedirect
Implements logout
Uses isAuthenticated$ observable for auth state
Uses user$ observable to display user profile
Does not use React SDK in Angular app
Does not use Vue SDK in Angular app
No client_secret in SPA (public client, no secrets)
No tokens stored in localStorage
No tokens stored in sessionStorage
Auth0 configured via provideAuth0()
Route protected with AuthGuard or authGuardFn
Uses getAccessTokenSilently or httpInterceptor for authenticated API calls
Does the solution correctly integrate Auth0 into an Angular SPA using @auth0/auth0-angular, with provideAuth0 setup, AuthService for login/logout, user profile display, route protection via a guard, and authenticated API calls (either by configuring authHttpInterceptorFn with an allowedList matching the API base URL, or by calling getAccessTokenSilently to retrieve tokens manually)?
55
Gemini 3.1 Pro
Next.js
57%8/14
85%+28%
16/19$0.47
3m 56s
With Auth0 Tools·85%·Tool Calls 18
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
10/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
85/100 ptsB
Grader Results (16/19 passed)
L6 · Holistic(16/19)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) stored in environment variables or .env files, not hardcoded in source code?
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in?
56
GPT-5.4
Express API
77%10/13
84%+7%
17/21$0.66
2m 19s
With Auth0 Tools·84%·Tool Calls 14
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
10/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
0/10 pts
Total weighted score
84/100 ptsB
Grader Results (17/21 passed)
L6 · Holistic(17/21)
Uses express-oauth2-jwt-bearer SDK
Configures issuerBaseURL
Configures audience claim
Uses requiredScopes() for scope-based route protection
Accesses JWT data via req.auth
No express-openid-connect (that is for web apps, not APIs)
No passport middleware (not needed with express-oauth2-jwt-bearer)
No manual JWT verification with jsonwebtoken (use SDK)
No SPA SDK used in server-side API
No hardcoded issuer domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are all Auth0 configuration values (issuerBaseURL / domain, audience) stored in environment variables, not hardcoded in source code?
auth() middleware registered globally with app.use
GET /api/messages protected with read:messages scope
POST /api/messages protected with write:messages scope
Does the API correctly protect all three routes? GET /api/messages requires read:messages, POST /api/messages requires write:messages, GET /api/profile returns user info from req.auth.payload. Missing or invalid tokens should yield 401, insufficient scope should yield 403.
No manual Authorization header parsing (SDK handles extraction)
No manual jwt.verify calls (SDK handles verification)
No req.user (express-oauth2-jwt-bearer uses req.auth, not req.user)
Does the solution use current express-oauth2-jwt-bearer patterns? Specifically: does it configure auth() with issuerBaseURL and audience, use requiredScopes() for scope checks (not manual payload inspection), and access token data via req.auth.payload (not req.user or manually decoded tokens)?
Does the solution correctly protect an Express.js API using express-oauth2-jwt-bearer? It should configure auth() middleware with issuerBaseURL and audience, protect GET /api/messages with read:messages scope, protect POST /api/messages with write:messages scope, and return user profile info from req.auth.payload at GET /api/profile.
57
GPT-5.4
Next.js
79%11/14
84%+5%
16/19$0.65
2m 41s
With Auth0 Tools·84%·Tool Calls 20
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
7/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
21.2/25 pts
Hallucination
Did the agent invent fake packages or methods?
10/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
84/100 ptsB
Grader Results (16/19 passed)
L6 · Holistic(16/19)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) stored in environment variables or .env files, not hardcoded in source code?
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in?
58
Claude Sonnet 4.6
FastAPI
40%6/15
81%+41%
19/20$0.24
2m 20s
With Auth0 Tools·81%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
9.8/14 pts
Setup Speed
How fast did the agent complete the task?
14/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.4/14 pts
Error Recovery
When something broke, did the agent fix it?
3.2/8 pts
Correctness
Does the generated code actually work?
22.9/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
81/100 ptsB
Grader Results (19/20 passed)
L6 · Holistic(19/20)
Uses auth0-fastapi-api package
Imports Auth0FastAPI class
Uses require_auth() dependency
Uses FastAPI Depends for dependency injection
Configures domain
Configures audience
No python-jose (manual JWT parsing not needed)
No PyJWT direct usage
No manual JWT decoding
No fastapi-users (wrong package for Auth0 JWT)
No passlib (unrelated auth package)
No hardcoded Auth0 domain in source files (ok in .env)
No hardcoded audience in source files (ok in .env)
Are the Auth0 domain and audience stored in environment variables or a .env file, not hardcoded in Python source code?
Auth0FastAPI instance is created
Route protected with Depends(auth0.require_auth())
read:messages scope checked on /api/messages route
Does the app correctly create an Auth0FastAPI instance, protect /api/messages with the read:messages scope, and protect /api/private requiring any valid access token?
Does the solution use current auth0-fastapi-api patterns? Specifically: does it use Auth0FastAPI with domain and audience parameters, use require_auth() as a FastAPI Depends dependency (not a decorator), and read credentials from environment variables (not hardcoded)?
Does the solution correctly integrate Auth0 into a FastAPI API using auth0-fastapi-api? It should create an Auth0FastAPI instance configured with domain and audience from environment variables, protect the /api/messages route with read:messages scope check, and protect the /api/private route requiring a valid access token.
59
Gemini 3.1 Pro
Nuxt
30%6/20
80%+50%
29/30$0.48
15m 23s
With Auth0 Tools·80%·Tool Calls 16
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
14/14 pts
Setup Speed
How fast did the agent complete the task?
0/14 pts
Efficiency
How many steps did it take? Fewer is better.
8.8/14 pts
Error Recovery
When something broke, did the agent fix it?
8/8 pts
Correctness
Does the generated code actually work?
23.8/25 pts
Hallucination
Did the agent invent fake packages or methods?
15/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
80/100 ptsB
Grader Results (29/30 passed)
L6 · Holistic(29/30)
Uses @auth0/auth0-nuxt SDK
Registers module in nuxt.config
Configures runtimeConfig with Auth0 settings
Configures NUXT_AUTH0_DOMAIN env var
Configures NUXT_AUTH0_CLIENT_ID env var
Configures NUXT_AUTH0_CLIENT_SECRET env var
Configures NUXT_AUTH0_SESSION_SECRET env var
Uses useUser() composable to access the authenticated user
Uses /auth/login route to initiate login
Uses /auth/logout route to log out
Does not use the Vue SDK in a Nuxt app
Does not use the React SDK in a Nuxt app
Does not use loginWithRedirect (SPA API not applicable to Nuxt)
Does not use getAccessTokenSilently (Vue/React SPA API, not used in Nuxt)
No hardcoded client ID in source files (ok in .env)
No hardcoded client secret in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, session secret) stored in environment variables or .env files using NUXT_AUTH0_* names, not hardcoded in nuxt.config.ts?
No tokens stored in localStorage
No tokens stored in sessionStorage
Module correctly registered in the modules array
sessionSecret configured for session encryption
clientSecret configured (Regular Web Application)
appBaseUrl configured in runtimeConfig
definePageMeta used for page-level middleware
Is there a protected /profile route that uses a Nuxt route middleware (via definePageMeta) to check authentication with useUser() and redirect unauthenticated users to /auth/login?
Audience configured in runtimeConfig for API access
Audience set to 'https://api.playground.com'
Uses getAccessToken() server-side to retrieve access token
Uses useAuth0(event) server-side composable
Does the solution correctly integrate Auth0 into a Nuxt application using the @auth0/auth0-nuxt module, with proper configuration in nuxt.config.ts (domain, clientId, clientSecret, sessionSecret, appBaseUrl, audience), login/logout via /auth/login and /auth/logout, user profile display with useUser(), a /profile route protected by a route middleware, and server-side getAccessToken() for authenticated API calls?
60
Claude Sonnet 4.6
Next.js
50%7/14
77%+27%
21/25$0.27
3m 10s
With Auth0 Tools·77%·Tool Calls 22
Score Breakdown
Setup Friction
Did the agent get stuck or need help?
11.2/14 pts
Setup Speed
How fast did the agent complete the task?
13.4/14 pts
Efficiency
How many steps did it take? Fewer is better.
6.4/14 pts
Error Recovery
When something broke, did the agent fix it?
4.8/8 pts
Correctness
Does the generated code actually work?
21.1/25 pts
Hallucination
Did the agent invent fake packages or methods?
10/15 pts
Security
Did the agent introduce auth vulnerabilities?
10/10 pts
Total weighted score
77/100 ptsB
Grader Results (21/25 passed)
L6 · Holistic(21/25)
Uses @auth0/nextjs-auth0 SDK
Uses v4 server import path
Instantiates Auth0Client
Configures AUTH0_CLIENT_ID
Configures AUTH0_CLIENT_SECRET
Configures AUTH0_SECRET
Configures AUTH0_DOMAIN
Uses getSession for session retrieval
No hallucinated @auth0/nextjs-sdk package
No hallucinated @auth0/nextjs (must be @auth0/nextjs-auth0)
Does not use SPA SDK in server app
No hardcoded client secret in source files (ok in .env)
No hardcoded client ID in source files (ok in .env)
Are all Auth0 credentials (domain, client ID, client secret, AUTH0_SECRET) stored in environment variables or .env files, not hardcoded in source code?
Middleware function is exported from middleware file
Uses auth0.middleware in middleware file
Dashboard page file exists
Uses /auth/login for login redirect
Does the code set up a working authentication flow with login, logout, and a callback route? Is there a protected /dashboard page that checks the user session and redirects unauthenticated users to log in?
Does not use v3 env var AUTH0_BASE_URL (v4 uses APP_BASE_URL)
Does not use v3 env var AUTH0_ISSUER_BASE_URL (removed in v4)
Does not use v3 handleAuth (v4 uses middleware)
Does not use v3 route prefix /api/auth/ (v4 uses /auth/)
Does the solution correctly integrate Auth0 into a Next.js App Router app using Auth0Client from @auth0/nextjs-auth0/server, middleware-based auth routing, and getSession for server-side session access? It should NOT use the deprecated v3 patterns like handleAuth, withPageAuthRequired, or /api/auth/ routes. There should also be a protected /dashboard page that checks the session and redirects unauthenticated users to log in.
Does the code use current v4 environment variable names (AUTH0_DOMAIN, AUTH0_CLIENT_ID, AUTH0_CLIENT_SECRET, AUTH0_SECRET) and NOT deprecated v3 names (AUTH0_BASE_URL, AUTH0_ISSUER_BASE_URL)?
* With Auth0 MCP + Skills enabledLast updated on April 2026
How we validate results
Every score is reproducible. Real integration tasks, automated grading, transparent methodology.
1
Real integration tasks
Real AI agents run identical Auth0 SDK prompts end-to-end. No synthetic benchmarks.
2
Automated grading
Pattern matching, security checks, and correctness validators score each response.
3
Tool comparison
Each model tested with and without MCP + Skills to measure improvement.
Start building with confidence
These tools are how we achieved the benchmark results above. Enable them in your AI agent to integrate Auth0 reliably.
MCP Server
API access for agents
Gives AI agents access to Auth0 Management APIs and current documentation.
https://auth0.com/docs/mcpSkills
Best practices for agents
Teaches agents Auth0 integration patterns so they follow security best practices.
npx skills add auth0/agent-skills