Test Runner & CI/CD

Automated quality testing with assertions, reporters, and CI/CD integration for Orka AI.

import { minScore, maxScore, maxLatency, ConsoleReporter, JUnitReporter } from 'orkajs';
 
const report = await orka.test({
  name: 'Regression Tests',
  dataset: [
    { input: 'What is Orka AI?', expectedOutput: 'A TypeScript framework.', knowledge: 'docs' },
  ],
  metrics: ['relevance', 'correctness', 'faithfulness'],
  assertions: [
    minScore('relevance', 0.7),
    maxScore('hallucination', 0.3),
    maxLatency(10000),
  ],
  reporters: [
    new ConsoleReporter(),
    new JUnitReporter('./test-results.xml'),
  ],
});
 
if (report.failed > 0) process.exit(1);

Assertions

minScore(metric, threshold)

Require metric >= threshold

maxScore(metric, threshold)

Require metric <= threshold

maxLatency(ms)

Response time must be under limit

maxTokens(n)

Token usage must be under limit

contains(text)

Output must contain text

notContains(text)

Output must NOT contain text

matchesRegex(pattern)

Output must match regex

customAssertion(name, fn)

Create any custom assertion

Reporters

ConsoleReporter

Formatted console output with pass/fail icons

JsonReporter(path?)

Export full report as JSON

JUnitReporter(path)

JUnit XML format for CI/CD platforms

GitHub Actions

name: AI Quality Tests
on: [push, pull_request]
 
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm install
      - run: npx tsx tests/ai-quality.ts
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      - uses: dorny/test-reporter@v1
        if: always()
        with:
          name: AI Quality Report
          path: test-results.xml
          reporter: java-junit