What can a confidence score tell you?
A V2 confidence score describes the evidence behind one inference. It does not measure the accuracy of the whole service or establish a person's gender identity. Read confidence together with confidence_kind, source and sample_count.
This page explains the current V2 response contract and a proposed evaluation protocol. Contract tests verify response behavior; they are not an independently labelled accuracy study. The page review date is not a benchmark date.
A separate database profile, observed on 26 September 2026, reports collection metadata counts and numeric consistency in 5,000 randomly sampled records per collection. Those checks do not compare predictions with independent reference labels, so they do not supply an accuracy percentage.
Two confidence kinds, two meanings
Illustrative arithmetic only: if a selected row stores male = 90, female = 10 and total = 100, its confidence is 0.9. These invented counts are not a GenderAPI benchmark or a claim that 90% of people with that name identify as male.
Stored counts are not published here as independent, representative observations. An AI score is not calibrated against those counts. Do not compare the two kinds as interchangeable probabilities or use a universal confidence cutoff without an evaluation for your use case.
| Response field | Dataset result | AI result |
|---|---|---|
source | dataset | ai |
confidence_kind | observed_frequency when a gender is returned | model_reported when a gender is returned |
confidence | max(stored male count, stored female count) / stored total when a gender is returned; null for an exact count tie | A model-reported score greater than 0.5 and at most 1 when a gender is returned; otherwise null |
sample_count | The total stored in the selected record; not an evaluation sample size | null; no dataset sample count is supplied |
match | The selected lookup candidate, matching method, and country/global scope | method is model_inference; name, scope and country are null |
Unknown, ambiguous and failed requests
A dataset result does not have a universal low-confidence rejection threshold. The AI response validator requires a non-null score to be greater than 0.5; that is a contract constraint, not proof of predictive quality. Preserve null rather than converting it to zero confidence or a binary guess.
| Outcome | V2 behavior | Evaluation treatment |
|---|---|---|
| Identified | result_status is identified and gender is male or female | Score against the reference label and count as answered |
| Unknown | gender and confidence are null; result_status is unknown | Keep in the eligible sample and count separately from a wrong prediction |
| Ambiguous dataset row | Equal stored male and female counts produce reason: ambiguous in dataset-only mode; fallback can replace that outcome with an AI result | Track the final outcome and the chosen AI mode; do not double-count ambiguity as another outcome |
| Insufficient AI evidence | An AI unknown uses source: ai and reason: insufficient_evidence | Count as unknown; a valid JSON response does not establish correctness |
| Request or item error | HTTP errors and batch item errors are operational outcomes, not gender labels | Report failures separately; publish their count and denominator instead of silently dropping them |
Measure the configuration you actually use
- Separate name, email and username inputs. An email or username workflow includes candidate extraction, so a name-only study does not measure its end-to-end performance.
- Separate ai_mode off, fallback and always. Without forceToGenderize, single requests default to fallback and batches default to off. Fallback tries the dataset and asks AI if no gender is returned. A combined score can hide a difference between dataset and AI outcomes.
- Evaluate forceToGenderize separately for all supported input types. It can infer an association from a nickname even when name is null; an alias is not evidence of the account owner's identity.
- Record whether country was supplied. Country rows are preferred for each candidate when present, with global lookup as fallback; inspect match.scope and match.country to see which evidence was selected.
- Report by country context, writing system and input type with the number of cases in each segment. A regional association is not nationality or residence.
- Choose any application confidence threshold before measuring the held-out sample. Report both the raw API result and application abstentions caused by that threshold, separately for each confidence_kind.
Metric definitions and denominators
For a planned evaluation, N is the number of eligible reference-labelled cases. A = C + W is the number with an answered gender: C correct, W incorrect. U is unknown and E is an operational error after the predefined execution policy. With one final outcome per case, N = C + W + U + E. Publish these counts before percentages.
For N = 0, these population rates are undefined. Do not publish an undefined rate as 0%. Include uncertainty intervals and the method used to calculate them; small or unrepresentative segments cannot support broad accuracy claims.
| Metric | Definition | Published measurement |
|---|---|---|
| Answer coverage | A / N | Not measured |
| Accuracy on answered cases | C / A; undefined if A = 0 | Not measured |
| Error rate on answered cases | W / A; undefined if A = 0 | Not measured |
| Correct answers across all eligible cases | C / N; do not label this as answered-case accuracy | Not measured |
| Unknown rate | U / N; ambiguity can be reported as a subset of U | Not measured |
| Operational error rate | E / N; also break down errors by stable error code | Not measured |
| Calibration | Compare score bands with observed correctness, separately for observed_frequency and model_reported; disclose binning, counts and uncertainty | Not measured |
What a reproducible evaluation must include
Do not create reference labels by asking another name-inference service for its guesses. A comparison measures agreement, not verified accuracy, unless an appropriate independent reference is available. If reference labels do not map to the API's binary output, document that limitation and the evaluation scope without relabelling people to fit the API.
| Record | Required disclosure | Current public evidence |
|---|---|---|
| Reference population | Origin and permitted use of reference labels; eligibility, exclusions, sampling, deduplication and segment definitions | No reviewed reference dataset is published here |
| Separation from source data | Check overlap and leakage; distinguish development fixtures from an independent held-out evaluation | No verified separation report is published here |
| Version and execution | Execution date, exact API release, dataset snapshot, model/provider revision when AI is used, prompts and request settings | No executed evaluation version is published here |
| Results | N, C, W, U, E, segment counts, confidence intervals and calibration analysis | No measured performance values are published here |
| Reproduction and review | Evaluation code, reproducible aggregate results, known limitations and identifiable review responsibility | Protocol only; no completed independent benchmark review |
Use the result with its limits
Name and alias associations vary across languages, countries and populations. A returned label cannot establish a person's gender identity. Preserve unknown outcomes and let information supplied by the person take precedence. Do not use inferred gender as the basis for consequential decisions about an individual.
Review history
| Date | Documentation revision | Change |
|---|---|---|
| 2026-09-26 | 1.0.1 | Aligned this English guide with V2 confidence kinds, result sources, unknown outcomes and evaluation denominators. No new accuracy result is claimed. |