Methodology: How We Measured Which DSA Patterns Engineers Fail Most

This post documents exactly how we computed the numbers in The DSA Patterns Engineers Actually Fail: data sources, the precise definitions used, the privacy rule that governs what gets published, what was excluded, and where the numbers are weaker than they look. It accompanies every original-research post Thita.ai publishes.

---

Data sources

Two tables, read-only, aggregate queries only:

• user_code_submissions — every real code submission run against a DSA problem's test cases. Only four columns were ever read: user_id, problem_id, status, submitted_at. No submitted_code, error_message, or submission_metadata column was queried.
• technical_problem_subpattern_mappings — the canonical mapping of each problem to its pattern and sub-pattern. Only the row marked is_primary = true for each problem was used, so no problem is double-counted across two pattern families.

Snapshot pulled 2026-09-07. At that time: 81,768 total rows in user_code_submissions, 409 DSA problems with a primary pattern mapping.

user_problem_progress (106,000+ rows) is a corroborating source for total attempt volume on the platform, but was not used in the pass-rate computation itself — it does not preserve per-submission ordering, which the first-attempt calculation requires.

Definitions

A "real attempt" is a submission whose status reflects an actual executed run against test cases: accepted, wrong_answer, runtime_error, timeout, compilation_error, execution_error, partial, failed, or injection_error. Excluded as non-attempts: analysis_mode, analysis_only, canvas_submitted, no_template, no_template_analysis — these are UI states where no code was actually executed against the problem's test cases.

First-attempt pass rate for a cell (a pattern or sub-pattern): take every (user, problem) pair with at least one real attempt on a problem mapped to that cell; order each pair's attempts by submitted_at; check whether the first one has status accepted (or passed / success, the platform's other historical acceptance statuses). The rate is the share of pairs where that first attempt passed.

Median attempts to pass: among the pairs that eventually reached an accepted attempt, the median 1-based position of that first accepted attempt in the pair's chronologically ordered attempt list. Pairs that never passed are excluded from this specific number (they have no "attempts to pass"), but they are still counted in the first-attempt pass rate denominator (a first attempt that never leads to a pass is still a real first attempt).

Pattern-level rollup: every sub-pattern cell under a pattern is pooled together before recomputing the rate, so pattern-level numbers are not an average of sub-pattern rates — they are recomputed directly from the union of (user, problem) pairs.

The suppression rule (k-anonymity)

Any cell — pattern or sub-pattern — backed by fewer than 50 distinct users is withheld from the published output entirely. Its rate, median, and problem count never appear in the committed data file or in the blog post. This is stricter than the k≥5 threshold Thita.ai's platform uses internally for other aggregate reporting, because this dataset is public.

In the 2026-09-07 snapshot: 15 of 16 DSA pattern families and 51 of 100 tracked sub-patterns passed the threshold. The Segment Tree & Fenwick Tree pattern family had no problems with a resolvable primary mapping and enough attempts in this snapshot, so it does not appear at all — that is a data-availability gap, not a suppression judgment call. 49 sub-pattern cells were below k=50 and were suppressed; none of their names, user counts, or rates appear in the published post or data file.

Exclusions

• Non-attempt submission statuses (listed above) — never executed code, so not a graded attempt.
• The one (user, problem) pair whose problem had no resolvable primary pattern mapping.
• Any cell below the k≥50 threshold.
• submitted_code, error_message, and every other free-text or per-user column on user_code_submissions — never read, never published.

Limitations

• Self-selected users. Everyone in this dataset chose to practice on Thita.ai and chose which problems to attempt. This is not a random sample of engineers; it is a sample of people actively preparing, which likely skews pass rates upward relative to a cold, unpracticed population.
• Difficulty mix varies by pattern. A pattern's first-attempt pass rate is a blend of its easy and hard problems in whatever ratio Thita.ai currently offers. A pattern with more Hard-rated problems mapped to it will show a lower rate for that reason alone, independent of true conceptual difficulty.
• Sub-pattern granularity means smaller n. Many sub-pattern cells sit close to the k=50 floor; a rate built on 50-100 users has a wider confidence interval than one built on the 1,000+ that back Two Pointers. Read close rankings (a percentage point or two apart) as roughly tied rather than strictly ordered.
• One snapshot in time. Pass rates can shift as the problem set, hint quality, or user mix changes. This is why the study is re-run on a fixed cadence rather than published once.

Cadence

This study is re-run quarterly, using the same script (backend/scripts/research/study1_hardest_patterns.py) against the same definitions, so numbers are comparable release to release. Each re-run gets its own snapshot date; this post and its companion data post are updated in place rather than forked into a new URL.

What we never publish

From this study or any other original-research post on Thita.ai:

• Submitted code, error messages, or any other free-text submission content.
• Session transcripts, recordings, or resume content.
• Per-user data of any kind — no leaderboards, no individual timelines, no "user X solved this in Y attempts."
• Community posts or platform feedback text.
• Any aggregate cell backed by fewer than 50 distinct users.

Questions about this methodology, or a request for a cut of the data not covered here, can go to the Thita.ai team through the contact page.