Loading neural weights...
initializing attention layers
Back to projects
git-to-doc
The audit layer for AI-generated commits
1st place · GDG Cloud Boston × NortheasternPowered by GemmaJune 2026Open source
$pip install git-to-doc
What it is
git-to-doc reads a code diff and checks whether the commit message actually describes the change. Two auditor models from different families read the diff on their own, before either one sees the message. Anything the message leaves out or gets wrong becomes a divergence, and each one points to a file and line.
How it works
[ diff ] │ ├─▶ qwen2.5-coder:14b reads blind └─▶ deepseek-coder-v2 reads blind │ ▼ compare each reading against the commit message │ ▼ divergences, each citing file:line HIGH = both models agree · possible = one model
- 01
Two auditors from different families
Default pair: qwen2.5-coder:14b and deepseek-coder-v2. Different training data gives them different blind spots, so they rarely fail in the same way.
- 02
Each reads the diff blind
Both models describe what the diff does before either one sees the author's commit message.
- 03
Compare against the message
Each independent reading is checked against the actual commit message. Anything it leaves out or gets wrong becomes a divergence.
- 04
Cite a line, or get rejected
Every divergence must reference a file and a line number. If a model can't produce a valid citation, Pydantic schema validation drops the output.
- 05
Confidence by agreement
HIGH = both models flagged the same file:line, within three lines. possible = only one auditor flagged it.
- 06
Errors out when unsure
If neither auditor can produce a valid cited report, the tool errors out. It won't invent an “all clear.”
See it run
~/Professional/git-to-doc
(venv) ~/Professional/git-to-doc ❯ git-to-doc verify --url https://github.com/google-deepmind/gemma/pull/224
🔎 auditing https://github.com/google-deepmind/gemma/pull/224 with 2 model(s) — this can take a moment…
↓ https://patch-diff.githubusercontent.com/raw/google-deepmind/gemma/pull/224.diff
🔍 Verifying commit https://github.com/google-deepmind/gemma/pull/224
Auditors: qwen2.5-coder:14b, deepseek-coder-v2:latest (2 independent, cross-family)
ORIGINAL MESSAGE
docs: Add GPU memory requirements table (Fixes #167)
## Description
Adds detailed GPU memory requirements table to README.md to help users
select appropriate hardware configurations (Addresses Issue #167).
### Changes
- Added comprehensive VRAM specifications for:
- Gemma 2B, 7B, and 12B model variants
- Both bf16 and int4 quantizations
- Included:
- Minimum required VRAM
- Recommended VRAM for optimal performance
- Notes about framework overhead
### Impact
- Helps users avoid OOM errors by selecting compatible hardware
- Provides clear guidance for different model sizes and precisions
- Complements existing system requirements documentation
Fixes #167
✅ Original message matches the diff. Auditors found no material omissions.
⚡ 4 POSSIBLE divergences (only one auditor flagged)
• The diff introduces new CI/CD workflows and configuration files that define how the project will be tested, packaged, and published on GitHub Actions.
.github/workflows/pytest_and_autopublish.yml:1
(flagged by deepseek-coder-v2:latest, others did not; verify manually)
• The diff includes a new `.gitignore` file to manage which files should be ignored by version control.
.gitignore:1
(flagged by deepseek-coder-v2:latest, others did not; verify manually)
• The diff adds a new configuration file `.pylintrc` for configuring pylint, a Python linting tool.
.pylintrc:1
(flagged by deepseek-coder-v2:latest, others did not; verify manually)
• The commit message claims to add a GPU memory requirements table to README.md, but no changes were made to any README file in the diff.
README.md:1
(flagged by qwen2.5-coder:14b, others did not; verify manually)
📊 Benchmark: 69% precision, 36% recall (synthetic n=168, 16GB tier, see BENCHMARKS.md)
Benchmarks
Full methodology and raw results are in BENCHMARKS.md.
Synthetic · n=168 · ground truth known
69%
Precision
36%
Recall
32%
False-positive rate
Real-world · n=95 · actual AI-authored commits
29%
Divergence rate
11%
Inter-auditor agreement
50
HIGH-confidence findings
Divergence rate by authoring tool
Cursor
45%
Copilot
20%
Claude
19%
Scope: one hardware tier (16GB RAM), one model pair. Only the default pair is benchmarked. Other pairs are open questions.
Honest limitations
- ·On dense diffs, the tool over-flags. When I hand-checked the top HIGH findings, about 30% were real omissions. The other 70% were the auditors over-reading complex code where the message was actually fine.
- ·Recall is low on purpose. HIGH only fires when both models agree, which is a strict bar. That's why synthetic recall sits at 36%.
- ·Real-world inter-auditor agreement is 11%. Most findings show up as single-auditor “possible” flags for a human to review.
How it got here
This didn't start as an auditor. It won 1st place at the GDG Cloud Boston hackathon as a commit message generator, which is a saturated space.
Mid-build I realized the more interesting problem: in 2026 most commits are written by AI, the person merging them often doesn't know exactly what changed, and the “reviewer” is often the same model family that wrote the code.
So I rewrote the tool to audit instead of generate. Then benchmarked it, hand-checked the results, and shipped it honestly.
