aimb-x/agentgit
repositories / aimb-x / agentgit

agentgit public

git clone https://git.169.58.159.131.nip.io/aimb-x/agentgit.git
README.md 7893 bytes 1 lines
1
# agentgit
2
3
GitHub for agents. Agents register over MCP; each one learns only from the projects
4
its owner chose. Everything is text from the real repositories, cited as `path:line`.
5
6
- MCP endpoint: `POST https://agents.169.58.159.131.nip.io/mcp`
7
- Dashboard: <https://agents.169.58.159.131.nip.io/> (admin: `/admin?token=…`, local only)
8
- Stack: Python 3.14 stdlib + **kilat**, our own storage/search engine (C11).
9
No SQLite, no Postgres, no vector DB, no frameworks.
10
11
## Why it exists
12
13
A coding agent knows whatever was in its context window. A team's knowledge lives in
14
repositories: decisions, workarounds, measured numbers, the reason a function looks
15
the way it does. agentgit gives every agent a *bounded, permissioned* view of those
16
repositories and a place to write down what it learned, so the next session starts
17
from evidence instead of guesses.
18
19
## How it works
20
21
```
22
owner (human) agent (MCP client)
23
│ ag_usr_… │ ag_agt_…
24
├─ agent_register ──► token │
25
├─ project_add (url/path) │
26
│ │ │
27
│ ▼ ▼
28
│ indexer worker ──► SQLite FTS5 ◄── knowledge_search / knowledge_doc
29
│ (git clone, walk, chunk) ◄── remember / recall
30
└─ project_list / status / summary
31
```
32
33
- **Indexing** — `git clone --depth 1` (or a local path), walk text files, skip
34
`node_modules`/`dist`/lockfiles/binaries, chunk to ~1400 chars aligned to blank
35
lines with a 6-line overlap, store in `docs` with a line range.
36
- **Search** — kilat's inverted index, BM25 (k1=1.2, b=0.75), OR terms with bounded
37
prefix expansion; Indonesian and English stopwords are dropped. Each hit carries
38
`project`, `path`, `lines` and a snippet with its offsets.
39
- **Memory** — `remember` writes per-agent notes (also FTS-indexed); `recall`
40
searches only that agent's notes.
41
- **Scoping** — a project is `owner` (only agents of its owner) or `public` (any
42
registered agent). Owner projects never appear in another agent's `project_list`.
43
44
## Tools
45
46
| Tool | Token | What it does |
47
|---|---|---|
48
| `whoami` | any | identity + counters |
49
| `agent_register` | user | create an agent, returns `ag_agt_…` once |
50
| `project_add` | user | add repo URL or local path, queues indexing |
51
| `project_status` | any | pending/running/ready/failed + stats |
52
| `project_list` | any | projects this token may learn from |
53
| `project_summary` | agent | languages, top dirs, last commit, largest files |
54
| `knowledge_search` | agent | search all visible projects, cited snippets |
55
| `knowledge_doc` | agent | read one file (or continue from a line) |
56
| `remember` | agent | save a lesson, optionally tied to a project |
57
| `recall` | agent | search this agent's own notes |
58
59
## Quickstart
60
61
```bash
62
# 1. owner + agent token (server side)
63
agentgit user bagas --email bagas@aimb-x.xyz # → ag_usr_…
64
agentgit agent bagas nara --model hermes # → ag_agt_…
65
66
# 2. add a project to learn (owner token, via MCP)
67
curl -s https://agents.169.58.159.131.nip.io/mcp \
68
-H "Authorization: Bearer $USER_TOKEN" -H 'content-type: application/json' \
69
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"project_add",
70
"arguments":{"name":"micrograd","url":"https://github.com/karpathy/micrograd",
71
"visibility":"public"}}}'
72
73
# 3. agent asks questions (agent token)
74
curl -s https://agents.169.58.159.131.nip.io/mcp \
75
-H "Authorization: Bearer $AGENT_TOKEN" -H 'content-type: application/json' \
76
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"knowledge_search",
77
"arguments":{"query":"topological sort gradient","limit":3}}}'
78
```
79
80
Client configs (Claude Code, Cursor, Hermes, Python): see
81
[`docs/mcp-clients.md`](docs/mcp-clients.md).
82
83
## Measured on this machine
84
85
| | |
86
|---|---|
87
| projects indexed | 5 (sinema, veloce-pages, kantor-ai, agentgit, micrograd) |
88
| chunks | 311 across 90 files |
89
| index time, small repo (60 KB) | **0.17 s** |
90
| search latency, in-process | **p50 1.06 ms** kilat vs 1.34 ms SQLite (engine only: 0.50 ms) |
91
| search latency, public HTTPS | p50 66 ms (TLS + nginx + hairpin) |
92
| database | **2.27 MB** (kilat; SQLite was 3.35 MB for the same 685 chunks) |
93
| service RSS | 13.5 MB |
94
| tests | **52 passing** (37 agentgit + 15 kilat) |
95
96
## Storage engine: kilat
97
98
Everything agentgit stores — users, agents, projects, jobs, documents, notes, events —
99
lives in one append-only file written by [`kilat`](../kilat), our own C11 engine:
100
101
```
102
[u32 len][u32 crc32][u8 type][u8 pad×3][payload]
103
put : [u32 cl][coll][u32 kl][key][u32 vl][val]
104
del : [u32 cl][coll][u32 kl][key]
105
meta: [u32 cl][coll] (collection marked for full-text indexing)
106
```
107
108
- Opening the file replays the log, verifies every checksum, rebuilds the key map and
109
the inverted index. A torn tail from a crash is truncated to the last valid record —
110
tested by appending garbage to a live database.
111
- `fsync` per record for durability; bulk reindexing switches it off and syncs once at
112
the end (`kg_set_sync` + `kg_sync`), which is the difference between 321 and 18,133
113
documents/second on this machine.
114
- Compaction rewrites only live records to `<file>.tmp`, fsyncs, renames, and replays.
115
116
Old SQLite data was migrated with `tools/migrate_sqlite.py`; token hashes were copied
117
verbatim, so every issued `ag_agt_…` token kept working.
118
119
Measured on the real dataset (685 chunks, 6 projects):
120
121
| | SQLite FTS5 | kilat total | kilat engine only |
122
|---|---|---|---|
123
| search p50 | 1.34 ms | **1.06 ms** | 0.50 ms |
124
| search p95 | 1.97 ms | **1.60 ms** | 0.76 ms |
125
| file size | 3.35 MB | **2.27 MB** | — |
126
| top-1 agreement | — | 6/9 query set | — |
127
128
Engine limits, stated plainly: one writer process (no cross-process locking), the
129
whole indexed corpus sits in RAM, prefix-only scans, manual compaction. That is sized
130
for agentgit today, not a claim of SQLite parity.
131
132
## Security model
133
134
- Two token kinds, SHA-256 at rest, constant-time compare by lookup on the hash.
135
- Public surface is `/mcp` and `/healthz` only; `/admin` is 404 from the internet and
136
requires `AGENTGIT_ADMIN_TOKEN` when reached locally.
137
- Rate limit `240 r/m` per IP with burst 60 on `/mcp`; request body capped at 1 MB
138
(nginx) and `1 MiB` in the app.
139
- Repository content is untrusted text: it is stored and returned verbatim, never
140
executed, and path traversal (`..`) is rejected in `knowledge_doc`.
141
- Tokens are shown once. There is no rotation or revocation endpoint yet.
142
143
## Limits (deliberate)
144
145
- **Lexical search only.** No embeddings, so synonyms are missed; OR + bounded prefix
146
plus stopword filtering covers the common cases.
147
- **Text files only.** Images, PDFs, notebooks and binaries are skipped.
148
- **One indexer worker**, jobs are FIFO, no per-owner quota.
149
- **No per-agent project allowlist** beyond `owner`/`public`.
150
- **No OAuth / multi-tenant UI**; registration is a CLI/API action by the owner.
151
152
Next steps that would pay off first: per-agent allowlists, incremental reindex on
153
webhook/poll, embeddings as a second ranking signal, and token rotation.
154
155
## Development
156
157
```bash
158
make test # 37 tests: JSON-RPC, auth, scoping, indexing, notes, HTTP, web
159
make serve # run locally on 127.0.0.1:8188
160
make deploy # rsync to /opt/agentgit + kilat .so + restart systemd unit
161
make engine # build kilat and install /opt/kilat/libkilat.so
162
make seed PROJ=/root/sinema NAME=sinema USER=bagas # add + index a local project
163
```
164
165
Layout: `agentgit/db.py` (collections on kilat) · `agentgit/kilat.py` (ctypes binding)
166
· `agentgit/indexer.py` (clone, walk, chunk) · `agentgit/mcp.py` (tools + JSON-RPC) ·
167
`agentgit/httpd.py` (HTTP, worker, web routes) · `agentgit/web.py` (GitHub-style UI) ·
168
`agentgit/cli.py` (ops).