3aa665352-linux

result.json

total: 63, pass: 43, fail: 20

update `tests/server_permission.py`

The test script is (almost) complete, but I found some issues in
push/clone implementation. They're supposed to use api keys to
authenticate, but they aren't.

• commit: 3aa6653521091d526a28065d96c5a7fa7f9e5ea1
• platform: Linux-5.15.0-136-generic-x86_64-with-glibc2.35
• ragit version: ragit 0.4.0-dev
• rustc version: rustc 1.87.0-nightly (78948ac25 2025-03-20)
• cargo version: cargo 1.87.0-nightly (6cf826701 2025-03-14)
• python version: 3.10.12
• tested at: 2025-05-23T00:18:04.093493Z (22 days ago)
• total elapsed time: 6,862,392 ms

TOC

1. cargo_tests  
2. add_and_rm  
3. add_and_rm2  
4. ignore  
5. recover  
6. clone  
7. clone_empty  
8. pull  
9. server  
10. server_permission  
11. cli  
12. outside  
13. archive  
14. many_chunks  
15. many_jobs  
16. ls  
17. meta  
18. symlink  
19. gh_issue_20  
20. ii  
21. cat_file  
22. generous_file_reader  
23. clean_up_erroneous_chunk  
24. images  
25. markdown_reader  
26. csv_reader  
27. real_repos  
28. real_repos_regression  
29. subdir  
30. tfidf  
31. merge  
32. external_bases  
33. end_to_end dummy  
34. end_to_end llama3.3-70b  
35. audit llama3.3-70b  
36. logs llama3.3-70b  
37. prompts dummy  
38. prompts gpt-4o-mini  
39. prompts gemini-2.0-flash  
40. prompts claude-3.5-sonnet  
41. empty dummy  
42. empty llama3.3-70b  
43. server_chat llama3.3-70b  
44. server_chat gemini-2.0-flash  
45. images2 gpt-4o-mini  
46. images3 gpt-4o-mini  
47. pdl gpt-4o-mini  
48. pdf gpt-4o-mini  
49. svg gpt-4o-mini  
50. web_images gpt-4o-mini  
51. images2 claude-3.5-sonnet  
52. extract_keywords dummy  
53. extract_keywords gpt-4o-mini  
54. orphan_process llama3.3-70b  
55. write_lock llama3.3-70b  
56. ragit_api command-r  
57. query_options llama3.3-70b  
58. query_with_schema llama3.3-70b  
59. models_init  
60. test_home_config_override  
61. migrate  
62. migrate2  
63. config  

Cases

elapsed time: 282,318 ms

history

elapsed time: 34,662 ms

history

elapsed time: 11,572 ms

history

elapsed time: 3,211 ms

history

elapsed time: 3,100 ms

history

elapsed time: 48,215 ms

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/clone.py", line 43, in clone
    create_user(id="test-user", password="password")
  File "/home/baehyunsol/Documents/ragit/tests/server.py", line 192, in create_user
    assert response.status_code == expected_status_code
AssertionError

history

elapsed time: 1,238 ms

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/clone_empty.py", line 20, in clone_empty
    create_user(id="test-user", password="87654321")
  File "/home/baehyunsol/Documents/ragit/tests/server.py", line 192, in create_user
    assert response.status_code == expected_status_code
AssertionError

history

elapsed time: 1,331 ms

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/pull.py", line 47, in pull
    create_user(id="test-user", password="password")
  File "/home/baehyunsol/Documents/ragit/tests/server.py", line 192, in create_user
    assert response.status_code == expected_status_code
AssertionError

history

elapsed time: 3,228 ms

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/server.py", line 38, in server
    create_user(id="test-user", password="12345678")
  File "/home/baehyunsol/Documents/ragit/tests/server.py", line 192, in create_user
    assert response.status_code == expected_status_code
AssertionError

history

elapsed time: 1,188 ms

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/server_permission.py", line 27, in server_permission
    create_user(id="admin", email="i.am@admin.com", password="strong-password")
  File "/home/baehyunsol/Documents/ragit/tests/server.py", line 192, in create_user
    assert response.status_code == expected_status_code
AssertionError

history

elapsed time: 5,653 ms

history

elapsed time: 3,013 ms

history

elapsed time: 198,702 ms

history

elapsed time: 560,287 ms

history

elapsed time: 158,774 ms

history

elapsed time: 73,539 ms

history

elapsed time: 1,728 ms

history

elapsed time: 3,087 ms

history

elapsed time: 2,731 ms

history

elapsed time: 170,254 ms

tfidf result on term 'search gpg annot select correspond' is not close enough. error: `approximation[2] not in answer`, answer: ['3e0d93ece16c10490435c08b7b755db9a57e53b818a9e62c0000000100000fa3', 'c5719c769542cb0cde49558784948082703f2da9618c29d80000000100000fb3', '6d1b2eeef26e5ce9672e62a7ca43412c66b86ad0e48d27620000000100000fa0', '606389435f969a017ad1cf63a7a30eba0d1a08c743efea9f0000000100000318', 'f386d96798aad5baf548b6985b367932bdc89483b756b515000000010000081f', 'c66345d5ab119b4cf05a6899472b54a4fd0041ee2b83b9f80000000100000fa2', 'bf8735875031f53ccd50e48e6674d9ac64c90f68bb0c7edb0000000100000fa0', '509b4b369f9f9729365a6947ce43335209d934562feeb7220000000100000fa2', '82ad9747a31109a3ef965e4168a0968cb56a448390416e290000000100000bf5', 'b632241f25a98c9320097079669e1acd10afd534e67ec2600000000100000fa2'], approximation: ['3e0d93ece16c10490435c08b7b755db9a57e53b818a9e62c0000000100000fa3', 'b632241f25a98c9320097079669e1acd10afd534e67ec2600000000100000fa2', '90a25e1efdafffab6369490140eecabb90ab0649108feeff0000000100000cd4', 'bf8735875031f53ccd50e48e6674d9ac64c90f68bb0c7edb0000000100000fa0', '5cdbfe828a4a84a4129bda3cc32bb8376914275561fa6a1a0000000100000da8', 'c5719c769542cb0cde49558784948082703f2da9618c29d80000000100000fb3', '0833e100c47da17ca6a2d202310483ed3c08f75ec2cfbf4a0000000100000c67', '1ff3d753fa4b857385f748c5d02a7371332241a8579211f9000000010000075c', '6f305111c4ab2bb2243ce34889afb4f72dff498303da56890000000100000c1e', '6d1b2eeef26e5ce9672e62a7ca43412c66b86ad0e48d27620000000100000fa0']
Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/ii.py", line 103, in ii_worker
    raise AssertionError(f"approximation[{i}] not in answer")
AssertionError: approximation[2] not in answer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/ii.py", line 49, in ii
    ii_worker()
  File "/home/baehyunsol/Documents/ragit/tests/ii.py", line 116, in ii_worker
    raise AssertionError(f"tfidf result on term '{term}' is not close enough. error: `{e}`, answer: {answer}, approximation: {approximation}")
AssertionError: tfidf result on term 'search gpg annot select correspond' is not close enough. error: `approximation[2] not in answer`, answer: ['3e0d93ece16c10490435c08b7b755db9a57e53b818a9e62c0000000100000fa3', 'c5719c769542cb0cde49558784948082703f2da9618c29d80000000100000fb3', '6d1b2eeef26e5ce9672e62a7ca43412c66b86ad0e48d27620000000100000fa0', '606389435f969a017ad1cf63a7a30eba0d1a08c743efea9f0000000100000318', 'f386d96798aad5baf548b6985b367932bdc89483b756b515000000010000081f', 'c66345d5ab119b4cf05a6899472b54a4fd0041ee2b83b9f80000000100000fa2', 'bf8735875031f53ccd50e48e6674d9ac64c90f68bb0c7edb0000000100000fa0', '509b4b369f9f9729365a6947ce43335209d934562feeb7220000000100000fa2', '82ad9747a31109a3ef965e4168a0968cb56a448390416e290000000100000bf5', 'b632241f25a98c9320097079669e1acd10afd534e67ec2600000000100000fa2'], approximation: ['3e0d93ece16c10490435c08b7b755db9a57e53b818a9e62c0000000100000fa3', 'b632241f25a98c9320097079669e1acd10afd534e67ec2600000000100000fa2', '90a25e1efdafffab6369490140eecabb90ab0649108feeff0000000100000cd4', 'bf8735875031f53ccd50e48e6674d9ac64c90f68bb0c7edb0000000100000fa0', '5cdbfe828a4a84a4129bda3cc32bb8376914275561fa6a1a0000000100000da8', 'c5719c769542cb0cde49558784948082703f2da9618c29d80000000100000fb3', '0833e100c47da17ca6a2d202310483ed3c08f75ec2cfbf4a0000000100000c67', '1ff3d753fa4b857385f748c5d02a7371332241a8579211f9000000010000075c', '6f305111c4ab2bb2243ce34889afb4f72dff498303da56890000000100000c1e', '6d1b2eeef26e5ce9672e62a7ca43412c66b86ad0e48d27620000000100000fa0']

history

elapsed time: 14,741 ms

history

elapsed time: 348,621 ms

history

elapsed time: 1,425 ms

history

elapsed time: 4,409 ms

history

elapsed time: 5,222 ms

history

elapsed time: 2,813 ms

history

elapsed time: 2,304,237 ms

history

elapsed time: 2,527 ms

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/real_repos_regression.py", line 157, in real_repos_regression
    assert count_files() == (len(reproductions), 1, len(reproductions) - 1)  # (total, staged, processed)
AssertionError

history

elapsed time: 9,216 ms

history

elapsed time: 10,369 ms

history

elapsed time: 14,407 ms

history

elapsed time: 78,276 ms

history

elapsed time: 35,250 ms

history

elapsed time: 1,046,879 ms

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 608, in <lambda>
    ("end_to_end llama3.3-70b", lambda: end_to_end(test_model="llama3.3-70b")),
  File "/home/baehyunsol/Documents/ragit/tests/end_to_end.py", line 137, in end_to_end
    assert chunk_count_prev == chunk_count
AssertionError

history

elapsed time: 95,491 ms

Command '['cargo', 'run', '--release', '--', 'query', 'Why is the sky blue?']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 609, in <lambda>
    ("audit llama3.3-70b", lambda: audit(test_model="llama3.3-70b")),
  File "/home/baehyunsol/Documents/ragit/tests/audit.py", line 35, in audit
    cargo_run(["query", "Why is the sky blue?"])
  File "/home/baehyunsol/Documents/ragit/tests/utils.py", line 77, in cargo_run
    result = subprocess.run(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', 'query', 'Why is the sky blue?']' returned non-zero exit status 1.

history

elapsed time: 3,834 ms

history

elapsed time: 4,684 ms

history

elapsed time: 94,332 ms

Expected 12~15 chunks, got 1.
Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 612, in <lambda>
    ("prompts gpt-4o-mini", lambda: prompts(test_model="gpt-4o-mini")),
  File "/home/baehyunsol/Documents/ragit/tests/prompts.py", line 40, in prompts
    raise Exception(f"Expected 12~15 chunks, got {chunks}.")
Exception: Expected 12~15 chunks, got 1.

history

elapsed time: 28,589 ms

history

elapsed time: 96,848 ms

Expected 12~15 chunks, got 1.
Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 614, in <lambda>
    ("prompts claude-3.5-sonnet", lambda: prompts(test_model="claude-3.5-sonnet")),
  File "/home/baehyunsol/Documents/ragit/tests/prompts.py", line 40, in prompts
    raise Exception(f"Expected 12~15 chunks, got {chunks}.")
Exception: Expected 12~15 chunks, got 1.

history

elapsed time: 2,706 ms

history

elapsed time: 93,618 ms

Command '['cargo', 'run', '--release', '--', 'query', "what's your name?"]' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 616, in <lambda>
    ("empty llama3.3-70b", lambda: empty(test_model="llama3.3-70b")),
  File "/home/baehyunsol/Documents/ragit/tests/empty.py", line 23, in empty
    cargo_run(["query", "what's your name?"])
  File "/home/baehyunsol/Documents/ragit/tests/utils.py", line 77, in cargo_run
    result = subprocess.run(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', 'query', "what's your name?"]' returned non-zero exit status 1.

history

elapsed time: 2,412 ms

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 617, in <lambda>
    ("server_chat llama3.3-70b", lambda: server_chat(test_model="llama3.3-70b")),
  File "/home/baehyunsol/Documents/ragit/tests/server_chat.py", line 32, in server_chat
    create_user(id="test-user", password="secure-password")
  File "/home/baehyunsol/Documents/ragit/tests/server.py", line 192, in create_user
    assert response.status_code == expected_status_code
AssertionError

history

elapsed time: 2,382 ms

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 618, in <lambda>
    ("server_chat gemini-2.0-flash", lambda: server_chat(test_model="gemini-2.0-flash")),
  File "/home/baehyunsol/Documents/ragit/tests/server_chat.py", line 32, in server_chat
    create_user(id="test-user", password="secure-password")
  File "/home/baehyunsol/Documents/ragit/tests/server.py", line 192, in create_user
    assert response.status_code == expected_status_code
AssertionError

history

elapsed time: 10,071 ms

history

elapsed time: 22,506 ms

history

elapsed time: 3,893 ms

history

elapsed time: 119,910 ms

Command '['cargo', 'run', '--release', '--', 'ls-chunks', 'landscape.pdf', '--json']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 622, in <lambda>
    ("pdf gpt-4o-mini", lambda: pdf(test_model="gpt-4o-mini")),
  File "/home/baehyunsol/Documents/ragit/tests/pdf.py", line 41, in pdf
    chunks = json.loads(cargo_run(["ls-chunks", pdf["name"], "--json"], stdout=True))
  File "/home/baehyunsol/Documents/ragit/tests/utils.py", line 77, in cargo_run
    result = subprocess.run(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', 'ls-chunks', 'landscape.pdf', '--json']' returned non-zero exit status 1.

history

elapsed time: 7,508 ms

Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 623, in <lambda>
    ("svg gpt-4o-mini", lambda: svg(test_model="gpt-4o-mini")),
  File "/home/baehyunsol/Documents/ragit/tests/svg.py", line 123, in svg
    assert "ragit" in cargo_run(["pdl", "test1.pdl"], stdout=True).lower()
AssertionError

history

elapsed time: 113,116 ms

Command '['cargo', 'run', '--release', '--', 'ls-images', 'double.md']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 624, in <lambda>
    ("web_images gpt-4o-mini", lambda: web_images(test_model="gpt-4o-mini")),
  File "/home/baehyunsol/Documents/ragit/tests/web_images.py", line 25, in web_images
    assert count_images(["double.md"]) == 1
  File "/home/baehyunsol/Documents/ragit/tests/utils.py", line 136, in count_images
    images = cargo_run(["ls-images"] + (args or []), stdout=True)
  File "/home/baehyunsol/Documents/ragit/tests/utils.py", line 77, in cargo_run
    result = subprocess.run(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', 'ls-images', 'double.md']' returned non-zero exit status 1.

history

elapsed time: 10,155 ms

history

elapsed time: 1,643 ms

history

elapsed time: 92,061 ms

Command '['cargo', 'run', '--release', '--', 'extract-keywords', 'How does ragit store chunks?']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 630, in <lambda>
    ("extract_keywords gpt-4o-mini", lambda: extract_keywords(test_model="gpt-4o-mini")),
  File "/home/baehyunsol/Documents/ragit/tests/extract_keywords.py", line 13, in extract_keywords
    result = cargo_run(["extract-keywords", "How does ragit store chunks?"], stdout=True)
  File "/home/baehyunsol/Documents/ragit/tests/utils.py", line 77, in cargo_run
    result = subprocess.run(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', 'extract-keywords', 'How does ragit store chunks?']' returned non-zero exit status 1.

history

elapsed time: 99,689 ms

history

elapsed time: 144,788 ms

Command '['cargo', 'run', '--release', '--', 'query', 'How do I see a history of a file in git?']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 632, in <lambda>
    ("write_lock llama3.3-70b", lambda: write_lock(test_model="llama3.3-70b")),
  File "/home/baehyunsol/Documents/ragit/tests/write_lock.py", line 69, in write_lock
    cargo_run(["query", "How do I see a history of a file in git?"])
  File "/home/baehyunsol/Documents/ragit/tests/utils.py", line 77, in cargo_run
    result = subprocess.run(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', 'query', 'How do I see a history of a file in git?']' returned non-zero exit status 1.

history

elapsed time: 1,098 ms

history

elapsed time: 92,386 ms

Command '['cargo', 'run', '--release', '--', 'query', 'In which city did Baehyunsol live?']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 670, in <module>
    test()
  File "/home/baehyunsol/Documents/ragit/tests/tests.py", line 634, in <lambda>
    ("query_options llama3.3-70b", lambda: query_options(test_model="llama3.3-70b")),
  File "/home/baehyunsol/Documents/ragit/tests/query_options.py", line 17, in query_options
    assert "ragit" in cargo_run(["query", "In which city did Baehyunsol live?"], stdout=True).lower()
  File "/home/baehyunsol/Documents/ragit/tests/utils.py", line 77, in cargo_run
    result = subprocess.run(args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', 'query', 'In which city did Baehyunsol live?']' returned non-zero exit status 1.

history

elapsed time: 2,073 ms

history

elapsed time: 289 ms

history

elapsed time: 134 ms

history

elapsed time: 152,785 ms

history

elapsed time: 46,942 ms

history

elapsed time: 24,291 ms

history