d283fd55c-mac

result.json

total: 36, pass: 30, fail: 6

update commit hash

• commit: d283fd55c3c6a8f0736f8641356858f7c150f02f
• platform: macOS-14.3.1-arm64-arm-64bit
• ragit version: ragit 0.3.0-dev
• rustc version: rustc 1.86.0-nightly (4a4309466 2025-02-02)
• cargo version: cargo 1.86.0-nightly (0e3d73849 2025-02-01)
• python version: 3.9.6
• tested at: 2025-02-20T10:48:05.861169Z (114 days ago)
• total elapsed time: 2,914,678 ms

TOC

1. cargo_tests  
2. add_and_rm  
3. recover  
4. clone  
5. clone2  
6. cli  
7. archive  
8. many_chunks  
9. many_jobs  
10. ls  
11. meta  
12. symlink  
13. ii  
14. cat_file  
15. images  
16. markdown_reader  
17. csv_reader  
18. subdir  
19. tfidf  
20. merge  
21. external_bases  
22. end_to_end dummy  
23. end_to_end gpt-4o-mini  
24. prompts dummy  
25. prompts gpt-4o-mini  
26. prompts claude-3.5-sonnet  
27. empty dummy  
28. empty gpt-4o-mini  
29. images2 gpt-4o-mini  
30. images2 claude-3.5-sonnet  
31. extract_keywords dummy  
32. extract_keywords gpt-4o-mini  
33. ragit_api llama3.2-11b-groq  
34. ragit_api command-r  
35. ragit_api phi-4-14b-ollama  
36. migrate  

Cases

elapsed time: 232,864 ms

history

elapsed time: 98,396 ms

history

elapsed time: 12,096 ms

history

elapsed time: 468,878 ms

history

elapsed time: 7,072 ms

history

elapsed time: 11,419 ms

history

elapsed time: 359,204 ms

history

elapsed time: 309,536 ms

Command '['cargo', 'run', '--release', '--', 'build']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/tests.py", line 377, in <module>
    test()
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/many_chunks.py", line 43, in many_chunks
    cargo_run(["build"], timeout=6.0)
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/utils.py", line 65, in cargo_run
    result = subprocess.run(args, **kwargs)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', 'build']' returned non-zero exit status 1.

history

elapsed time: 46,517 ms

history

elapsed time: 237,872 ms

history

elapsed time: 6,907 ms

history

elapsed time: 8,155 ms

history

elapsed time: 123,430 ms

tfidf result on term 'annot ancestor bad itself init' is not close enough. error: `approximation[0] not in answer`, answer: ['25226859944c801d09de829ad4ca44afb20e25ab709e86bd0000000100000fa2', '6eafab8a99414c6c52474a26d2419b03e89928e3e54693620000000100000fac', '5118fde6d2871a5856f3d287538d0cf70713b1010b05184c0000000100000fa9', 'eb1a1fefec4525c2ce2d696361ca967bca6fda8bfc89e3720000000100000fa6', '21bb29a2476b976295323f6fc152b141576c7034daabb2ad0000000100000fa1', 'a77115d715f5a8a104c40a149ccb07541ca095990dc0faad0000000100000b88', '3f990903a44f08a159b0eca30f7f7d226ff1c5510cacbca10000000100000fa3', '4c29234ba368d52883cbc6aad4122c93db39f676867e2eca0000000100000e4b', 'c1606aa75f56209996cd146d7a75cccb217c0aa649b4ecce0000000100000fa4', 'ab52ab66f1279ff1b0eead3f0d28f288fa8168ea66c331880000000100000fa0'], approximation: ['6517016b86562ad1656ea34a5c11dcb13b85370fd9f3cf490000000100000fa2', '3f990903a44f08a159b0eca30f7f7d226ff1c5510cacbca10000000100000fa3', '4c29234ba368d52883cbc6aad4122c93db39f676867e2eca0000000100000e4b', '819e297a454322c2ccb99775279fbc1c132c7c6a0f8d75710000000100000c73', '25226859944c801d09de829ad4ca44afb20e25ab709e86bd0000000100000fa2', '5118fde6d2871a5856f3d287538d0cf70713b1010b05184c0000000100000fa9', 'aeea5a2e7e7a4280b8c71ba2884f61a25100d20d087213f10000000100000fa2', '1b22f8022cd852eb695a8c16c184a432511196301d0fd4380000000100000fa7', '9e66317fe2fafdf78ef919ef9a968179b3768fc0059266e20000000100000f03', '9303416434dc1e2abe87253e22e414c669483ac4577f22660000000100000820']
Traceback (most recent call last):
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/ii.py", line 105, in ii_worker
    raise AssertionError(f"approximation[{i}] not in answer")
AssertionError: approximation[0] not in answer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/tests.py", line 377, in <module>
    test()
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/ii.py", line 51, in ii
    ii_worker()
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/ii.py", line 118, in ii_worker
    raise AssertionError(f"tfidf result on term '{term}' is not close enough. error: `{e}`, answer: {answer}, approximation: {approximation}")
AssertionError: tfidf result on term 'annot ancestor bad itself init' is not close enough. error: `approximation[0] not in answer`, answer: ['25226859944c801d09de829ad4ca44afb20e25ab709e86bd0000000100000fa2', '6eafab8a99414c6c52474a26d2419b03e89928e3e54693620000000100000fac', '5118fde6d2871a5856f3d287538d0cf70713b1010b05184c0000000100000fa9', 'eb1a1fefec4525c2ce2d696361ca967bca6fda8bfc89e3720000000100000fa6', '21bb29a2476b976295323f6fc152b141576c7034daabb2ad0000000100000fa1', 'a77115d715f5a8a104c40a149ccb07541ca095990dc0faad0000000100000b88', '3f990903a44f08a159b0eca30f7f7d226ff1c5510cacbca10000000100000fa3', '4c29234ba368d52883cbc6aad4122c93db39f676867e2eca0000000100000e4b', 'c1606aa75f56209996cd146d7a75cccb217c0aa649b4ecce0000000100000fa4', 'ab52ab66f1279ff1b0eead3f0d28f288fa8168ea66c331880000000100000fa0'], approximation: ['6517016b86562ad1656ea34a5c11dcb13b85370fd9f3cf490000000100000fa2', '3f990903a44f08a159b0eca30f7f7d226ff1c5510cacbca10000000100000fa3', '4c29234ba368d52883cbc6aad4122c93db39f676867e2eca0000000100000e4b', '819e297a454322c2ccb99775279fbc1c132c7c6a0f8d75710000000100000c73', '25226859944c801d09de829ad4ca44afb20e25ab709e86bd0000000100000fa2', '5118fde6d2871a5856f3d287538d0cf70713b1010b05184c0000000100000fa9', 'aeea5a2e7e7a4280b8c71ba2884f61a25100d20d087213f10000000100000fa2', '1b22f8022cd852eb695a8c16c184a432511196301d0fd4380000000100000fa7', '9e66317fe2fafdf78ef919ef9a968179b3768fc0059266e20000000100000f03', '9303416434dc1e2abe87253e22e414c669483ac4577f22660000000100000820']

history

elapsed time: 32,910 ms

history

elapsed time: 8,798 ms

history

elapsed time: 14,647 ms

history

elapsed time: 7,641 ms

history

elapsed time: 22,391 ms

history

elapsed time: 26,858 ms

history

elapsed time: 49,638 ms

history

elapsed time: 90,372 ms

history

elapsed time: 74,141 ms

history

elapsed time: 199,052 ms

history

elapsed time: 12,414 ms

history

elapsed time: 72,061 ms

history

elapsed time: 88,146 ms

history

elapsed time: 5,624 ms

history

elapsed time: 6,610 ms

history

elapsed time: 14,047 ms

history

elapsed time: 13,482 ms

history

elapsed time: 3,127 ms

history

elapsed time: 13,275 ms

history

elapsed time: 2,241 ms

Command '['cargo', 'run', '--release', '--', '--model', 'llama3.2-11b-groq', '--input', 'hello_world.pdl']' returned non-zero exit status 101.
Traceback (most recent call last):
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/tests.py", line 377, in <module>
    test()
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/tests.py", line 346, in <lambda>
    ("ragit_api llama3.2-11b-groq", lambda: ragit_api(test_model="llama3.2-11b-groq")),
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/ragit_api.py", line 11, in ragit_api
    subprocess.run(["cargo", "run", "--release", '--', '--model', test_model, '--input', 'hello_world.pdl'], check=True)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', '--model', 'llama3.2-11b-groq', '--input', 'hello_world.pdl']' returned non-zero exit status 101.

history

elapsed time: 409 ms

Command '['cargo', 'run', '--release', '--', '--model', 'command-r', '--input', 'hello_world.pdl']' returned non-zero exit status 101.
Traceback (most recent call last):
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/tests.py", line 377, in <module>
    test()
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/tests.py", line 347, in <lambda>
    ("ragit_api command-r", lambda: ragit_api(test_model="command-r")),
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/ragit_api.py", line 11, in ragit_api
    subprocess.run(["cargo", "run", "--release", '--', '--model', test_model, '--input', 'hello_world.pdl'], check=True)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', '--model', 'command-r', '--input', 'hello_world.pdl']' returned non-zero exit status 101.

history

elapsed time: 5,391 ms

Command '['cargo', 'run', '--release', '--', '--model', 'phi-4-14b-ollama', '--input', 'hello_world.pdl']' returned non-zero exit status 101.
Traceback (most recent call last):
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/tests.py", line 377, in <module>
    test()
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/tests.py", line 348, in <lambda>
    ("ragit_api phi-4-14b-ollama", lambda: ragit_api(test_model="phi-4-14b-ollama")),
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/ragit_api.py", line 11, in ragit_api
    subprocess.run(["cargo", "run", "--release", '--', '--model', test_model, '--input', 'hello_world.pdl'], check=True)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', '--model', 'phi-4-14b-ollama', '--input', 'hello_world.pdl']' returned non-zero exit status 101.

history

elapsed time: 168,843 ms

Command '['cargo', 'run', '--release', '--', 'migrate']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/tests.py", line 377, in <module>
    test()
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/migrate.py", line 90, in migrate
    cargo_run(["migrate"])
  File "/Users/baehyunsol/Documents/Rust/ragit/tests/utils.py", line 65, in cargo_run
    result = subprocess.run(args, **kwargs)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['cargo', 'run', '--release', '--', 'migrate']' returned non-zero exit status 1.

history