Historical tool output | archived output | JSONL line 4955 | 2026-09-07T12:54:19.780Z

=== error ===
RuntimeError: ca
=== phases w/ timestamps ===
12:49:19 [default_loader.py:430] Loading weights took 449.29 seco
12:50:07 [default_loader.py:430] Loading weights took 496.33 seco
12:51:18 [default_loader.py:430] Loading weights took 117.46 seco
12:51:18 [model_runner.py:407] Model loading took 65.0 GiB memory a
12:52:33 [default_loader.py:430] Loading weights took 144.20 seco
12:52:33 [model_runner.py:407] Model loading took 65.0 GiB memory a
12:52:38 [encoder_runner.py:120] Encoder cache will be i
12:52:59 [backends.py:1155] Dynamo bytecode tra
12:52:38 [encoder_runner.py:120] Encoder cache will be i
12:53:08 [core.py:1346] Traceback (most rece
12:52:59 [backends.py:1155] Dynamo bytecode tra
=== top memory users (from Ray msg) ===
=== .115 memlog last 3 min (min avail) ===
08:53:19,0.0,2.8,0.0,3.9:ray::RayWorkerP
08:53:27,4.7,2.8,0.0,2.8:ray::RayWorkerP
08:53:09,7.8,10.8,0.0,6.0:ray::RayWorkerP
=== .115 oom-guard ===
2026-09-07T08:53:19-04:00 二级触发：avail=0G < 12G，终止 vLLM
2026-09-07T08:53:34-04:00 二级处理后 avail=113G
=================== .98 ===================
--- kernel OOM kills (this boot) ---
[Mon Sep  7 07:33:05 2026] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from rmStatus @ system_mem.c:345
[Mon Sep  7 08:53:08 2026] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_des
[Mon Sep  7 08:53:14 2026] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_des
[Mon Sep  7 08:53:14 2026] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from rmStatus @ system_mem.c:345
--- memlog: 3 lowest avail rows in last 4 min ---
08:53:14,1.9,2.7,0.0,5.9:ray::RayWorkerP
08:53:07,14.6,17.3,0.0,6.2:ray::RayWorkerP
08:52:27,36.7,38.8,0.0,6.2:ray::RayWorkerP
--- oom-guard ---
2026-09-07T08:53:14-04:00 二级触发：avail=1G < 12G，终止 vLLM
2026-09-07T08:53:29-04:00 二级处理后 avail=115G
--- worker log tail ---
(Worker_TP1_EP1 pid=234) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(Worker_TP1_EP1 pid=234) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=T
(Worker_TP1_EP1 pid=234) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(Worker_TP1_EP1 pid=234) [transformers] Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}
(Worker_TP1_EP1 pid=234) [rank1]:W0907 12:52:59.864000 234 torch/_inductor/utils.py:1953] Not enough SMs to use max_autotune_gemm mode